Andre Henriques
50bb49f822
All checks were successful
continuous-integration/drone/push Build is passing
693 lines
38 KiB
TeX
693 lines
38 KiB
TeX
%%% Preamble
|
|
\documentclass[11pt, a4paper, twoside]{article}
|
|
|
|
|
|
\include{settings.tex}
|
|
|
|
% Write the approved title of your dissertation
|
|
\title{Image Classification as a Software Platform}
|
|
|
|
% Write your full name, as in University records
|
|
\author{Andre Henriques\\Univerity of surrey}
|
|
|
|
\date{}
|
|
|
|
%%% Begin document
|
|
\begin{document}
|
|
\include{start}
|
|
\include{intro}
|
|
\include{lit}
|
|
\include{sanr}
|
|
\include{design}
|
|
|
|
\section{Service Implementation} \label{sec:si}
|
|
|
|
This section will discuss how the service followed some possible designs to achieve a working system.
|
|
The design path that was decided matches what made sense for the scale and needs of the project.
|
|
|
|
\subsection{Structure of the Service}
|
|
|
|
The structure of the service matches the designed structure, as it can be seen \ref{fig:simplified_service_diagram}.
|
|
|
|
\begin{figure}[h!]
|
|
\centering
|
|
\includegraphics[height=0.4\textheight]{system_diagram}
|
|
\caption{Simplified diagram of the service}
|
|
\label{fig:simplified_service_diagram}
|
|
\end{figure}
|
|
|
|
The implementation contains: Web App; Web server, which serves the Web App; API; Training Runners; Model Runners;
|
|
|
|
The implementation contains an extra nginx reverse proxy server that allows for the API and the webpage to be accessible from them same domain.
|
|
|
|
The rest of this section will go into details on how every tier of the structure was implemented.
|
|
|
|
\subsection{Web Application} \label{web-app-design}
|
|
|
|
The web application (WEB App) is the chosen user interface to control the service.
|
|
|
|
This subsection discusses details of the user flows and implementation of the application.
|
|
|
|
\subsubsection*{Implementation Details}
|
|
|
|
The Web APP is a single-page application (SPA).
|
|
The SPA architecture is one of the most prevalent architectures that exists nowadays.
|
|
It allows for the fast transitions between pages without having a full reload of the website.
|
|
|
|
Since this implementation separated the API and the Web App, it makes the use of server-side rendering more complex and less efficient.
|
|
As, the server would have to first request the API for information to build the web page and then send it to the users' device.
|
|
Therefore, the system will use client-side rendering only, allowing for the users' device to request the API directly for more information.
|
|
|
|
There exist currently many frameworks to create SPAs.
|
|
I selected Svelte \cite{svelte} for this project.
|
|
I selected Svelte because it's been one of the most liked frameworks to work with in the last years, accordingly to the State of JS survey \cite{state-of-js-2022}.
|
|
It's also one of the best performant frameworks that is currently available that has extremity good performance \cite{js-frontend-frameworks-performance}.
|
|
I also already have experience with Svelte.
|
|
|
|
I will be using Svelte with the SvelteKit framework \cite{svelte-kit} which greatly improves the developer experience.
|
|
|
|
SvelteKit allows for the easy creation of SPAs with a good default web router.
|
|
|
|
The static adapter will be used to generate a static HTML and JavaScript files, and they will be hosted by an NGINX proxy \cite{nginx}.
|
|
|
|
The web application uses the API to control the functionality of the service.
|
|
This implementation allows users of the application to do everything that the application does with the API, which is ideal in a SaaS project.
|
|
|
|
\subsubsection*{Service authentication} \label{sec:impl:service-auth}
|
|
|
|
\begin{figure}[h!]
|
|
\centering
|
|
\includegraphics[width=\textwidth]{service_authentication}
|
|
\caption{Simplified Diagram of User Authentication}
|
|
\label{fig:simplified_auth_diagram}
|
|
\end{figure}
|
|
|
|
The user uses an email and password to Sign In or Register with the application.
|
|
This is sent to the server and stored in a user account.
|
|
|
|
The Password is stored hashed using bcrypt \cite{bycrpt}.
|
|
In the future, other methods of authentication might be provided; like using Googles' OAuth.
|
|
Once logged In, the user will be able to use the application and manage tokens that were emitted for this application.
|
|
This allows the user to manage what services have access to the account. % and the usage that those services have used.
|
|
|
|
On the web app, the user can manage existing tokens.
|
|
Guaranteeing that only the clients that should be accessing the information are.
|
|
|
|
In the management screen, which can be seen in Fig. \ref{fig:token_page}, the user can remove, and create tokens.
|
|
|
|
\begin{figure}[H]
|
|
\centering
|
|
\includegraphics[width=0.5\textwidth]{token_page}
|
|
\caption{Screenshot of web application on the token control section.}
|
|
\label{fig:token_page}
|
|
\end{figure}
|
|
|
|
\subsubsection*{Model Management}
|
|
|
|
\begin{figure}[h!]
|
|
\centering
|
|
\includegraphics[width=\textwidth]{models_flow}
|
|
\caption{Simplified Diagram of Model management}
|
|
\label{fig:simplified_model_diagram}
|
|
\end{figure}
|
|
|
|
The diagram \ref{fig:simplified_model_diagram} shows the steps that the user takes to use a model.
|
|
|
|
First, the user creates the model.
|
|
In this step, the user uploads a sample image of what the model will be handling.
|
|
This image is used to define what the kinds of images the model will be able to intake.
|
|
This is done in the page shown in Fig. \ref{fig:create_model}, the user provides a name for the model and an image and then presses the button create.
|
|
|
|
\begin{figure}[H]
|
|
\centering
|
|
\includegraphics[width=0.9\textwidth]{create_model}
|
|
\caption{Screenshot of web application on the page that allows the creation of a new model.}
|
|
\label{fig:create_model}
|
|
\end{figure}
|
|
|
|
The user is then shown the model page, which contains all the information about a model, which can be seen in Fig. \ref{fig:model_page}.
|
|
|
|
\begin{figure}[H]
|
|
\centering
|
|
\includegraphics[width=0.9\textwidth]{base_model_page}
|
|
\caption{Screenshot of web application on the page shows basic information about the model.}
|
|
\label{fig:model_page}
|
|
\end{figure}
|
|
|
|
This page contains a set of tabs a the top.
|
|
Each tab gives different insight abou the model.
|
|
The ``Model'' tab, contains only relevent actions ot the most pressing action that the user take.
|
|
In Fig. \ref{fig:model_page}, the user has created a model but has not added training data so the page shows a section where the user can input training data.
|
|
The ``Model Data'' tab contains a more detailed view about data that has been updated.
|
|
|
|
Currently, the system does not support resizing of images that are different from the one uploaded at the creation step.
|
|
This was done to guarantee that the image that the user wants to classify is unmodified.
|
|
Moving the responsibility of cropping and resizing to the user.
|
|
In the future, systems could be implemented that allow the user to select how an image can be cropped.
|
|
|
|
The second step is uploading the rest of the dataset.
|
|
This can be done via the ``Model'' tab or via the ``Model Data'' tab that becomes available when the data of the model is first uploaded.
|
|
In this tab, the user can add and remove images, as well as create new classes for the model.
|
|
The page also shows some useful information, such as the distribution of the dataset, which can be seen in \ref{fig:model_data_tab}.
|
|
|
|
\begin{figure}[H]
|
|
\centering
|
|
\includegraphics[width=0.9\textwidth]{model_data_tab}
|
|
\caption{Screenshot of web application part of the ``Model Data'' tab}
|
|
\label{fig:model_data_tab}
|
|
\end{figure}
|
|
|
|
This information can be useful to more advanced users that might decide to gather more data to balance the dataset.
|
|
|
|
To upload the reset of the data set, the user can upload a zip file that contains a set of classes and images corresponding to that class.
|
|
That zip file is processed and images and classes are created.
|
|
The user is given instruction on how create the zip file so that the system can esaly process the data, the upload set can be seen in \ref{fig:upload_data_section}.
|
|
|
|
This process was original slow as the system did not have the capabilities to parallelize the process of importing the images, but this was implemented, and the import process was improved.
|
|
The improved process now takes a few seconds to process and verify the entirety of the dataset, making the experience for the end user better.
|
|
|
|
Alternatively, the user can use the API to create new classes and upload images.
|
|
|
|
\begin{figure}[H]
|
|
\centering
|
|
\includegraphics[width=0.9\textwidth]{upload_data_page}
|
|
\caption{Screenshot of web application upload zip step of the model page.}
|
|
\label{fig:upload_data_section}
|
|
\end{figure}
|
|
|
|
After all the images that are required for training are uploaded, the user can go to the training step.
|
|
This step will appear both in the main tab of the model page and in the dataset tab. Once the user instructs the system to start training, the model page will become the training page, and it will show the progress of the training of the model.
|
|
During this step, the system automatically trains the model.
|
|
After the system trains a model that meets the specifications set by the user, the service will make the model available for the user to use.
|
|
|
|
When the model is finished training, the user can use the model to run inference tasks on images.
|
|
To achieve this, the user can either use the API to submit a classification task or use the tasks tab in the web platform.
|
|
|
|
In the tasks tab, which can be seen in Fig. \ref{fig:update_data_section}, the user can see current and previous tasks.
|
|
The users can see what tasks were performed and their results.
|
|
The user can also inform the service if the task that was performed did return the correct results.
|
|
This information can be used to keep track of the real accuracy of the model.
|
|
The information can be used to see if the model needs refinement.
|
|
The system can add the classes that failed to return the correct result to a list of the original data, to be used in case of retraining the model.
|
|
|
|
\begin{figure}[H]
|
|
\centering
|
|
\includegraphics[width=0.6\textwidth]{model_task_tab}
|
|
\caption{Screenshot of web application on the tasks tab.}
|
|
\label{fig:upload_data_section}
|
|
\end{figure}
|
|
|
|
\subsubsection*{Advanced Model Management}
|
|
|
|
\begin{figure}[H]
|
|
\centering
|
|
\includegraphics[width=\textwidth]{models_advanced_flow}
|
|
\caption{Simplified Diagram of Advanced Model management}
|
|
\label{fig:simplified_model_advanced_diagram}
|
|
\end{figure}
|
|
|
|
The diagram \ref{fig:simplified_model_advanced_diagram} shows the steps that the user takes to use a model.
|
|
|
|
The steps are very similar to the normal model management.
|
|
The user would follow all the steps that are required for normal model creation and training.
|
|
|
|
At the end of the process, the user will be able to add new data to the model and retrain it.
|
|
To achieve that, the user would simply go to the data tab and create a new class, which the Fig. \ref{expand_class_part} shows.
|
|
Once a new class is added, the webpage will inform the user that the model can be retrained.
|
|
The user might choose to retrain the model now or more new classes and retrain later.
|
|
|
|
\begin{figure}[H]
|
|
\centering
|
|
\includegraphics[width=0.6\textwidth]{expand_class_part}
|
|
\caption{Screenshot of web application on the expand part of the ``Model Data'' tab.}
|
|
\label{fig:expand_class_part}
|
|
\end{figure}
|
|
|
|
During the entire process of creating new classes in the model and retraining the model, the user can still perform all the classifications tasks they desire.
|
|
|
|
\subsubsection*{Task Management}
|
|
|
|
Task management is the section of the website is where uses can manage their tasks. This includes training and classification tasks.
|
|
|
|
Users in this tab can see what is the progress, and results of their tasks.
|
|
The webpage also provides nice, easy to see statistics on the task results, allowing the user to see how the model is performing.
|
|
Which is shown on Fig. \ref{fig:upload_data_section}
|
|
|
|
On the administrator, users should be able to change the status of tasks as well as see a more comprehensive view on how the tasks are being performed.
|
|
Administrator users can see the current status of runners, as well as which task the runners are doing.
|
|
|
|
\begin{figure}[H]
|
|
\centering
|
|
\includegraphics[width=0.6\textwidth]{runner_page}
|
|
\caption{Screenshot of web application on the runners administrator Page.}
|
|
\label{fig:runner_page}
|
|
\end{figure}
|
|
|
|
|
|
\subsection{API}
|
|
|
|
The API was implemented as a multithreaded go \cite{go} server.
|
|
The application, on launch, loads a configuration file and connects to the database.
|
|
After connecting to the database, the application performs pre-startup checks to make sure no tasks that were interrupted via a server restart and were not left in an unrecoverable state.
|
|
Once the checks are done, the application creates workers, which will be explained in section \ref{impl:runner}, which when completed the API server is finally started up.
|
|
|
|
Information about the API is shown around the web page so that the user can see information about the API right next to where the user would normally do the action, providing a good user interface.
|
|
As the user can get information about right where they would normally do the action, as it can be seen in Fig. \ref{fig:code_demo}.
|
|
|
|
\begin{figure}[H]
|
|
\centering
|
|
\includegraphics[width=0.6\textwidth]{code_demo}
|
|
\caption{Screenshot of web application that shows the explanation of the API call}
|
|
\label{fig:code_demo}
|
|
\end{figure}
|
|
|
|
This server will take JSON and multipart form data requests, the requests are processed, and answered with a JSON response.
|
|
|
|
The multipart requests are required due to JSON's inability to transmit binary data, which will make the uploading of images extremely inefficient.
|
|
Those images would have to be transformed into binary data and then uploaded as a byte array or encoded as base64 and uploaded.
|
|
Either of those options is extremely inefficient.
|
|
Therefore, there is a need to use multipart form requests are required to allow the easy uploading of binary files.
|
|
|
|
Go was selected as the language to implement the backend due to various of its advantages.
|
|
Go has extremely fast compilations which allows for rapid development, and iteration.
|
|
It has a very minimal runtime which allows it to be faster, than heavy runtime languages such as JavaScript.
|
|
It is also a simple language, which helps maintain the codebase.
|
|
|
|
% TODO cite cgo tensorflow and torch
|
|
The Go language integrates well with C libraries, which allows it access to machine learning libraries like TensorFlow or Lib Torch.
|
|
|
|
\subsubsection*{Authentication}
|
|
|
|
The API allows users to login, which emits a token, and once logged in, manually create tokens.
|
|
While using the web application, this is done transparently, but it can also be manually done via the respective API calls.
|
|
|
|
During the login process, the service checks to see if the user is registered and if the password provided during the login matches the stored hash.
|
|
Upon verifying the user, a token is emitted.
|
|
|
|
Once a user is logged in they can then create more tokens as seen in the section \ref{sec:impl:service-auth}.
|
|
While using the API the user should only use created tokens in the settings page as those tokens are named, and have controllable expiration dates.
|
|
This is advantageous from a security perspective, as the user can manage who has access to the API.
|
|
If the token gets leaked, the user can then delete the named token, to guarantee the safety of his access.
|
|
|
|
The token can then be used in the ``token'' header as proof to the API that the user is authenticated.
|
|
|
|
|
|
\subsection{Generation and Training of Models}
|
|
|
|
Model generation happens on the API server, the API server analyses what the image that was provided and generates several model candidates accordingly.
|
|
The number of model candidates is user defined.
|
|
The model generation subsystem decides the structure of the model candidates based on the image size, it prioritizes the smaller models for smaller images and convolution networks with bigger images.
|
|
The depth is controlled both by the image size and number of outputs, models candidates that need to be expanded are generated with bigger values to account for possible new values.
|
|
It tries to generate the optimal size if only one model is requested.
|
|
If more than one is requested then the generator tries to generate models of various types and sizes, so if there is possible smaller model it will also be tested.
|
|
|
|
Model training happens in a runner, more information about runners will be explained in section \ref{impl:runner}.
|
|
|
|
% TODO explore this a bit more
|
|
Model training was implemented using TensorFlow. % \cite{tensorflow}.
|
|
Normally, when using go with machine learning, only the prediction is run in go and the training happens in python.
|
|
The training system was implemented that way.
|
|
|
|
The runner, when it needs to perform training it generates a python script tailored to the model candidate that needs to be trained, then runs the that python script, and monitors the result of the python script.
|
|
While the python script is running, it takes use of the API to inform the runner of epoch and accuracy changes.
|
|
|
|
The during train, the runner takes a round-robin approach.
|
|
It trains every model candidate for a few epochs, then compares the different models candidates.
|
|
If there is too much difference in accuracy, from the best model to the worst model, then the system might decide not to continue training a certain candidate and focus the training resources on candidates that are performing better.
|
|
Once one candidate archives the target accuracy, which is user defined, the training system stops training the models candidates.
|
|
The model candidate that achieved the target accuracy is then promoted to the model, and the other candidates are removed.
|
|
The model now can be used to predict the labels for any image that the user decides to upload.
|
|
|
|
\subsubsection*{Expandable Models}
|
|
|
|
Expandable models follow mostly the same process as the normal models.
|
|
First, bigger model candidates are generated.
|
|
Then the models are training using the same technic.
|
|
At the end, after the model candidate has been promoted to the full model, the system starts another python process that loads the just generated model and splits into a base model and a head model.
|
|
|
|
With these two separate models, the system is now ready to start classifying new images.
|
|
|
|
|
|
\subsubsection*{Expanding Expandable Models}
|
|
|
|
During the expanding process, the generation system creates a new head candidate that matches the newly added classes.
|
|
The base model, that was created in the original training process, is used with all available data to train the head candidate to perform the classification tasks.
|
|
The training process is similar to the normal training system, but this uses a different training script.
|
|
Once the model has finished training and the system meets the accuracy requirements, then makes the new head available for classification.
|
|
|
|
|
|
\subsection{Model Inference}
|
|
|
|
Model inference also runs inside a runner.
|
|
However, inference runs internally on go, instead of creating tailored scripts for training using python.
|
|
|
|
Once a classification request is received by the API, the uploaded image is checked to see if the model will accept it.
|
|
If the model is capable of accepting the image, it is temporarily saved to disk and then a classification task is created.
|
|
|
|
Eventually, a runner will pick up the classification task.
|
|
Upon pickup, the runner will load the model and run the image through it, the results are then matched with the stored classes in the database.
|
|
The system then stores the class with the highest probability of matching the image, according to the model, in the results of the task.
|
|
|
|
The user then can finally use the API to obtain the results of the model.
|
|
|
|
\subsubsection*{Expandable Models}
|
|
|
|
For expandable models, the inference step is very similar to the normal models.
|
|
The runner first loads the base model and runs the image through the model, the resultant features are then stored.
|
|
Once the features are obtained, the system then runs those features to the various possible heads, and the results are then matched with the stored classes, and the one with the height probability is then selected.
|
|
|
|
\subsection{Runner} \label{impl:runner}
|
|
|
|
Runners are the name used to reference to the software that can perform CPU or GPU intensive tasks without halting the main API server.
|
|
|
|
Architecturally, they were implemented as a controller and worker pattern.
|
|
When the application that runs the main application starts, the system creates an orchestrator, this orchestrator is a piece of software that decides what work each runner is doing.
|
|
The orchestrator runs on go routine created at startup.
|
|
During the startup, the orchestrator by obtaining values from the configuration file.
|
|
Those values define the number of local runners.
|
|
|
|
These runners, which are started up by the orchestrator, act as local runners, runners that are running on the same machine as the main server.
|
|
Local runners are useful when running the main server on a machine that also has GPU power available to it, and in testing.
|
|
|
|
Local runners, run inside a go routine, this allows the runners and the orchestrator to communicate using go channels, which are the easiest way to communicate between two go routines.
|
|
The orchestrator is constantly waiting to receive either for a timer-based event or a runner-based event.
|
|
|
|
Timer-based events happen when the orchestrator internal clock informs it of needing to check if tasks are available to run.
|
|
The time at which this clock ticks in configured in the settings of the app.
|
|
Upon receiving a timer-based event, the orchestrator then checks if there is a new task available for it to run and if there are any runners available for the task to run on.
|
|
If there are tasks available, then the orchestrator instructors the runner to pick up the task and run it.
|
|
|
|
Runner-based events happen when a runner finishes running a task or crashes while trying to do it.
|
|
Upon receiving a runner event, the orchestrator checks if it is a success or a failure message.
|
|
|
|
If it is a failure message and the runner is a local runner, then the orchestrator just restarts the runner.
|
|
Upon restart, it adds the runner to the list of available runners.
|
|
If the runner is a remote runner, the orchestrator marks the runner as failed and stops sending messages to the runner until the runner informs the service again that is available.
|
|
|
|
If the message is of success, then the orchestrator just adds the runner to the list of viable runners, independently if the runner is remote or not.
|
|
|
|
% TODO talk more about remote runners
|
|
% TODO talk about how the runner loads images
|
|
|
|
|
|
\subsection{Conclusion}
|
|
This section went into the details of how the designed was implemented.
|
|
The design was envisioned to be the best possible version of this service, but scope was restrained to the necessities of the system while it was being developed.
|
|
And possible features that would make the implemented application closer to the ideal design could have been implemented if there was higher need during the development timeline.
|
|
This will be more discussed in the section about a critical review of the work.
|
|
|
|
|
|
\pagebreak
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\section{Legal, Societal, Ethical and Professional Considerations} \label{sec:lsec}
|
|
This section will address possible legal, societal, ethical and professional issues that might arise from the deployment of the software being designed.
|
|
|
|
|
|
The Self-Assessment for Governance and Ethics (SAGE) form has addressed, and it is submitted along with the report.
|
|
|
|
\subsection{Legal Issues}
|
|
|
|
Legal issues can occur due to the data being stored by the service.
|
|
|
|
The service collects, the least amount of sensitive information, from the users who directly use the service.
|
|
That data that is collected while being sensitive is required to be able to authenticate the user, such as name, email, and password.
|
|
To safeguard that information, the system will be using industry standards to guarantee data security of that data.
|
|
|
|
|
|
Legal issues might occur due to image uploaded images. For example, those images could be copyrighted, or the images could be confidential. The service is designed to provide ways to allow users to host their images without having to host the images itself, moving the legal requirement to the management of the data to the user of the system.
|
|
|
|
\subsubsection{GDPR}
|
|
The General Data Protection Regulation (GDPR) (GDPR, 2018) is a data protection and privacy law in the European Union and the European Economic Area, that has also been implemented into British law.
|
|
The main objective of the GDPR is to minimise the data collected by the application for purposes that are not the used in the application, as well as giving users the right to be forgotten.
|
|
|
|
The application collects only personal data needed to authenticate the user, and data that is generated during the normal usage of the application.
|
|
|
|
All the data that is related to the user can be deleted.
|
|
The system will prevent any new work that is related to the data, that was requested to be deleted.
|
|
Once the there is no more work that requires the data being done, the system will remove all relevant identifiable references to that data.
|
|
|
|
\subsection{Social Issues}
|
|
The web application was designed to be easy to use and there tries to consider all accessibility requirements.
|
|
|
|
% TODO talk about this
|
|
|
|
% The service itself could raise issues of taking jobs that are currently done by humans.
|
|
% This is less problematic as time has shown that the jobs just change, instead of manually classifying the images, the job transforms from the classifying all the images that are needed to maintain and verifying that the data being input to the model is correct.
|
|
|
|
\subsection{Ethical Issues}
|
|
While the service itself does not raise any ethical concerns. The data that the service will process could raise ethical complications.
|
|
|
|
For example, if the service gets acquired by a company that also wants to use the data provided to the system for other reasons.
|
|
|
|
\subsection{Professional Issues}
|
|
As a member of the British Computer Society (BCS), it is important to follow the Code of Conduct practices. The code of conduit contains 4 key principles.
|
|
|
|
\subsubsection*{Public interest}
|
|
This project tries to consider the public health, privacy, and security of third parties and therefore follows the principle of public interest.
|
|
|
|
\subsubsection*{Professional Competence and Integrity}
|
|
This project has been an enormous undertaking that pushed the limits of my capabilities.
|
|
I am glad that I was able to use this opportunity to learn about distributed systems, image classification, go, and Svelte.
|
|
During this project, I also followed the best practices of software development such as using source control software and having an audit to tasks and issues.
|
|
|
|
\subsubsection*{Duty to Relevant Authority}
|
|
For the duration of the project, all the guidelines provided by the University of Surrey were followed.
|
|
|
|
\subsubsection*{Duty to the Profession}
|
|
During the research, design, and implementation, and report state all interactions with the supervisor of the project have been professional, respectful, and honest.
|
|
To the best of my ability, I tried to design a system that would contribute to the field.
|
|
|
|
|
|
\pagebreak
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\section{Service Evaluation} \label{sec:se}
|
|
This section will discuss how the service can be evaluated from a technical standpoint and its results.
|
|
|
|
With the goals of the project, there are two kinds of tests that need to be accounted for.
|
|
User testing tests that relate to the experience of the user while using the project and tests that quantitive test the project.
|
|
|
|
Such as accuracy of the generated models, response time to queries.
|
|
|
|
\subsection{Testing the model}
|
|
To test the system, a few datasets were selected.
|
|
The datasets were selected to represent different possible sizes of models, and sizes of output labels.
|
|
|
|
The ImageNet\cite{imagenet} was not selected as one of the datasets that will be tested, as it does not represent the target problem that this project is trying to tackle.
|
|
|
|
The tests will measure:
|
|
\begin{itemize}
|
|
\item Time to process and validate the entire dataset upon upload
|
|
\item Time to train the dataset
|
|
\item Time to classify the image once the dataset has been trained
|
|
\item Time to extend the model
|
|
\item Accuracy of the newly created model
|
|
\end{itemize}
|
|
|
|
The results will be placed in the results table.
|
|
|
|
\subsubsection{MNIST}
|
|
|
|
The MNIST \cite{mnist} dataset was selected due to its size. It's a small dataset that can be trained quickly and can be used to verify other internal systems of the service.
|
|
|
|
During testing only the 9 out 10 classes are trainged and the 10th is added during the retraining process.
|
|
|
|
\subsubsection{CIFAR-10}
|
|
|
|
The MNIST \cite{mnist} dataset was selected due to its size. It's a small dataset that can be trained quickly and can be used to verify other internal systems of the service.
|
|
|
|
During testing only the 9 out 10 classes are trainged and the 10th is added during the retraining process.
|
|
|
|
\textbf{TODO add image}
|
|
|
|
\textbf{TODO add more datasets}
|
|
|
|
\subsubsection{Results}
|
|
|
|
|
|
\textbf{TODO add more data}
|
|
|
|
|
|
\begin{longtable}{ | c | c | c | c | c | c |}
|
|
\hline
|
|
Dataset & Import Time & Train Time & Classification Time & Extend Time & Accuracy \\ \hline
|
|
MNIST & $8s$ & $2m$ & $1s$ & $50s$ & $98\%$ \\ \hline
|
|
CIFAR-10 & $6s$ & $41m 38s$ & $1s$ & $1m 11s$ & $95.2\%$ \\ \hline
|
|
\caption{Evaluation Results}
|
|
\label{tab:eval-results}
|
|
\end{longtable}
|
|
|
|
\subsubsection{Conclusions}
|
|
The service can create models that represent what the users want in a reasonable amount of time without much interaction from the user.
|
|
The models created have the target accuracy required by the users, and the amount of time it takes for the models to train and expand is reasonable and within the margins that meet the success criteria for the project.
|
|
|
|
\pagebreak
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\section{Critical Review of Project Outcomes} \label{sec:crpo}
|
|
|
|
This section will go into details to see if the project was able to achieve the goals set forth in the introduction.
|
|
|
|
The section will be analysing if the goals of the project were met, then shortcomings and improvements off the implementation will be discussed. After analysing shortcomings and improvements, possible future work that be done to the project will be discussed.
|
|
The section will end with a general statement about the state of the project.
|
|
|
|
\subsection{Project Objectives}
|
|
|
|
In the introduction section of this project, some objectives were set for this project.
|
|
|
|
By the end of the project, the developed solution can achieve the goals set forth.
|
|
|
|
\subsubsection*{A system to upload images that will be assigned to a model}
|
|
|
|
This goal was achieved.
|
|
One of the abilities of both the API and the webpage are to be able to upload images to the service.
|
|
Which means that a system was created that allows users to upload images that will be linked with a model.
|
|
|
|
\subsubsection*{A system to automatically train and create models}
|
|
|
|
This goal was achieved.
|
|
The designed server can create models based only using the data provided by the user without any human interaction.
|
|
The model creation system is not as efficient, this inefficient will be discussed more in a future subsection it could be but can still achieve the desired goal.
|
|
|
|
\subsubsection*{Platform where users can manage their models}
|
|
|
|
This goal was achieved.
|
|
A web-based platform was developed where users can manage all the data related to machine learning models that were created.
|
|
The platform that was implemented allows users to create models, upload images related to the model, and then manage the submitted classification tasks.
|
|
|
|
The platform allows managing any models easily they create with within, meaning that the developed solution can achieve the first goal of the project.
|
|
|
|
\subsubsection{A system to automatically expand models without fully retraining the models}
|
|
|
|
This goal was achieved.
|
|
A system was created that allows users to add more images and classes to models that were previously created.
|
|
And this is done without having to fully retrain the model.
|
|
|
|
\subsubsection*{An API that users can interact programmatically}
|
|
|
|
This goal was achieved.
|
|
The API implementation allows users to programmatically access the system.
|
|
The efficacy of the API is proved by its use in the front end application.
|
|
The front end application uses the API to fully control the service.
|
|
This means that everything that can be done in the frontend can be done via the API.
|
|
Which means that the API can satisfy every need that a possible user might have; therefore this goal was accomplished.
|
|
|
|
\subsection{Project Shortcomings and Improvements}
|
|
|
|
Although the project was able to achieve the desired goals, the project has some shortcomings that can be improved upon in future iterations.
|
|
This section will analyse some of those shortcoming and ways to improve the service.
|
|
|
|
\subsubsection*{Model Generation}
|
|
The model generation system is a complex, and due to all the moving parts that make the system work, it requires a large amount of to work to maintain.
|
|
It is also very inefficient due to the having to generate custom tailored python scripts, that cause the data to be reloaded every time a new a round-robin round needs to happen.
|
|
|
|
A way more efficient way is to perform all the training directly in go server.
|
|
Running the training directly in go would allow the service to be able to keep track of memory and GPU usage, move data from the GPU and CPU effortlessly between runs, and would remove uncertainty from the training system.
|
|
|
|
The model generation was originally implemented with TensorFlow, this ended up limiting the generation of the models in go as the bindings for TensorFlow were lacking in the tools used to train the model.
|
|
Using Lib Torch libraries would allow more control over data, and allow that control to be done in go, which would improve both control and speed of the process.
|
|
Unfortunately, when a version of the service was attempted to be implemented using Lib Torch, the system was too unstable.
|
|
Problems were encountered with the go bindings for the Lib Torch library or, the Lib Torch library was causing inconsistent behaviour with between runs.
|
|
That compounded with time limitations make it impossible for a Lib Torch implementation to come to fruition.
|
|
|
|
Having a full go implementation would make the system more maintainable and fast.
|
|
|
|
|
|
\subsubsection*{Image storage}
|
|
|
|
The image storage is all local, while this does not currently affect how remote runner works.
|
|
%TODO improve this
|
|
This is less problematic when the runner is on the same network as the main server, but if a possible user would like to provide their runners.
|
|
This would require a lot of bandwidth for the images to be transferred over the network every time the model needs to run.
|
|
|
|
A better solution for image storage would allow user provided runners to store images locally.
|
|
During the upload time, the API, instead of storing the images locally, would instruct the users' runner to store the images locally, therefore when the runner would need to perform any training tasks with local data instead of remote data.
|
|
|
|
This would not also not require modification of the current system.
|
|
The system was designed and implemented to be expanded.
|
|
The dataset system was designed to be able to handle different kinds of storage methods in the future, such as remote storage and Object Buckets, like Amazon S3.
|
|
|
|
\subsection{Future Work}
|
|
This section will consider possible future work that can be built upon this project.
|
|
|
|
\subsubsection*{Image Processing Pipelines}
|
|
The current system does not allow for images of different sizes to be uploaded to the system, an interesting project would be to create a new subsystem that would allow the user to create image processing pipelines.
|
|
|
|
This new system would allow users to create a set of instructions that images would go through to be added to the system.
|
|
For example, automatically cropping, scaling, or padding the image.
|
|
|
|
A system like this would add versatility to the system and remove more work from the users of the service as they don't have to worry about handling the image processing on their side.
|
|
|
|
\subsubsection*{Different Kinds of Models}
|
|
The runner system could be used to train and manage different kinds of models, not just image classification models.
|
|
|
|
If the system was modified to have different kinds of models, it would allow the users to run different kinds of models.
|
|
Such as Natural Language Processing Models, or Multi Model Models.
|
|
This would increase the versatility of the service, and it would allow users to automate more tasks.
|
|
|
|
|
|
\subsection{Conclusion}
|
|
|
|
With the increase in automation recently, having a system that allows users to quickly build classification models for their tasks, would be incredibly useful.
|
|
This project provides exactly that, a simple-to-use system that allows the user to create models with ease.
|
|
|
|
There are more features to be added to the service, that would improve the quality of the project.
|
|
The service is in a state that it would be possible to run it in a production environment, making this project successful.
|
|
|
|
|
|
\pagebreak
|
|
\section{Appendix}
|
|
\begin{figure}[h!]
|
|
\begin{center}
|
|
\includegraphics[height=0.8\textheight]{expandable_models_simple}
|
|
\end{center}
|
|
\caption{Contains an overall view of the entire system}\label{fig:expandable_models_simple}
|
|
\end{figure}
|
|
|
|
\begin{figure}
|
|
\begin{center}
|
|
\includegraphics[height=0.8\textheight]{expandable_models_generator}
|
|
\end{center}
|
|
\caption{Contains an overall view of the model generation system}\label{fig:expandable_models_generator}
|
|
\end{figure}
|
|
|
|
|
|
|
|
\pagebreak
|
|
\section{References}
|
|
\printbibliography[heading=none]
|
|
|
|
% TODO add my job title
|
|
\end{document}
|