andr3/fyp-report
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 3 Wiki Activity

chore: updated report
All checks were successful
continuous-integration/drone/push Build is passing
Details

Browse Source
This commit is contained in:
Andre Henriques 2024-04-28 14:06:02 +01:00
parent 5fa89aec1d
commit 15d4de3b45
1 changed files with 297 additions and 74 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

371
report/report.tex
View File

@ -1,5 +1,5 @@
%%% Preamble
\documentclass[11pt, a4paper]{article}
\documentclass[11pt, a4paper, twoside]{article}
\usepackage[english]{babel} % English language/hyphenation
\usepackage{url}
@ -7,6 +7,7 @@
\usepackage{pdfpages}
\usepackage{float}
\usepackage{longtable}
\usepackage{multicol}
\usepackage{graphicx}
\usepackage{svg}
@ -30,10 +31,35 @@
%%% Custom headers/footers (fancyhdr package)
\usepackage{fancyhdr}
\pagestyle{fancyplain}
\fancyhead{} % No page header
\fancyfoot[L]{} % Empty
\fancyfoot[C]{\thepage} % Pagenumbering
\fancyfoot[R]{} % Empty
% \fancyhead{}
\fancypagestyle{my_empty}{%
\fancyhf{}
\renewcommand{\headrulewidth}{0pt}
\renewcommand{\footrulewidth}{0pt}
}
\fancypagestyle{simple}{%
\fancyhf{}
\renewcommand{\headrulewidth}{0pt}
\renewcommand{\footrulewidth}{0pt}
\fancyfoot[L]{} % Empty
\fancyfoot[C]{\thepage} % Pagenmbering
\fancyfoot[R]{} % Empty
}
\fancypagestyle{full}{%
\fancyhf{}
\renewcommand{\headrulewidth}{0.5pt}
\renewcommand{\footrulewidth}{0pt}
\fancyfoot[L]{} % Empty
\fancyfoot[C]{\thepage} % Pagenmbering
\fancyfoot[R]{} % Empty
\fancyhead[RO,LE]{Andre Henriques}
}
\renewcommand{\headrulewidth}{0pt} % Remove header underlines
\renewcommand{\footrulewidth}{0pt} % Remove footer underlines
\setlength{\headheight}{13.6pt}
@ -45,7 +71,7 @@
\addbibresource{../main.bib}
% Write the approved title of your dissertation
\title{Classify: Image Classification as a Software Platform}
\title{Image Classification as a Software Platform}
% Write your full name, as in University records
\author{Andre Henriques\\Univerity of surrey}
@ -58,10 +84,11 @@
\pagenumbering{gobble}
\maketitle
\pagestyle{my_empty}
\begin{center}
\includegraphics[height=0.5\textheight]{uni_surrey}
\end{center}
\end{center}
\begin{center}
\monthyeardate\today
@ -70,6 +97,8 @@
\NewPage
\pagenumbering{arabic}
\pagestyle{simple}
\begin{center}
\vspace*{\fill}
\section*{Declaration of Originality}
@ -109,6 +138,8 @@
\tableofcontents
\newpage
\pagestyle{full}
\section{Introduction} \label{sec:introduction}
% This section should contain an introduction to the problem aims and obectives (0.5 page)
@ -159,15 +190,15 @@
\subsection{Project Structure}
The report on the project shows the development and designs stages of the project. With each section addressing a part of the design and development process.
\begin{longtable}{rp{0.35\textwidth} rp{0.45\textwidth}}
\hyperref[sec:introduction]{Introduction} & The introduction section will do a brief introduction of the project and its objectives. \\
\hyperref[sec:lit-tech-review]{Literature and Technical Review} & The Literature and Technical Review section will introduce some current existing projects that are similar to this one, and introduce some technologies that can be used to implement this project. \\
\hyperref[sec:sanr]{Service Analysis and Requirements} & This section will analyse the project requirements. The section will define design requirements that the service will need to implement to be able to achieve the goals that were set up. \\
\hyperref[sec:sdai]{Service Design and Implementation} & This section discusses transforming the requirements defined in the previous section and implementing them, to obtain a working application. \\
\hyperref[sec:lsec]{Legal, Societal, and Ethical Considerations} & This section will cover potential legal societal and ethical issues that might arise from the service and how they are mitigated.\\
\begin{longtable}{p{7cm} p{8cm}}
\hyperref[sec:introduction]{Introduction} & The introduction section will do a brief introduction of the project and its objectives. \\
\hyperref[sec:lit-tech-review]{Literature and Technical Review} & The Literature and Technical Review section will introduce some current existing projects that are similar to this one, and introduce some technologies that can be used to implement this project. \\
\hyperref[sec:sanr]{Service Analysis and Requirements} & This section will analyse the project requirements. The section will define design requirements that the service will need to implement to be able to achieve the goals that were set up. \\
\hyperref[sec:sd]{Service Design} & This section will discuss how a service could be designed that it matches the requirements of the service. \\
\hyperref[sec:sd]{Service Implementation} & Information on how the design of the system was turned into software is in this section. \\
\hyperref[sec:lsec]{Legal, Societal, and Ethical Considerations} & This section will cover potential legal societal and ethical issues that might arise from the service and how they are mitigated. \\
\hyperref[sec:crpo]{Critical Review of Project Outcomes} & In this section, the project goals will compare to what was achieved. Then according to the results, the project will either be deemed successful or not.
\caption{Project structure}
\label{tab:project-structure}
\end{longtable}
@ -265,7 +296,7 @@
\subsection{Conclusion}
The technical review of current systems reveal that there are current systems that exist that can perform image classification tasks, but they are not friendly in ways to easily expand currently existing models.
The technical review of current systems reveals that there are current systems that exist that can perform image classification tasks, but they are not friendly in ways to easily expand currently existing models.
The current methods that exist for image classification seem to have reached a classification accuracy and efficiency that make a project like this feasible.
@ -398,7 +429,7 @@
It also requires keeping track of computational resources that are available to it, so it does not cause deadlocks. For example, using all of its GPU recourses to train a model while there are classification tasks to be done.
The next section will go thought the process of the implementation of an application that implements a subset of this design requirements, with some limitations that will be explained.
The next section will go thought the process of the implementation of an application that implements a subset of these design requirements, with some limitations that will be explained.
\pagebreak
@ -412,16 +443,147 @@
\section{Service Design and Implementation} \label{sec:sdai}
\section{Service Design} \label{sec:sd}
This section will discuss the design of the service.
The design on this section is an ideal design solution, where no time limitations or engineering limitations were considered.
This section tries to provide a description of a designed solution that would allow for the best user experience possible.
This section will discuss the design decisions for the web application, API.
The design proposed on this section can be viewed as a scope version of this project, and the \hyperref[sec:si]{Service Implementation} section will discuss how the scope was limited so that the service would achieve the primary goals of the project while following the design.
\subsection{Structure of the Service}
The service is designed to be a 4 tier structure:
\begin{itemize}
\item{Presentation Layer}
\item{API Layer}
\item{Worker Layer}
\item{Data Layer}
\end{itemize}
This structure was selected because it allows separation of concerns, to happen based on the resources required by that layer.
The presentation layer requires interactivity of the user, and therefore it needs to be accessible from the outside, and be simple to use.
The presentation can be either implemented as webpage working directly on the server that is running the API, or it can be implemented as separate web app that uses the API to interface directly.
The API layer, is one of the most important part of the service. As it's going to be most used way to interact with the service.
The user can use the API to control their entire model process from importing, to classification of images.
The Worker layer, consists of a set of servers available to perform GPU loads.
The Data layer, consists of stored images, models, and user data.
\subsection{Interacting with the service}
As a software platform, this project requires a way for users to interact with the service.
This interface is mainly intended to be as control and monitoring interface.
The user would use the interface to set up and manage the models, then most of the interactions would happen via the API.
There are no specific restrictions on what the interface can be.
The interface can either be a webpage, a desktop application or a CLI tool.
It makes the most sense for the application to be a WEB application, since this project is software as a service.
Most of the interactions will be made by the users' services programmatically via the API, which will be an HTTPS REST API.
Independently of the kind of the application is it needs to allow users to fully control their data in an easy to use and understand way.
The application should allow users to:
%TODO add more
\begin{multicols}{2}
\begin{itemize}
\item Manage access tokens.
\item Upload images for training.
\item Delete images.
\item Request training model.
\item Delete models.
\item Classify Images.
\item See previous classification results
\item Keep track of model accuracy
\end{itemize}
\end{multicols}
\subsection{API}
As a software as a service, one of the main requirements is to be able to communicate with other services.
The API provides the simplest way for other services to interact with this service.
The API needs to enable users to perform any task that are required for the service is to work.
It needs to be able to:
\begin{multicols}{2}
\begin{itemize}
\item Upload images for training.
\item Delete images.
\item Classify Images.
\item See previous classification results
\end{itemize}
\end{multicols}
The API should be implemented to be able to handle large amounts of simultaneous requests, and respond to those requests as fast as possible.
API should be implemented such that it can be expanded easily, so that future improvements can happen.
The API should be consistent and easy to use, information on how to use the API should also be available to possible users.
As it was mention in \ref{sec:anal-api} most services use REST JSON APIs to communicate with each other. Therefore, to make this service as compatible with each other services, this service should also implement an REST JSON API.
\subsection{Generation of Models}
The service should use any means available to generate models, such means can be:
\begin{multicols}{2}
\begin{itemize}
\item Templating.
\item Transfer Learning.
\item Database Search.
\item Pretrained Models with classification heads.
\end{itemize}
\end{multicols}
\subsection{Models Training}
% The Training process follows % TODO have a flow diagram
Model Training should be independent of image classification. A model training should not affect any current classification. The system could use multiple ways to achieve this such as:
\begin{multicols}{2} % TODO think of more ways
\begin{itemize}
\item Separating the training to different machines.
\item Control the number of resources that training machine can utilize
\item Control the time when the shared training and inference machine can be used for training.
\item Allow users to have their own ``Runners'' where the training tasks can happen.
\end{itemize}
\end{multicols}
\subsection{Conclusion}
This section introduced multiple possible designs options for a service, that intends to achieve automated image classification, can follow to implement a robust system.
The next section will be discussing how the system was implemented and which of the possible design options was chosen when implementing the system.
\pagebreak
\section{Service Implementation} \label{sec:si}
This section will discuss how the service followed some possible designs to achieve a working system.
The design path that was decided matches what made sense for the scale and needs of the project.
\subsection{Structure of the Service}
The structure of the service matches the designed structure as it can be seen \ref{fig:simplified_service_diagram}.
\begin{figure}[h!]
\centering
\includegraphics[height=0.4\textheight]{system_diagram}
@ -429,30 +591,17 @@
\label{fig:simplified_service_diagram}
\end{figure}
The service is designed to be a 4 tier structure:
\begin{itemize}
\item{Presentation Layer}
\item{API Layer}
\item{Worker Layer}
\item{Database Layer}
\end{itemize}
The implementation contains: Web App; Web server, which serves the Web App; API; Training Runners; Model Runners;
This structure was selected because it allows separation of concerns to happen based on the resources required by that layer.
The implementation contains an extra nginx reverse proxy server that allows for the API and the webpage to be accessible from them same domain.
The presentation layer requires interactivity of the user, and therefore it needs to be accessible from the outside, and be simple to use.
The presentation layer consists of a webpage that interacts with the API layer, to manage both the resources allocated to users and administrators of the system.
More details of the implementation can be found in \ref{web-app-design}.
The rest of this section will go into details on how every tier of the structure was implemented.
The API layer, controls the system, it's the interface that both the webpage and users' servers used to interact with the system.
The Worker layer, consists of a set of servers available to perform GPU loads.
\subsection{Web application} \label{web-app-design}
\subsection{Web Application} \label{web-app-design}
The web application (WEB App) is the chosen GUI to control the service.
This subsection discusses details of the workflows and implementation of the application.
This subsection discusses details of the user flows and implementation of the application.
\subsubsection*{Implementation Details}
@ -492,12 +641,17 @@
This is sent to the server and stored in a user account.
The Password is stored hashed using bcrypt \cite{bycrpt}.
In the future other methods of authentication might be provided; like using Googles' OAuth.
In the future, other methods of authentication might be provided; like using Googles' OAuth.
Once logged In, the user will be able to use the application and manage tokens that were emitted for this application.
This allows the user to manage what services have access to the account and the usage that those services have used.
This allows the user to manage what services have access to the account. % and the usage that those services have used.
The User can emit new tokens that can be used in the users services to request the classification of images.
On the web app, the user can manage existing tokens.
Guaranteeing that only the clients that should be accessing the information are.
In the management screen, the user can remove, and create tokens.
\textbf{TODO add image}
\subsubsection*{Model Management}
@ -514,22 +668,54 @@
In this step, the user uploads a sample image of what the model will be handling.
This image is used to define what the kinds of images the model will be able to intake.
\textbf{TODO add image}
The user is then shown the model page, which contains all the information about a model.
This page will contain some tabs, each page gives different inside about the model.
The main page is designed to contain only actions relevant to the task it is trying to achieve.
For example, if there are no images added to the model, the user will be prompted to add an image.
Or if the model has been trained and the user can submit webpages, then the user will have an option to submit and image.
\textbf{TODO add image}
Currently, the system does not support resizing of images that are different from the one uploaded at this step during evaluation.
This was done to guarantee that the image that the user want to classify is unmodified.
This was done to guarantee that the image that the user wants to classify is unmodified.
Moving the responsibility of cropping and resizing to the user.
In the future, systems could be implemented that allow the user to select how an image can be cropped.
The second step is uploading the rest of the dataset.
This can be done via the main page or via the dataset tab that becomes available when the data of the model is first uploaded.
In this tab, the user can add and remove images, as well as creating new classes for the model.
The page also shows some useful information, such as the distribution.
This information can be useful to more useful users that might decide to gather more data to balance the dataset.
The next step in the training process is for the user to upload the rest of the dataset.
The user uploads a zip file that contains a set of classes and images corresponding to that class.
That zip file is processed and images and classes are created.
Alternatively, the user can use the API to create new classes and upload.
This process was original slow as the system did not have the capabilities to parallelize the process of importing the images, but this was implemented, and the import process was improved.
The improved process now takes a few seconds to process and verify the entirety of the dataset, making the experience for the end user better.
Alternatively, the user can use the API to create new classes and upload images.
\textbf{TODO add image}
After all the images that are required for training are uploaded, the user can go to the training step.
This step will appear both in the main tab of model page. Once the user instructs the system to start training, the model page will become the training page, and it will show progress of the training of the model.
During this step, the system automatically trains the model.
After the system trains a model that meets the specifications set by the user, the system will make the model available for the user to use.
After the system trains a model that meets the specifications set by the user, the service will make the model available for the user to use.
When the model is finished training, the user can use the model to run inference tasks on images.
To achieve this, the user can either use the API to submit a classification task or use the tasks tab in the web platform.
In the tasks tab, the user can see current, previous tasks.
The users can see what tasks were performed and their results.
The user can also inform the service if the task that was performed did return the correct results.
This information can be used to keep track of the real accuracy of the model.
The information can be used to see if the model needs refinement.
The regiment would use the new data that was uploaded with the new data that was uploaded.
\subsubsection*{Advanced Model Management}
@ -543,9 +729,16 @@
The diagram \ref{fig:simplified_model_advanced_diagram} shows the steps that the user takes to use a model.
The steps are very similar to the normal model management.
There exists a new step where the user can upload new images and create new classes, then the user can request the retraining of the model.
The user would follow all the steps that are required for normal model creation and training.
During the expanding and training of new classes, the user can still use the inference step.
At the end of the process the user will be able to add new data to the model and retrain it.
To achieve that, the user would simply to the data tab and create a new class.
Once a new class is added, the webpage will inform the user that the model can be retrained.
The user might choose to retrain the model now or more new classes and retrain later.
\textbf{TODO add image}
During the entire process of creating new classes in the model and retraining the model, the user can still perform all the classifications tasks they desire.
\subsection{API}
@ -553,34 +746,47 @@
The API provides the simplest way for other services to interact with this service.
The API provides a various HTTPS REST JSON endpoints that will allow any user of the service to fully control their model using only the API.
Information about the API is shown around the web page so that the user can see information about the API right next were the user would normally do the action providing a good user interface.
As the user can get information about right where they would normally do the action.
\textbf{TODO add image}
\subsubsection*{Implementation Details}
The API will run a go \cite{go} http server.
This server will take JSON and multipart form data requests, those requests will be processed, and they will respond with JSON.
A go \cite{go} http server will run the application.
This server will take JSON and multipart form data requests, the requests are processed, and answered with a JSON response.
The multipart requests are required due to JSON's inability to transmit binary data, which will make the uploading of images extremely inefficient.
Those images would have to be transformed into binary data and then uploaded as a byte array or encoded as base64 and uploaded.
Either of those options is extremely inefficient.
Therefore, there is a need to use multipart form requests are required to allow the early uploading of binary files.
Therefore, there is a need to use multipart form requests are required to allow the easy uploading of binary files.
Go was selected as the language to implement the backend due to various of its advantages.
Go has extremely fast compilations which allows for rapid development, and iteration.
It has a very minimal runtime which allows it to be faster, than heavy runtime languages such as JavaScript.
It is also a simple language, which helps maintain the codebase.
The Go language integrates well with C libraries, which allows it access to machine learning libraries like TensorFlow.
% TODO cite cgo tensorflow and torch
The Go language integrates well with C libraries, which allows it access to machine learning libraries like TensorFlow or Lib Torch.
\subsubsection*{Authentication}
For a user to be authenticated with the server, it must first log in.
During the login process, the service checks to see if the user is registered and if the password provided during the login matches the stored hash.
Upon verifying the user, a token is emitted.
The tokens can also be created in the settings page.
The advantage of creating the tokens in the settings page is that they are named, and their lifetime is controllable.
That token can be used as the header ``token'' as proof that the user is authenticated.
\subsection{Generation of Models}
\subsection{Generation and Training of Models}
In able to be able to provide specialized models to the user the service needs to be able to generate the models first.
The service requires the generation of models \ref{fig:expandable_models_generator}.
@ -607,21 +813,32 @@
Since the AutoML approach would be more computational intensive, it would be less desirable to run. Therefore, the approach would be for the database search to happen first, where known possibly good models would be first tested. If a good model is found, then the search stops and if no model is found, the system would resort to AutoML to find a suitable model.
\subsection{Models Training}
% The Training process follows % TODO have a flow diagram
\subsection{Model Training}
% The Training process follows % TODO have a flow diagram
The training of the models happens in a secondary Training Process(TP).
The training of the models happens in a secondary Training Process(TP).
Once a model candidate is generated, the main process informs the TP of the new model.
The TP obtains the dataset and starts training.
Once the model finished training, it reports to the main process with the results.
The main process then decides if the model matches the requirements.
If that the case, then the main process goes to the next steps; otherwise, the service goes for the next model that requires training.
Once a model candidate is generated, the main process informs the TP of the new model.
The TP obtains the dataset and starts training.
Once the model finished training, it reports to the main process with the results.
The main process then decides if the model matches the requirements.
If that the case, then the main process goes to the next steps; otherwise, the service goes for the next model that requires training.
The TP when training the model decides when the training is finished, this could be when the training time has finished or if the model accuracy is not substantially increasing within the last training rounds.
The TP when training the model decides when the training is finished, this could be when the training time has finished or if the model accuracy is not substantially increasing within the last training rounds.
During the training process, the TP needs to cache the dataset being use.
This is because to create one model, the service might have to generate and train more than one model, during this process, if the dataset is not cached then time is spent reloading the dataset into memory.
During the training process, the TP needs to cache the dataset being use.
This is because to create one model, the service might have to generate and train more than one model, during this process, if the dataset is not cached then time is spent reloading the dataset into memory.
\subsection{Model Inference}
\subsubsection*{Implementation Details}
The model definitions are generated in the go API and then stored in the database.
The runner then loads the definition from the API and creates a model based on that.
\subsubsection*{Inferencing images}
TODO
\subsection{Conclusion}
This section discussed the design and implementation specifications for the system.
@ -659,16 +876,17 @@
That data that is collected while being sensitive is required to be able to authenticate the user, such as name, email, and password.
To safeguard that information, the system will be using industry standards to guarantee data security of that data.
Legal issues might occur due to image uploaded images. For example, those images could be copyrighted, or the images could be confidential. The service is designed to provide ways to allow users to host their images without having to host the images itself moving the legal requirement to the management of the data to the user of the system.
Legal issues might occur due to image uploaded images. For example, those images could be copyrighted, or the images could be confidential. The service is designed to provide ways to allow users to host their images without having to host the images itself, moving the legal requirement to the management of the data to the user of the system.
\subsubsection{GDPR}
The General Data Protection Regulation (GDPR) (GDPR, 2018) is a data protection and privacy law in the European Union and the European Economic Area, that has also been implemented into British law.
The main objective of the GDPR is to minimise the data collected by the application for purposes that are not the used in the application, as well as giving users the right to be forgotten.
The application collects only personal data need to authenticate the user, and data that is generated during the normal usage of the application.
The application collects only personal data needed to authenticate the user, and data that is generated during the normal usage of the application.
All the data that is related to user can be deleted.
The system will prevent any new work that is related with the data, that was requested to be deleted.
All the data that is related to the user can be deleted.
The system will prevent any new work that is related to the data, that was requested to be deleted.
Once the there is no more work that requires the data being done, the system will remove all relevant identifiable references to that data.
\subsection{Social Issues}
@ -682,7 +900,7 @@
\subsection{Ethical Issues}
While the service itself does not raise any ethical concerns. The data that the service will process could raise ethical complications.
For example, if the service gets acquired by a company that also wants to use the data provided to system for other reasons.
For example, if the service gets acquired by a company that also wants to use the data provided to the system for other reasons.
\pagebreak
@ -700,13 +918,13 @@
\section{Evaluating the Service}
This section will discuss how the service can be evaluated from a technical standpoint and its results.
With the goals of the project there are two kinds of tests that need to be accounted for.
With the goals of the project, there are two kinds of tests that need to be accounted for.
User testing tests that relate to the experience of the user while using the project and tests that quantitive test the project.
Such as accuracy of the generated models, response time to queries.
\subsection{Testing the model}
To test the system a few datasets were selected.
To test the system, a few datasets were selected.
The datasets were selected to represent different possible sizes of models, and sizes of output labels.
The ImageNet\cite{imagenet} was not selected as one of the datasets that will be tested, as it does not represent the target problem that this project is trying to tackle.
@ -724,17 +942,22 @@
\subsubsection{MNIST}
The MNIST\cite{mnist} dataset was selected due to its size. It's a small dataset that can be trained quickly and can be used to verify other internal systems of the service.
The MNIST \cite{mnist} dataset was selected due to its size. It's a small dataset that can be trained quickly and can be used to verify other internal systems of the service.
\subsection{Results}
\subsubsection{Results}
\begin{longtable}{ | c | c | c | c | c | c |}
\hline
MNIST & 0ms & 0ms & 0ms & 0ms & $98\%$ \\ \hline
Dataset & Import Time & Train Time & Classification Time & Extend Time & Accuracy \\ \hline
MNIST & 0ms & 0ms & 0ms & 0ms & $98\%$ \\ \hline
\caption{Evaluation Results}
\label{tab:eval-results}
\end{longtable}
\subsubsection{Conclusions}
The service can create models that represent what the users want in a reasonable amount of time without much interaction from the user.
The models created have the target accuracy required by the users, and the amount of time it takes for the models to train and expand is reasonable and within the margins that meet the success criteria for the project.
\pagebreak
@ -748,7 +971,7 @@
\section{Results} % TODO change this
\section{Critical Review of Project Outcomes} \label{sec:crpo}
As it was stated during the introduction, this project has multiple objectives.