fyp-report/report/review.tex

\section{Critical Review of Project Outcomes} \label{sec:crpo}

	This chapter will go into details to see if the project was able to achieve the goals set forth in the introduction. 
	The chapter will be analysing if the goals of the project were met, then shortcomings and improvements off the implementation will be discussed. After analysing shortcomings and improvements, possible future work that be done to the project will be discussed.
	The section will end with a general statement about the state of the project.

	\subsection{Project Objectives}

		In the introduction section of this project, some objectives were set for this project.

		By the end of the project, the developed solution can achieve the goals set forth.

		\subsubsection*{A system to upload images that will be assigned to a model} 

			This goal was achieved.
			One of the abilities of both the API and the webpage are to be able to upload images to the service.
			Which means that a system was created that allows users to upload images that will be linked with a model.

		\subsubsection*{A system to automatically train and create models}

			This goal was achieved.
			The designed server can create models based only using the data provided by the user without any human interaction.
			The model creation system is not as efficient, this inefficient will be discussed more in a future subsection it could be but can still achieve the desired goal.

		\subsubsection*{Platform where users can manage their models}

			This goal was achieved.
			A web-based platform was developed where users can manage all the data related to machine learning models that were created.
			The platform that was implemented allows users to create models, upload images related to the model, and then manage the submitted classification tasks.

			The platform allows managing any models easily they create with within, meaning that the developed solution can achieve the first goal of the project.

        \subsubsection*{A system to automatically expand models without fully retraining the models}

			This goal was achieved.
			A system was created that allows users to add more images and classes to models that were previously created.
			And this is done without having to fully retrain the model.

		\subsubsection*{An API that users can interact programmatically}

			This goal was achieved.
			The API implementation allows users to programmatically access the system.
			The efficacy of the API is proved by its use in the front end application.
			The front end application uses the API to fully control the service.
			This means that everything that can be done in the frontend can be done via the API.
			Which means that the API can satisfy every need that a possible user might have; therefore this goal was accomplished.

	\subsection{A retrospective analysis of the development process}

		This project was complex to implement, with many interconnected systems working together to achieve the goals of the project.
		This complexity was a result of open-ended design and scope expansion.
		If the scope of the project had been more limited, the project could have achieved higher overall results.

		While there were no technical setbacks done during the development process. 
		There were times when software updates of libraries made the implementation unusable, which slowed considerably the development velocity, as those issues required fixing.
		If actions such as creating OCI containers were done in the earlier stages of development, issues such as this could have been prevented.
		One of these software updates, made it so that images were not being able to classified.
		This then prompted me to try to use a different library to train and classify the images, but this ended up not being achievable, and ended up with just fixing the original library problem.
		While the time used to try to integrate the different machine learning library helped the project improve, most of the effort put into this possible transition as spent inefficiently.
		
		As far as tools aiding the development, this project followed industries norms by having the source code tracked in Git and issues tracked in an issue tracker.
		Which greatly helped in the development process, by having one centrailized repository of both code and known issues of that code.


	\subsection{Project Shortcomings and Improvements}

		Although the project was able to achieve the desired goals, the project has some shortcomings that can be improved upon in future iterations.
		This section will analyse some of those shortcoming and ways to improve the service.

		\subsubsection*{Model Generation}
			The model generation system is a complex, and due to all the moving parts that make the system work, it requires a large amount of to work to maintain.
			It is also very inefficient due to the having to generate custom tailored python scripts, that cause the data to be reloaded every time a new a round-robin round needs to happen.

			A way more efficient way is to perform all the training directly on go server.
			Running the training directly in go would allow the service to be able to keep track of memory and GPU usage, move data from the GPU and CPU effortlessly between runs, and would remove uncertainty from the training system.

			The model generation was originally implemented with TensorFlow, this ended up limiting the generation of the models in go as the bindings for TensorFlow were lacking in the tools used to train the model.
			Using Lib Torch libraries would allow more control over data, and allow that control to be done in go, which would improve both control and speed of the process.
			Unfortunately, when a version of the service was attempted to be implemented using Lib Torch, the system was too unstable.
			Problems were encountered with the go bindings for the Lib Torch library or, the Lib Torch library was causing inconsistent behaviour with between runs.
			That compounded with time limitations make it impossible for a Lib Torch implementation to come to fruition.
			Having a full go implementation would make the system more maintainable and fast.
		
			
		\subsubsection*{Image storage}

			The image storage is all local, while this does not currently affect how remote runner works.
			%TODO improve this
			This is less problematic when the runner is on the same network as the main server, but if a possible user would like to provide their runners.
			This would require a lot of bandwidth for the images to be transferred over the network every time the model needs to run.

			A better solution for image storage would allow user provided runners to store images locally.
			During the upload time, the API, instead of storing the images locally, would instruct the users' runner to store the images locally, therefore when the runner would need to perform any training tasks with local data instead of remote data.

			This would not also not require modification of the current system.
			The system was designed and implemented to be expanded.
			The dataset system was designed to be able to handle different kinds of storage methods in the future, such as remote storage and Object Buckets, like Amazon S3.

		\subsubsection*{User Interface}

			The user interface is simplistic, this helps new users use the program but limits what advanced users might want to do.
			The user interface also might need an overhaul as it not visually appealing.
			A future improving for this project is definitely getting a professional graphical designer that can create a better-looking and recognizable application.

	\subsection{Future Work}
		This section will consider possible future work that can be built upon this project.

		\subsubsection*{Image Processing Pipelines}
			The current system does not allow for images of different sizes to be uploaded to the system, an interesting project would be to create a new subsystem that would allow the user to create image processing pipelines.

			This new system would allow users to create a set of instructions that images would go through to be added to the system.
			For example, automatically cropping, scaling, or padding the image.

			A system like this would add versatility to the system and remove more work from the users of the service as they don't have to worry about handling the image processing on their side.

		\subsubsection*{Different Kinds of Models}
			The runner system could be used to train and manage different kinds of models, not just image classification models.

			If the system was modified to have different kinds of models, it would allow the users to run different kinds of models.
			Such as Natural Language Processing Models, or Multi Model Models.
			This would increase the versatility of the service, and it would allow users to automate more tasks.
		

	\subsection{Conclusion}

		With the increase in automation recently, having a system that allows users to quickly build classification models for their tasks, would be incredibly useful.
		This project provides exactly that, a simple-to-use system that allows the user to create models with ease.

		The implemented system is able to accept images provided by the user, then create and train that model, and then allow user to classify the images with that created model.
		To achieve this, the developed software is large and complex.
		Developing such large and complex systems comes with compromises.
		In this case the model generation, training, and classfication; and API systems were prioritized over other systems such as file management systems.

		While there are still improvements that can be made, and more features, that can be added to the service to make it event better, such as image processing pipelines and diferent kinds of models.
		The service is in a state that It could be deployed in a production enviroment and work.
		Therefore this project is successful.