In this chapter, we discussed how to deploy machine learning models, especially neural networks, to mobile and cloud platforms. We examined that, on these platforms, we usually need a customized build of the machine learning framework that we used in our project. Mobile platforms use different CPUs, and sometimes, they have specialized neural network accelerator devices, so you need to compile your application and machine learning framework in regards to these architectures. The architectures that are used for cloud machines differ from development environments, and you often use them for two different purposes. The first case is to use powerful machine configuration with GPUs to accelerate the machine learning training process, so you need to build your application while taking the use of one or multiple GPUs into account. The other case is using a cloud machine for inference only. In this case, you typically don't need a GPU at all because a modern CPU can satisfy your performance requirements.

In this chapter, we developed an image classification application for the Android platform. We learned how to connect the Java module with the native C++ library through JNI. Then, we examined how to build the PyTorch C++ library for Android using the NDK and saw what limitations there are to using the mobile version.

Finally, we implemented and deployed the HTTP image classification service for the Google Compute Engine platform. We learned how to create and configure VM instances, as well as how to configure the environment on a particular VM instance machine so that we could build our application and its third-party dependencies. We saw that different configurations for VM instances might require different runtime binaries for machine learning frameworks, so we built the PyTorch C++ library for the selected configuration. This custom build allowed us to utilize the maximum performance of the CPU-only machine. We also implemented a simple HTML client for our service for testing purposes.