Machine learning is a technology that has traditionally resided in the cloud and has powered everything from search engines to your recommended playlist on your favorite streaming service. Running a machine learning inference often required a lot of processing power, especially to train the algorithm. Machine learning, therefore, has mostly lived in the cloud and has been completely off the radar to embedded system developers until just recently.

Machine learning started to find its uses for IoT edge devices, but the processing wasn't being done on the edge device. The first machine learning applications used the edge device as a sensor node to collect the information that was needed, the processing was done in the cloud, and then the result was transmitted back down to the edge device. While this saved the need to have a heavy-duty processor at the edge, machine learning is beginning to shift from the cloud to the edge for several reasons, including the following:

• Bandwidth: As more devices are connected to the IoT, it becomes unrealistic to have trillions of devices constantly connecting back to the cloud and transferring large amounts of data. There is limited bandwidth that can be used through the internet and, more importantly, that bandwidth costs money. The less bandwidth that is used the better, and this is helping to drive machine learning from the cloud to the edge.

• Power: Power is an important factor because when a data packet is sent to the cloud, devices may have to stay awake in order to receive the processed response. This means that the device will be unable to go into a low-power state and will use more energy. While this may not be an issue for a device that is connected to the power grid, many IoT devices are battery-operated devices, and the longer a Wi-Fi module is powered up, the faster that battery will be drained. Processing as much as possible at the edge can therefore potentially improve the energy usage for the device.
• Cost: The cost to run a machine learning algorithm in the cloud can also get expensive. Cloud-based machine learning has a monthly cost associated with it. Developers have to pay for access to the following:
  • The cloud
  • The bandwidth they use
  • The specific machine learning features they use
  • The number of devices connecting to that service, and so on.

  Moving to the edge won't necessarily remove all these costs, but it can help to potentially decrease them dramatically.

• Latency: This is an important issue to consider when using cloud-based machine learning. Every time an edge device has to send data to the cloud and wait for a response, there is going to be a non-deterministic latency associated with that transaction. Network communications are inherently inconsistent, which means that running a real-time edge node that meets its deadlines can be nearly impossible. Again, the more that can be done at the edge, the lower the latency and the response times.
• Reliability: The reliability for an edge device can also be improved by removing its reliance on the cloud. If the cloud were to go down, the connection was to be severed, or even the cloud APIs were updated, the edge device's reliability could easily be compromised. The fewer device resource dependencies there are, the better it is for the system and the less complicated it will be to design and test.
• Security: Finally, the security for a system can be improved by keeping the system as self-contained as possible. Pushing extra data up to the cloud and processing it there increases the attack surface for a would-be hacker. For IoT devices, security is a critical component in system design.

As you can see, the need to move machine learning from the cloud to the edge is quite important to embedded system developers. Now that we understand this importance, let's define some requirements for our own machine learning project.