Chapter 3. Edge-to-Enterprise Analytics

The technical challenges brought on by the advent of the IoT require new system architectures in order to capture and process the data. The first challenge is capturing all of the data from the edge; in other words, gathering information from the IoT and devices and transferring it to the enterprise datacenter. When you consider there are potentially millions of devices involved, it quickly becomes apparent that making direct connections from the edge devices to the enterprise datacenter is not a workable solution.

Additionally, depending on the use case, the number of edge devices can change, up or down, over time as new devices come online and others are retired. Even devices that will persist in sending data can drop off the network and come back online later. To address this, a transfer layer (commonly called an IoT gateway) is inserted between the edge devices and the enterprise datacenter, as depicted in Figure 3-1.

Figure 3-1 is, admittedly, a very simple diagram, but it illustrates the point. The message brokers between the edge and the enterprise act as a temporary holding area. Edge devices transfer data to the brokers as they generate it, and the enterprise consumes it from the brokers. The enterprise communicates only with the brokers. Each edge device needs to know only the address of the broker. The message broker needs to be highly available, fault-tolerant, resilient, and capable of handling large volumes of data (Apache Kafka comes to mind).

Figure 3-1. Layers from edge to enterprise

In addition to dealing with the variability in the number of IoT devices, this approach has other benefits. It creates a buffer between the devices generating the data and the enterprise. Thus, if the enterprise systems go offline for scheduled maintenance, the brokers continue to accept data from the edge devices, queueing it up for when the enterprise systems come back online.

This also deals with issues related to geography. for cases in which devices are spread over great distances or even all over the world, many brokers can be used, with the IoT devices communicating with whatever broker is closest (and thus has the lowest latency).

Another technical challenge is dealing with cases in which a fast response is required by the device generating the data. There often is not time to ship the data over the network to the enterprise datacenter, analyze it, formulate a response and send it back.

In this case, the analysis and response must occur closer to where the data is being generated. In Figure 3-2, IoT devices send data to an edge node (or an IoT gateway system) where analysis and response occur in real time. The edge node then sends the data to the enterprise datacenter. Here, it is aggregated with all of the data from every device and analyzed for use in understanding the business, and in the creation, modification or refinement of the analysis to be performed by the edge node.

Figure 3-2. Edge-to-enterprise analytics

The edge node is close to the edge devices such that limited bandwidth or network latency do not affect the timely arrival of the data and receipt of any response. The edge node can be a server dedicated for this purpose, or even embedded in the network router. The latter is a particularly elegant solution because that device is needed anyway to route the network traffic.

Finally, the edge node can be programmed to first filter¹ the data before sending it to the enterprise datacenter (note that this will be done via a message broker—not shown in the diagram for clarity’s sake). This addresses the issue for which it is impractical to send everything due to the sheer volume of data involved as well as cases for which there is a lot of “noise” in the data, which will just be deleted after it reaches the enterprise data center.

In the enterprise, typically at the enterprise data center, all of the data from all the edge devices is combined to create a global context. This context is used for real-time response on the macro scale, as well as traditional historical analysis. It is also used to build and refine the analyses that are implemented at the edge.

We now turn our attention to how the explosion in connections to the internet and how applied analytics, at the edge and in the enterprise, will affect our cities, the energy sector, manufacturing, healthcare, and transportation.