The overall hype around IoT has unexpectedly hindered the understanding of how it works. If you ask 20 people about how it works, you will get 20 answers. Most of those answers would cover how outcomes of IoT or manifested IoT solutions work; not the way IoT works. There is a lot of technology under the hood that makes IoT possible.
The key building blocks of an IoT solution
A detailed block diagram of an IoT platform
How blocks work together in a meaningful way
A proposed approach for building our platform
These topics will help us identify how it all works, and then we can plan the building of our IoT platform in an effective way.
Let’s first discuss some of the various terminologies, which are often used interchangeably. There is a difference between an IoT solution and an IoT application. An IoT solution usually means an end-to-end product, service, or a mix of both; whereas an IoT application usually refers to IT software or a mobile application, or a combination of both. Clearly, IoT solutions encompass many more things than an IoT application. A lot of business context, customer context, and geopolitical context influence IoT solutions.
Functional blocks of an IoT solution
The Functional Blocks of an IoT Solution
At a high level, we can identify IoT solutions comprising four major functional blocks. If any of these blocks are missing, then it is not prudent to call it an IoT solution.
Devices (a.k.a. “things”) are physical sensors and actuators. They measure various parameters and translate them into electrical or digital data. These sensors are either connected to the host devices (typical for legacy upgrades) or integrated into the host devices (modern). These devices are critical nodes of an IoT application and are required to deliver full-solution functionality by acting as inputs, outputs, or both. Typical examples of such devices are thermostats, intelligent mousetraps, connected refrigerators, and so forth.
Gateways are edge devices that can communicate with the upstream system in one of two ways: with or without a gateway. Some devices have the capability to communicate directly over Internet Protocol (IP) using various communication protocols, such as REST, MQTT, AMQP, CoAP, and so forth. These capabilities are usually a result of integrated communication modules, such as Wi-Fi or GSM chips, which enable a device to connect to network gateways, such as Wi-Fi routers and mobile towers, and communicate with the upstream layer directly. In these cases, routers and mobile towers perform the job of the gateway.
However, not all devices are capable of direct Internet connectivity and do not have the necessary hardware built in. In these cases, they need to piggyback on some other device to help their data get pushed to the upstream layer. Gateways help devices do this. Usually, hardware gateways are built with dual communication technologies, which enable them to communicate with downstream devices with one type of channel and with upstream layers with another type of channel. Typical examples of such gateway capabilities include GSM and RF, GSM and Bluetooth, Wi-Fi and Bluetooth, Wi-Fi and XBee, LoRaWAN and Ethernet, and so forth. In some cases, smartphones are used as gateways, which is more prominent with Bluetooth Low Energy (BLE) devices.
In addition to providing a transport mechanism, a gateway can also provide optional functions, such as data segregation, clean up, aggregation, deduplication, and edge computing.
An IoT platform is the orchestrator of the whole IoT solution and is often hosted in the cloud. This block is responsible for communicating with downstream devices and ingesting large amounts of data at a very high speed. The platform is also responsible for storage of the data in a time series and structured format for further processing and analysis.
Depending upon the sophistication built into it, a platform may support deep data analyses and other operations. However, the core of the IoT platform is as an orchestrator of the whole system.
In most scenarios, applications are the front face of the whole solution; it must be presented to the end user in a meaningful way. These applications are desktop based, mobile based, or both. Applications also enrich the data from the platform in various ways and present it to the users in a usable format. Additionally, these applications integrate with other systems and applications at the interface level and enable interapplication data exchange. A typical example of such an operation is inventory-tracking devices equipped with tracking mobile applications to the users, and the data fed to the ERP system for stock keeping.
The Detailed Block Diagram of an IoT Platform
Block diagram of a typical IoT platform
Interconnecting arrows indicate the data and information flow between each block. Each block is indicative of the major functional component of the platform. The platform is installed on a virtual cloud machine or VPS (virtual private server) . It is highly recommended to use a Linux-based operating system, such as Ubuntu, Centos, Debian, OpenWRT, or LEDE, for better performance, security features, and overall control of the platform. The concept and block-level architecture does not change for any of these operating systems.
Edge Interface, Message Broker, and Message Bus
This module deals and talks with the physical world, especially heterogeneous devices and sensors. Since devices could be communicating over a multitude of communication technologies, such as Wi-Fi, Bluetooth, LoRaWAN, GPRS, and so forth, this module needs to cater to all of them. We can achieve this in a modular format where each type of communication protocol is dealt with separately. As an example, a Wi-Fi-capable device can be a REST API, which caters to the constrained devices. It could be an MQTT-based message broker, which enables communication in a pub/sub manner. For LoRaWAN (Long Range Wide Area Network) –based devices, there is another plugin to the main message broker, which talks with LoRaWAN network servers and performs decoding of packets.
Note
Pub-sub refers to the publish-and-subscribe paradigm of communication. It is explained in Chapter 6.
This module decouples the entire platform from devices in an effective way. Many edge interfaces and protocols are supported for modern IoT devices. Regardless of the medium of communication, network type used, and protocols in play, the message broker’s job is to consolidate the data in a unified manner and push it to the common message bus. All the other functional blocks share this message bus for further operation. The broker acts as a coordinator and consolidator of messages.
Message Router and Communication Management
Once the messages are available on the main message bus, the message may need to include more context or refinement to be useful to other modules. Some messages need feature enrichment and additional information to be appended or added separately, which depends on the context of the device deployment and application requirements. The functionality of enriching existing data messages, rebroadcasting them to the message bus, publishing additional contextual information and other messages after the main message arrives, and tagging them as appropriate is the job of the communication management module. Communication management functions coordinate with the message broker and the rule engine block and interacts with the device manager, as required.
In addition, the communication management module performs the duties of format conversions; for example, it translates data from CSV to JSON, or binary to text format, and so forth. We can also task it to perform certain operations, like deduplication of messages. Deduplication is the process of eliminating or discarding multiple duplicate messages or redundant data packets from the devices, as they may not be of any use. Deduplication schemes are dependent on device or sensor types, and we need to implement them on a case-by-case basis, although the methodology remains the same. As a communications router, this module can control further messaging and communication on the platform .
Time-Series Storage and Data Management
As the name suggests, this block stores all the received and parsed data that is available on the message bus in sequential (i.e., time-series style). While data storage is not the core function of the IoT platform, modules outside the platform handle it; although, it is an essential activity for coordination and orchestration perspective. Very often, communication and routing modules, or the message broker itself, need recent data for specific functional purposes; this storage comes in handy for all such instances.
For many IoT applications, users prefer to extract the data away from the IoT platform and store it in an application data warehouse for further processing. Therefore, it is often utilized for interim storage of the device data and is not meant for large-sized dataset storage.
Rule Engine
In my view, this is a very powerful block and provides enhanced capabilities to the platform. The rule engine is the execution block that monitors the message bus and events across the platform and takes action based on set rules.
For example, a typical rule engine function may look like this: “Trigger and broadcast alert message when the downstream device sends a data packet containing the keyword ka-boom.” The rule engine is constantly listening to the message bus broadcasts. When the communication block puts up a decoded data packet from the downstream device on to the message bus, a rule triggers. The rule engine broadcasts another message (alert) to the message bus. Since this happens all within the IoT platform and among closely coordinated modules, execution speed is quite fast.
The rule engine also helps with building modular rules for decoding and enriching existing or received data from devices, and therefore, augments the communication module’s functionality. In addition to that, it is easy to implement callbacks to other modules, applications, programs, and systems.
The REST API Interface
Restful APIs are useful for support functions and utilities that do not need constant or real-time connectivity and access. Although typically used by upstream programs and applications, downstream devices can also access these APIs when needed.
A classic example of such a use case is a temperature sensor with Wi-Fi___33 connectivity that sends readings every 15 minutes. Due to such a long time between two subsequent readings, a real-time connection or an always-on connectivity is undesired. A simple HTTP operation can do the data-sending job relatively more efficiently. In this case, the sensor can send the data over REST API to the platform. The REST API works with the message broker and communications manager to present the received data post to the main message bus; it may also use time-series database records to send back the response to the sensor. This response may contain additional information for the sensor to do its job in a certain way for the next round.
This API block can also support data aggregation and bulk operational functionalities, such as querying multiple records by the upstream application. This way, upstream applications and systems remain decoupled from the core platform blocks, thereby maintaining the partition of functions and ensuring security. Various role-based authentications can be built in for access to the API.
The REST API block can also feed into the rule engine and allow applications to configure or trigger specific rules at any given point in time. This also makes it possible for downstream devices to utilize the same functionality, which could be handy when devices need to initiate certain workflows automatically in place of application triggers. A good example is a smart lock; for instance, when there is activity at the front door that needs the homeowner’s attention when she is away from home. An upstream application may notify the user when the smart lock reports activity, and then expects the user to respond or react for further steps. If the user is not available, then the application can trigger the rule for predefined actions. If the severity of the alert is relatively high, then the device may be configured to not wait for user action or response, but directly trigger the default workflow (e.g., notifying security, etc.). These functionalities can come in handy when designing and operating an autonomous and intelligent fleet of devices.
Microservices
Besides data management, manipulation, and exchange functionalities, the IoT platform also needs certain support functions to function effectively. Services such as text messaging or email notifications, verifications, captcha, social media authentications, or payment services integration are a few examples of these auxiliary services. These services are bundled in the microservices block.
In case of frequent use of certain functionalities within the platform, it can be bundled and packaged under this block to separate it from the mainstream platform. Once separated and packaged, it then can be exposed to the blocks within and outside the platform for reuse.
Device Manager
When the platform starts to host approximately 50 or more devices, things could become difficult to manage. It becomes necessary to have some type of central control in place for managing things (a.k.a. devices). This is where the device manager block helps. It essentially provides the generic functionality of managing devices as assets. This includes listing all the devices, their active-inactive status, battery levels, network conditions, access keys, readings, stored data access, device details, session information, and other similar things.
The device manager also helps with managing over-the-air updates for a fleet of devices, or central monitoring functions for system admins. In certain use cases, devices also need access rights, and users may be assigned certain access rights to a set of devices. Management of such an accessibility matrix becomes easy with the device manager.
Application and User Management
This block provides functionalities similar to the device manager. The difference is that it provides functionalities for upstream applications and users. Typical user management functions, such as passwords and credentials, access keys, logins, and rights are managed through this block. For upstream applications and various other integrated systems, API keys, credentials, and access can be managed through the same block.
While it may appear to be more of an application-level functionality, it remains in an IoT platform’s interest to bind it as a platform function, so that it is integrated tightly with the overall architecture and set of things. IoT is the system of systems, and heterogeneous systems are a fact of this phenomenon. Letting these system functions get out of sync is the last thing that you want to happen with IoT solutions.
Is Everything from this Block Architecture Mandatory?
No. While eight of the blocks define a very well-architected IoT platform, not all of them are mandatory or necessary. A specific use case or industry vertical may define this situation differently. You may not need all blocks at the outset, and they may be added later in the life cycle of the platform development.
The core functional blocks—the device interface and message broker, the message router and communications module, data storage, device management, and the rule engine are critical for the effective functioning of an IoT platform. Other blocks—REST APIs, microservices, and application and user management—are good to have and often make life easy but are not mandatory and do not obstruct functionality of the IoT platform.
When developing our IoT platform from the ground up, we will keep these functionalities on the back burner and will only implement them if time permits and resources are available.
What Is the Proposed Approach?
To develop an IoT platform in the quickest amount of time, we will not only develop it in modular form but will also do it in an agile way. Each module will be planned with functions and features set out, developed, and then deployed on the cloud for testing. Once we test an individual module and find it to be working as expected, we can go to the next module.
As a first step, we will set up the cloud environment for the platform. This is followed by setting up the essential components to develop for our first module: the edge interface and the message broker. The logical next step is to set up time-series data storage. Then we will develop basic REST APIs for the platform, followed by message router functionality.
Some of the microservices are developed after we have set up a fundamental wireframe of the platform. We will then iterate through all of these blocks a few more times to make a stable core for the platform.
Once we are happy with the core functionalities, the rule engine can be set up, followed by the device management functions. Application and user management is reviewed at the end because it is among the non-essential modules.
Summary
In this chapter, we discussed the functional blocks of an IoT platform, and we decided on the approach that we want to take toward building our own platform. In the next chapter, we discuss the essential requirements for building a platform. The detailed specifications of required elements, and how and where to get them, are covered. Chapter 3 also expands on the functional block diagram of platforms in the context of our planned work.