The “Things” in the Internet of Things are edge devices connected to centralized computing resources through external networks. Sometimes, the devices are also connected to each other at the edge via local networks. Data might be gathered from sensors in state-of-the-art complex devices such as smart meters, industrial barcode and RFID readers, programmable logic controllers (PLCs), and robotic machinery. Sometimes, the equipment that you need to gather data from lacks sensors in critical locations. Simple sensor kits might be applied to legacy equipment lacking needed sensors. Such kits might also be installed in new locations such as upon city infrastructure, buildings, mobile vehicles, or even drones.
A variety of edge devices can be connected to the Microsoft cloud backend, most often through Azure IoT Hub (explored in greater depth in the next chapter). Communications protocols supported by the IoT Hub include the Advanced Message Queuing Protocol (AMQP), the Message Queue Telemetry Transport (MQTT) protocol, and HTTPS. However, devices using other protocols can be deployed by providing protocol translation at the edge.
Edge sensor and device selection
The Azure IoT edge runtime
The Azure IoT device catalog
Edge Sensor and Device Selection
Deploying a successful IoT solution often begins with an assessment of the data needed and types of actions that must be taken in response. You might begin by evaluating whether existing edge devices gather the right data and whether sensors are in the right locations to get the measurements needed.
If the data gathered is inadequate, you could then be faced with deciding whether to retrofit existing equipment with additional sensors or perform wholesale replacement of equipment with newer IoT-ready versions. When evaluating the acquisition of new equipment, criteria considered often includes the accuracy of measurements provided as well as cost.
Additional physical considerations can include component durability, physical size, and mounting options. Critical components in the edge devices might need to function in difficult environments so understanding normal operating temperature ranges, acceptable moisture levels, the presence of acids or chemicals, and amount of electrical noise can play a part in selection.
Devices and sensors could require minimal voltage and be battery powered. Or they might need local power drops to be adequately powered. In an environment where high availability is a must, an uninterrupted power supply (UPS) could be required.
Physical and software security provided by the proposed devices and deployment in secured areas also warrants consideration. Communication protocols supported by the devices and their connectivity (wired or wireless) should be consistent with the planned overall architecture. Evaluations that touch on all these considerations are key to simplifying integration and ongoing operational management.
The support model for devices should be developed and tested as part of scalability testing prior to full deployment. Key areas that receive testing should include initial provisioning, centralized management of connections, and troubleshooting based on diagnostics. The plan should include details regarding ongoing engineering support after deployment.
Edge devices are sometimes connected directly to the Azure IoT Hub in the cloud for communications. More often, they are networked together in their remote location via local area networks (such as over Ethernet) or via wireless connections (such as Wi-Fi, 3G, 4G LTE, 5G, and Bluetooth). Remote IoT networks at the edge are connected locally to an IoT gateway device to transmit and receive data to/from the Azure IoT Hub in the cloud.
As noted previously in this book, the Azure IoT Hub supports MQTT, MQTT over WebSockets, AMQP, AMQP over WebSockets, and HTTPS for connectivity into the Azure cloud. Other protocols, such as OPC UA, are supported through protocol translation. Most organizations seek to standardize on a single protocol for device connection to the cloud to simplify their architecture.
IoT protocols in OSI and TCP/IP models
AMQP is a popular choice where field and cloud gateways are deployed since connection multiplexing is supported. Multiple devices with unique credentials communicating using MQTT and HTTPS cannot share the same TLS connection. However, the MQTT and HTTPS protocols can run in devices with fewer resources, so they are sometimes your only option for such devices.
WebSockets come into consideration when ports needed by AMQP (using port 5671) or MQTT (using port 8883) are closed to non-HTTPS protocols. Hence you would deploy AMQP over WebSockets or MQTT over WebSockets in such a scenario if you did not want to rely on HTTPS as the protocol.
AMQP and MQTT both support a server push of messages from the cloud to the device. HTTPS devices must poll the server to determine if messages are present, thus generating additional traffic over the network. For HTTPS devices, it is recommended that such polling should be limited to frequencies of every 25 minutes or more.
AMQP and MQTT are binary protocols. Payloads are more compact when using these protocols compared to HTTPS giving yet another reason why they are often favored in IoT deployment.
For Industrial Internet of Things (IIoT) implementations , the OPC UA protocol has gained a great deal of acceptance. Microsoft was an early supporter and adopter of OPC UA. For example, OPC UA is the industrial protocol in Microsoft’s Connected Factory solution accelerator. OPC UA is implemented through protocol translation in the Azure IoT Edge module, as described in the next section of this chapter. Microsoft provides a protocol translation sample in GitHub that provides useful guidance for setup.
OPC UA deployed in manufacturing
The OPC UA Publisher also supports direct calls in the IoT Edge that provide general information, diagnostic information on OPC sessions, subscriptions, monitored items, and diagnostic information on IoT Hub messages and events. The last 100 lines of the log that is maintained can be read.
HPE IoT System EL20. A rugged performance-optimized edge gateway with compute capabilities designed for collecting and transmitting data in high-volume deployments.
Siemens SIMATIC S7-1500 Advanced Controller. A programmable logic controller (PLC) with integrated motion controls.
Mitsubishi Electric MELSEC iQ-R series PLC. A multidiscipline PLC with motion CPUs to control positioning, speed, torque, advanced synch, and other functions.
Leuze Electronic IoT Ready Barcode Reader. An industrial quality barcode scanner.
Beckhoff IoT Controller. Features temperature and fan control and speed sensors to demonstrate the ability to control cooling.
HARTING Ha-VIS UHF RFID reader RF-R310. An industrial RFID reader.
Rockwell Automation Control Logic Controller. A PLC for multi-axes motion control often used in mixing ingredients and filling containers.
Schneider Electric IoT Controller. A PLC sometimes used in measuring liquid levels in tanks and for similar applications.
Honeywell Elster Alpha Smart AS300P. A device that enables an advanced metering infrastructure (AMI) and manages data from electricity, gas, and water meters.
OPC UA connected equipment in Microsoft Technology Centers
There are other protocols that might be supported in your industrial equipment, especially in equipment that predates OPC UA. For example, Modicon (now Schneider Electric) published Modbus as a protocol in the late 1970s, and it gained some widespread adoption. Prior to OPC UA, Microsoft promoted the usage of its OPC Classic.
The Azure IoT Edge Runtime
Microsoft’s Azure IoT Edge is runtime software that can perform communications and module management functions on edge devices. From an edge device, communications can occur to further downstream devices, modules present in the device, and to the Azure cloud. Management functions available include the ability to install and update workloads on the device, maintain IoT Edge security standards on the device, ensure that IoT Edge modules are running, and report on module health in the device enabling monitoring.
The IoT Edge runtime consists of two modules: the IoT Edge Hub and the IoT Edge Agent. The IoT Edge Hub serves as a local proxy for the Azure IoT Hub and supports MQTT and AMQP as protocols to the IoT Hub. It optimizes connections to the cloud and relies on the Azure IoT Hub for authentication when connections are first established.
The IoT Edge Agent instantiates modules, ensures that they are running, and reports on status back to the Azure IoT Hub. The IoT Edge Agent uses its module twin to store configuration data.
Deployment of the IoT Edge runtime begins with deployment of an Azure IoT Hub (described in the next chapter). You then register an IoT Edge device to the Azure IoT Hub and install and start the IoT Edge runtime on the device. It is then possible to deploy a module remotely to the IoT Edge device.
Create the Azure resources.
Set up Azure DevOps services.
Define continuous integration (using tokens).
Create a release pipeline and smoke test.
Add a scalable integration test to the release pipeline.
Monitor the devices with Application Insights.
The IoT Edge Device As a Gateway Device
IoT Edge devices can serve as transparent, protocol translation, or identity translation gateways to the Azure IoT Hub. The gateways can be used to provide downstream device isolation, connection multiplexing, traffic smoothing, and limited offline support (temporarily storing messages and twin updates at times when they cannot be delivered to the IoT Hub).
IoT Edge device as a transparent gateway
IoT Edge device as a protocol translation gateway
IoT Edge device as an identity translation gateway
Deployment of Containers
As noted in Chapter 2, Azure Machine Learning, certain Azure Cognitive Services, Azure Event Grid, Azure Functions, Azure Stream Analytics, and SQL Server can be deployed in the IoT Edge within Docker containers. Docker containers are lightweight executable software packages that include needed system libraries and settings, system tools, runtime, and code needed to run the Azure service. Code is typically built in Azure or Visual Studio and placed into a Docker image. The Docker containers are registered to an Azure container registry.
Deployment of the containers onto the device relies on the two modules present in the Azure IoT Edge runtime – the IoT Edge Hub and the IoT Edge Agent. The IoT Edge Agent instantiates modules, ensures that they continue to run, and reports the status of modules back to the IoT Hub. The IoT Edge Agent uses its module twin to store configuration information.
A deployment manifest is created as a JSON document and stored on the IoT Hub. It describes the IoT Edge Agent module twin including the container image for each module, credentials needed to access private container registries, and instructions on how modules should be created and managed. The deployment manifest also describes the IoT Edge Hub module twin defining how messages flow between modules and to the IoT Hub.
Upon device startup, an IoT Edge security daemon starts the IoT Edge Agent. This agent retrieves its module twin from the IoT Hub and the contents in the deployment manifest. The named modules are then started on the device as module instances.
Key modules in container deployment at the edge
In the spring of 2019, Microsoft began to preview Azure SQL Database Edge, a variation of the Azure SQL Database designed to run on ARM-based or x64-based edge devices that are configured with only about 1 GB of memory. The installation procedure for the database onto the device is through containers.
Azure SQL Database Edge has a streaming engine built on Azure Stream Analytics that enables queries to connect through the Azure IoT Hub or Azure Event Hub on to backend Azure cloud services. Bidirectional data movement is supported. The database can be utilized in disconnected as well as cloud-connected scenarios.
For local machine learning applied to data in the edge device, both R and Python are supported through external procedures. The predictive functionality provided by Azure SQL Database Edge is the same as that in SQL Server. There is support for processing and storing time series, graph, and JSON data. The database can be accessed using popular business intelligence tools such as Microsoft’s Power BI.
Azure IoT Edge and Device Security
Microsoft defines an Azure IoT Edge security framework that provides a foundation for secure deployment of devices. Anchoring the framework is secure silicon that is a tamper-resistant root of trust. At middle layers of the foundation, highly assured booting of the device and secured execution environments relying on proper authentication, authorization, and attestation become critical. Runtime integrity monitoring of applications completes the foundation.
Realizing security principles in IoT Edge devices
Foundation Principles | How the Principles are Realized |
|---|---|
Application runtime integrity monitoring | Azure IoT Edge |
Systems resource access control and privileged actions | Secured operating system on device |
High assurance bootstrapping and resiliency | Azure IoT Edge Security Manager |
Tamper resistant root of trust/secure silicon | Secure silicon/device provided by manufacturer |
At the root, a best practice is to specify devices that meet minimal physical requirements. Physical features, such as USB ports, are to be avoided if they are considered optional as they can expose the device to attack. Devices might also be protected from physical tampering through secure enclosures and other means. A Software Guard Extension (SGX) in device processors can assure normal processes are in enclaves that can’t be overridden.
Secured and measured bootstrapping of the device
Device identity provisioning and transition of trust
Hosting and protection of the Device Provisioning Service
Securely provisioning IoT Edge modules with unique identities
Serving as a gatekeeper to device hardware root of trust
Monitoring the integrity of IoT Edge operations at runtime
Device manufacturers are responsible for providing Trusted Platform Module (TPM) drivers, the TPM itself, and any custom hardware security modules (HSMs) and drivers. The Trust Computing Group provides specification of a Device Identifier Composition Engine (DICE) that can be used for creating cryptographic representations of device identities.
Further protection is provided by securing the operating system on the device through systems resource access control and establishing privileged actions. Runtime integrity monitoring provided by the Azure IoT Edge software helps protect the general computing environment for the device.
Authentication is used in the foundation to assure that only trusted parties, modules, and devices have access. Certificate-based authentication is derived from standards governing public key infrastructure (PKi) by the Internet Engineering Task Force (IETF) and is the primary means of authentication. Where devices or components do not support certificate-based authentication, extensibility in the security framework can be utilized to provide needed authentication.
Authorization refers to the permission scope that actors, modules, and devices are granted. It is usually configured at a least privilege level that provides just enough access to the resources and data needed to deliver the designed business solution. Authorization can be managed through certificate signing rights or role-based access control (RBAC) for some scenarios.
Attestation assures the integrity of software at boot-up of the device, during runtime, and during software updates. During secure boot-up, integrity of all software on the device is assured including the operating system, the runtimes present, and the configuration information. Runtime attestation detects malware injections, improper physical access, and improper configuration changes, with countermeasures provided by the device and security framework to combat these threats. Software attestation assures secure software patching and updates through measured and signed packages.
A device can be considered as trusted by meeting standards such as ISO/IEC 11889 that specifies the architecture, data structures, command interface, and behavior of a TPM. A TPM device is trustworthy for storage, measurement, and reporting. Trust in these three elements is assured through authorization by using certificates and through attestation, thus providing evidence of the accuracy of information. The platform also offers protected locations for keys and data objects and can provide integrity measurements of platform state.
You likely will also consider protecting devices from external threats that could be initiated using cloud resources. For example, Arm TrustZones could be established to secure boundaries between edge devices or IoT gateways and the cloud.
Note
One way to monitor transmissions from the devices is through a Security Information and Event Management (SIEM) tool. Microsoft began to preview its Sentinel tool as this book was being published and promoted its utilization of AI to provide an early warning of unusual events, including in IoT devices. In addition, Azure Security Center can be used to help find missing security configurations in IoT devices.
The Azure IoT Device Catalog
To better understand which devices can most easily be implemented in the Microsoft IoT architecture and enable the device manufacturers to verify this, the Azure IoT certification service (AICS) was created. Once certified, the devices are listed in the Azure IoT device catalog (https://catalog.azureiotsolutions.com/).
Some of the equipment listed in the catalog are build-your-own-device Microsoft Azure IoT Starter Kits. These kits typically include a breadboard, some sensors, LEDs, resistors, jumper wires, and other parts needed.
Azure IoT device catalog certified devices and starter kits page
To become certified, device partners first create a company profile and indicate the devices they wish to certify. For each device, the type of operating system for the platform and programming languages supported are provided in order to receive a certification test that can be run on the device. The vendor runs compatibility tests and provides packaging for installing agent on device as well as an example of using device with Azure IoT.
Note
A wide variety of operating systems can be run on the devices that are tested for certification. At the time this book was published, versions of Raspbian-stretch, CentOS, Debian, RHEL, Ubuntu, Ubuntu Server, Wind River, Windows 10 IoT Core, Windows 10 IoT Enterprise, Windows 10 Server, and Yocto had certification tests available.
Since 2018, Microsoft also provides a certification test for support of the Azure IoT Edge software on devices. During testing, IoT Hub primitives such as device-to-cloud, cloud-to-device, direct method, and device twin properties are validated, as is the presence of the EdgeAgent module on the device. A test is also run to ensure that a sample Edge module is successfully deployed to the device.
Level 1. Custom security
Level 2. Azure Device SDK
Level 3. Azure Device SDK, FIPS 140-2 Level 2, and Common Criteria EAL 3+
Level 4. Azure Device SDK, FIPS 140-2 Level 3, and Common Criteria EAL 4+
Level 2 can be reached through validation of base security processes at the Edge and all transactions monitored in accordance with a deployment risk assessment with no secure hardware in place. Level 4 can be reached by providing a secure element that includes a stand-alone security processor, secure hardware protection of storage, session key generation, authentication, and certificate processing in place. Level 4 can also be reached by providing a secure enclave containing an integrated security processor, providing secure element features, featuring protection of the execution environment, and providing metering, billing, secure I/O, and secure logging.
In 2019, Microsoft introduced IoT Plug and Play, an open modeling language to connect IoT devices to the cloud without having to write embedded code. IoT Plug and Play devices are defined by a device capability model in a JSON-LD document. Device properties including attributes (such as firmware version), device settings, telemetry sensor readings and alerting events, and available device commands are described in the model.
IoT Plug and Play. Certified for this capability?
Microsoft Azure IoT Starter Kit. Yes or no?
Azure IoT Edge. Certified for this capability?
Chip Manufacturer. Intel, Microchip, Espressif, Qualcomm, Broadcom, Texas Instruments, NXP/Freescale, Nvidia, STMicroelectronics, VIA Technologies, or other.
Cloud Protocol. Supports AMQPS, AMQPS over WebSockets, MQTT, MQTT over WebSockets, and/or HTTPS.
Connectivity. Bluetooth, LAN, WIFI, LTE, 3G, and/or other.
Device Security Services. Managed PKI (CSR, CRL, etc.), Symmetric Key Provisioning, firmware update and integrity, secure hardware attestation, secure hardware disablement, authentication and data protection, identity management, device management, and/or others.
Device type. Gateway, industry tablet, mobile POS, or other.
Geo availability. Worldwide, Europe, APAC, America, and/or Africa.
I/O hardware interfaces. GPIO, I2C/SPI, COM (RS232, RS485, RS422), USB, and/or others.
Industrial protocols. CAN bus, EtherCAT, Modbus, OPC Classic, OPC UA, PROFINET, ZigBee, PPMP, and/or others.
Industry. CityNext (Smart Cities), discrete manufacturing, government, health, hospitality, insurance, media and cable, power and utilities, process manufacturing, retail and consumer goods, banking and capital markets, or others.
Industry certification. Yes or no.
Operating system. Windows IoT Core, Windows IoT Enterprise, Windows 8/10, Debian, Arduino, Windows Server, Ubuntu, iOS, Mbed, Yocto, RTOS, Fedora, Android, Raspbian, RHEL CentOS, Wind River, no OS, or others.
Programming languages. C, C#, Java, JavaScript (node), and/or Python.
Secure hardware. TPM, DICE, SIM and eSIM, Smartcard, and/or others.
Tested built-in sensors. GPS, touch, LED, light, gas, noise, proximity, temperature, humidity, pressure, accelerometers, weight, soil alkalinity, vibrations, image capture, motion detection, chemical/compound presence, no built-in sensors, and/or others.
Note
The first starter kits for Azure Sphere microcontroller units (MCUs) providing a hardware-based root of trust were being released as this book was being published. For example, the Avnet Azure Sphere Starter Kit contains a MediaTek MT3620 MCU and includes a three-axis accelerometer, three-axis gyro, temperature sensor, and ambient light sensor.
The MCU features the Microsoft Pluton security subsystem, a full memory management unit for compartmentalization of processes, real-time operating system, Wi-Fi radio, multiplexed I/O, hardware firewall, and integrated RAM and flash memory. The software stack features a security monitor protecting the hardware and custom Linux kernel. Operating system services include a device authentication client, over-the-air (OTA) update client, application management, and networking management. Custom applications are typically developed in C using Visual Studio.
When looking for devices or gateways, you can search the catalog using these parameters. For example, you might want to gather and transmit data from computer numerical control (CNC) machinery on your factory floor that automates control of machining tools used in milling, turning, and grinding raw materials. You can search for IoT gateways in the catalog that support industrial protocols such as ABB, FANUC, or Modbus, and you will see the vendors of such equipment present in the catalog.
Throughout this chapter, we’ve focused on edge devices and Azure IoT Edge software but have frequently mentioned the critical role of the Azure IoT Hub in communicating and managing these devices. We next will look at the Azure IoT Hub in more detail in Chapter 4.