Chapter 5. Current IoT Architecture Design and Challenges

The number of IoT platforms is growing at a rate similar to the growing number of approaches to IoT platform design and the variety of capabilities on offer. The IoT Platform Companies List (2017), from IoT Analytics, found more than 450 different current IoT platform offerings. This demonstrates a huge increase of more than 50 percent in a single year, growing from 300 in 2016. This number will likely continue to grow rapidly, as technology and use cases continue to drive new opportunities. More scope will thus be available for platform providers to innovate and for end users and operators of platforms to benefit from.

Many of the known large technology vendors in IoT, including IBM, Amazon, Cisco, Google, Microsoft, and Intel, have launched platforms offering their own approaches and ecosystems. Interoperability has not been a focus. Although some common capabilities are offered, the difference lies in how these platforms are built and the architectural standards for IoT. The real value of IoT is making sense of data and services to provide business value; this will happen globally only by following standards.

For platform providers looking to develop an IoT platform, and for end users and operators looking to build or buy one, multiple factors affect choice, including architectural philosophy. Chapter 4, “IoT and Security Standards and Best Practices,” provided an overview of the evolving standardization landscape of IoT, where developments continue to focus on system architecture, integration, and security for IoT systems and platforms. As standards and technologies mature, IoT places itself in a better position to deliver the promise of creating new business value. To achieve this, a solid, open, extensible, and future-proof (as much as is possible) architectural approach to IoT platforms needs to be adopted.

With a multitude of IoT architectures and platforms available today, choosing which direction to focus on is a major challenge. Organizations can take a technological, legacy system, or deployment environment perspective, but they also have unclear business objectives to address. In any case, platforms should be based on a solid foundational architecture, preferably one that has been created with IoT in mind. Unfortunately, this is not what we see in practice.

There is agreement that taking an architectural approach as a foundation makes sense if interoperability and development of IoT is a goal. Combining this with security built in from the ground up should be a minimal requirement, especially to provide confidence in investing in IoT solutions. However, there is no agreement on what the appropriate architectural approach is to achieve this goal. Several tools, guidelines, best practices, models, and frameworks are available to help architects and system designers address the required components of an IoT system. In practice, we see more proprietary, closed systems, often with tie-ins to hardware or software from platform vendors.

Most IoT platforms came into existence trying to address particular technology or market/vertical needs as IoT has evolved. As a result, most solutions today are custom ones and address specific parts of the full IoT stack that is often outlined in standardized IoT reference architectures. However, end operators and users are realizing that this is not enough. To best enable their business and use case needs, a more holistic approach needs to be taken.

This chapter provides an overview of the main architectural approaches to building out IoT systems, including the benefits and drawbacks that existing architectures and platforms still need to address. The aim is not to go deeply into each architectural approach (you can turn to the specific websites and documents produced by the standards groups and consortia for that), but instead to highlight elements that are required to build a robust IoT platform with security at its foundation. The guidance in this chapter provides a series of consistent messages on the architectural elements needed to design a next-generation IoT platform. In many IoT system designs, more than one approach likely is needed. They also highlight how many of these recommendations are missing in today’s platform solutions. This chapter looks at what has already been proposed, to set the context and foundations for what still needs to be done, as discussed in subsequent chapters.

What, Why, and Where? A Summary

With any architectural approach, understanding the elements that compose the system is essential. Chapter 3, “IoT Security Fundamentals,” introduced the layers of the IoT architecture hierarchy (see Figure 5-1). As Chapter 4 highlighted, the standardized architecture has not yet been accurately defined or universally accepted. It continues to evolve. Although many different models and architectures have been proposed, these layers remain consistent, to some degree.

■   Things and endpoints: These are the physical or software devices, sensors, actuators, and controllers that send and receive data and information in the IoT system.

■   Infrastructure and transport: This is where data and information are moved throughout the IoT system. It ensures that devices and hardware are connected via a network. Core networks are high-speed, highly scalable, and reliable communication networks that connect multiple edge or fog networks together, as well as to the center and back end in the cloud or data center. Edge and fog networks consist of a decentralized infrastructure where data, computing, storage, and applications are distributed in the most logical and efficient place between the data source and the cloud. This architecture extends cloud computing and services to the network edge. Gateways and edge nodes are devices such as switches, routers, and computing platforms that act as intermediaries between the endpoints and the higher layers of the IoT system.

■   Center/back end: The cloud (private or public), data centers, or hybrid cloud infrastructure provides large pools of servers and storage networked together, usually virtualized. Centralized processing, orchestration, and management occur at this level. This is where the IoT platform back end resides. It centrally manages the control and data elements of the IoT system and includes a control user interface for the system. In addition, it provides data integration and interfaces to other systems.

■   Business processes and services: This layer consists of a collection of related, structured activities or tasks that produce a specific service or product to add business value or competitive advantage. It includes operational or business applications that take advantage of the data produced by the IoT platform, plus data visualization and reporting user interfaces.

■   Data pipeline and processing: The focus of this layer is collecting, handling, and moving data to where it can be appropriately processed. This includes capabilities such as protocol translation, data transformation, and data normalization.

■   Integration: Leveraging APIs, adapters, and SDKs to integrate with third-party applications, devices, or systems.

■   Security: Security encompasses all elements of the stack and should be built in versus bolted on. It includes protection for all devices, users, applications, and data via mechanisms such as authentication, encryption, identity, certificate management, and so on.

In addition to the architectural layers, several common nonfunctional requirements relate to the implementation and operation of the IoT itself:

■   Openness, interoperability, and heterogeneity: Seamless interoperability between each device, user, or application

■   Scalability: The capability to support a large range of applications, devices, and services varying in size, complexity, and workload

■   Reliability: Reliable operation of the hardware and software of the IoT system, generally measured in uptime

■   Availability: Continuous management and orchestration of the IoT system, usually measured in uptime

To translate these requirements into a consistent view that can be acted upon, a typical set of processes occurs. A reference model is created to provide a common understanding of the IoT domain, followed by a reference architecture to explain the IoT domain building blocks, then a reference implementation, and finally specific system implementations.

A reference model embodies basic aims or concepts, to be used as a reference point for various purposes. It divides functionality into elements, together with the flow of data between those elements. A well-known example is the OSI model. It includes a structure that allows the modules and interfaces of a system to be described in a consistent manner. It provides an abstract framework for understanding relationships between things in an environment and for developing consistent standards or specifications that support the environment. A reference model can represent the component parts of any consistent idea, including business and technical ideas, as long as it covers a full view. This frame of reference can then be used to clearly communicate ideas.

A reference architecture provides a high-level blueprint of an architecture for a specific domain, bringing components together. It indicates how an abstract set of elements and relationships achieves a predetermined set of requirements, and it provides a more complete picture that includes what is involved in realizing the modeled entities. It provides a template, typically based on a set of solutions that have been generalized and composed to describe one or more architecture structures in multiple successful implementations. It shows how to bring these parts together into a solution and can be instantiated for a particular domain or for specific projects.

A reference implementation is a best-practice instantiation of a reference architecture. This is derived from the reference architecture, which makes it possible for architects to select functional components, capabilities, protocols, and so on. Figure 5-2 shows the flow from reference architecture to implementation reference architectures, to solution deployments. In the context of this book, this moves from reference architecture to best practice implementation architecture, to IoT platform implementation.

IoT platform design should be an outcome of a reasoned process to address the reference architecture and reference model requirements created early in a process. The IoT platform is the ultimate enabler in IoT, bringing together the component parts of an IoT system in a uniform, scalable, manageable, and secure way. The platform has the potential to deliver the entire IoT capability stack, but not every use case or deployment scenario needs each element. IoT platforms should be seen as a delivery mechanism in response to solving customer or industry use cases through the use of technology. However, as Gartner points out in its guide for IoT system architects (2015), there is no single, complete, and consistent solution for IoT. Instead, Gartner believes it is possible to provide a foundational IoT platform that is capable of delivering most aspects of the IoT stack in a uniform and standardized way.

The Boston Consulting Group (2017) differentiates between an IoT platform and partial solutions, arguing that partial solutions should not be termed IoT platforms at all. Products that manage connectivity or enable applications should be termed connectivity management platforms or application enablement platforms, not IoT platforms. An IoT platform fundamentally connects devices, applications, and data so that users and organizations can focus on business outcomes.

The IEC provides a comprehensive definition of an IoT platform as follows:

[An IoT platform is] the central hub of domains that collectively constitute the physical realization of the functional view of an architecture encompassing one or more aggregated edge environments. The IoT platform is an integrated physical/virtual entity system employing various applications and components to provide fully interoperable IoT services and management of those services. This includes, but is not limited to, networks, IoT environments, IoT devices (sensors, controllers, actuators, tags and tag readers, gateways) and the attached physical devices, IoT operations and management, and external connectivity with suppliers, markets and temporary stakeholders of the IoT system.

The IoT platform itself can be located in the cloud, located on premise, or involve a combination of the two. Regardless of physical location or architecture, the domains that comprise the IoT platform—operations, information, application, and aspects of business and control—contain multiple data and control flows with one another, with the back end applications of the business domain, and with the physical systems/control domain that resides in the edge. Additional services of the IoT platform can include resource interchanges to enable access to resources outside of the IoT system, network services, cloud integration services and many other services as defined by the individual platform provider or user.

To address this type of complexity, interoperability is needed. That interoperability must be based on taking standardized and agreed-upon approaches to architecting and building IoT systems and platforms that enable them.

Approaches to IoT Architecture Design

As technology and use cases have continued to evolve, so have thoughts about how to break down an IoT system into representative architectures. With applicability across multiple markets and industries (including energy, manufacturing, healthcare, transportation, fitness, education, and home automation), it is no surprise there are myriad efforts. No single consensus exists regarding an architecture for IoT, and this is perhaps the only universally accepted viewpoint.

To better discuss the different options for IoT platform architectures, and to provide examples of some of the most successful in operation today and what capabilities they offer, it is necessary to explore some of the key IoT system approaches and architectures that have been developed. In theory, platforms should follow (or try to address) system reference architecture because the IoT platform is a technology solution created to address or enable the capabilities that the IoT system requires. We review the approaches to IoT architecture from three different perspectives:

■   The X-centric approach, in which architectures focus on a particular element of the IoT system (such as a gateway or user) where the main capabilities should reside.

■   The system viewpoint or perspective approach, which centers on a particular slice of the full IoT system, such as the cloud, where the main capabilities of the system should reside.

■   The universal or umbrella approach, which addresses all elements of the IoT stack without specifically focusing on a particular element and slice. End-to-end comprehensive architectures are built from this perspective.

Before examining each of these approaches, note that one interesting aspect applies across all three: the approach of IoT systems being delivered in an “over the top” (OTT) way instead of in an integrated manner. This is applicable to both architectural and platform designs. Most IoT deployments today are OTT, with the IoT services decoupled from the underlying infrastructure (meaning no integration with the network). This is a challenge when use cases are mission critical or need to be highly reliable, as in remote healthcare or industrial control, where latency and guaranteed delivery are essential. Decoupling the services from the network also impacts other areas, such as security and service assurance to enforce KPIs. Therefore, when choosing the OTT or decoupled approach, be sure to carefully consider the ramifications.

A fundamental requirement of an IoT platform is to provide effective security and privacy capabilities. Gartner’s 2016 IoT Backbone Survey reported that 32 percent of business leaders cite security as a top barrier to IoT success; IDG advises that 57 percent of organizations see it as the greatest challenge in their IoT projects. With no universally accepted architectures or standards for IoT, securing IoT deployments is a challenging area.

The placement of the platform (private versus public), whether the platform is designed to address the full IoT stack or slices of it, and the type of use case or customer all have an impact on security and how security is designed. As discussed throughout this chapter, most IoT platforms today are delivered in an OTT way. The majority are cloud based and decouple IoT services from the underlying infrastructure. Although these platforms have a number of security mechanisms built in (such as authentication and identity management, access control, and encryption and cryptography), the location in the public shared infrastructure leads to potential security issues:

■   They are subject to traditional Internet and network attacks such as denial of service (DoS, DDoS), spoofing, man-in-the-middle, and eavesdropping.

■   The question of who owns the data (customer, platform provider), is responsible for the data, and has control over where and when data is placed can create issues in legal, intellectual property, and financial perspectives. Organizations must protect business data and information while also limiting liability caused by privacy breaches. Users also require control over their individual data.

■   Privacy, trust, and data storage protection (including the need to protect identity and transaction history) are harder to address. Whether through personal choice, due to regulation, or due to a fear of losing control, many organizations are unable to store data or types of data outside their private data center. Cloud- and public-based solutions increase the risk of potential unauthorized access and exposure to private, competitive, and sensitive information.

■   Sending data to the cloud means more data is exposed to security threats.

■   Governance, internal, and regulatory compliance becomes more challenging. Organizations have processes for compliance monitoring and enforcement in their private infrastructure, through existing security and information management procedures. Leveraging OTT IoT platforms means changes to processes. This can involve passing elements of monitoring and enforcement to external parties, who themselves are not subject to the same requirements.

■   The borders of responsibility, liability, and reliability blur. The reliability of systems, particularly in industrial environments where human or environmental health is a major concern, is essential. Understanding where security issues are and who is responsible and liable for addressing them is a key element of detecting and resolving security threats. Decoupling the platform from the underlying infrastructure removes reliability mechanisms such as QoS and introduces a scenario in which people do not know where the threat is or who should resolve it.

■   Ensuring a timely response to security issues can become a challenge with external dependencies to address threats.

These factors are applicable in any IoT deployment, but industrial environments such as oil and gas, manufacturing, and transportation face an even more serious challenge. Most organizations follow a strict segmented architecture such as that proposed by the Purdue Model of Control and IEC 62443 (see Chapter 4 and later sections in this chapter). In these scenarios, most operational data must remain private and not be exposed to the enterprise or external systems. Cloud OTT platforms are not an option for many operational IoT use case deployments.

Based on these challenges, it is clear that a different approach is needed. Organizations need consistent and integrated security that not only addresses authentication, access control, encryption, privacy, data storage and exchange, trust management, compliance, governance, and reliability, but also provides deep and controlled integration into an organization’s existing security systems and procedures. The type of deployment impacts where these security capabilities can be enforced (see Figure 5-3).

To most effectively monitor and enforce security, the infrastructure must be capable of effectively reacting to needs dynamically, based on the platform requirements. This cannot be effectively achieved by decoupling the platform from the infrastructure; an integrated approach is required that tightly combines both. This enables end-to-end service assurance to effectively monitor, manage, and secure services or capabilities that are deployed. By combining infrastructure, applications, message brokers, users, and devices into a common and consistent orchestrated framework, security itself can be deployed as a fully integrated capability and leveraged to make contextual security decisions and enforcement stemming from increased visibility and control. This is not achievable through OTT platforms, where these capabilities are disparate.

An X-Centric Approach

In 2014, Gartner outlined five approaches to creating architecture for IoT systems. These continue to be relevant, although they might be considered simplistic and generic in nature. A number of X-centric approaches also have been developed since then, particularly from human and social aspects. Choosing the right approach very much depends on the use case to be implemented.

Gartner’s five approaches include thing-, mobile-, gateway-, cloud-, and enterprise-centric views. In each of these views, the IoT services (such as the user interface, applications and logic, data handling, analytics and processing, and control or decision points) shift their positions of importance (see Figure 5-4).

In the thing-centric approach (Figure 5-5), the endpoints are intelligent and self-sufficient. They have their own processing, storage, and logic capabilities onboard. Communication is from the endpoint directly to a central cloud or data center location only for coordination and analysis. As such, the endpoints need to have local processing, storage, and logic. In these scenarios, the focus is on providing IoT services capabilities at the edge of the deployment. Data processing and decisions are located throughout the IoT system instead of centrally located, and actions can be made locally. An example is an oil rig setting where terabytes of sensor data are produced but cannot be sent to central locations because of bandwidth and latency challenges. The thing/endpoint could have the capacity to make local decisions. This architectural approach would be favored in industrial IoT deployments and in the transportation industry with smart and autonomous cars.

The thing-centric approach offers these advantages:

■   Achievable real-time functionality and response because resources are local

■   Increased reliability, with less reliance on communication networks

■   Devices that are more independent and autonomous instead of reliant on external services that might have problems

■   Reduced security and privacy attack surface by not transmitting over networks

■   Reduced operational costs because only minimum data is transmitted

At the same time, disadvantages must be considered:

■   Increased cost and complexity of devices and supporting edge infrastructure

■   Poor design leading to islands of disconnected or disaggregated devices and resources

■   Advanced centralized capabilities that are not leveraged; difficulty getting a centralized system view

■   Device-specific knowledge required, which might be more difficult to obtain

In the mobile-centric approach (see Figure 5-6), a mobile device (such as a smartphone or tablet) houses processing, storage, and logic capabilities, to coordinate and analyze endpoints it is responsible for. Similar to a gateway, the mobile device acts as an intermediary point between endpoints that it is responsible for and the central location in the data center or cloud. Using this approach, endpoints do not need to be intelligent devices, and the IoT services move up a level to the mobile device. Use cases include mobile technicians in an industrial plant environment, with all application and system access controlled from a mobile device; home automation, such as the capability to control thermostats; and fitness wearables, such as a smartwatch or step monitor.

The mobile-centric approach offers these advantages:

■   Enables the mobile device to be the endpoint or thing while also providing gateway services

■   Provides ubiquitous cover, carried by most employees

■   Offers good computational power through the device

■   Typically provides Internet support directly, minimizing infrastructure needs

■   Reduces the cost and complexity of user interfaces and device training

■   Involves minimal power consumption and reduces weight/size constraints

At the same time, some disadvantages must be considered:

■   Mobile devices are often not standardized.

■   A smartphone is needed and has a cost associated with it.

■   Security is based on the security capabilities of the device itself. Although there are strong access control capabilities, such as Touch-ID, many devices are weak in this area.

In the gateway/hub-centric approach (see Figure 5-7), one or more endpoints connect to a gateway that acts as an intermediary device to a central location. The gateway contains processing, storage, and logic to coordinate and analyze the endpoints it is responsible for. The gateway communicates with the central cloud or data center where it is coordinated and analyzed. Attached endpoints do not need to be intelligent devices, and the IoT services move up a level to the gateway. The gateway can also act as a buffer to store information if backhaul connectivity to the back end is lost, and it provides local and transport security services. This approach can be seen in building control systems, in street lighting and traffic control regulation in smart cities, and within the smart home.

The gateway-centric approach offers these advantages:

■   Reduces the cost and complexity of end devices

■   Can provide proxy and translation services for devices with nonuniform capabilities, protocols, and standards

■   Can aggregate multiple edge devices and send uniform required data sets to a central location

■   Increases security via specific technologies such as firewalling, VPNs, and access control lists

■   Can link to both new and legacy IoT end devices and technologies

■   Can provide NAT capabilities

At the same time, some disadvantages must be considered:

■   Adds an extra tier to the network, which increases integration and operational costs

■   Provides a single point of failure for potentially multiple devices connecting to it

■   Increases the cost of the gateway itself

■   Offers minimal local capabilities, compared to centralized resources

In the cloud-centric approach (see Figure 5-8), the cloud provides central connectivity for gateways or endpoints, delivering processing, storage, and logic to coordinate and analyze any endpoints it is responsible for. Endpoints and gateways do not need to be intelligent devices, and the IoT services move up a level to the cloud. The cloud provides multiple centralized IoT services, including analytics, storage, applications, logic, and security. Many consumer IoT use cases are cloud based and involve non-real-time control, such as with fleet management. In IIoT scenarios, this is very much use case dependent, based on who owns the architecture (IT or OT). Real-time use cases that involve control, with data that is not allowed to leave the organization, will not leverage this approach. On the other hand, optimization services such as machine or asset health monitoring can use this architecture. The next section takes a more detailed look at cloud compute architectures because of its growing importance in IoT.

The cloud-centric approach offers these advantages:

■   Scalability through leveraging centralized capabilities such as virtualization

■   Solution cost efficiencies through shared resources

■   A centralized approach providing a system-level view

At the same time, some disadvantages must be considered:

■   A consistent and persistent Internet connection required for all connected devices

■   Challenges for high-data and low-jitter/-latency applications to operate

■   An expensive option if large amounts of data need to be sent from device to cloud

■   No real-time capabilities for applications

■   Security challenges in sending data across public connections, and having resources and computing in a shared infrastructure

Finally, in the enterprise-centric approach (see Figure 5-9), all elements of the IoT deployment, from endpoints through to the data center, are located inside a private and secure perimeter. IoT services (including analytics, storage, applications, logic, and security) are provided by the organization via organization-owned assets. The IoT requirement for external access outside the perimeter is limited. The focus of this approach is keeping data and decisions securely within an organization’s control. Use cases typically are those in which data should not leave an organization’s control or where decisions must remain inside an organization’s boundary.

The enterprise-centric approach offers these advantages:

■   Better opportunities to enforce security and privacy by minimizing external connectivity to the enterprise and keeping data stored on-premises

■   Better resource control and the capability to respond more quickly to faults or changes

■   Better manageability

At the same time, some disadvantages must be considered:

■   Not always well suited to deployments where many devices and systems are geographically dispersed

■   More capital expense because equipment and infrastructure needs to be purchased

■   More operational costs from management and maintenance

■   More skill sets required of staff

In these Gartner-defined centric views, the approach typically emerges from a device or infrastructure perspective. The people and information/data elements are missing. Combining the device and infrastructure with people and information adds value. As highlighted in Chapter 1, “Evolution of the Internet of Things (IoT),” in IoT, merely connecting devices and producing data is not enough: The data needs to be consumed in the right way; at the right time; by the correct device, system, or human; and in a way that is meaningful and can be acted upon appropriately. This is what delivers the true business value. As a result, we have seen developing X-centric views on which to build the architectural foundations for IoT.

The People-/User-Centric IoT Approach (Internet of People and Social IoT)

People-centric/user-centric views, combined with advances in technology and changing needs of consumers and business, have opened the possibility of creating a range of people-centric and social services. This approach centers on how IoT data should be processed differently than data in the Internet today. The focus is things instead of users as the main data consumers and producers. The premise is that systems will be able to gain information about and learn to solve real-world problems via the data produced by things. The goal is for IoT devices and systems to sense and react to the real world in a similar way to humans. Everyday objects are identified and networked, to make independent decisions. IoT provides the potential to merge real and virtual worlds through the deployment of embedded devices.

Combining ever-cheaper IoT technologies, such as people sensors (for example, the Fitbit) and mobile devices, with the explosion of social networks and virtual worlds allows the creation of the “Internet of people” (IoP) or “social IoT” (SIoT). The people-centric approach is a dynamic system of connected smart objects enabling people-centric applications. Unlike thing-centric architectures, the people-centric approach combines the real, sensory world with the virtual world, for the benefit of people. Typical use case scenarios include e-health, social networking, sustainable mobility, smart transportation, and bespoke individual needs.

Technology areas such as machine learning (ML) and artificial intelligence (AI) are advancing the people- and user-centric approach. Examples include the following:

■   Integrating semantics into mobile devices, to leverage the user’s location and locally generated information.

■   Integrating reasoning engines into mobile devices, providing context-aware services that can interpret the current situation of the person based on different sensor measurements from the mobile device.

■   Using ontology-based applications for mobile devices. An example is an in-vehicle application that displays vehicle information such as fuel consumption, provides driving safety suggestions based on weather conditions, and detects the driver’s awareness levels.

The people-centric approach is in the early stages with regard to standardized architectures, but it is rapidly progressing. Not only are semantic recommendations provided to users via these technologies, but capabilities also include real-time automated intelligent control based on the selected sensor data. The approach still requires elements to be worked through, including limited reasoning engines, no consideration for external sensors belonging to different domains, no dynamic discovery of sensors, and a lack of consistent definitions or standards. However, development is moving quickly and merits a lot of interest.

Most architectures proposed are based on a server and object divide (see Figure 5-10). The server side connects to all the interconnected components, aggregates the services, and acts as a single point of service for users.

The server-side architecture typically has three layers. The base layer contains databases that store details of all devices, their attributes, their meta-information, and their relationships. The component layer contains the software code to interact with devices, query their status, and use a subset of them to effect a service. The application layer provides services to the users.

The device side consists of two layers. The object layer allows a device to connect to other devices, talk to them (via standardized protocols), and exchange information. The object layer passes information to the social layer, which manages the execution of users’ applications, executes queries, and interacts with the application layer on the server.

This approach comes with some potential challenges for IoT solutions:

■   Transactional decisions: With data being updated and spread across potentially hundreds of thousands of devices with differing update policies, it can be difficult to define what a transaction or decision is and where it should be handled.

■   Semantics: With things and machines providing more important roles than humans, and needing to be able to both understand and process data, machines need capabilities for semantic enrichment and semantic event processing. Machines need to effectively understand raw data coming from sensor data streams and other types of data streams, to enable them to make automated context-based decisions that would usually be carried out by a person. These semantic techniques are still developing.

■   Usefulness: In these information-centric systems, distinguishing between relevant and useful information, particularly when machines are leveraging information and making decisions, can be a challenge.

■   Privacy: Many of the devices contain personal embedded sensors, so individual details such as location and activity could potentially become available to other people. This personal data could be used to detect personal activities, hobbies, home and work locations, and more.

The Information-Centric IoT Approach

The information-centric network (ICN) is not a new concept. It also goes by other names, such as content-centric networking. Its potential application to IoT use cases has given it a resurgence, with an IoT-specific draft RFC submitted in July 2017 to the IETF. From the ICN perspective, the Internet is inefficient in emerging use cases that include IoT, and a new approach is needed to address the shortcomings. The Internet infrastructure of today leverages a set of disparate technologies and distribution services that are deployed in silos. Using these methods, it is not possible to uniquely and securely identify named information independently of the channel of distribution. Different distribution approaches are typically implemented OTT, leading to inefficiency.

ICN aims to evolve the approach to an Internet infrastructure, making it more appropriate for developing use cases such as IoT by introducing uniquely named data as a core Internet principle (see Figure 5-11). Following such an approach, data becomes distinct from location, application, storage, and communications, enabling in-network caching and replication. The expected benefits are improved efficiency, better scalability with respect to information/bandwidth demand, and better robustness in challenging communication scenarios that are highly distributed, are highly scalable, and consist of a range of endpoint capabilities, such as would be found in IoT.

ICN has three main features:

■   Objects (including mobile devices, content, services, or context) are identified by name instead of IP address.

■   It utilizes a hybrid name/address scheme for routing.

■   It provides delay-tolerant transport.

These features make it appropriate for a number of IoT scenarios, including mobility, cloud or edge computing, and security. Figure 5-12 provides an overview of the information-centric approach for IoT.

The ICN-IoT system architecture (see Figure 5-13) is a general abstraction of an IoT system and contains the following main components:

■   Embedded systems (ES): The embedded system contains sensing and actuating functions and can relay data from other sensors to the aggregator.

■   Aggregator: Interconnects entities in the local IoT network and provides device discovery, service discovery, and name assignment. Aggregators can interact with one another directly or through a local service gateway.

■   Local service gateway (LSG): The LSG provides the administrative boundary (for example, the edge of the enterprise or the edge of the factory), connecting the local IoT system to the global IoT system. It also assigns ICN names to local sensors, enforces data access policies for local IoT devices, and provides context processing services to generate application-specific information (instead of raw data) to the IoT server.

■   IoT server: The IoT server is centralized and maintains subscription memberships plus lookup service for subscribers. The IoT server is involved only in the control path from publishers to subscribers, not the data path, potentially reducing data bottlenecks.

■   Authentication manager (AM): The AM enables authentication of the embedded devices at onboarding and when their identities need to be validated at the system level.

■   Services/consumers: These are applications that interact with the IoT server to fetch, or be notified of, anything of interest within the scope of the IoT service.

ICN-IoT middleware is leveraged to bridge the gap among underlying ICN functions, IoT applications, and devices, to deliver self-organizing capabilities. Figure 5-14 shows the main middleware functions:

■   Device onboarding and discovery: Often no differentiation takes place between device onboarding and device discovery. In ICN, the objective of onboarding is to connect new devices and enable them to operate in the ecosystem.

■   Detailed discovery process: A device can be an embedded device, a virtual device, a process, or a service.

■   Naming service: The objective is to ensure that the device or service is authenticated. The naming service assigns and authenticates embedded systems and device names.

■   Service discovery: This function learns IoT services that are hosted by an aggregator or neighbor aggregators.

■   Context processing and storage: To facilitate context-aware communications and data processing, contextual processing is required for the IoT system. The objective is to expose the embedded system’s low-level context information to upstream aggregators and local service gateways.

■   Publish-subscribe management: Data publish/subscribe (pub/sub) provides IoT information resource sharing and management.

■   Security: Security spans all middleware functions, with the objective of ensuring that devices connecting to the IoT system are authenticated, services are authenticated, and generated data from devices and services is authenticated and kept private.

Although ICN promises some interesting advantages for IoT environments, some potential challenges must be considered:

■   Scalability: Moving away from a focus on nodes or devices to information objects (where many can be part of one device) raises scalability to a new level. In 2016, the Internet contained approximately 10⁹ nodes, but the number of addressable ICN objects is several orders of magnitude higher. To support this, name-based routing and name resolution (especially through IoT considerations such as mobility) need to be designed to scale accordingly.

■   Security: Identity, trust, and security mechanisms are bound to information objects instead of nodes. New information-centric models are needed in place of the host-centric models we see today.

■   Existing technology solutions: New technologies and interfaces might be required for applications to interact with the ICN.

■   Data leveraged data from named content: New data transport methods and protocols (such as receiver-oriented pull mechanisms) are required to access the data. In addition, once named objects exist in multiple caches or locations, mechanisms to provide coordinated updates to ensure synchronization and only one source of truth could be needed.

■   Migration challenges: Today’s infrastructure is built very differently. A cost is involved in migrating to ICN architectures, in terms of technology, the redevelopment of existing applications and business processes, training of staff, and support.

The Data-Centric IoT Approach

The final X-centric view is data centric. In IoT, uniform connectivity and uniform information exchange is perhaps the greatest challenge. Today’s Internet and enterprise-class technologies cannot deliver the guaranteed performance, scalability, reliability, and redundancy that many IoT systems need. As mentioned already, IoT systems focus on the data that can be intelligently gathered and leveraged. This leads to the term data-centricity.

A data-centric architecture (see Figure 5-15) has no concept of the traditional interactions between applications and data sources or data users. Instead, it provides a series of interlinked data buses at each level of a system architecture that focus on data and information exchange in a uniform way. When applied to IoT, data centricity overcomes traditional scalability, interoperability, and extensibility problems associated with point-to-point system integration. It aims to provide simple connectivity integration, scalability, and high performance. Data centricity decouples data from communications, allowing applications to focus on the processing of data. The data-centric architecture also simplifies collaboration between developers and suppliers.

Data centricity has often been defined as middleware, in that it is providing a uniform interface and communications bus between different systems, applications, and users. Where it differs from traditional middleware approaches is that it does not focus on sending information between applications and systems. Instead, it ensures that all messages throughout a system include the contextual information an application needs to understand any data it receives. A data-centric architecture understands what data it stores and controls how data is shared. In traditional systems, developers write code that must include provisions to send messages. In a data-centric system, developers create code that specifies how and when to exchange data and then share their data values. The data-centric architecture is responsible for the coordinated, synchronized, managed, and secure sharing of the data. A data-centric system is responsible for locating data, reliably sending data, ensuring that data is fresh, and securely controlling the data.

A data-centric system focuses on the data model (user-defined data), with data values being the unit of exchange. The data-centric middleware understands data context (throughout the full architectural hierarchy), ensuring that all interested subscribers have a current, correct, and consistent view of the data. In the data-centric architecture, applications interact with only the data and the properties of data. Because multiple applications might interact with the data independently, data centricity also provides natural redundancy. A data-centric platform is responsible for maintaining the state of the overall system, even in case of failure, so that the latest consistent state of the system will always be known and available.

A data-centric architecture is designed to be scalable from small edge devices all the way to the cloud and large implementations. As such, it is potentially suitable for IoT, with high-speed scalability to millions of endpoints. It also helps minimize system complexity by providing a single, standard communications and data layer.

A leading standard for data centricity in IoT is the Data Distribution Service (DDS). DDS can address multiple IoT system requirements, including those for real-time systems in industrial settings. Additional information can be found at http://portals.omg.org/dds/.

Each architectural approach comes with advantages and disadvantages, depending on the needs of the organization. The approach itself is not a hard model; models can be combined according to end user needs, which is why each model term is appended with the word -centric. An example is a gateway-centric architecture that still relies significantly on cloud capabilities for analytics. Value generated by the platform typically accumulates as an implementation moves from an entry-level endpoint deployment to full business service integration.

This section provided an overview of established and developing X-centric views to architectural approaches for IoT. The reality is that although some of these approaches work for isolated or basic implementations of an IoT system, they cannot address the full IoT stack. Based on research of what the next generation IoT system needs to include, they deliver only a slice—or a few slices, if combined. This means a more comprehensive approach is required (see the next section).

System Viewpoint: A Cloudy Perspective

The previous section covered a range of X-centric views for IoT architecture that have arisen over the last few years and will continue to develop. The system viewpoint focuses on architecting the full IoT system, but with a particular emphasis on where the main capabilities of the system should reside. The architecture is then built from this perspective.

Cloud Computing

Centralized cloud computing as a concept has been around for decades. It likely arose from networks diagrams that represented multiple devices as a cloud; only in the 2000s did it converge on the standardized meaning of today, which refers to applications and services hosted and delivered from the Internet cloud. By the late 2000s, cloud technologies really took off (with examples such as Google docs, Salesforce, and Microsoft SharePoint) as consumers and enterprises realized the benefits of cloud-based services.

Cloud computing provides shared computer resources (see Figure 5-16), such as processing, storage, and data, to users and devices on demand. The basic premise is to enable on-demand access to a shared pool of resources that can be quickly scaled up or down automatically, or with minimum effort. It allows individual users or organizations to process and store information in a cloud data center that can be located anywhere.

The main characteristics of cloud computing include a shared infrastructure leveraging virtualization technologies that deliver shared physical services, applications, storage, and networking capabilities. This optimizes physical resources across users; provides automated, dynamic provisioning of services to users based on real-time demand and the need to scale up or down; delivers secure network access via standards-based APIs, from a range of endpoints including computers and mobile devices such as tablets and phones; enables reliability via the use of multiple redundant sites, making it suitable for disaster recovery and business continuity; allows multitenancy, for single systems shared between multiple organizations; and focuses on security due to centralization of data and security-focused resources. (This last feature is also seen as a major drawback because security is provided by a third party. In many IoT scenarios, this is not an option.)

Cloud computing can be deployed in a variety of individual or combined ways (see Figure 5-17). Three main types of deployment exist, although there can be others (such as a community cloud, in which organizations with similar requirements and interests share the same infrastructure).

■   Private cloud: Infrastructure is deployed, operated by, and maintained for a single organization. It can be deployed inside an organization or externally, and can be managed by the organization itself or a third party. Significant investments and process changes are required. When deployed and managed internally, many of the benefits of cloud computing do not materialize.

■   Public cloud: The infrastructure is available on a commercial basis by a cloud service provider. Users of a public cloud can create and deploy services with minimal initial financial outlay. The infrastructure support costs are included in the commercial contract. From an architectural perspective, there could be very little difference between public and private deployments. However, the security considerations are different for services supported by a public cloud provider, particularly because communication is established over nontrusted public networks. Large public cloud providers such as Google, Microsoft, and Amazon generally own and operate the infrastructure at their data center, and access is generally via the Internet.

■   Hybrid cloud: The infrastructure combines two or more clouds (private, public, or community) that remain distinct entities but are connected, offering the benefits of multiple deployment models. Through the cloud interfaces, data and applications can move from one cloud to another. For some organizations, a combination of private and public clouds supports the requirement to retain some data in an organization and also offer services in the cloud. This type of model is often seen in IoT deployments.

NIST defines three main service deployment models (see Figure 5-18). These models have different focus areas, business models, and applicability depending on organizational requirements. They are often considered as a layer in a stack providing infrastructure (IaaS), platform (PaaS), and software (SaaS) services. These services do not need to be related or rely on one another for deployment. For example, SaaS can be implemented on bare-metal servers without any underlying PaaS or IaaS layers.

Software as a service (SaaS) allows an end user to consume the provider’s applications running on a cloud infrastructure. Applications can be accessed typically through a thin client interface, such as a web browser, or a program interface. The cloud service provider manages all of the infrastructure and applications.

Platform as a service (PaaS) allows an end user to deploy applications onto the cloud infrastructure using tools supported by the provider. The cloud service provider manages all of the infrastructure, but the user has control over the deployed applications and associated configuration settings in the application hosting environment.

Infrastructure as a service (IaaS) allows the user to provision the infrastructure computing resources to deploy and run applications. The cloud service provider manages the underlying cloud infrastructure, but the user has control over operating systems, storage, and deployed applications, as well as potentially specific hardware such as a firewall.

Several benefits are associated with cloud computing as an architecture:

■   Cost advantages: Both capital and operational expenditures can be reduced. This lowers the entry barrier to markets, including IoT, and minimizes the staff needed to support projects.

■   Scalability and extensibility: Deployments can be scaled up or down rapidly, providing the opportunity to launch solutions on a small scale and then expand as success increases.

■   Reliability and availability: Redundancy and multisite deployments are characteristics of cloud computing. This supports business continuity and disaster recovery.

■   Accessibility: Cloud services can be accessed from any device wherever a connection to the Internet exists. This increases productivity and flexibility.

■   Reduced support and maintenance: For individual users or organizations with minimal support staff, the cloud provider looks after the infrastructure and services.

■   Environmental benefits: When many users efficiently share large systems, this translates into lower power consumption and less equipment used.

Of course, certain challenges are associated with cloud computing, although some can be resolved with careful consideration at the planning stage:

■   Lack of cloud interoperability standards: Although documented interfaces exist and the Open Cloud Consortium and Open Grid Forum are working on efforts to resolve this, the lack of a standardized approach means that most clouds are not interoperable.

■   A continuous evolution: Technology and end user requirements are continually changing, meaning that clouds are also continually changing to adapt or provide competitive differentiation services. This can pose a challenge if services and applications need to be updated or changed. In addition, the possibility exists that a cloud provider might stop supporting a product or system that forms a key part of a user’s or organization’s business.

■   Compliance need: EU data protection directives, as well as the Sarbanes–Oxley Act (SOX) in the United States, mean specific considerations are required in the cloud depending on the type of data and application. This often results in hybrid cloud deployments and can drive up cost.

■   Cloud service provider dependence: Although SLAs exist for services in the cloud, if something goes wrong, the user or organization is dependent on the service provider to resolve issues and restore services.

■   Security and privacy: Finally, the big one! Security and privacy are often the two major issues that arise when using cloud computing services because storing and securing data become the responsibility of the cloud service provider. Privacy is a key concern because the service provider can access cloud stored data at any time; thus, the data could be accidentally or deliberately altered. Data can also be shared by the service provider to third parties as part of the contract. The Cloud Security Alliance sees insecure interfaces and APIs, data loss or leakage, and infrastructure failure as the three main security issues in the cloud. These issues are often barriers to deploying IoT services in the cloud. It is possible to address these challenges by storing specific information inside an organization and allowing it to be accessed from and used in the cloud. However, this increases the complexity of security mechanisms between the organization and the cloud. Users and organizations can also encrypt data that is processed or stored in the cloud, to minimize the chance of unauthorized access.

With centralized cloud-based architectures seen as a key enabler for IoT by organizations such as McKinsey, Gartner, and Forrester, what does this mean for IoT? First, IoT requirements and cloud capabilities have a lot of synergy. IoT requires broad connectivity and accessibility—cloud can deliver this. IoT requires broad orchestration of devices, users, and applications—cloud can deliver this. IoT requires optimized resource utilization—cloud can deliver this. IoT requires the connectivity and integration of a broad set of heterogeneous devices and objects—cloud can deliver this. IoT requires the handling and processing of huge amounts of data—cloud can deliver this. IoT requires dynamic scalability of resources up and down—cloud can deliver this. IoT requires reliability in terms of centralized policy and business continuity—cloud can deliver this. IoT requires the virtualization of physical devices—cloud can deliver this. IoT might require multitenancy to enable the sharing of resources among different users, different departments of an organization, or different organizations—cloud can deliver this. IoT requires location independence so that devices and users can connect and access services from anywhere, using centralized or distributed methods—cloud can deliver this.

From a practical perspective, cloud can provide a number of positive capabilities to accelerate IoT adoption:

■   Solution acceleration: Creating and deploying new services often involve technical and financial challenges. This can include hardware, communications, and storage, as well as the software applications to facilitate data collection, processing, and delivery. It also includes the core infrastructure to host services, glue the component parts together, and make them accessible to customers. Cloud computing has been designed to facilitate these services without the cost of ownership or the challenges of scaling and supporting solutions. IoT-specific cloud services now exist, such as Amazon’s IoT platform and Microsoft Azure. These services support the rapid deployment of new IoT capabilities, speed development, and time to market, and are chargeable on a pay-as-you-consume basis.

■   Capability to deal with Big Data: IoT devices continue to generate vast amounts of data, and those amounts will grow as businesses look to take advantage of increased visibility. IoT devices do not always deliver data in a consistent way, and mechanisms are needed to help with the variance in volume. Organizations need an efficient way to access, process, and store data, and sometimes to do this in a way that scales up or down. Cloud services have been designed to dynamically scale and cope with ingesting and storing vast amounts of data. Users and organizations thus do not need to invest in and support these capabilities themselves.

■   Lack of interoperability and standards: As mentioned in Chapter 4, this is one of the big hurdles preventing the adoption of IoT. Cloud-based services support the ingestion, normalization, and sharing of data in a consistent (although not always standardized) way. This allows uniform access to services by different users or organizations, as well as the integration of services between different users or organizations.

■   Security: Security will likely remain one of the biggest barriers, or threats, for IoT. The cloud is able to provide centrally managed and consistent access to services, along with advanced security mechanisms such as secure remote access, IPS/IDS, and firewalling. These services can be provided at a fraction of the cost of building and maintaining them.

Although a cloud-based architecture for IoT can overcome several challenges, a number of aspects need to be addressed. These are especially important in industries or markets that require data to make real-time decisions, or where high amounts of data are produced and need to be analyzed but first must be moved to a centralized location.

■   The need for real-time processing. Many applications, particularly in the industrial or healthcare sectors, require real-time feedback to make decisions. Moving large amounts of data to the cloud might not be practical, or possible, and latency could become a factor.

■   Availability through the challenge of losing connectivity to the cloud. Edge devices and users need access to centralized systems in a cloud-based model. If issues with connectivity to the cloud arise (as with Amazon in March 2017 because of human error and British Airways in May 2017), services can be impacted for hours (Amazon) or days (British Airways).

■   QoS challenges in general, including little bandwidth, latency, jitter, and packet loss. Moving data to the cloud and having devices managed remotely (including aspects such as software/firmware upgrades) might mean that limited bandwidth makes devices or users unable to carry out services.

■   Dynamic orchestration and resource management.

■   Geographic legislation about where an organization stores data or accesses services.

■   Security, security, security!

Cloud and IoT are closely aligned for many use cases and industries, particularly where real-time, critical, or nonsecure data is involved. The amount of data produced by IoT devices has rapidly expanded, and this needs to be processed, stored, and accessed. Cloud computing technologies were designed to address these needs; for many use cases, cloud is the main solution today for Big Data and high-computational analytics requirements. IoT and cloud are likely to see growth in new monitoring services, effective processing of data streams, and even moves toward real-time control for noncritical applications. The goal of IoT is to make sense of data and turn this into business insight—the cloud can deliver against this, depending on the use case. This use case consideration is ultimately the decision point for cloud suitability. For real-time, critical, privacy-sensitive use cases, the lack of reliability due to minimal (if any) end-to-end QoS, service assurance, security, and the OTT deployment scenarios means cloud does not fit all IoT architectural requirements or deployments.

Fog/Edge Computing

A recent development to address many of the challenges highlighted for cloud services is fog or edge computing. Fog computing can address those problems by providing resources and services to end devices and users at any level of the IoT stack, although it is more commonly (and incorrectly) positioned for the edge of network only.

The terms fog computing and edge computing are often used interchangeably, and this might make sense. However, a difference does exist. Edge computing is not a new concept, but it has been modernized as part of the fit with IoT. Edge computing pushes most of the data processes to the edge of the network, as close to the source as possible. In the edge computing model, data is still centrally stored; as a result, all data is moved to centralized systems for permanent storage and future processing. Edge computing therefore means we replicate centralized processing and data storage close to the source, but the architecture is master/slave in nature (see Figure 5-19), with the edge processing merely an extension of a centralized system.

Whereas edge computing refers to performing processing at or close to the edge of the IoT system, a fog computing architecture performs processing anywhere from the center to the edge of the IoT system. Many IoT applications are latency sensitive, nonfixed in location, and geographically coordinated, and this presents a challenge for cloud or purely edge-based solutions. Many use cases require mobility support and geodistribution in addition to location awareness and low latency, and this is where fog comes in. Whereas cloud computing provides centralized computing, storage, and application resources over public and private networks, fog moves these resources to the most appropriate locations, anywhere from edge to cloud.

Fog computing therefore extends the cloud computing paradigm closer to the things that produce and act on IoT data (see Figure 5-20). These devices, called fog nodes, can be deployed anywhere with a network connection and can be any device with computing, storage, and network connectivity (such as industrial controllers, switches, routers, embedded servers, and video surveillance cameras). Fog provides distributed computing, storage, and network services; as a result, it enables a new breed of applications and services. Fog can be thought of as cloud for IoT.

Fog is not an adapted version of cloud computing, but is a complementary technology that enables a new breed of applications and services. Interaction will take place between the cloud and the fog, particularly when it comes to data management and analytics. Whereas fog technologies provide localized computing and correlation enabling low latency and contextual awareness (although we will see coordination, federation, and orchestration in fog soon), cloud provides centralization, coordination, federation, and orchestration. Many applications require both fog localization and cloud centralization, particularly for analytics, optimization services, and Big Data. An interesting aspect of fog is the capability to allow devices to independently communicate and make localized decisions based on these actions (see Figure 5-21).

Fog as an IoT architecture is often considered when the following requirements present themselves:

■   Minimal latency: This involves normalizing and analyzing data as close to the device as possible, for real-time or delay-sensitive applications.

■   Bandwidth restrictions: Transporting vast amounts of data from the edge to the cloud is not always practical—nor is it necessary, because many critical analyses do not require cloud-scale processing and storage. As an example, an oil well running acoustic optical sensors could produce 2 TB of data per day. The vast majority of this might not need to be sent to a central location; local analysis with aggregation might be more appropriate.

■   Security concerns: IoT data should be secured at rest and in transit. This requires monitoring and automated response across the entire attack continuum: before, during, and after.

■   Operational reliability: IoT data is increasingly used for real-time processes and decisions, safety, and critical environments.

■   Capability to collect and secure data across a wide geographic area with different environmental conditions: IoT devices can be distributed over hundreds or more square miles. Devices deployed in harsh environments such as roadways, railways, utility field substations, and vehicles might need to be ruggedized. That is not the case for devices in controlled, indoor environments.

■   Geographic considerations: Environments can include widespread geographic distribution with a large number of nodes and scalability requirements.

From an operational perspective (see Figure 5-22), IoT applications are deployed in fog nodes typically close to the network edge. The fog nodes closest to the edge ingest data from sensors and devices. The fog application has the capability to consume data, normalize data, and send different types of data to the optimal place for analysis. The approach typically follows:

■   The most time-sensitive data should be analyzed on the fog node closest to the things generating the data.

■   Data that can wait seconds or minutes for action is passed along to an aggregation node for analysis and action.

■   Data that is less time sensitive is sent to the cloud for historical analysis, Big Data analytics, and long-term storage.

We cover reference architectures for fog in the next section as we look at the OpenFog Consortium Reference architecture in detail.

Leveraging a fog-based architecture offers multiple advantages (remember, it is end-to-end, not just at the edge):

■   Proximity: Fog brings data and processing closer to where it needs to be consumed. Instead of housing information at data center sites far from the endpoint, fog aims to place the data close to the end user. A good example is industrial environments such as oil and gas and manufacturing, where edge sensors or devices can generate real-time data through continuously monitoring equipment (for example, robots, pumps, drives, and sensors). In oil and gas, distributed acoustic sensors at an oil well can generate 1–2 TB of data per well per day. Processing the data at the edge can enable real-time use case monitoring, leading to predictive analytics. Instead of sending this data (most of which only confirms that equipment is operating as expected) back to a centralized location, fog allows for it to be processed at the edge. If any issues occur, data can immediately be sent to a central location for action.

■   Speed: Big Data analysis can be achieved quicker and with more predictable results. This also enables real-time analytics, increasing processing speed at the source and allowing true real-time processing versus today’s near-real-time.

■   Mobility: By controlling data and compute capabilities at multiple points throughout the system, users and mobile endpoints such as cars can offload computation to nearby fog nodes and move computation between fog nodes.

■   Cross-vertical replicability: Fog is already being adopted and implemented across multiple market segments, with industrial, enterprise, and commercial application.

■   Heterogeneity: Fog extends to cover nodes and gateways from any vendor, making it suitable for IoT deployments because they are typically mixed-vendor systems.

■   Cost: Reducing the amount of data that needs to be transmitted across networking links also reduces bandwidth or transmitted data. This minimizes cost and also frees up network resources for other services.

■   Security and governance: Minimizing the frequency by which data is transmitted, as well as the distance it must travel, reduces the potential attack surface. From a compliance and governance perspective, some countries or industries do not allow data to move between different geographic locations or to be handled in the cloud. Fog reduces these challenges by processing and storing locally.

In practice, fog also brings some potential challenges:

■   Proximity: Placing cloud-type services such as computing and storage in geographically dispersed locations renders unavailable much of the powerful shared computing and virtualization of services offered by cloud. Overall solution costs could well rise. Uniform access to services and resources, regardless of location, also disappears. For some mobility use cases, this could be a challenge. The design and placement of resources must be carefully considered.

■   Security: The sheer disparity of fog equipment and protocols means a potentially wide and varied attack surface, making it difficult to provide a uniform way to cover everything. Fog nodes and edge gateways are particularly vulnerable to attack because of their distributed geographic locations and often limited device resource capability to provide security mechanisms. This is even more of a challenge because different organizations often maintain and own different fog hardware and do not take the same approaches to security. This also includes physical access to devices across multiple locations. Providing a comprehensive risk assessment across potentially hundreds of thousands of devices is itself a major challenge. This is even harder when mobility and services can move between different fog nodes.

■   Reliability: Geographically dispersed deployments make easily and quickly locating failed or compromised fog nodes more difficult. In addition, the edge fog infrastructure and communication links might not be as reliable as centralized or well aggregated locations. Fog nodes might not be able to provide autonomy of services if they are unreachable, even if the node itself is still fully operational. Standard administrative tasks such as hardware auditing and updates are more difficult. The diagnosis of and tolerance to faults in a highly scaled fog architecture can increase failure probability. Software or hardware issues not picked up in small-scale testing are likely to occur and have a negative effect on system performance and reliability. With the heterogeneity and complexity of IoT systems, it is likely that different types of fault combinations will occur that cannot be predicted through smaller-scale testing.

■   Cost: Fog is more expensive than cloud architectures, which use centralized virtualized shared services, optimized power, and centralized administrative management and support processes.

■   Complexity: Examples include a more complex architecture and challenges in deploying and scheduling services consistently. Applications might be spread across multiple levels of the hierarchy, including edge, core, and cloud, in addition to being mobile. Deciding where and when to schedule computational tasks can be more difficult.

Middleware

The previous sections outlined system views centered more from the top (cloud) and bottom (fog) perspectives. Another approach to IoT architecture is centered in the middle and builds the architecture from this perspective.

IoT systems often consist of heterogeneous devices and networks and disparate applications. Defining and enforcing a common architectural approach among the diverse de-vices and applications is difficult because they might also belong to different vendors or business domains. Heterogeneity and complexity can potentially increase with new technologies or use cases. A common approach that binds these disparate things together is to develop a middleware layer for IoT that acts as the glue, bringing together the mixed applications, communication interfaces, and devices. A middleware architecture can act as a consistent standard among the diverse sensors, endpoints, and applications that have their own numerous communication protocols, data sets, and transport requirements.

An IoT middleware architecture typically consists of four main components (see Figure 5-23): interface protocols to define and enforce a common standard; device abstraction joining heterogeneous components together; centralized management, control, and contextualization; and application abstraction for physical layer communications and required services to the applications.

It is important not to confuse middleware with an IoT platform, even though it does provide the interfacing between lower and higher levels. They both perform different roles and have different architectural elements. Despite the name, middleware also pervades the entire IoT architecture.

Lambda Architecture

The Lambda architecture differs from the approaches presented so far because it does not address any aspect of an IoT implementation except for handling the data flows for analytics. The Lambda architecture (see Figure 5-24) is a data-processing architecture designed to handle massive quantities of data, as might be expected in IoT, by using both batch processing and stream processing methods. This idea is to balance latency, throughput, scaling, and fault tolerance by using batch processing to provide comprehensive and accurate views of batch data, while simultaneously using real-time stream processing to provide views of online data. If needed, the two view outputs can be joined before presentation. This provides a way to bridge the gap between historical versions of the truth and the current real-time solution. Combining traditional batch processing with stream consumption achieves the needs of both in one approach.

In IoT, this is important when applications need to handle real-time data at tremendous speed but might also later need to correct or further analyze the data for optimization or archival purposes.

The term Lambda architecture was first coined by Nathan Marz to describe a generic, scalable, and fault-tolerant data processing architecture. The architecture was created to satisfy the needs for a robust, fault-tolerant system that can stand up to both hardware failures and human mistakes. It should be able to serve a wide range of workloads and use cases where low-latency reads and updates are required. Any resulting system should be linearly scalable and should scale out rather than up.

Full IoT Stack/Universal

Taking a universal approach aims to cover the elements of the IoT stack throughout the hierarchy, without a specific focus on a particular element or viewpoint. This has become more popular in recent years, particularly because customers are maturing their IoT deployments to cover more than point use cases and also to provide more standardization and interoperability. The following examples highlight more popular and mature approaches to IoT architecture.

General Approaches

The following standards or guidelines can be considered general approaches to the full IoT stack.

Internet of Things Architecture Reference Architecture (IoT-A RA)

The European Lighthouse Integrated Project worked for three years to develop the first mainstream Internet of Things architecture and set of foundational IoT building blocks, IoT-A, in 2010. The aim, using a top-down approach based on existing architectural principles and guidelines, combined with prototyping and simulation, was to better understand the technical consequences of the architectural choice for IoT and to leverage this learning to foster emerging IoT use cases. The architectural model is now maintained by the IoT Forum.

Figure 5-25 explains the aims of IoT-A, although this should be viewed as a general depiction. The roots represent a set of communication protocols (6LoWPAN, Zigbee, IPv6) and device technologies (sensors, actuators, tags); the leaves represent the IoT applications that can be built from the sap (data and information) coming from the roots. The trunk is the most important element, representing the architectural reference model (ARM). The ARM is the combination of the reference model and the reference architecture, the set of models, guidelines, best practices, views, and perspectives. The combination should allow fully interoperable IoT architectures and systems to be created.

The goal of the reference architecture is for system designers and architects to adopt it. This is seen in the focus on linking it to existing models, views, and perspectives, making the work of systems designers easier. Starting with existing architectures and solutions means generic baseline requirements can be extracted and used as an input to the design. The IoT-A ARM consists of four parts (see Figure 5-26):

■   The vision provides the rationale for providing an architectural reference model for the IoT. It highlights how the ARM can be used, the methodologies applied to architecture modeling, and the business scenarios and stakeholder viewpoints addressed.

■   Business scenarios, defined as stakeholder requirements, drive the architecture. A holistic view of IoT architectures can be derived from understanding business goals. This also means a concrete instance of the reference architecture can be validated against selected business scenarios.

■   The IoT reference model provides the highest abstraction level and aims to provide a common understanding of the IoT domain. It includes a general discussion of what the IoT domain is and provides an IoT domain model as a top-level description, an IoT information model explaining how IoT information will be modeled, and an IoT communication model to understand specifics about communication among many heterogeneous IoT devices and the Internet as a whole.

■   The IoT reference architecture is the reference for building compliant IoT architectures. It provides views and perspectives on different architectural aspects that are of concern to stakeholders of the IoT, and it centers on abstract sets of mechanisms instead of concrete application architectures. ARM aims to provide best practices so that organizations can create compliant IoT architectures in different application domains, including architectural design choices.

The IoT-A proposes an entity-based reference model (see Figure 5-27).

This model includes the following components:

■   Physical entities: These are the real-world things (machines, equipment, people) that are the essential part of an IoT system. Tags of various types can be attached to physical entities to aid in their monitoring and identification.

■   IoT devices: These devices connect the physical entities to the IoT system. IoT devices consist of sensors that monitor or scan the physical entities to retrieve some information and actuators that act on or change some properties of the physical entities based on digital instructions.

■   Network: IoT devices communicate via a network.

■   Gateways: Gateways provide a connection between the local proximity network(s) and the wide-area access network. They usually contain a management agent that provides remote management capabilities. IoT gateways can contain other entities and provide a wider range of capabilities, including a device data store that can support local edge or fog processing capabilities or that can be a means of dealing with intermittent communications networks.

■   Applications and services: Various kinds exist in most IoT systems, with associated data stores.

■   Operation and management: Other applications, services, and data stores are de-voted to the operation and management of the IoT system itself. These include the device registry data store and an associated device identity service, which provides lookup capabilities for applications and services. A device management application provides monitoring and administration capabilities for the IoT devices in the system. An operational support system provides various capabilities relating to the monitoring and management of the overall IoT system, including administration capabilities offered to users.

■   User interaction: Users gain access to the capabilities of the IoT system through the access and interchange entities. Those entities provide controlled interfaces for service capabilities, administration capabilities, and for business capabilities. Users can include both human users and digital users. Human users typically interact with the IoT system using some kind of user device. Digital systems can use IoT systems, providing for autonomic use of the system. Both user devices and digital users communicate with the rest of the IoT system via the user network.

■   Peer systems: These can be other IoT systems or non-IoT systems. They interact with the IoT system through the network.

■   Security and privacy: Various elements apply across the complete IoT system, including authentication, authorization, certificates, encryption, key management, logging and auditing, and data protection.

The model is then translated into a more detailed functional view (see Figure 5-28) that provides horizontal, inside-domain functions as well as vertical, end-to-end, cross-domain functions. These functional capabilities can then be leveraged to build out the specific vertical or market architectures.

■   The SCD consists of a set of common functional components whose implementation complexity depends on the infrastructure of IoT systems.

■   The ASD domain represents the collection of functions implementing application logic that realizes specific business functionalities for the service providers in the ASD. The ASD has components such as logic and rules, functional components, APIs, and portal functional components.

■   The OMD domain represents the collection of functions responsible for lifecycle management, business support, security and safety management, and regulation management. The management functions enable management centers to issue a suite of management commands to the control systems or the corresponding devices. Lifecycle management provides several types of functional components for the IoT system operations: provisioning, deployment, monitoring, maintenance, prognostics, diagnostics, optimization, and billing.

■   The resource and interchange domain contains the main functional components, including resource management, analytics, resource interchange, and access. This domain performs interchange of resources for the whole IoT system with other systems.

■   The main function of the UD is to provide access to IoT services and information on how to use them. Here the functional components are users and HMIs that provide the interface for users to access, subscribe to, and receive the services provided by the application service domain.

■   PED has sensed physical objects and controlled physical objects, which are the subject of functions in other domains.

■   Cross-domain functions exist in all six domains and include security, safety, resilience, trust and privacy, interoperability, dynamic composition, and automated interoperability.

ITU-T Y.2060

ITU-T Y.2060 provides an overview of IoT, clarifying the concept and scope and identifying fundamental characteristics and high-level requirements of IoT. It contains an IoT reference model that consists of four layers (see Figure 5-29), as well as management capabilities and security capabilities within the four layers.

The application layer contains IoT applications.

The service support and application support layers consist of two main groups. The generic support capabilities are common capabilities that IoT applications can use, such as data processing or data storage; they can be invoked by specific support capabilities. Specific support capabilities cater to the requirements of diversified applications that provide different support functions to different IoT applications.

The network layer includes two types of capabilities. Networking capabilities provide the control functions of network connectivity, such as access and transport resource control, mobility management, and AAA. Transport capabilities provide connectivity for the transport of IoT service and application-specific data, as well as the transport of IoT-related control and management information.

The device layer has two kinds of capabilities. Device capabilities include direct interaction with the communication network, where devices have two-way direct interaction with the communication network, and indirect interaction with the communication network, where devices have two-way interaction with the communication network indirectly through a gateway.

The gateway layer includes several capabilities, including multiple interfaces support and protocol conversion.

The model addresses required management capabilities and security capabilities for each layer. Management capabilities cover the traditional fault, configuration, accounting, performance, and security (FCAPS) classes (fault management, configuration management, accounting management, performance management, and security management).

Generic security capabilities are independent of applications, including mechanisms such as AAA, antivirus, confidentiality, and integrity. Specific security capabilities are closely coupled with application-specific requirements and mobile payment security requirements.

IoT World Forum (IoTWF) Reference Model

The IoT model was created in 2014 as a result of collaboration among the 28 members of the IoTWF’s Architecture, Management, and Analytics Working Group (including Cisco, Intel, GE, SAP, and Oracle). The aim was to define an open system that organizes IoT components into layers and provide a graphical representation of an entire system. In an effort to address the fragmented IoT landscape, the model aims to provide common IoT terminology, bring clarity to how data flows and is processed, and create a framework for a unified IoT industry. An open system is a first step toward IoT product interoperability.

The IoT model is designed to have the following characteristics:

■   Simplification: Breaks down complex systems so that each part is more understandable

■   Clarification: Provides additional information to precisely identify levels of the IoT and to establish common terminology

■   Identification: Identifies where specific types of processing are optimized across different parts of the system

■   Standardization: Provides a first step in enabling vendors to create IoT products that work with each other

■   Organization: Makes the IoT real and approachable instead of simply conceptual

Instead of being a detailed, specific guide on how to deploy an IoT system, the model provides a loosely coupled set of recommendations on how various elements should be considered, to provide a single architectural frame of reference. The model allows processing capabilities throughout the system to be simple, complex, or a mixture. It does not restrict the scope or location of the architectural components; instead, it aims to cover complete functionality for an IoT system through a seven-layer model (see Figure 5-30). The fundamental idea is to provide a level of abstraction and supporting functional interfaces to provide a coherent model of an IoT system. It needs to be a logical and consistent IoT architecture to allow data to be processed contextually, make information meaningful, manage the large scalability, and design responses that provide insight.

Level 1 consists of the physical devices and controllers that might control multiple devices. Elements at this level are physical things, such as sensors and actuators. Among the capabilities devices might have are analog-to-digital and digital-to-analog conversion, data generation, and the capability to be queried and/or controlled remotely.

Level 2 is the connectivity layer. From a logical point of view, this level enables communication between devices and the network, between devices at the same level, and between devices and the processing that occurs at Level 3. This level consists of networking devices, including routers, switches, gateways, and firewalls that make up local- and wide-area networks and that provide Internet connectivity. Level 2 enables devices to communicate with one another and also to communicate, via upper logical levels, with application platforms. The reference model explains that communications and processing are executed by existing networks and do not require or indicate the creation of new or different types of networks. Some legacy devices are not based on IP, and some require proprietary controllers. In these scenarios, gateways or controllers provide connectivity to the network. The layer focuses on reliable delivery, protocol support and translation, security, and network-based analytics.

Level 3 is the fog or edge computing layer that was outlined earlier in this chapter. This level converts network data flows into information suitable for storage and processing at Level 4. Level 3 focuses on high-volume data analysis and transformation, fulfilling a goal of the model in which the most intelligent system initiates information processing as early and as close to the edge of the network as possible. Services at this level include data handling, content inspection, combined network- and data-level analytics, baselining, and event generation.

Level 4 is the data accumulation level, where data in motion coming from multiple devices is converted to data at rest that can be leveraged by higher levels. Applications can access the data as needed on a non-real-time basis. Upper levels operate on a query or transaction basis, whereas the lower three levels operate on an event basis. Level 4 converts event-based data to query-based processing by converting data-in-motion to data-at-rest and reformatting the network packets to database relational tables. This dramatically reduces data through filtering and selective storing.

Level 5 provides data abstraction functions, rendering data and its storage in ways that enable the development of simpler, performance-enhanced applications. This level is responsible for reconciling multiple data formats, ensuring consistent semantics, consolidating data into one place, protecting the data, and normalizing and indexing data.

Level 6 is the application level. It contains any type of application that uses IoT input or controls IoT devices. There is no strict application definition—applications vary widely, based on the industry or market and business needs. Typically, applications interact with data at rest in Level 5, but provisions should be available for applications to interact directly with lower layers.

Level 7 is the collaboration and processes level. It directly reflects the comments from Chapter 1 that IoT (or IoE) not only includes people, but must produce useful data. The IoT system and information it generates are of little value unless they yield actions. The objective is not to just deliver data to the application; it is to empower people to do their work better using the applications that give people the right data, at the right time, in the right way, so they can do the right thing.

Running throughout the model are aspects of security for each level (see Figure 5-31) and for the movement of data between levels. The model emphasizes that security must be end to end and should fulfill a minimum set of capabilities, including securing each device or system, providing security for all processes that occur at each level, and allowing secure movement and communications between and within each level.

The IWF positions the reference model as an industry-accepted framework aimed at standardizing the concepts and terminology associated with IoT. The framework provides an interesting approach by addressing the business and application sides in addition to the technology aspects. It aims to address the full IoT stack. This model is different from others, such as the ITU-T Y.2060 documents that focus primarily on the device and gateway level, with minimum attention paid to the upper layers.

The model is particularly useful in describing the types of functionality required in an IoT system architecture. However, this is where the usefulness stops. It does not go on to recommend interoperability or standards that can be leveraged to fulfill the functions it outlines, nor does it point to any reference designs or reference implementations that provide details and guidelines on how to achieve the functionality. It is also very data centric. Key elements outlined by the research for the next-generation platform include orchestration, automation, and services-led approaches.

oneM2M Reference Architecture

The oneM2M standard focuses on a single, horizontal platform architecture to exchange data between applications. It was formed in 2012 and is seen as the leading global standards body for M2M for IoT. It has support from the world’s leading standards development organizations (SDO), five global information and communications consortiums, and more than 200 companies. The goal is to create a distributed software layer that provides a unification framework for different technologies.

The platform architecture delivers a three-layer model consisting of applications, services, and networks (detailed later in this section). The outcome is a distributed software layer that provides a framework for the internetworking and interoperability of different technologies. oneM2M aims to reuse whatever is already available wherever possible and its goal is to be uniformly relevant to multiple industries and sectors. Derived value is seen through combining M2M communications with Big Data to create new functionality, with benefits coming from the insights instead of the communication itself.

oneM2M does not aim to standardize the full IoT stack; it seeks to standardize inter-faces to make them relevant to an entire IoT stack ecosystem by providing secure interoperability with any vendor, device, or application in the system. The architecture focuses on being network independent, not network agnostic. Network independence involves knowledge and awareness of the network to provide the best service or experience. Being network agnostic means a solution will work regardless of the type of network—however, it does not mean that the solution will work well or that a solution can be optimized, depending on the type of network being leveraged.

oneM2M provides both interoperability and integration of data between different devices, applications, networks, and industries. This is achieved through abstracted heterogeneous data and common tools. Future work focuses on also introducing semantic exchanges to provide more context.

From a security perspective, oneM2M attempts to create solutions that focus on an industry’s security needs instead of the needs of stakeholders in a system.

The oneM2M functional architecture focuses on the service layer aspects and takes an underlying network-independent view of the end-to-end services. The underlying network is used for the transport of data and potentially other services. Figure 5-32 shows the oneM2M layered model for supporting end-to-end (E2E) M2M Services. This consists of three layers: the application layer, the common services layer, and the underlying network services layer.

The oneM2M functional architecture (see Figure 5-33) consists of the following functions:

■   Application entity (AE): The AE is an entity in the application layer that implements an M2M application service logic. Each application service logic can reside in a number of M2M nodes and/or more than once on a single M2M node. Each execution instance of an application service logic is termed an application entity and is identified with a unique AE identifier. Examples include a vehicle tracking application, a remote healthcare application, and an oil well–monitoring application.

■   Common services entity (CSE): The CSE represents a set of common service functions in an M2M environment. Service functions are exposed to other entities through predefined reference points. The reference points are used to access underlying network service entities. Each CSE is identified with a unique CSE identifier. Examples include data management, device management, and location services. These subfunctions offered by a CSE can be logically and informatively conceptualized as common services functions (CSF).

■   Underlying network services entity (NSE): This provides services from the underlying network to the CSEs. Examples include device management, location services, and device triggering. Underlying networks provide data transport services between entities in the oneM2M system. Such data transport services are not included in the NSE.

The common services layer includes a set of common service functions (CSFs; see Figure 5-34). The CSFs provide services to the AEs and other CSEs via the reference points. CSFs contained inside a CSE can also interact with each other.

oneM2M does not specify vertical-specific data formats for exchange between application entities. However, this can be achieved by interworking with other technologies.

More details can be found at http://www.onem2m.org/images/files/deliverables/TS-0001-Functional_Architecture-V1_6_1.pdf.

IEEE P2413 IoT Architecture

The IEEE P2413 working group was formed in 2014 to promote cross-domain interaction and to facilitate system interoperability and functional compatibility in IoT. It brings together multiple organizations from industry, telecommunications, compute, standards, and academia. The group also is in the process of defining an architectural framework for the IoT, including abstractions and a common vocabulary. The aim is to create a “blueprint for data abstraction and the quality quadruple (protection, security, privacy, and safety).”

The architectural framework seeks to accelerate the growth of the IoT market. The IEEE believes a unified approach to the development of IoT systems will reduce industry fragmentation and create a critical mass of multi-stakeholder activities around the world. The standard defines an architectural framework for IoT, including descriptions of various IoT domains, definitions of IoT domain abstractions, and commonalities between different IoT domains (see Figure 5-35). It also provides a reference model that defines relationships among various IoT verticals (transportation, healthcare, and so on) and common architecture elements.

In addition, this framework provides a reference architecture that builds upon the reference model. The reference architecture covers the definition of basic architectural building blocks and their capability to be integrated into multitiered systems. The reference architecture also addresses how to document and, if desired, mitigate architecture divergence. This standard leverages existing applicable standards and identifies planned or ongoing projects with a similar or overlapping scope.

The OpenFog Consortium Reference Architecture

The OpenFog Consortium was founded by ARM, Cisco, Dell, Intel, Microsoft, and Princeton University in November 2015. Membership consists of technology and networking companies, entrepreneurs, and academic bodies, with the aim of providing innovation through its reference architecture.

The reference architecture is driven by a set of core principles (see Figure 5-36) that outline the belief, approach, and intent for the architecture. Each principle (known as a pillar) represents a key capability an IoT system needs to deliver for a horizontal, system-level architecture that provides the distribution of computing, storage, control, and networking functions as close as possible to the data source anywhere from the thing to the cloud. Fog is often misrepresented as edge focused; it covers the full IoT stack in what it terms the “cloud-to-thing continuum.”

The OpenFog Consortium believes a new approach is needed to enable effective and well-architected IoT systems and sees many limitations in cloud technologies. Performance, security, bandwidth, and reliability are concerns that make cloud-only solutions incapable of working for some IoT use cases. The focus of the consortium is to define a detailed architecture to address the infrastructure and connectivity challenges. This architecture centers on information processing and intelligence at the logical edge via fog computing and seeks to meet the automation and scalability requirements posed by IoT deployments.

Consider these descriptions of the pillars:

■   Security: Many IoT applications have privacy-critical, mission-critical, and even life-critical aspects. Any security compromise in the fog network can have severe consequences. Security in the reference architecture is not a one-size-fits-all approach, but it describes mechanisms that can be applied to make a fog node as secure as possible for hardware and software. The security requirements can vary, depending on factors such as market, vertical use case, and node location, but certain foundational parts of the architecture must be in place. The foundational security elements need an approach to discover, attest, and verify all connected endpoints and things before trust can be established. The aim is to create a secure end-to-end compute environment, covering node, network, management, and orchestration security.

■   Scalability: This pillar aims to address the dynamic scalability needed in IoT and is relevant across all fog computing applications and verticals. Scalability considerations include performance, capacity, reliability, security, hardware, and software. Scalability includes the hierarchical properties of fog and its location at logical edges of networks, as well as individual nodes scaling through additional hardware or software. Fog networks’ storage and analytics services should be capable of scaling up, down, and out on demand.

■   Openness: Seen as an essential requirement throughout the ecosystem for IoT platforms and applications, the openness pillar is needed to architect solutions that are fully interoperable even among multiple vendors. Openness includes composability, interoperability, open communications, and location transparency.

■   Autonomy: Autonomy enables fog nodes to continue to deliver their services and functionality even when they lose connectivity to higher levels. All nodes in the IoT architecture should support this concept. Autonomy at the network edge means local nodes can still be used to fulfill their required tasks. Autonomy covers discovery, orchestration and management, security, and operation.

■   Programmability: The programmability pillar allows for a flexible deployment through the support of fully automated programming at the software and hardware layers.

■   Reliability, availability, and serviceability: These are fundamental operational capabilities needed throughout successful system architectures for hardware, software, and operations.

■   Agility: In many IoT scenarios, it is not possible for people to analyze generated data at the speed needed for many business and operational decisions. This pillar focuses on transforming data into actionable insights and deals with the highly dynamic nature of fog deployments and the need to respond quickly to change.

■   Hierarchy: Computational and system hierarchy might not be needed for all IoT architectures, but it is still seen in most deployments.

The reference model can be seen as a logical hierarchy based on the functional requirements of an end-to-end IoT system. Depending on the scale and nature of the scenario being addressed, the hierarchy could be a network of smart and connected partitioned systems arranged in physical or logical layers, or it could collapse into a single physical system (scalability pillar), as in Figure 5-37. Fog nodes can be linked to form a mesh to provide load balancing, resilience, fault tolerance, data sharing, and minimization of cloud communication.

Figure 5-38 shows the abstract architecture. It consists of both perspectives and viewpoints. The perspectives are indicated by the gray vertical bars on the sides of the architecture, representing end-to-end need. These perspectives include the following:

■   Performance: Many IoT use cases require low latency, and many requirements and design considerations are needed to achieve this. Included are elements such as critical computing, time sensitive networking (TSN), and network time protocols.

■   Security: End-to-end security is essential across all IoT fog computing deployment scenarios.

■   Manageability: This covers all aspects of fog deployments, including reliability, scalability, and availability. Automation and orchestration are recognized as essential elements.

■   Data analytics and control: For fog nodes to be autonomous, localized data analytics must be coupled with control. The actuation/control needs to occur at the correct tier or location in the hierarchy, as dictated by the given scenario. It is not always at the physical edge, but it could be at a higher tier.

■   IT business and cross-fog applications: In a multivendor ecosystem, applications must be capable of migrating and properly operating at any level of a fog deployment’s hierarchy. Applications should also have the capability to span all levels of a deployment and not be confined to a single tier.

In addition to perspectives, the abstracted architecture includes stakeholder views:

■   Software view: Spans the top three layers and includes application services, application support, node management (IB), and the software backplane.

■   System view: Covers the middle five layers and includes hardware virtualization down to the hardware platform infrastructure.

■   Node view: Represented by the bottom two layers. Includes the protocol abstraction layer and sensors, actuators, and control.

In the reference architecture, a fog node covers both hardware and software. One or more fog nodes constitutes a solution for a given use case or market.

In an IoT fog infrastructure, security must span everything from the edge of the network to the cloud. Security starts with the fog node hardware. If the node is not designed with appropriate security trustworthiness, it is not possible to build a trustworthy end-to-end IoT fog solution. Once fog nodes are trusted and deployed, a secure fog network can be layered on top of the node infrastructure, providing the basis for secure node-to-node, node-to-thing, and node-to-cloud communication. Figure 5-39 shows the fog security layers.

The goal of the OpenFog Consortium is to increase the number of market segments (use cases) for fog computing and its business value. This covers IoT. The consortium will create testbeds to enable the high-level architecture to be adapted for real-world deployments in different market segments. This is a difference compared to the IoTWF and ITU-T approaches.

Alliance for the Internet of Things Innovation (AIOTI)

AIOTI was launched by the European Commission in 2015 in an attempt to break the silos between leading vertical IoT applications and promote the IoT ecosystem in Europe. AIOTI builds on the work of the IoT European Research Cluster (IERC) by expanding activities toward innovation within and across industries and addressing legal issues and obstacles. The focus is on future IoT research and innovation, standardization, and policy.

The AIOTI WG03 created an IoT reference architecture, consolidating reference architectures from many sources (including IoT-A, IEEE P2413, oneM2M, and the ITU-T).

The functional model of AIOTI consists of three layers (see Figure 5-40). Note that the term layer is used in the software architecture sense, with each layer simply representing a grouping of modules that offers a cohesive set of services:

■   Application layer: This layer contains the communications and interface for process-to-process communications.

■   IoT layer: This layer groups IoT-specific functions, exposing them to the application layer via API interfaces. This layer leverages services from the network layer.

■   Network layer: Services from this layer can be grouped into data plane services (which provide short- and long-range connectivity and data forwarding between entities) and control plane services (such as location, device triggering, QoS, and determinism).

The model describes functions and interfaces between the different elements of an IoT system. Functions do not mandate any specific implementation or deployment and are more loosely coupled. There is not necessarily a direct correspondence to a physical entity in an operational deployment. Grouping multiple functions in a single device or piece of physical equipment is also possible. Functions include the following:

■   App entity: This is an entity in the application layer that implements IoT application logic. It can reside in devices, gateways, or servers and can be centralized or distributed. Examples include a fleet-tracking application entity and a remote equipment health application entity.

■   IoT entity: This entity exposes IoT functions to app entities or IoT entities via an interface. IoT functions might include data storage, data sharing, subscription and notification, firmware upgrade of a device, access right management, location, and analytics. An IoT entity makes use of the underlying network data plane interfaces to send or receive data. It can also be used to access control plane network services such as location or device triggering.

■   Networks: This function refers to different IP-based network technologies (PAN, LAN, WAN, and so on) and consists of different interconnected administrative network domains.

The aim of the AIOTI architecture is to enable the digital representation of physical things in IoT entities. These representations typically support discovery of things by app entities and enable related services such as control or monitoring. To achieve semantic interoperability, the representation of things contains data and metadata. The metadata provides semantic descriptions of the things in line with the domain model and can be enhanced or extended with knowledge from specific vertical domains. The representation of things in IoT Entities is typically provided by app entities or IoT entities residing in devices, gateways, or servers.

Security and management are fully recognized as important features in the AIOTI HLA. Whereas some standards organizations specify security and management as separate features, they are seen as intrinsic to interface specifications. All depicted interfaces support authentication, authorization, and encryption at a hop-by-hop level. End-to-end application level security is achieved via securing interfaces.

Cloud Customer Architecture for IoT

In 2016, the Cloud Standards Council released its Cloud Customer Architecture for IoT. It contains a number of elements needed for any IoT solution, grouped across five domains (see Figure 5-41). These domains include the user layer, the proximity network, public networks, provider clouds, and enterprise networks.

The user layer is independent of any specific network domain. It can be inside or outside any specific domain.

The proximity network domain includes networking capabilities that typically extend the public network domain. Devices and physical entities are part of the proximity network domain. The devices communicate for both data flow and control flow either via a gateway and edge services or directly over the public network via edge services.

The public network and enterprise network domains contain data sources that feed the entire architecture. Data sources come from the enterprise, coupled with new sources from IoT. The public network includes communication with peer clouds.

The provider cloud domain captures data from devices, peer cloud services, and other data sources. It can use integration technologies or stream processing to transform, filter, and analyze this data in real time, and it can store the data into repositories where further analytics can be performed. The processes data is used to generate business intelligence and insight. Insights are leveraged by users and enterprise applications, and can also trigger actuator actions.

Results are delivered to users and applications using transformation and connectivity components that provide secure messaging and translations to and from systems of engagement, enterprise data, and enterprise applications.

The cloud components, which take the domain functions, are positioned within a three-tier architecture consisting of edge, platform, and enterprise tiers. Figure 5-42 shows the capabilities and relationships for supporting IoT using cloud computing. The tiers consist of the following:

■   The edge tier includes proximity networks and public networks where data is collected from devices and transmitted to devices. Data flows through an IoT gateway or, optionally, directly from and to the device through edge services into the cloud provider.

■   The platform tier is the provider cloud, which receives, processes, and analyzes data flows from the edge tier and provides API management and visualization. It also provides the capability to initiate control commands from the enterprise network to the public network.

■   The enterprise tier includes the enterprise network, which consists of enterprise data, an enterprise user directory, and enterprise applications. The data flow to and from the enterprise network takes place via a transformation and connectivity component.

Security and privacy in the architecture aim to address both IT and OT security elements. Furthermore, the level of attention to security and the topic areas to address vary, depending on the application environment, business pattern, and risk assessment. In addition to security considerations, connecting IT systems with physical systems brings with it the need to consider the safety impact of the IoT system. IoT systems must be designed, deployed, and managed so that they can always bring the system to a safe operating state, even when disconnected from communications with other systems that are part of the deployment. The architecture considers several areas of security, including identity and access management, data protection, security monitoring, analysis, and response. These are all applicable to system, application, and solution lifecycle management.

More details can be found at http://www.cloud-council.org/deliverables/CSCC-Cloud-Customer-Architecture-for-IoT.pdf.

Open Connectivity Foundation and IoTivity

The Open Connectivity Foundation (OCF) is an industry group focused on delivering specification standards, interoperability guidelines, and a certification program for IoT devices.

The OCF delivers a framework to enable its deliverables via a specification, an open source reference implementation called IoTivity, and a certification program.

The key goals of IoTivity follow:

■   Common solution

■   Established protocols

■   Security and identity

■   Standardized profiles

■   Interoperability

■   Innovation opportunities

■   Necessary connectivity

In addition to providing architectural components (see Figure 5-43 for an overview), the IoTivity open source software framework aims to implement the OCF standards by ensuring seamless device-to-device connectivity to address the emerging needs of IoT. It is licensed under Apache License Version 2.0 and is available on TIZEN, Android, Arduino, and Linux (Ubuntu) platforms. It supports peer-to-peer, remote access, and cloud-based intelligent services topologies.

IoTivity includes a discovery subsystem and various messaging methods, including connectivity abstraction, remote access over XMPP, and CoAP over TCP. In addition, it has a set of programmable APIs and security features, including onboarding, ownership transfer, provisioning, and access control.

It also has some primitive services (see Figure 5-44) that focus on enabling easier and simpler APIs for application developers. These are mostly designed to run on smart controllers or devices and leverage the IoTivity Base APIs.

Industrial/Market Focused

The focus of the following groups is on a particular domain, industry, or market implementation.

The Industrial Internet Consortium (IIC)

The IIC was founded in March 2014 by AT&T, Cisco, General Electric, IBM, and Intel. It is not a standards organization, but instead was formed to bring together key industry players (vendors, technology innovators, governments, and academia) to accelerate the development, adoption, and use of industrial IoT technologies. The aim of the consortium is to develop the ecosystem to provide interoperability via open standards, reference architectures, and security frameworks, in addition to providing testbeds to ratify solutions. The Industrial Internet reference architecture (IIRA) was released in June 2015, and a corresponding security framework has also been developed (discussed later in this chapter).

The IIRA is for industrial environments (manufacturing, oil and gas, power utilities, transportation, and so on) and specifies an Industrial Internet Architecture Framework (IIAF) that consists of viewpoints and concerns to aid in the development, documentation, and communication of the IIRA. The reference architecture uses a common vocabulary and a standards-based framework to describe business, usage, functional, and implementation viewpoints that it has defined.

The architecture can be functionally decomposed into domains that contain key building blocks applicable across multiple industrial verticals. These functional domains and associated blocks serve as a starting point to formulate a specific implementation architecture. Specific system and use case requirements determine how the functional domains are decomposed, what additional functions can be included or omitted, and what functions can be combined and further decomposed. The architecture (see Figure 5-45) divides the IoT system into five functional domains: control, operations, information, application, and business.

■   The control domain represents common functions performed by industrial control systems. The core of these functional is sensing and actuation. Data is read from sensors, rules and logic are applied, and control is implemented in the physical system. Components or systems implementing these functional components are typically situated close to the physical systems they control and are likely to be geographically distributed. This means they might not be easily accessible by maintenance personnel, and physical security of these systems might require special consideration. Implementation of the functions might involve various levels of complexity and sophistication, depending on system or use case requirements; in some scenarios, some components might not exist at all.

■   The operations domain represents the collection of functions needed for the provisioning, management, monitoring, and optimization of systems in the control domain. These can exist for a single plant environment or across assets.

■   The information domain represents the collection of functions for gathering data from various domains and transforming, sending, or analyzing data so that insight or intelligence can be gathered about the overall system.

■   The application domain represents the collection of functions implementing application logic that delivers the specific business functionalities. Functions in this domain apply application logic, rules, and models to enable optimization in a global scope.

■   The business domain functions enable end-to-end operations of IoT systems by integrating them with traditional or new types of IoT business functions, including those supporting business processes and procedural activities. Example business functions include ERP, CRM, MES, product lifecycle management (PLM), human resources management (HRM), asset management, service lifecycle management, billing and payment, work planning, and scheduling systems.

The implementation viewpoint is concerned with the technical representation of an industrial IoT system. This includes the technologies and system components required to implement the activities and functions prescribed by the usage and functional viewpoints.

From an implementation perspective for the domains, the IIC recommends a three-tier pattern (see Figure 5-46), connected by three distinct networks and supporting an interlinked layered data bus.

The edge tier collects data from the edge nodes, using the proximity network, and implements most of the control domain. The platform tier receives, processes, and forwards control commands from the enterprise tier to the edge tier and implements most of the information and operations domains. The enterprise tier receives data flows from the edge and platform tier, issues control commands to the platform tier and edge tier, and implements most of the application and business domains.

The proximity network connects the sensors, actuators, devices, control systems, and assets, collectively called edge nodes. It typically connects these edge nodes, as one or more clusters related to a gateway that bridges to other networks. The access network (which could be a private network or could span a public infrastructure) enables connectivity for data and control flows between the edge and the platform tiers. The service network enables connectivity between the services in the platform tier and the enterprise tier, and the services within each tier. It can be an overlay private network over the public Internet or the Internet itself.

The architecture promotes a layered data bus (see Figure 5-47), which is common across IIoT systems in multiple industries. It focuses on providing low-latency, secure, peer-to-peer data communications across logical layers of the system. A data bus is a logical connected communication space that implements a common schema. Each layer of the architecture implements a common data model and data bus, allowing interoperable communications between endpoints at that layer and between applications and devices.

Industry 4.0

Industry 4.0 (Industrie 4.0) dates to 2012. A set of implementation recommendations were presented to the German government, focusing on automation and data exchange in manufacturing technologies (including IoT, cyber-physical systems, cloud computing, and cognitive computing).

Industry 4.0 focuses on digital manufacturing and creates a smart factory. In the smart factory, cyber-physical systems monitor physical processes, create a virtual copy or digital twin of the physical world, and make decentralized decisions. Leveraging IoT technologies, cyber-physical systems communicate and cooperate with each other and with humans in real time. Industry 4.0 accomplishes the following:

■   Connects and merges production with information and communications technology

■   Merges customer data with machine data

■   Allows machines to communicate with machines

■   Takes a decentralized approach, allowing components and machines to autonomously manage production

■   Has security built in by design; security is a precondition and enabler

The aim is not to replace the traditional business requirements in manufacturing: productivity, error prevention, and flexibility. Instead, the goal is to provide completeness of interoperability to better support overall equipment efficiency, predictive maintenance, and product/process quality. Four design principles in Industry 4.0 support companies in identifying and implementing Industry 4.0 scenarios:

■   Interoperability: Machines, devices, sensors, and people can connect and communicate with each other as well as possible.

■   Information transparency: Information systems create a virtual copy of the physical world by supplementing digital plant models with sensor data.

■   Technical assistance: Assistance systems can support humans by comprehensibly aggregating and visualizing information to support making informed decisions and solving problems. Cyber-physical systems can physically support humans by conducting a range of tasks that are unsafe or undesirable for humans.

■   Decentralized decisions: The capability of cyber physical systems to make decisions on their own and perform tasks autonomously.

Industry 4.0 includes a services-oriented reference architecture model called RAMI 4.0. This is a three-dimensional map (see Figure 5-48) that shows how to approach manufacturing in a structured manner, breaking down complex processes such as data privacy and security into easy-to-understand packages.

This three-layered distributed architecture takes into account the requirements for autonomy and self-sufficiency of each production site and balances the workload among the edge, the plant, and the enterprise.

The architecture defines communication structures and facilitates a common language with its own signs, alphabet, vocabulary, syntax, grammar, semantics, pragmatics, and culture.

The aim is to move from the current Industry 3.0 architecture, which is hardware focused (see Figure 5-49). Functions are bound to hardware, communication is based on hierarchy, and products are isolated. Industry 4.0 seeks to move toward flexible systems and machines, with functions being distributed throughout the network, participants interacting across hierarchy levels, communication occurring among all participants, and product becoming part of the network.

OPC Unified Architecture (OPC UA)

The OPC Unified Architecture (UA) was released in 2008, providing a platform-independent service-oriented architecture that integrates the functionality of the individual OPC Classic specifications into one extensible framework. The standard is specified in the IEC 62541 standard series and provides capabilities through which systems and devices can communicate by sending messages between clients and servers over various networks. The primary components are robust and secure communication transport and standard data modeling.

The secure transport defines different mechanisms optimized for different use cases. The first version of OPC UA defined an optimized TCP protocol for high-performance private network communication, in addition to mapping to accepted Internet standards such as Web Services, XML, and HTTP. It uses a message-based security model known from Web Services. The abstracted communication model is not protocol specific and allows new protocols to be added in the future. Currently, two communication patterns are specified: client/server and publish/subscribe. For the publish/subscribe model, a new mapping to Time-Sensitive Networking (TSN) is in development. This enables deterministic networking and supports use cases that need strict QoS requirements.

The data modeling defines the rules and building blocks needed to expose an information model. It defines entry points into the address space and base types used to build a type hierarchy.

The service-oriented aspects provide the interface between servers, as suppliers of an information model, and clients, as consumers of the information model. Services are defined in an abstract manner and leverage the transport mechanisms to exchange the data between client and server.

This concept is to enable a client to access only the pieces of data it requires, without needing to understand the whole model that could be exposed by complex systems. Figure 5-50 demonstrates how the different layers of information models defined by OPC map to those from vendors or other organizations.

Information is conveyed using OPC UA–defined and vendor-defined data types. Servers define object models that clients can dynamically discover. Servers can provide access to current and historical data, as well as alarms and events to notify clients of important changes.

OPC UA provides the protocol and services for exchanging complex data between applications that were developed independently. It also enables syntactic interoperability between clients and servers exchanging information.

This multilayered approach aims to create an architecture that delivers platform independence, is secure and extensible, and provides comprehensive data modeling:

■   Platform independence: OPC UA functions on traditional PC hardware, cloud-based servers, PLCs, and microcontrollers. It runs in Microsoft Windows, Apple OSX, Android, and Linux. This provides the necessary IoT infrastructure for interoperability across the enterprise, from machine to machine, machine to enterprise, and everything in between. Several open source implementations exist.

■   Security: OPC UA provides a suite of controls, including transport, session encryption, message signing, sequenced packets, authentication, user control, and auditing.

■   Extensibility: The multilayered architecture provides a “future-proof” framework. New technologies, protocols, standards, and applications can be incorporated into OPC UA while maintaining backward compatibility.

■   Information modeling framework: OPC UA turns data into information. With complete object-oriented capabilities, even complex multilevel structures can be modeled and extended.

OPC UA has been suggested as an appropriate framework for IoT because systems consist of a series of technologies that have traditionally not been connected or integrated but now are connected to an IP-based network. The interoperability aspects of OPC UA promote IoT interoperability where standardized communication and data modeling is needed. This includes interoperability at scale across all layers of an IoT architecture.

OPC UA has been gaining traction as a standard for connecting operations to the business domain. It continues to pursue interoperability in industrial automation by creating and maintaining open specifications that standardize communication of acquired process data, alarm and event records, historical data, and batch data to multivendor enterprise systems and between production devices. The aim is to continue to push for interoperability for moving information vertically from the factory floor through the enterprise of multivendor systems, as well as providing interoperability between devices on different industrial networks from different vendors.

As an example of this, Microsoft announced that it is working with the OPC Foundation to enable IIoT scenarios through interoperability between the applications and industrial equipment that is compliant with the OPC UA standard (see Figure 5-51). Microsoft will enable its IIoT customers to connect to a range of manufacturing equipment and software that can even cover both new and legacy equipment with support of the OPC UA open source software stack.

The OPC Foundation and W3C have signed a memorandum of understanding in which both organizations agree to closely cooperate to ensure IoT interoperability. The collaboration provides the infrastructure for the Industry 4.0 reference architecture, facilitating the smart factory vision.

The OPC Foundation and the OpenFog Consortium have also announced a liaison to collaborate on technical specifications, whitepapers, guidelines, and processes to enable systemwide IoT interoperability around manufacturing. Deliverables from this collaboration should result in a cohesive set of standards for digital factories.

OPC UA is the standard protocol for M2M in Industry 4.0.

In addition to the broader industry architectures we have just discussed, there are a number of market- or domain-specific implementations for IoT environments. We provide a couple examples from Cisco to look at the manufacturing and power utility segments and demonstrate how the more conceptual models translate into specific industry requirements.

Cisco and Rockwell Automation Converged Plantwide Ethernet

The Converged Plantwide Ethernet (CPwE) Design and Implementation Guide represents a collaborative development effort between Cisco Systems and Rockwell Automation. It provides an Ethernet and IP networking–based architecture for industrial Ethernet applications. This effort applies across many industrial verticals (with a focus on manufacturing) in which organizations are seeking to integrate or upgrade their Industrial Automation and Control System (IACS), including making them IIoT/IoT ready.

The purpose of the CPwE architecture, as a set of manufacturing-focused reference architectures, is to accelerate the successful deployment of standard networking technologies and the convergence of manufacturing and enterprise/business networks. The solution architecture and relevant design and implementation guidelines help provide the confidence and background necessary to successfully deploy standard networking technologies and integrate IACS and business networks. By adopting the solution architecture, the manufacturing process will operate at higher levels of performance, efficiency, and uptime than under the previous solutions. At the same time, the solution must safely and securely integrate the IACS into the broader manufacturing environment; only at this point will all the benefits be available to the manufacturing enterprise.

More information can be found at https://www.cisco.com/c/en/us/solutions/enterprise/design-zone-manufacturing/landing_ettf.html.

Cisco Smart Grid Reference Model: GridBlocks

The Cisco GridBlocks reference model provides a forward-looking view into how the electrical grid can be integrated with digital communications across the entire power delivery chain. The model is a starting point for creating utility-specific designs, and it offers guidance on deployment of grid-specific applications. It also lays out a framework for designing and deploying comprehensive management and security solutions across the grid.

More information can be found at https://www.cisco.com/c/dam/en_us/solutions/industries/docs/energy/overview_gba.pdf.

Although many industry- and market-focused approaches exist, and there is an ever-growing cooperation between IT and OT, a number of challenges still need to be addressed for industrial and heavy industries as architectural approaches become more standardized:

■   Common known IT security issues must be prevented from infiltrating the operational environment.

■   Reliability and stability are needed for critical operational activities, including very short and stable latency times and real-time closed loop control.

■   The integrity of production processes must be maintained.

■   Industrial intellectual property must be protected.

■   The appropriate skills shortage in the OT environment must be addressed.

■   Culture and ethos will not change overnight.

■   As processes become increasingly automated, this might lead to the loss of jobs for more manually oriented tasks.

NFV- and SDN-Based Architectures for IoT

More recently, interest has swelled in leveraging NVF and SDN for IoT. No standardized reference architectures, models, or frameworks exist, but these are underway. Several proposed approaches have been published, some of which are included in the following list. These technologies are seen as the only real way to be able to scale, orchestrate, automate, and provide the reliability that IoT implementations demand.

■   SDN and Virtualization Solutions for the Internet of Things: A Survey, by the IEEE

■   A Highly Scalable IoT Architecture Through Network Function Virtualization (2017), in the Open Journal of Internet of Things

■   A General SDN-Based IoT Framework with NVF Implementation, in Science Direct

■   An SDN-based Architecture for Horizontal Internet of Things Services, by the IEEE

Vendors such as Ericsson, Cisco, Juniper, and Huawei are building out NFV- and SDN-based architectures for both industrial and enterprise IoT scenarios. We have also seen multiple service providers following this approach, including AT&T, British Telecom, Verizon, and Etisalat. Although NFV and SDN have traditionally been the backbone architecture for the service provider IoT networks, we have seen the first service provider implementations being proposed from back-end platforms to edge devices. Figure 5-52 shows an example from Etisalat. This is becoming increasingly important because multiple technologies (including SDN, NFV, and 5G) are converging to offer scalable and secure service-based approaches to IoT. Part II of this book, “Leveraging Software-Defined Networking (SDN) and Network Function Virtualization (NFV) for IoT,” discusses this convergence in detail. It is only a matter of time before we see many more implementations and a move toward standardization in this space.

Leveraging virtual network functions (VNF) enables the use of standardized, flexible, multipurpose hardware to create a more cost-effective and programmable network that IoT demands. Combining NFV and SDN can provide the capabilities to manage a full range of IoT use cases in one network infrastructure, with resources allocated flexibly and in real time.

This section covered a number of approaches to architecting IoT systems. Although there is a lot of variety among them, they also have a lot of consistency. A key message is that IoT platform capability requirements are evolving, mainly because of technology advancements and customer maturity. It is key to note that the end operator requirements are becoming much more business outcome oriented, and architectures and corresponding platform implementations need to reflect this. The IEC 2020 Platform Whitepaper summarizes this well:

The smart and secure IoT platform will allow omnidirectional data flow to/from devices, products and the edge through the network, allowing the collection, storage and analysis of “thing”-related information and the integration of enterprise and IoT-specific applications. This platform will provide the means for processing and lifecycle management of the mass of endpoint data in an integrated way—across different data types and processing technologies. It will enable development of new/innovative applications by combining key platform services/capabilities, and will allow the remote management of smart and secure products and services (configuration changes, software updates, remote control) and connections. Moreover, as an important advancement in the way IoT systems work, the smart and secure IoT platform will support decentralized data (pre-) processing at the gateway, product or device (edge and fog computing).

In summary, we can see common themes in the different architectural approaches. These need to be considered when evaluating the choice of platform to help bring together the disparate IoT components into a solution that meets business objectives:

■   The platform must be capable of orchestrating and automating any of its component parts at scale. This includes devices, applications, services, infrastructure, and users.

■   To best deliver services in a predictable and expected way, the platform should follow an integrated versus OTT approach, with the underlying infrastructure and services running over it and tightly coupled.

■   The platform should be reliable and available.

■   Scalability should be up and down, without changes impacting underlying performance.

■   It should enable an end-to-end data handling mechanism, through an advanced data bus.

■   The platform should reside at any level of the IoT hierarchy, or at multiple levels.

■   The platform should be built around the data the business needs. This can come from a sensor, a machine, or a human.

■   It should be based on common data models and languages, and it should leverage common protocols. Interoperability is key to success.

■   Data should flow north and south, as well as east and west.

Of course, security must be built in inherently at every level.

Approaches to IoT Security Architecture

In this section, we look at several different approaches that have been implemented to some degree in IoT deployments. The first is a general approach that we have seen for many industrial environments and is based on the Purdue Model of Control. The second is the IIC security framework, again focused more on industrial applications. The third comes from the Cloud Security Alliance and is more general in nature. The fourth is the general IoT security framework considerations from OWASP. Finally, we provide an overview of the Cisco approach to IoT security architecture and frameworks.

Before diving into the individual approaches, it is essential to revisit the IT and OT viewpoints mentioned throughout this book. The different perspectives and mindsets from these domains mean that architectures have typically been different and separated. This section outlines the traditional divide and, more important, areas of convergence, particularly driven by IoT/IIoT.

Organizationally, convergence is increasing between historically separate IT and OT teams and tools. This has led to more IT-centric technologies being leveraged for operational activities. Whereas information derived from OT systems is typically used to make physical decisions (such as closing a valve), IT information is typically leveraged to make business decisions (such as process optimization). Regardless of the type of technology or information, the business must treat all security challenges in a similar manner. As the borders blur between these traditionally separate domains, strategies should be aligned and teams must work more closely together to ensure end-to-end security. This includes standards-based and emerging architectures, management, administration and policy, and infrastructure.

OT and IT security solutions cannot simply be deployed interchangeably. The same technology might be leveraged, but how it is architected and implemented could be different. And although IT and OT teams might be part of the same organization, they can have different priorities and often skill sets. Because of these changing needs, in conjunction with the growing amount of regulatory legislation aimed at critical infrastructure protection, there is an urgent need for stronger cybersecurity in OT environments. This should be designed with a defense and detection in-depth approach to mitigate potential damage. It must involve a multilayered, multitechnology, and multiparty (IT, OT, and vendors) strategy to protect critical assets. The reality is that most organizations still have a gap between IT and OT; some are very siloed and others are much more closely aligned. This is unlikely to disappear in the short term. Gartner argues that a shared set of standards, platforms, and architectures across IT and OT will reduce both risk and cost, as well as cover external threats and internal errors.

A properly designed standards-based architecture is critical to secure use cases and systems and to bring together the operational and enterprise domains. Architecture provides an understanding of all components of a use case, maps these elements together in a structured way, and shows how they interact and work together. Architecture not only brings together IT and OT technologies, but it also should include vendors and third-party components to complete the full system architecture—in other words, it should secure the ecosystem. Gartner believes security can be enhanced if IT security teams are shared, seconded, or combined with OT staff to plan a holistic security strategy.

As reference and solution architectures are developed, they must provide a foundation that takes an end-to-end approach, with technologies designed to operate together to minimize risk and operational complexity. These requirements should be implemented across both existing and new use cases, for control systems and emerging technologies such as IoT, Big Data, mobility, virtualized infrastructure, and collaboration.

Purdue Model of Control Hierarchy Reference Model

IEC 62443 is the most widely adopted cybersecurity standard globally for industrial control systems. It evolved from the 1990s, when the Purdue Model of Control reference model (see Figure 5-53) was used by ISA95 to emphasize a security architecture using segmented levels for control system deployments. This was further developed through ISA99 and IEC 62443, which brought additional risk assessment and business processes focus. This segmented, layered architecture forms the basis of many industrial system architectures and industrial control system security architectures.

This model describes various “levels” of applications and controls in a manufacturing enterprise. It describes components from the physical levels of the plant (Level 0) through control equipment and local supervisory control (Level 2).

Level 3 is the manufacturing control level. These are applications that “control” operations and provide a system-level view. This level is often referred to as operations management, based on the various applications inside this level. This is also the domain of manufacturing IT professionals. Level 3 and below are the OT domain.

Levels 4 and 5 are referred to as the enterprise or business levels, and they are the domain of corporate IT.

The Purdue model also describes a hierarchical data flow model, with sensors and other field devices connected to the control system. The control system serves the dual purpose of controlling processes or machines and also serving processed data to the operations management level of applications. Level 3 applications feed information to the enterprise business system level.

An additional level has informally been integrated into the model, mainly because of the divide between OT and IT domains. The Level 3.5 Industrial DMZ (IDMZ) provides a strict segmentation zone and creates boundaries between domains. However, services and data need to be exchanged between the enterprise network and the operational domains. Systems located in the Industrial DMZ, such as a shadow historian, bring all the data together for company personnel in a near-real-time system, exposing near-real-time and historical information to the enterprise for better business decision making.

No direct communications are allowed between the enterprise and the operational domain. The IDMZ provides a point of access and control for the provision and exchange of data between these two entities. The IDMZ provides termination points for the enterprise and the PCD and hosts various servers, applications, and security policies to broker and police communications between the two domains.

Even with the Level 3.5 IDMZ, this foundational approach imposes potential challenges for an IoT/IIoT environment. The data of today, and the data of the future, is not hierarchical in nature. It has many sources and many clients that will leverage the data. We are already seeing systems that can automatically leverage data and initiate processes, and there are federated data structures with storage all over the organization, not just in central locations. Although this approach has been the de facto way to architect industrial networks, it divides IT/enterprise and OT services, with physical or heavily virtualized segmentation implemented to achieve separation. This separation is blurring, and use cases often need a combination of IT and OT services to provide the maximum business benefit.

IT/OT convergence is inevitable, although the extent of this convergence and which parts of the business are affected is not well understood. To deliver transformational operational use cases, such as real-time analytics for machine health monitoring, the full IoT stack (infrastructure, OS, applications, data pipeline, service assurance, security, and so on) is required. This means integrated IT-centric services must be deployed alongside OT services to generate business value. As such, IT capabilities are becoming operationalized and are pushing the boundaries of traditional security architectures such as the Purdue Model of Control and IEC 62443. We are seeing more converged standards based on easier sharing of data.

These newer approaches focus on open architectures for peer-to-peer, scalable systems that support edge analytics and local monitoring, decision making, and control. Recent standardized approaches such as the Open Process Automation (see Figure 5-54) mentioned in Chapter 4 are defining open architectural approaches. In many traditional industrial environments today, there are multiple control systems from different vendors, each with its own protocols, philosophies, interfaces, development ecosystems, and security. This makes the introduction of new technologies we might see in IoT challenging, as well as more expensive to integrate and implement. The Open Process Automation approach is to develop a single IIoT-based architecture to address these challenges. At the heart of it, and in other modern approaches such as OpenICE in healthcare, is an integrated real-time service bus providing open architecture software applications and a device integration bus. This aligns closely with some of the earlier IoT architecture approaches we discussed.

The central focus of the architecture in these approaches is a software integration bus, which acts as a software backplane that facilitates distributed control. Building out the rest of the architecture are capabilities that include end-to-end security, system management, automated provisioning, and distributed data management.

From a practical perspective, this brings up another consideration. Most security technologies deployed in operational environments are IT centric, typically enforced as part of IEC 62443-3-3 (system security requirements and security levels). This poses a skill set challenge for operational staff, with IT-centric skills needed to deploy the networking, infrastructure, and compute technologies that OT needs to operate. This creates challenges surrounding who (IT or OT) owns which use cases and services, and where the IT and security skills should reside.

Industrial Internet Security Framework (IISF) IIC Reference Architecture

One of the best-known approaches to IoT for the industrial environment is the IIC Security Framework document and the associated reference architecture. An underlying assumption implies that IT and OT have aspects that interact with each other but can often conflict. To be most successful, these different aspects must converge and be reconciled together into overall system trustworthiness. This document seeks to understand the different perspectives and address the gaps between both viewpoints. It is an integrated element of the IIC reference architecture described earlier in this chapter (see Figure 5-55).

The IISF is a guide for industrial environments that addresses security, privacy, safety, and reliability issues caused by adding new connectivity to historically disconnected devices. The main goal is to set a significantly higher security level than that seen for consumer devices.

The new framework articulates a standard for “trustworthiness” in industrial IoT systems and provides standard definitions for concepts such as risk, threats, metrics, and performance indicators. Unlike traditional industrial control systems, IIoT systems are inherently connected to other systems and people, increasing the potential complexity and attack surface.

The framework divides the IoT environment into three areas: component builders, who create hardware and software; system builders, who build solutions on top of the hardware and software; and the owners and operators of those systems. To ensure end-to-end security, industrial users must assess the level of trustworthiness of the complete system, including all three areas.

The document describes characteristics that affect the trust decisions of an IIoT deployment:

■   Key system characteristics include security, safety, reliability, resilience, and privacy.

■   Nonkey characteristics include scalability, usability, maintainability, portability, and composability.

The focus of the architecture is to provide assurance of the key system characteristics. Assurance requires collecting and analyzing evidence that supports the design, construction, deployment, and testing of an IoT system, as well as the activities that take place during operation. The correct blend of innate system capabilities and compensating security controls must be demonstrable through evidence. Assurance includes conducting risk analysis to identify hazards and prevent incidents or accidents, including rigorous design and validation testing. The following list defines the key system characteristics:

■   Security is seen as protecting the system from unintended or unauthorized access, change, or destruction. It is seen as a continuum. The document recognizes that no IoT system can be fully secure in every context, so the design or risk assessment must explicitly outline specific contexts deemed relevant, along with the mitigating security controls that the stakeholders expect.

■   Safety is defined as the system operating without causing (directly or indirectly) unacceptable risk of physical injury or damage to the health of people or to the environment.

■   Reliability is seen as the capability of the full system, or parts of it, to perform as specified under normal operating conditions for a determined period of time. System availability is also called out, taking into consideration planned scheduled maintenance, updates, repairs, and backups. When these tasks take place, reliability is reduced because the system is not operating; however, they do not affect reliability if they are properly scheduled.

■   Resilience refers to how a system behaves to avoid, absorb, and manage dynamic challenging conditions while still completing predefined activities. It is recommended to design the system so that failures are compartmentalized, meaning that a single function failure does not impact other functions.

■   Privacy is defined as the right of an individual or group to control or influence what information related to them can be collected, processed, and stored. This also includes who is collecting the information and who has access to it.

In addition, the reference architecture includes several recommendations:

■   The architecture of the system must provide end-to-end security from the edge endpoint, to the cloud. This includes endpoint hardening and onboarding, protection of communications, policy and update management, secure remote access, and analytics that monitor the entire security process. Wherever possible, security and real-time situational awareness should cover IT and OT without impacting operational business processes.

■   Security should be fully integrated into the architecture and system design, not added as an afterthought. Security must be built into the design and risks should be evaluated early instead of trying to bolt on security as an afterthought. This is easier for greenfield deployments but much more difficult for existing scenarios.

■   Any design should include an ecosystem approach. Anyone who is part of the system (vendors, systems integrators, and equipment owner/operators) must be involved in any security risk assessment, architecture, or design.

■   No single “best way” exists for implementing security and achieving adequately secure behavior. Technological building blocks should support a defense-in-depth strategy that maps logical defensive levels to security tools and techniques.

■   No “one size fits all” solution exists. Multiple subnetworks and differing functional zones can have different operating technologies and security requirements. Security tools and techniques built for IT environments might not always be well suited for OT environments.

■   An essential element is that IIoT system security should rely on automation as much as possible, with users able to interact with the security implementation to monitor status, review analytics, make decisions when needed, and plan modifications and improvements. Humans can cause security challenges through misconfiguration and errors.

The functional viewpoint of the security framework consists of six interacting building blocks (see Figure 5-56). They are organized into three layers. The top layer includes the four core security functions of security configuration management, security monitoring and analysis, communications and connectivity protection, and endpoint protection. These functions are supported by a data protection layer and a systemwide security model and policy layer.

The document describes the role of each of these functions in creating and maintaining the overall security of the system. Endpoint protection implements defensive capabilities on devices wherever they are located in the system and emphasizes physical security of the device, cybersecurity techniques, and device identity through an identity management system or authority.

The communications and connectivity protection leverages the authoritative identity capability to authenticate and authorize data traffic. Cryptographic and information flow control techniques should be used to protect communications and connectivity.

When endpoints are protected and communications are secured, system state must be preserved throughout its operational lifecycle, through security monitoring and analysis and also controlled security configuration management for all system components.

The data protection function covers data-at-rest in the endpoints, as well as data-in-motion in the communications. It also encompasses any data generated through the monitoring and analysis function, and all system configuration and management data.

The security model and policy layer orchestrates how the functional elements work together to deliver an end-to-end security strategy. This layer governs how security is implemented and focuses on associated policies to ensure that confidentiality, integrity, and availability of the system are maintained throughout the lifecycle.

These six building blocks provide guidance for implementing security end to end across IIoT systems in the context of trustworthiness. To translate these blocks into a specific implementation, the reference architecture recommends eight design principles. Any number of these can be applied when implementing the functional building blocks:

■   Principle of economy of mechanism: Keep the design as simple and small as possible.

■   Principle of fail-safe defaults: Base access decisions on permission instead of exclusion.

■   Principle of complete mediation: Every access to every object must be checked for authority.

■   Principle of open design: A design should not be secret. The mechanisms should not depend on the ignorance of potential attackers; they should be based on the possession of specific, easily protected keys or passwords.

■   Principle of separation of privilege: Where feasible, a protection mechanism that requires two keys to unlock it is more robust and flexible than one that allows access to the presenter of only a single key.

■   Principle of least privilege: Every program and every user of the system should operate using the least set of privileges necessary to complete the job.

■   Principle of least common mechanism: Minimize the number of mechanisms common to more than one user and depended on by all users.

■   Principle of psychological acceptability: It is essential that the human interface be designed for ease of use so that users routinely and automatically apply the protection mechanisms correctly.

Cloud Security Alliance Security Guidance for IoT

The Cloud Security Alliance (CSA) document “Security Guidance for Early Adopters of the Internet of Things” was released in 2015 to provide specific control guidelines for building IoT solutions. The guide is aimed at the overall security approach for early adopters in the IoT, but it also includes some specific guidance instead of just general security insights. It also raises some good points and highlights areas in which GlobalSign makes a strong partner to beef up the security controls in your IoT ecosystem or application. It highlights seven security controls (see Figure 5-57) that are tailored to IoT environments, with the goal of mitigating risks associated with the new technologies and environments.

This document is intended to provide guidance on the secure implementation of IoT-based systems, with traditional enterprise security solutions that have not been seen as capable to address the security needs of IoT. The CSA sees IoT introducing such new challenges as the following:

■   Increased privacy concerns that are often confusing

■   Platform security limitations that make basic security controls challenging

■   Ubiquitous mobility that makes tracking and asset management a challenge

■   Mass quantities that make routine update and maintenance operations a challenge

■   Cloud-based operations that make perimeter security less effective

The document provides a generic set of security controls for IoT but recognizes that each industry, and each implementation, needs to be evaluated for security weaknesses. Some level of customization is always required for each IoT implementation.

The CSA believes that adopters of IoT face the following top challenges. Therefore, the CSA provides a mapping to the recommended security controls detailed earlier in Figure 5-55:

■   Many IoT systems are poorly designed and implemented, using diverse protocols and technologies that create complex configurations.

■   The IoT encompasses edge devices, messaging and transport protocols, application programming interfaces (API), data analytics, storage, software, and various other technology concepts. #2: Apply a secure systems engineering approach to architecting and deploying a new IoT system.

■   Few mature IoT technologies and business processes exist. #2: Apply a secure systems engineering approach to architecting and deploying a new IoT system.

■   Limited guidance is available for lifecycle maintenance and management of IoT devices. #5: Define lifecycle controls for IoT devices.

■   The IoT introduces unique physical security concerns. #3: Implement layered security protections to defend IoT assets.

■   IoT privacy concerns are complex and not always readily evident. #1: Analyze privacy impacts to stakeholders and adopt a privacy-by-design approach to IoT development and deployment.

■   Limited best practices are available for IoT developers. #2: Apply a secure systems engineering approach to architecting and deploying a new IoT System.

■   Few standards exist for the authentication and authorization of IoT edge devices. #6: Define and implement an authentication/authorization framework for the organization’s IoT deployments.

■   No best practices apply to IoT-based incident response activities. #5: Define lifecycle controls for IoT devices.

■   Audit and logging standards are not defined for IoT components. #7: Define and implement a logging/audit framework for the organization’s IoT ecosystem.

■   Restricted interfaces are available to interact IoT devices with security devices and applications. There is no focus yet on identifying methods for achieving situational awareness of the security posture of an organization’s IoT assets. #3: Implement layered security protections to defend IoT assets; #6: Define and implement an authentication/authorization framework for the organization’s IoT deployments; #7: Define and implement a logging/audit framework for the organization’s IoT ecosystem.

■   Security standards for platform configurations that involve virtualized IoT platforms supporting multitenancy is immature. #3: Implement layered security protections to defend IoT assets.

CSA describes in detail the following security controls for organizations implementing IoT capabilities. These controls have been tailored to IoT-specific characteristics (see Figure 5-58) to allow early adopters of IoT to mitigate many of the risks associated with this new technology. The controls map directly to the seven security controls previously outlined.

In addition to identifying measures to mitigate known IoT security threats, the CSA highlights a series of areas for the future that should form part of a security architecture and design but that are currently seen as barriers to adoption and holistic approaches.

■   A lack of standardization affects all aspects of IoT.

■   Situational awareness of the IoT security posture. Industry must work on ways to identify security-relevant data from the streams of operational data provided by and analyzed within IoT.

■   Information sharing is needed. Because of the scale of devices within the IoT and the newness of IoT technology in general, a stream of zero-day attacks could be likely, to exploit weaknesses discovered in IoT implementations. To reduce the period of exposure to these new exploits, organizations should consider joining an information sharing and analysis center for IoT.

■   Better understanding of the software defined perimeter and IoT is needed. With many use cases in IoT leveraging the cloud, the notion of a physical security perimeter is becoming obsolete.

■   Privacy in the IoT environment must be better examined. Significant concern centers on the issue of data capture from sensors the consumer is unaware of. In these instances, an organization or another individual could be watching or tracking someone without that person’s knowledge.

Open Web Application Security Project (OWASP)

OWASP has created a number of IoT architecture and framework initiatives since 2014. The OWASP Internet of Things Project is designed to help manufacturers, developers, and consumers better understand the security issues associated with IoT. It also seeks to enable users in any context to make better security decisions when building, deploying, or assessing IoT technologies. This includes recommendations such as an IoT framework; a set of security guidelines for manufacturers, developers, and consumers; a series of IoT security principles and testing guides; and a set of potential IoT attack areas. A few of these areas are outlined in this section. For more details, visit https://www.owasp.org/index.php/OWASP_Internet_of_Things_Project.

The OWASP IoT Framework Security Considerations focus on leveraging a secure framework to ensure that developers and architects of IoT systems do not overlook security considerations. This should also expedite development. Any framework should have security built in at every level, making it easier for the architects and developers to concentrate on capabilities.

The framework outlines a vendor-agnostic set of evaluation criteria that architects can leverage to measure security strengths of IoT development frameworks. The evaluation criteria are divided into four sections that represent typical IoT system architectural control points: edge, gateway, cloud platform, and mobile. Each section addresses specific security concerns and appropriate mitigation techniques.

Cisco IoT Security Framework

Cisco has developed a framework for IoT security that serves as a useful complement to the IoT World Forum Reference Model discussed earlier in this chapter. Figure 5-59 illustrates the security environment related to the logical structure of an IoT, with the aim of addressing the highly diverse IoT environment and related security challenges.

The model is a simplified version of the IoT World Forum Reference Model and consists of four levels: smart objects/embedded systems, fog/edge network, core network, and data center/cloud. Various mechanisms span these levels.

■   Role-based security: Role-based access control (RBAC) systems assign access rights to roles instead of individual users. In turn, users are assigned to different roles, either statically or dynamically, according to their responsibilities. RBAC enjoys widespread commercial use in cloud and enterprise systems and is a well-understood tool that can be used to manage access to IoT devices and the data they generate.

■   Anti-tamper and detection: This function is particularly important at the device and fog network levels, but it also extends to the core network level. All of these levels can involve components that are physically outside the area of the enterprise that is protected by physical security measures.

■   Data protection and confidentiality: These functions extend to all levels of the architecture.

■   Internet Protocol Protection: Protecting data in motion from eavesdropping and snooping is essential at all levels.

The framework also defines four components of a security capability for IoT that encompasses all the levels (see Figure 5-60)—authentication, authorization, network enforced policy, and secure analytics: visibility and control.

■   Authentication: Used to provide and verify the identity information of an IoT entity. When connected IoT endpoints need access to the IoT infrastructure, the trust relationship is initiated based on the identity of the device. Unlike typical enterprise devices, which can be identified by a human credential (username, password, token, biometrics, and so on), in IoT, store and present identity information could be substantially different for the IoT devices. Identifiers can include RFID, a shared secret, X.509 certificates, and the MAC address of the endpoint, among others.

■   Authorization: Controls a device’s access throughout the IoT system and leverages the identity information of an entity to enforce access control. With authentication and authorization components, a trust relationship is established between IoT devices to exchange appropriate information. This allows IoT related services to be carried out.

■   Network enforced policy: Encompasses all elements that securely route and transport endpoint traffic (control, management, or data) over the infrastructure.

■   Secure analytics, including visibility and control: Defines the services by which all elements (endpoints and network infrastructure, including data centers) can participate to provide telemetry, for the purpose of gaining visibility and eventually controlling the IoT system. Combining this with analytics provides the capability to conduct real statistical analysis on the security data to pick out anomalies. It includes all elements that aggregate and correlate the information, including telemetry, to provide reconnaissance and threat detection.

With the increasing complexity of IoT systems, the scale of many projects, and the human factor, it is necessary to orchestrate the visibility, context, and control to drive accurate intelligence. This includes all the functions required for central management and orchestration of IoT devices and services. Orchestration provides visibility of IoT devices, meaning that central management services are securely aware of the distributed IoT device and service collection, including the identity and attributes of each device throughout their lifecycle. Building on this visibility is the capability to exert control, including configuration, patch updates, and threat countermeasures, and to do so at speed through automation.

An important concept related to this framework is a trust relationship, as we have seen in other models, such as the IISF. In this context, the trust relationship refers to the capability of two entities in an exchange to have confidence in the identity and access rights of the other. The authentication component of the trust framework provides a basic level of trust, which is expanded with the authorization component.

The IEC sees future IoT systems providing “holistic security capabilities” that span the whole lifecycle of an IoT system and its components, covering design, development, operation, and maintenance. It firmly believes that, in the industrial space, new capabilities will take into account the interdependencies between IT and OT. From a security perspective, the IEC sees platforms as needing an essential set of security capabilities that include the following:

■   New threat analytics and risk management, as well as self-healing capabilities to detect and defeat potential attacks.

■   Improved system resilience due to trustworthy security collaboration management systems that will span devices, platforms, and different enterprises. These security collaboration systems will include new threat intelligence mechanisms to exchange security-related information in a trustworthy manner between organizations.

■   Advanced capabilities to identify the “things” involved, such as sensors, devices, and services, as well as to ensure data integrity, data ownership, and data privacy.

■   A new federated identity and access management for collecting, integrating, and processing heterogeneous data from different sensors, devices, and systems. New capabilities are also needed to ensure controllable data ownership across enterprise boundaries.

■   New data analytics algorithms and cryptographic methods are required, to leverage the massive volume of data while preserving the privacy of customers and enterprises.

Although these ideas can be considered in parts, no IoT platform has these capabilities today.

The IoT Platform Design of Today

So just where do we start when choosing an IoT platform approach to meet the requirements of the IoT platform of the future IoT? An IoT Analytics report in 2017 found more than 450 platforms on offer, with vastly varying capabilities. Key highlights from the report include the following:

■   IoT platforms remains a fragmented market, with more than 450 vendors.

■   Leading providers are growing, at more than 50 percent.

■   Most platforms (32 percent) focus on industrial use cases (see Figure 5-61).

■   M&A activity is sharply increasing (17 deals last year).

■   Startup funding remains insignificant compared to other industries ($330 million in 2016).

The IoT platform comparison looked at five types of IoT platforms: application enablement platforms, IoT device management platforms, IoT cloud storage platforms (IaaS), IoT analytics platforms, and IoT connectivity back-end platforms. IoT platforms are emerging as the central backbone of IoT deployments and are essential for the development of scalable IoT applications and services. The report found that, despite the huge diversity among platform offerings, U.S. vendors dominated the market in 2017. The United States is also currently the biggest market for IoT deployments using IoT platforms. The prediction, however, is that Asia will emerge as the biggest market by 2021. IoT Analytics predicts that the market for IoT platforms will grow at a compound annual growth rate (CAGR) of 33 percent over the 2015–2021 forecast period, with annual revenue reaching $1.6 billion in 2021 (see Figure 5-62). However, it does not yet break out what percentage of the opportunity each platform type makes up.

One of the main challenges the research uncovered is the confusion over what an IoT platform is. The report describes an IoT platform as an IoT application enablement platform that addresses the full IoT stack. This includes connectivity and normalization, device management, databases, processing and action management, analytics, visualization, a developer ecosystem, orchestration, open external interfaces, and built-in security by design.

The report also describes four other types of platforms that are often referred to as IoT platform but do not meet this definition:

■   Connectivity/M2M platforms that focus mainly on the connectivity of IoT devices via telecommunication networks (for example, via SIM cards) but rarely on the processing and enrichment of different sets of sensor data. (An example of a connectivity platform is the Sierra Wireless AirVantage.)

■   Infrastructure-as-a-service (IaaS) back ends that provide hosting space and processing power for applications and services. These back ends previously were optimized for desktop and mobile applications, but IoT is now also in focus. (An example is IBM Bluemix, not to be confused with the IBM IoT Foundation.)

■   Hardware-specific software platforms that include connected devices and a proprietary software back end. Vendors like to refer to the back end as an IoT platform here; however, because the platform is not open to anyone else on the market, this terminology is not entirely accurate. (An example is Google Nest.)

■   Consumer/enterprise software extensions in which existing enterprise software packages and operating systems such as Microsoft Windows 10 are allowing the integration of IoT devices. Currently, these extensions are often not advanced enough to classify as a full IoT platform—but they might get there soon.

Maturity is one of the interesting aspects of IoT platforms that supports the findings in the IoT Analytics research. Belief in the capabilities offered by IoT, changes in how business are able to operate by taking advantage of IoT data, a more realistic understanding of the security and risk threat, and advances in technologies and offerings from IoT platform providers all mean that end user expectations of the capability of an IoT system have shifted. Figure 5-63 shows the progress in technology maturity.

IoT Analytics identified three major lenses to differentiate IoT platforms: technological depth, segment focus, and implementation/customization approach.

Technological depth refers to the capabilities offered and level of integration an IoT platform provides. Typically, three different levels of technology depth exist. The most basic level is the connectivity platform that provides data collection and transformation, along with a message bus. The next level is the action platform that manages connectivity and allows action-based triggers based on specific events or predetermined rules. The most advanced platform includes capabilities from the lower-level versions as well as other features, such as modularity, open external interfaces, support for a wide range of IoT protocols, and scalability (in terms of devices, services, and data processing). In addition, it is standards based wherever possible.

In 2016, IoT Analytics research estimated that around 75 percent of IoT platforms offered were at the basic connectivity platform level, providing not much more than a messaging bus. However, this is not necessarily a bad thing. An end user of a platform needs to carefully assess what type of platform will provide the right technological depth for the planned use cases of today and the future. This future planning is becoming more of an essential element in platform choice and the architectures needed to support it. Designers of platforms are no longer focused on just delivering experiments or platforms against a single use case. They are planning on standardized transformational IoT platforms that can be leveraged holistically in an organization and that can be tied to business goals and objectives. Figure 5-64 shows the progress of customer maturity.

Other factors will also play a part. Different markets or industries require fundamentally different options; different levels of customization might be required, and companies might have a philosophy to build a platform rather than buy one.

From an IoT platform provider perspective, companies also have various ways to approach building out solutions. IoT Analytics described these strategies in its research:

■   Organic bottom-up approach: Starting with connectivity and building out platform features from the bottom up (for example, Ayla Networks)

■   Organic top-down approach: Starting with analytics and building out platform features from the top down (for example, IBM IoT Foundation)

■   Partnership approach: Creating alliances to offer the full package (for example, GE Predix and PTC Thingworx)

■   Mergers and acquisitions approach: Making targeted acquisitions (for example, Amazon with 2lemetry) or performing strategic mergers (for example, Nokia and Alcatel-Lucent)

■   Investment approach: Making tactical investments throughout the IoT ecosystem (for example, Cisco)

■   New services: Selling the main product as a by-product, while making the business model around the data into the main product

All of this means that we have a fragmented market today, and it is likely to continue for some time. Regardless of the approach a vendor or end user/operator wants to take, the market will continue to develop offerings as technologies and standards develop. The reality is that today’s offerings fall short of the IEC vision of the platform of 2020, and there is some way to go. However, the journey has to start somewhere. The following list describes some of the top IoT platforms for 2018, as ranked by the Internet of Things Wiki:

1. The Amazon Web Services (AWS) IoT platform includes a registry for recognizing devices, a software development kit (SDK) for devices, device shadows, secure device gateways, and a rules engine for inbound message evaluation. This is a cloud-only platform.

2. Microsoft Azure IoT includes device shadowing, a rules engine, an identity registry, and information monitoring. This is a cloud-only platform.

3. Google Cloud Platform focuses on data processing and analytics. This is a cloud-only platform.

4. ThingWorx IoT Platform features connectivity of devices to the platform, IoT application development, a sharing platform among developers for rapid development, and integrated machine learning for automating complex Big Data analytics. It can be deployed in the cloud, embedded or on-premises.

5. IBM Watson includes device management, secure communications, real-time data exchange, data storage, and enhanced data sensor and weather data services. This is a cloud-only platform.

6. Samsung Artik focuses heavily on security.

7. Cisco IoT Cloud Connect is based on Jasper and supports capabilities for voice and data connectivity, SIM lifecycle management, IP session control, and customizable billing and reporting. This is cloud only and is available only to service providers.

8. Hewlett-Packard Enterprise Universal of Things Platform delivers secure monetization, data analytics, and a developer community and marketplace. It follows an M2M approach.

9. The Salesforce IoT Cloud focuses on customer engagement with modules for sales, services, marketing, and inventory applications.

10. BSquare DataV features predictive failure, adaptive diagnostics, device management, condition-based maintenance, asset optimization, and asset utilization. It can be deployed in the cloud, embedded or on-premises.

11. Siemens MindSphere offers machine data applications, data analytics, and security. It is cloud based.

12. Ayla IoT Platform includes embedded agents, a cloud service, application libraries, and a developer ecosystem.

13. Bosch IoT Suite is delivered as PaaS, including analytics. It is cloud based.

14. Carriots IoT Platform is delivered as PaaS and includes device management, an SDK application engine, debug and logs, API key management, a data export feature, custom alarms, customer hierarchy levels, user management, and a custom control panel. It is cloud based only.

15. Oracle Integrated Cloud focuses on Big Data analysis for real-time IoT data, device virtualization, endpoint management, and high-speed messaging. It is cloud based only.

16. General Electric Predix is delivered as PaaS, with a two-tier architecture. It includes Digital Twins for introducing small changes and a Predix Machine that offers a communications layer that brings together sensors, controllers, cloud, and analytics.

17. ARM mbed includes an operating system, cloud services, and developer tools focused on commercial products.

18. LTI Mosaic is a cloud-based platform focusing on secure Big Data and analytics for industrial environments.

19. Mocana by Mocana addresses the full IoT stack, includes a device-to-cloud security model, and is military grade.

20. Kaa is a middleware platform that delivers cloud enablement software for connected devices, customizable middleware, and transport-agnostic links.

In addition to the top 20, the following feature in a number of top 10 lists:

■   Intel IoT Platform: https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/iot-platform-reference-architecture-paper.pdf

■   SAP Cloud Platform: https://eaexplorer.hana.ondemand.com/rest/mimeRepositories/12578/file/IoT%20Reference%20Architecture%202.0.pdf

■   Telit Application Enablement Platform: http://www.insight.tech/retail/iot-middleware-delivers-web-based-application-enablement

■   IBM Bluemix: https://www.ibm.com/cloud-computing/bluemix/internet-of-things

■   RelayR IoT Middleware Platform: https://relayr.io/en/iot-middleware-platform/

■   EdgeX Foundry: https://www.edgexfoundry.org/

■   Cisco Kinetic IoT Platform: https://www.cisco.com/c/en/us/solutions/internet-of-things/iot-kinetic.html

Today there is no one-size-fits-all best platform for every application. Nor is there one platform that can meet all the requirements of a specific vertical or industry. Nor is there a platform that fully meets all the requirements of a particular IoT architectural standard. The market is saturated and likely to be so for some time; it is difficult to say which platforms will survive in the coming years and which will fail. More than 30 of the companies included in the Global IoT Platform Companies List 2016 report now cease to exist because they went out of business or were acquired. The market will need years to settle down and become more standardized.

Platform development is similar to the standards landscape discussed in Chapter 4: It is highly fragmented. With the standards landscape so fragmented, it is understandable that the platform market is, too. Therefore, platforms being developed (and platform choice for end users/operators) are more likely to be based on use case, industry, or business requirements than architecture. However, the architectural choice is essential because it will be the gating factor for either success or limited capability.

As mentioned throughout this book, IoT needs interoperability to succeed. Interoperability is derived from standardization. For the foreseeable future, platform providers and end users/operators will need to assess which standards/parts of standards are most suitable for their business needs and what architectural philosophy they want to adopt. Then they must make appropriate choices and assumptions. Fundamentally, if platforms are proprietary and not looking at open standards and architectures, they will not likely succeed.

The majority of the platforms highlighted in the list are cloud based. Similarly, most are delivered in an OTT way, decoupling them from the underlying infrastructure. Depending on the market or vertical, as well as the use case, this approach might not be suitable, especially when considering the security requirements that end operators are adopting.

Security for IoT Platforms and Solutions

As the proliferation of platforms increases, the expected rise in security technologies to secure them and the underpinning architectures is occurring. As with platform maturity, security technologies and end user/operator security needs have matured. Clearly, not all spending in IoT is platform related, but it clearly is influential. A 2017 IoT Analytics research article describes IoT security as any element that protects IoT solutions against virtual or physical damage at the device, communication, cloud, or lifecycle management layers, or a combination. Clearly, these are aspects of the IoT platform.

Research estimates global 2017 spending on IoT security at $703 million USD; this is predicted to grow at a compound annual growth rate (CAGR) of 44 percent, to become a $4.4 billion USD market opportunity by 2022 (see Figure 5-65). This is based on IoT security revenue from the leading technology companies in the IoT market, across 12 industries and 21 technology areas. As expected, security is outgrowing other IoT markets for these reasons:

■   Increasingly, IoT projects are moving from proof of concept or pilot to deployment and have an additional need for security.

■   Increased regulation is pushing for higher security standards.

■   Connecting legacy equipment to the system requires additional and different security solutions.

■   New threats continue to emerge that were not previously considered, and these need to be secured.

The report also found the IoT security market to be a wide-ranging mix of established technology vendors, start-ups, chip manufacturers, infrastructure providers, and cloud and enterprise software companies. The leading vendor is Cisco systems, but with the market so fragmented, that company holds only a 7 percent market share. More than 40 percent of the market is shared by the top ten vendors. As the complexity of IoT platforms and solutions increases, this number is only likely to grow.

What really stands out in the research is that no vendor has an end-to-end offering for IoT security. Most vendors in this space today have a background in either traditional IT security or OT security. Although all are adapting their security solutions for IoT, no single vendor has an offering that covers all analyzed technology elements of an end-to-end IoT security solution. Even the most comprehensive vendors covered only 12 of the 21 needed security elements. This points to a clear gap in the types of methodologies, recommendations, and guidelines outlined by the standards groups and consortia through their reference models and designs. This also shows a very real need for a different approach that focuses on a standardized architecture to deliver a foundational IoT platform architecture with security designed in at every level and automated wherever possible.

Challenges with Today’s Designs: The Future for IoT Platforms

The next part of the book covers the foundational technologies and architectures needed to develop the next-generation IoT foundational platform. Chapter 8, “The Advanced IoT Platform and MANO,” looks more deeply at how a comprehensive platform approach can be achieved. A new approach to IoT system and platform architecture and technology design is needed to address several limitations that we see with today’s platforms to easily deliver comprehensive security. The following areas need improvement. Although they are being addressed, they are typically approached in an isolated way.

■   Platforms continue to be built without being based on architectural and technological standards, whether IoT based or not. This results in proprietary protocols and architectural approaches, and it limits interoperability, which is a key success factor for IoT. Systems are typically lacking in both interoperability at the protocol and semantic levels and, importantly, composability, which allows a system to adapt and reconfigure dynamically and autonomously according to changes in its environment. A 2017 Gartner article recommends a new architectural approach based on real-time, event-driven processing. Gartner believes an IoT platform must provide situational awareness that can deliver instant understanding and real-time, self-service, unconstrained analytics.

■   Platforms are typically OTT, meaning that IoT applications could be impacted by issues with the underlying infrastructure. Integration between both is needed to best provide the reliability, predictability, and consistency that many IoT applications need.

■   Orchestration and automation capability is limited. To successfully deploy, manage, and maintain thousands or millions of endpoints, services, and applications, we need a simplified way to achieve automation at scale. In addition, it is essential that this address the IoT stack, not just parts of it.

■   Resiliency continues to be an issue, particularly for use cases such as those in industry and healthcare that require predictability, high availability, and reliability. Platforms are typically made up of multiple component parts, but resiliency is typically designed only between similar components (for example, one device fails and another device takes over), not between different components (for example, an application is not performing properly, this is identified through service assurance mechanisms, and the underlying infrastructure automatically adapts to rectify the problem). Reliability is not just about operating consistently; it is also being able to bear changing conditions, security problems, and technology or process changes. Reliability needs to be guaranteed in all aspects of software and hardware at each layer of the IOT system hierarchy.

■   Today’s standards do not necessarily take technology into account, and technology is often the key enabler for use cases. We are only now starting to see technologies such as NFV and SDN being investigated by the IoT standards bodies and consortia, even though both NFV and SDN are mature.

■   Integration with enterprise solutions and existing technologies is lacking in the IoT environments. Platforms are typically designed and built as separate entities, and they ignore much of what an organization has already designed from a security perspective. This means that technologies and processes that have been designed to support a business might not be part of the IoT platform. This can significantly impact the levels of security.

■   Too much focus still centers on collecting data for the sake of collecting data instead of ensuring that the right data is collected and delivered to the right place for it to be consumed, in a format that is correct for the consumer and at the right time.

■   There is a distinct lack of data semantics and data context to enhance understanding of the information so that it can be leveraged most effectively. Better understanding of the IoT environment is needed. This extends to devices, sensors, and human and environmental conditions, and it should be understood and represented via understandable metadata.

■   Platform solutions are still not built to provide the scale needed for massive deployments. They also typically lack easy methods to turn scale up and down, depending on need.

■   Platforms are typically designed to solve a single use case or a limited set of use cases instead of focusing on being a foundational IoT enabler with a consistent set of core capabilities that can be developed on top of.

■   In industrial settings, consolidation of the different requirements of IT and OT is not addressed. This includes domain-specific requirements such as safety and privacy.

■   Most platforms are cloud based. However, in many markets and industries, cloud is not an option because of technology constraints (mainly speed and variability), perceived security weaknesses, and regulation.

■   Mobility is a key enabler for many IoT use cases. However, today’s platforms are often not dynamic enough to cope with mobility needs (including roaming partnerships, security, and identity and billing management). To make networks more agile and resilient to IoT demand, NFV, SDN and the cloud are key enablers (especially when combined) that make networks and services more elastic to demand and allow IoT traffic to be segmented when required.

■   Measuring the performance of IoT can be a challenge. This cannot be evaluated using simple mechanisms because it depends on multiple technologies’ components and performance. In addition huge amounts of data and network traffic are hard to measure. These need to be considered in architecture and platform design, with robust monitoring and service assurance capabilities to enforce KPIs.

■   Security is still an afterthought and is not built in by design. This is true not just in terms of the technologies implemented to protect the system, but also in terms of architectural design to ensure privacy, identity, and trust. IoT does not adhere to common enterprise security standards and architectures, with IoT platforms connecting enormous amounts of heterogeneous devices. This results in increased vulnerability because of the increased number of malware entry points. Therefore, traditional security architectures cannot fully satisfy the security requirements of IoT. With platforms not following a recognized IoT architecture, security challenges are likely to arise because things are done in a proprietary way. Heterogeneous devices or applications from multiple vendors might not be designed to work together as securely as when following an approved process.

■   Certifications based on agreed-upon standards are lacking. This is especially true in terms of security. Only certified devices must be allowed on IoT networks and be permitted to connect to these systems and services.

Clearly, end users/operators cannot wait for all of these challenges to be addressed. A Boston Consulting Group article emphasizes some key choices for platform selection today, understanding that a sound architectural approach is needed but does not fully exist today. BCG believes the first priority is to select a vendor that offers all three IoT capabilities to make up a true IoT platform: application enablement, data aggregation and storage, and connectivity management. Second, the risk an organization is willing to take needs to be considered. Should the platform come from an established vendor that is solid but slow in development? Or should it come from a startup that is flexible but untested? Third, platform flexibility needs to be considered in terms of how existing company technology investments can be integrated and also how new use cases and applications can be developed and integrated. Fourth, openness is seen as a key feature when the IoT platform must meet planned business needs while also allowing deployment with minimal disruption to existing systems. Attention to these factors can make the difference between a fast deployment and one plagued by lengthy and costly delays. Fifth, a balance must exist between horizontal platform requirements for repeatability and vertical-specific needs to deliver against market or industry use cases.

The IEC summarizes its platform of the future as delivering the following fundamental capabilities:

■   Supports expanded sensing capabilities and sensor fusion across multiple interwoven IoT systems and with an eye to tomorrow’s analytics. This will significantly enhance and expand algorithms created to support existing and new concepts for IoT productivity.

■   Leverages new data context mechanisms and data semantics based on new standards. This will provide for enhanced understanding of the information and enable fundamentally enhanced analytics.

■   Offers enhanced security that addresses complex issues such as maximized protection of the device and data privacy requirements from a variety of geopolitical entities.

■   Supports the key security capabilities and policies for IT and OT systems, such as Plan-Do-Check-Act (PDCA), Observe-Orient-Decide-Act (OODA), and cooperative security

As Gartner points out, when it comes to IoT platforms, you must think to the future and think big (see Figure 5-66)!

Summary

For long-term success, vendors and operators need IoT platforms that are scalable, flexible, and inherently secure. This chapter explored the main architectural approaches that underpin design choices and requirements for IoT systems, along with the main guidelines for securing the system. The chapter then looked at some of the leading IoT platforms and a number of challenges that still need to be addressed in their design. The following chapters move on to discuss core capabilities that can enable the foundational IoT platform of the future.

References

Build Your Blueprint for the Internet of Things, Based on Five Architecture Styles. Gartner, 2014. https://www.gartner.com/doc/2854218/build-blueprint-internet-things-based

Gartner Says the Worlds of IT and Operational Technology Are Converging. Gartner, 2011. http://www.gartner.com/newsroom/id/1590814

A General SDN-Based IoT Framework with NVF Implementation, in Science Direct

A Highly Scalable IoT Architecture Through Network Function Virtualization (2017), in the Open Journal of Internet of Things

How to Leverage Events to Empower Decisions in Business Real Time. Gartner, 2017.

https://internetofthingswiki.com/top-20-iot-platforms/634/

https://iot-analytics.com/iot-security-company-list-2017/

https://iot-analytics.com/worldwide-iot-platform-market-reach-1-6-billion-2021/ IoT Analytics IoT Platform market TAM.

https://opcfoundation.org/wp-content/uploads/2016/05/OPC-UA-Interoperability-For-Industrie4-and-IoT-EN-v5.pdf

http://portals.omg.org/dds/what-is-dds-3/ DDS Architecture.

https://www.bcg.com/publications/2017/technology-industries-technology-digital-who-will-win-the-iot-platform-wars.aspx

http://www.cisco.com/c/en/us/about/security-center/secure-iot-proposed-framework.html

http://www.iec.ch/whitepaper/pdf/iecWP-loT2020-LR.pdf

https://www.devteam.space/blog/10-best-internet-of-things-iot-cloud-platforms/

http://www.meet-iot.eu/deliverables-IOTA/D1_3.pdf IoT-A Architecture

IEC 62443 Standard, http://isa99.isa.org/Public/Information/The-62443-Series-Overview.pdf

IoT 2020: Smart and Secure IoT Platform, http://www.iec.ch/whitepaper/pdf/iecWP-loT2020-LR.pdf

IT and Operational Technology: Convergence, Alignment and Integration. Gartner, 2015. http://www.gartner.com/technology/research/infrastructure-operations-management.jsp

Key Findings of the IoT Platform Comparison: https://iot-analytics.com/iot-platform-comparison-how-providers-stack-up/

Mineraud, J., et al. “A Gap Analysis of Internet-of-Things Platforms.” Computer Communications (2016): 22–32. http://dx.doi.org/10.1016/j.comcom.2016.03.015

Purdue Model of Control. Cisco, 2015. http://www.cisco.com/c/en/us/td/docs/solutions/Verticals/CPwE/CPwE_DIG/CPwE_chapter2.html

An SDN-based Architecture for Horizontal Internet of Things Services, by the IEEE

SDN and Virtualization Solutions for the Internet of Things: A Survey, by the IEEE

Study Internet of Things. IDG Research Services, 2016. https://www.vmware.com/ciovantage/wp-content/uploads/2017/01/IDG-IoT-Studie-2016-EN.pdf?src=WWW_70134000001MD80_Worldwide_Radius_2016IoTStudy

Survey Analysis: 2016 Internet of Things Backbone Survey. Gartner, 2016. https://www.gartner.com/doc/3563218/survey-analysis--internet-things

VMware, How to Plan for IoT Success. https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/iot/vmware-planning-for-iot-success-tech-target-whitepaper.pdf