Who Is This Book For?

Programming the Internet of Things is written for programmers by a programmer. It’s also designed for any technologist (programmer or not) interested in learning about, and building, end-to-end IoT solutions, as well as college-level instructors teaching their own connected devices course.

This is not designed as a reference book, nor does it provide an exhaustive deep dive into and analysis of the various protocols commonly used for communicating amongst IoT devices and cloud services. There are extremely well-written specifications for this purpose (although we’ll certainly dive into some of these details to better understand these protocols).

Those who are interested in practicing what is learned in each chapter can move straight into the exercises at the end of each chapter, while those interested more in the concepts and general knowledge can simply move on to the next section. I’ve organized this book so that each of these three groups - Programmers, Instructors, and Information Technology (IT) Executives - can benefit.

To the Programmer

If you’re embarking on your own IoT learning journey as a practitioner, I’m assuming you’re mostly interested in expanding your skillset, as you’ve witnessed the growth of IoT opportunities and want to be part of this important technology evolution. Over the years, I found the step-by-step, “build from the ground up” approach to implementing integrated and open IoT solutions to be most helpful in understanding the complexities of this area, and so I follow this model throughout the book.

The programming examples you’ll encounter throughout this book are constructed from my own journey in learning about the IoT, and have evolved to represent many of the lab module assignments from the graduate-level Connected Devices course I teach as part of Northeastern University’s Cyber Physical Systems program.

Each chapter and exercise builds upon the previous, so if you’re starting out with the IoT, I’d recommend walking through each as-is, in the order given. If you’re experienced in this space and are using this book as a reference guide, you may consider incorporating specific lessons of interest into your own application.

If you’re new to the IoT and unfamiliar with how an end-to-end IoT system comes together through software, I’d recommend walking through the book in order, working through each exercise, and consider application customizations only after you’ve mastered each chapter.

To the Instructor

The contents that form the underpinnings of this book have been used successfully in my graduate-level Connected Devices course for the past three years. This book is the formalization of these lecture notes, presentations, examples, and lab exercises, structured in much the same way as my course.

The original class was designed as an introduction to the IoT, but with student input and suggestions from my Teaching Assistants, it quickly morphed into an advanced, project-oriented software development course designed as one of the final required classes for students wrapping up their Master’s in the Cyber Physical Systems department of Northeastern University. The goal of the course is to establish a strong baseline of IoT knowledge to help students move from academia into industry, bringing foundational IoT skills and knowledge into their respective organizations.

This book is structured so it can be used as a reference guide or even as a major component of a complete curriculum to help you with your own IoT-related course. The content is focused on constructing an end-to-end, open and integrated IoT solution, from device to cloud, using a ‘learn as you build’ approach to teaching. Since each chapter builds upon the previous, you can use this book to guide your students in building their own platform from the ground up.

You can remain up to date on exercises and other relevant content useful for teaching and explaining concepts on the book’s website (https://programmingtheiot.com/programming-the-iot-book/).

To the Technology Manager and Executive

The contents of this book should help you better understand the integration challenges inherent with any IoT project and provide insight into the skillsets required to help your technology team(s) succeed across your IoT initiatives.

If you’re part of this group, my assumption is that you’re mostly concerned with understanding this technology area as a whole – its integration challenges, dev team setup requirements and needs, team skillsets, business user / stakeholder concerns around the IoT, and the potential change management challenges you may encounter as you embark on your organization’s IoT journey.

What Do I Need To Know?

Although the exercises in this book assume that you have experience as a programmer, most do not require sophisticated programming skills or a formal computer science background. However, if you intend to complete the exercises at the end of most chapters, you’ll need to possess a basic level of comfort working with both Python and Java as coding languages to build simple applications comfortably; working in an Integrated Development Environment (IDE); reading, writing, and executing Unit Tests; and configuring Linux-based systems via a shell-based command line.

All exercises are preceded by a to-be design diagram for the specific task at hand, which provides specific detail on the way any new logical components you build are to work together along with the existing components you’ve already developed. Most of these diagrams will follow the guidelines specified by the Unified Modeling Language (UML)¹, and incorporate other non-UML design constructs in order to make the design as clear as possible. A legend is included within those diagrams where I deemed it will add sufficient value, and excluded where I felt it just adds noise.

For technology executives and managers, you don’t need to implement the exercises yourself, but it will be helpful to read through the entire book so you understand the challenges your team will likely encounter.

For business stakeholders interested mostly in understanding what the IoT entails, I recommend reading – at a minimum – the Overview section of each chapter, then focusing your efforts on the final chapter, which discusses a handful of practical cases, scenarios, and implementation suggestions.

How Is This Book Arranged?

This book will take you through building an end-to-end, full-stack, and integrated IoT solution using various open source libraries and software components that you’ll build – step-by-step. Each of these components will be part of the larger system, and you’ll see how they interconnect with and map into an end-state architecture as you work through each chapter’s exercises.

If you’re not planning on implementing the exercises and just want to get your feet wet with various IoT technologies, feel free to skip around to those sections that are most interesting to you and relevant for your needs.

I’ve grouped like chapters together and used this scheme to establish the four parts of the book: Part I - Getting Started, Part II - Connecting to the Physical World, Part III - Connecting to Other Devices, and Part IV - Connecting to the Cloud. Let’s review each in a bit more detail.

Part I – Getting Started

In this section, we build our initial foundation for IoT development. You’ll start by creating a development and testing environment, and wrap-up the section by writing two simple applications to validate our environment is working properly.

• Chapter 1 is the longest chapter in the book as it lays the groundwork for how you’ll build your end-to-end solution. It will help you establish a baseline of IoT knowledge and set up your workstation and development environment so you can be productive as quickly as possible. I’ll cover some basic IoT terms, create a simple problem statement, define core architectural concepts, and establish an initial design approach that I’ll use as the framework for each subsequent exercise.

Note

The IoT consists of a plethora of heterogeneous devices and systems, and there are many tools and utilities available to support development and system automation. The tools and utilities I use throughout the book fall into the open source category, and represent a small subset of those available to you. These shouldn’t be taken as carte blanche recommendations, of course, as you may have your own that you prefer to work with and are best suited for your specific needs. My goal is only to keep the content well-bounded, and to help inform a generalized development and automation approach to help you be successful implementing the exercises in this book.

• Chapter 2 covers your development environment setup and requirements capture approach, and then moves into coding. In this chapter, you’ll create your first two IoT applications - one in Python and the other in Java. These will be quite simple, but set the stage for subsequent chapters. Even if you’re an experienced developer, it’s important to work through the exercises as given so we’re working from the same baseline going forward.

Part II – Connecting to the Physical World

This section covers a fundamental characteristic of the IoT – integration with the physical world. You’ll learn how to read sensor data and trigger an actuator both virtually using emulators and on an actual IoT device.

• Chapter 3 explores ways you can collect data from the physical world (sensing), and trigger actions based on that data (actuation). You’ll start by building a set of simple emulators which you may continue using throughout each subsequent exercise. These simple emulators will help you simulate real sensors and permit you to trigger actuation events.

• Chapter 4 builds upon Chapter 3 by discussing ways to connect to a small sampling of real sensors and actuators if you choose to build and run your applications on one or more physical devices. This chapter focuses on the Raspberry Pi, and assumes the hardware is running the Raspbian operating system.

• Chapter 5 discusses telemetry and data formatting. I’ll discuss ways to structure your data so it’s easy to store, transmit, and understand by both humans and machines. This will serve as the foundation for your interoperability with other ‘things’.

Part III – Connecting to Other Things

This is where the rubber meets the road, so to speak. This section focuses on integration across devices, as to be truly integrated, you’ll need a way to get your telemetry and other information from one place to another. The chapters in this section focus on learning about and utilizing application layer protocols designed for use within IoT ecosystems. I’ll assume your networking layer is already in place and working, although I’ll discuss a few wireless protocols along the way.

• Chapter 6 introduces Publish / Subscribe protocols - specifically Message Queuing Telemetry Transport (MQTT) and other related standards often used in IoT applications. I’ll walk through a select set of specifications and explain how you can begin building out a simple abstraction layer that allows you to easily interface with common open source libraries.

• Chapter 7 is nearly identical to Chapter 5, except it focuses on Request / Response protocols - specifically the Constrained Application Protocol (CoAP) and other standards, also commonly used in IoT applications.

• Chapter 8 explores the use of both Publish / Subscribe and Request / Response protocols together. You’ll integrate MQTT and CoAP as part of your solution, and see how these two protocols complement one another.

• Chapter 9 introduces yet another protocol - the OPC UA (Open Platform Communications Unified Architecture) - giving you the ability to use both the publish / subscribe and request / response communications paradigms through a single protocol standard.

Part IV – Connecting to the Cloud

Finally, at the ‘top’ of the integration stack, you’ll learn how to connect all your IoT device infrastructure to the cloud by using your gateway application to serve as the go-between for your cloud functionality and all your devices.

This section covers basic cloud connectivity principles, and lightly covers the various cloud services that can store, analyze, and manage your IoT environment. In each case, you’ll build the same simple cloud application across each platform.

• Chapter 10 considers various IoT-centric cloud services that are built on open source platforms that can either be re-hosted in your own environment or accessed via a free or paid tier.

• Chapter 11 provides an overview of various commercial cloud services that have IoT-specific capabilities. As there are many books and tutorials available for these platforms, I won’t go into detail on building anything specific; this chapter will simply serve as a review of these capabilities.

• Chapter 12 examines a few simple IoT use cases that I’ve found particularly helpful in preparing my Connected Devices course at Northeastern University. Specifically, I’ll cover the overall problem statement, expected outcome, design approach, and target solution for a home environmental monitoring system, pet monitoring system, and hydroponic garden management system.

In each chapter, I’ll provide a brief introduction to the topic along with some helpful background material, which will include some pertinent definitions that should be useful in understanding the concepts I’ll discuss. I’ll also summarize why each of these topics is important and what you can expect to learn. Each chapter ends with some hands-on implementation exercises to help you take the information presented and apply it to your own application.

My hope is that this approach will allow you to understand, and create, an integrated IoT system - end to end. It’s also designed so you can skip to the chapters that are most relevant for you as more of a reference guide, so you can skip the exercises if you’re not interested in coding a solution, although I do recommend you give them a try! If you do decide to delve into each lab module, please keep in mind that they are cumulative in that the work you do for each chapter will generally be used – or referenced – in a later chapter.

Some Background on the IoT

It may be helpful to look at just what enabled these ecosystems, and so here’s a brief summary of how we got to this point. Computing took a big step forward with the invention of the transistor in the 1950’s, followed in the 1960’s by Gordon Moore’s paper describing the doubling of transistors packed in the same physical space (later updated in the 1970’s) ².

With modern computing came modern networking, and the beginnings of the Internet with the invention of the ARPAnet in 1969 ³. This led to new ways to chunk, or packetize, data using the Network Control Protocol (NCP) and Transmission Control Protocol (TCP) via the Internet Protocol (IP) and leveraging existing wired infrastructure in the 1970’s. This was useful for industry, allowing electrical industrial automation to move down the path of centralized management of distributed, connected systems. Supervisory Control and Data Acquisition (SCADA) systems emerged from their proprietary roots, and Programmable Logic Controllers (PLC’s) were developed to take advantage of TCP/IP networking and related equipment standards.⁴

Eventually, we arrived at the 1980’s with the introduction of User Datagram Protocol (UDP) ⁵ and the birth of what many of us experienced as the early modern Internet - the World Wide Web (WWW) - invented in the late 1980’s by Tim Berners-Lee ⁶.

I’m sure you’ve noticed a common theme: A problem is followed by a technology innovation (often proprietary in nature) to address the challenge, which then becomes standardized (or is superseded by one or more standards), leading to wide adoption and further innovation.

This brings us to the era of the IoT, where in the late 1980’s and early 1990’s we begin to see the first connected devices emerge, including the demonstration of an Internet-connected toaster, demonstrated by John Romkey and Simon Hackett at 1990’s Interop show ⁷. Shortly thereafter, a web camera was set up to monitor the coffee pot near the Trojan Room at the Computer Lab at the University of Cambridge in 1991 ⁸. Because who wants to make a trip to the coffee machine and find the coffee pot empty? I

More devices followed, of course, and I’d guess even more were built and connected as experiments in college labs, dorms, homes, apartments, and businesses. All the while, computing and networking continued to become more inexpensive, powerful, and - of course - smaller. In 1999, Kevin Ashton presented the “Internet of Things” at Proctor & Gamble ⁹, which is widely believed to be the first coining of the phrase.

Fast forward to 2005, and the Interactive Design Institute in Italy gives us the inexpensive and designed-for-novices Arduino Single Board Computer (SBC) ¹⁰, opening the door for more people to start building their own sensing and automation systems. Add easily accessible storage and the ability to analyze data through services reachable from anywhere on the Internet, and we have the underpinnings of an IoT ecosystem; that is, a lot of different things that may be individually unique, but can be connected to each other in such a way as to serve a larger purpose.

In this sense, I believe it’s best not to view the IoT as a bunch of things that connect the physical world and the Internet to do useful work. The essence of the IoT is heterogeneity: lots of dissimilarity in terms of device types, features and capabilities, and purposes as well as implementation approaches, supported protocols, security, and management techniques.

Complexity Redefined

So, what exactly is the ‘Internet of Things’? It’s a complex set of technology ecosystems that connect the physical world to the Internet using a variety of edge computing devices and cloud computing services.

For the purposes of this book, edge computing devices represent the embedded electronics, small computers and software applications that either interact directly with the physical world through sensors and actuators or provide a gateway, or bridge, for those devices and applications to connect to the Internet. Cloud computing services represent the computing systems, software applications, data storage, and other computing services that live within one or more data centers and are always accessible via the Internet.

To be sure, this is an oversimplification of two complex, and often overloaded, technical terms. My intent is to provide a relatively simple categorization of the key computing resources for which to discuss the IoT and the integrated system you’ll build as you work through the exercises in this book.

With that said, and for the remainder of the book, I’ll focus on the IoT in terms of architecture and purpose; that is, how an IoT system is designed, and the outcome the design is expected to achieve.

The first step is to map these two technical terms of edge computing devices and cloud computing services into architectural tiers, where edge computing devices are part of the edge tier, and cloud computing services are part of the cloud tier. This provides both physical and logical separation of the key functionality of an IoT system, meaning, for example, that all sensing and actuation will take place in the edge tier, and all long-term storage and complex analytics will take place in the cloud tier.

This separation of tiers represents a high-level depiction of the systems architecture, or plan, that I’ll use going forward to describe the composition of the various physical and logical components comprising an integrated IoT system.

The real value in any IoT system is its ability to provide measurement and analysis. When sufficient measurements are collected, and with sufficient granularity and samples, an analysis can be performed to determine how a system is performing. The insight this analysis provides helps determine if the system is achieving its desired outcomes, or if a correction needs to be made.

If, for example, I can measure the inside temperature of a refrigerator every minute, I can determine how long the items stored in the refrigerator have been exposed to a given temperature. If I sample the inside temperature only once a day, I don’t have sufficient detail to make that determination.

Of course, the data that I collect needs to be understandable by other systems, otherwise it’s pretty much useless. Building integratable IoT solutions requires the developers of each part of an IoT system to think carefully about how their design (and data) might interact with other systems. This is why, although individually these parts may be unique and NOT co-dependent on another, it’s important for you as the developer to consider how other systems (and other developers) may need to interact with your software and data to serve this larger purpose we’ve been discussing.

The nuances you as the developer will need to deal with at each step along the way of building an integrated IoT system are indeed significant – we can’t expect plug ‘n play, or even consistent behaviors from the system you receive data from. And, to make matters worse, your code may not always work across every platform even with the best intentions of writing it generically enough to function the same way from one hardware device to another.

It’s not possible to cover all specialized platforms, of course, nor is it easy to write consistent, semi low-level code at the device level that doesn’t need to be optimized for every device. As such, I’m going to work with the understanding that, while every device may be a bit different, and have differences that we need to account for when we create (and test) our solutions, the code samples will be largely portable (with some minor exceptions) and usable across most systems that can run a Java Virtual Machine or Python 3 Interpreter.

Living on the Edge

Recall we have two architectural tiers: edge tier and cloud tier. In terms of outcome analysis, the keys to the kingdom lie in the cloud tier. Scalability is what gives the IoT its true power, and is the ability for a system to handle as much (or as little) as we want. For example, a scalable cloud system that supports the IoT is one where I can have a single gateway device sending it data, or thousands (or millions, or billions) without it failing.

You’ve likely read numerous reports describing the IoT as being gazillions of connected devices which will drive some exorbitant amount of business value by {pick your favorite year post-2020}. In the cloud, I (mostly) don’t care about where my services run, as long as they’re always available and can handle any workload I throw at them, no matter how many devices I have within my edge tier.

With all the scalability the cloud tier gives us, an IoT solution generally exhibits the greatest complexity at the edge, which is where most, and often all, of the heterogeneity lives. The two categories of devices this book will focus on are constrained devices and gateway devices. I’m leaving out the nomenclature of “smart devices” on purpose, since it’s becoming less clear how to best define “smart” versus “not so smart” devices. To grossly over-simplify, constrained devices have some limitations from a power, communications, and processing perspective, whereas gateway devices do not.

An example of a constrained device is a low-power (sometimes battery-operated), single-board computer (SBC), that either reads data from the environment (such as temperature, pressure, humidity, etc.), or triggers a valve to open or close when instructed to do so. An example of a gateway device is also an SBC, but much more powerful and likely requires a direct-connect power supply, has the ability to communicate with many different constrained devices, and also has enough processing power to aggregate the data from these other devices, perform some simple analytics, and then determine when any relevant data is to be sent to the cloud for further storage and processing.

Figure P-1 envisions a notional IoT systems architecture that represents the relationships between these device types within the edge tier and the services and other functionality that live within the cloud tier.

Figure P-1. - Notional IoT System Architecture

I find it easiest to qualify these devices as follows:

• Constrained devices only handle sensing, actuation, or perhaps both, while only processing messages for themselves, passing messages along if the right communications protocol is implemented, and sending messages to a gateway device. In short, they’re limited in their abilities, and don’t connect directly to the Internet – only via a gateway device.

• Gateway devices are a super set – they can potentially do the same work as a constrained device (in that they may have sensor(s) or actuator(s) attached), but also have the ability to perform ‘at the edge’ analytics, determine how messages should be routed (if at all), and of course, connect directly to the Internet and the various cloud services that make the IoT useful for business stakeholders.

Can a constrained device connect directly to the Internet? Sure, if it contains a TCP/IP stack, has a routable IP address that’s accessible to and from the public Internet, and of course has the appropriate communications hardware to talk to, and is connected with, the Internet. For the purposes of this book, however, I’ll narrow the category of constrained devices to these two limitations:

• They do not support packet routing directly to or from the public Internet (although we’ll assume they support both TCP/IP and UDP/IP) and must interact with a gateway device in order to be part of any IoT ecosystem.

• They do not contain adequate computing resources to intelligently determine complex courses of action based on the data they collect.

What it all means is this: As we make better computing devices that are smaller, faster, and cheaper; use them to interact with the physical world, and connect them (or their data) to the Internet for processing using cloud services, we can derive insights that help deliver better business outcomes.

At the end of this book, I’ll explore some use cases like the one just discussed, but in more detail. Each use case will include an implementation exercise that will allow you to explore the concept using the knowledge you’ve gleaned from previous chapters and the code you’ll develop as part of the exercises.

One last point before delving into our first section: Each chapter begins with a haiku that attempts to capture the essence of what you’ll learn and some of the challenges you’ll likely encounter. How did this come to be? From my early days as a software developer, one of the team’s I worked with had a policy that if you committed code that caused the nightly build to break, you’d have to write a haiku related to the issue and e-mail it to everyone on the team. You may not encounter ‘broken nightly builds’ much by working through the exercises in this book, but the haiku’s should at least provide some levity before you dig into each chapter.

Thanks for reading!

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.

Tip

This element signifies a tip or suggestion.

Note

This element signifies a general note.

Warning

This element indicates a warning or caution.

Using Code Examples

Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/oreillymedia/title_title.

If you have a technical question or a problem using the code examples, please send email to [email protected].

This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Programming the Internet of Things by Andrew King (O’Reilly). Copyright 2021 Andrew King, 978-1-492-08141-8.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at [email protected].

O’Reilly Online Learning

Note

For more than 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help companies succeed.

Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

• O’Reilly Media, Inc.
• 1005 Gravenstein Highway North
• Sebastopol, CA 95472
• 800-998-9938 (in the United States or Canada)
• 707-829-0515 (international or local)
• 707-829-0104 (fax)

Email [email protected] to comment or ask technical questions about this book.

For news and information about our books and courses, visit http://oreilly.com.

Find us on Facebook: http://facebook.com/oreilly

Follow us on Twitter: http://twitter.com/oreillymedia

Watch us on YouTube: http://www.youtube.com/oreillymedia