If you have had success with the projects thus far in the book, you have at your disposal several forms of sensor and data-aggregate nodes. In essence, you have the basic building blocks for constructing a sensor network to monitor and record temperature data. It would not take much more work to add nodes for other environmental sensors such as humidity or barometric pressure. Indeed, the basic sensor node you have built can host a variety of sensors.
If you have run the example projects and experimented with the challenges, no doubt you have noticed that a lot of data is being generated. What do you do with that data? Is it meaningful only at the instant it is generated, or do you think it is more likely that you would want to store the data and examine it later? For example, if you want to know the temperature range for your workshop on a monthly basis throughout the year, logically you need data from an entire year1 to tabulate and average.
Arduino boards don’t have built-in storage devices in general (but some specialized variants do). Raspberry Pi boards come with a secure digital (SD) drive and can accept USB-based storage devices where you can store data, but what do you do with the data from your Arduino-based nodes?
This chapter examines the available storage methods and gives examples of how to store data using those methods. Sample projects are provided to illustrate the mechanisms and code, but I omit the sensor-specific code for brevity.
Storage Methods
Sensor data can come in several forms. Sensors can produce numeric data consisting of floating-point numbers or sometimes integers. Some sensors produce more complex information that is grouped together and may contain several forms of data. Knowing how to interpret the values read is often the hardest part of using a sensor. In fact, you saw this in a number of the sensor node examples. For example, the temperature sensors produced values that had to be converted to scale to be meaningful.
Although it is possible to store all the data as text, if you want to use the data in another application or consume it for use in a spreadsheet or statistical application, you may need to consider storing it either in binary form or in a text form that can be easily converted. For example, most spreadsheet applications can easily convert a text string like “123.45” to a float, but they may not be able to convert “12E236” to a float. On the other hand, if you plan to write additional code for your Arduino sketches or Raspberry Pi Python scripts to process the data, you may want to store the data in binary form to avoid having to write costly (and potentially slow) conversion routines.
But that is only part of the problem. Where you store the data is a greater concern. You want to store the data in the form you need but also in a location (on a device) that you can retrieve it from and that won’t be erased when the host is rebooted. For example, storing data in main memory on an Arduino is not a good idea. Not only does it consume valuable program space, but it is volatile and will be lost when the Arduino is powered off.
The Raspberry Pi offers better options. You can easily create a file and store the data on the root partition or in your home directory on the SD card. This is nonvolatile and does not affect the operation of the Raspberry Pi operating system. The only drawback is that it has the potential to result in too little disk space if the data grows significantly. But the data would have to grow to nearly two gigabytes (for a 2GB SD card) before it would threaten the stability of the operating system (although that can happen).
So, what are your options for storing data with Arduino? Are there any other possibilities with the Raspberry Pi? There are two types of storage to consider: local and remote. Local storage includes any method that results in the data being stored with the node, for example, storing data on the SD card on the Raspberry Pi. Remote storage includes any method where the data is stored on a device or medium that is not directly connected to the node, for example, storing data on a different node or even on a server connected to the Internet.
Neither the Arduino nor the Raspberry Pi has a real-time clock (RTC) on board. If you want to store your sensor data locally, you have to either store the data with an approximate date and timestamp or use an RTC module to read an accurate date/time value.
Fortunately, there are RTC modules for use with an Arduino or the Raspberry Pi. If your Raspberry Pi is connected to the Internet and you have enabled the network time synchronization feature, you do not need the RTC module. However, if your Raspberry Pi is not connected to the Internet, and you want to store accurate time data, you should consider using the RTC module.
The following sections examine the various local and remote storage options available for the Arduino and Raspberry Pi.
Local Storage Options for the Arduino
Although it is true that the Arduino has no onboard storage devices, there are two ways you can store data locally for the Arduino. You can store data in a special form of nonvolatile memory or on an SD card hosted via either a special SD card shield or an Ethernet shield (most Ethernet shields have a built-in SD card drive).
If you are truly inventive (or perhaps unable to resist a challenge), you can use some of the communication protocols to send data to other devices. For example, you could use the serial interface to write data to a serial device.
The following sections discuss each option in greater detail. Later sections present small projects you can use to learn how to use these devices for storing data.
Nonvolatile Memory
The most common form of nonvolatile memory available to the Arduino is electrically erasable programmable read-only memory (EEPROM—pronounced “e-e-prom” or “double-e prom”). EEPROMs are packaged as chips (integrated circuits). As the name suggests, data can be written to the chip and is readable even after a power cycle but can be erased or overwritten.
Most Arduino boards have a small EEPROM where the sketch is stored and read during power-up. If you have ever wondered how the Arduino does that, now you know. You can write to the unused portion of this memory if you desire, but the amount of memory available is small (512KB for some boards). You can also use an EEPROM and wire it directly to the Arduino via the I2C protocol to overcome this limitation.
Writing to and reading from an EEPROM is supported via a special library that is included in the Arduino IDE. Due to the limited amount of memory available, storing data in the EEPROM memory is not ideal for most sensor nodes. You are likely to exceed the memory available if the data you are storing is large or there are many data items per sample.
You also have the issue of getting the data from the EEPROM for use in other applications. In this case, you would have to build not only a way to write the data but also a way to read the data and export it to some other medium (local or remote).
That is not to say that you should never use EEPROM to store data. Several possible reasons justify storing data in EEPROM. For example, if your sensor node is likely to be isolated, or connectivity to other nodes is limited, you may want to use an EEPROM to temporarily store data while the node is offline. In fact, you could build your sketch to detect when the node goes offline and switch to the EEPROM at that time. This way, your Arduino-based sensor node can continue to record sensor data. Once the node is back online, you can write your sketch to dump the contents of the EEPROM to another node (remote storage).
SD Card
You can also store (and retrieve) data on an SD card. The Arduino IDE has a library for interacting with an SD drive. In this case, you would use the library to access the SD drive via an SD shield or an Ethernet shield.
Storing data on an SD card is done via files. You open a file and write the data to it in whatever format is best for the next phase in your data analysis. Examples in the Arduino IDE and elsewhere demonstrate how to create a web server interface for your Arduino that displays the list of files available on the SD card.
Compared to EEPROMs, SD cards store many times more data. You can purchase high-density SD cards that exceed 128GB of storage space. That’s a lot of sensor data!
You may choose to store data to an SD card in situations where your sensor node is designed as a remote sensor with no connectivity to other nodes, or you can use it as a backup-logging device in case your sensor node is disconnected or your data-aggregator node goes down. Because the card is removable and readable in other devices, you can read it on another device when you want to use the data.
Using an SD card means you can move the data from the sensor node to a computer simply by unplugging the card from the Arduino and plugging it in to the SD card reader in your computer.
Project: Saving Data in Nonvolatile Memory
Recall that you can use the local EEPROM on an Arduino to store data. There are some excellent examples in the Arduino IDE that I encourage you to experiment with at your leisure. They are located under the Examples menu under the EEPROM submenu. You need only an Arduino and your laptop to experiment with writing to and from the EEPROM on the Arduino.
Rather than rehash the example sketch for using the built-in EEPROM, this section outlines a project to use an external EEPROM to store data. Unlike the local EEPROM, which uses a dedicated library to interact with, an external EEPROM uses the I2C communication protocol.
Hardware Setup
I2C EEPROM chip (courtesy of SparkFun)
The pushbutton will allow you to reset the memory on the chip. Doing so erases the data values stored, resetting the memory configuration for reuse. You will find this feature particularly handy when using the sketch for the first time, debugging problems, and reusing the chip once memory has been read and stored on another medium.
Pinout of the I2C EEPROM
Setting the Address of the I2C EEPROM
Address | A0 | A1 | A2 |
|---|---|---|---|
0x50 | Ground | Ground | Ground |
0x51 | Ground | Ground | +5V |
0x52 | Ground | +5V | Ground |
0x53 | Ground | +5V | +5V |
0x54 | +5V | Ground | Ground |
0x55 | +5V | Ground | +5V |
0x56 | +5V | +5V | Ground |
0x57 | +5V | +5V | +5V |
Now that you understand how to address the chip, let’s connect it to your Arduino. Begin by placing the chip in a breadboard with the half circle pointing to the left. This establishes pin 1 as the upper-right pin. Connect a ground wire to all four pins on the top side of the chip. These are pins 1–4, as shown in Figure 7-2.
Wiring the EEPROM to the Arduino
If you are using the Leonardo board, you need to use the SDC and SCL pins located near the USB port. For the Uno board, they are located at A4 and A5 and on the Mega 2560, they are on pins 20 and 21. Check the hardware pinout for your board to ensure you use the correct I2C interface connections.
Software Setup
With the wiring in place, you are ready to start writing a sketch to read and write data. Rather than write a script to simply store data, in this example, you write a sketch to let you write data to and read it from the chip. You also include a reset operation to allow you to overwrite any memory.
You add the read methods so that you can create additional sketches to read data, should you wish to review the data, move the chip (data) to another Arduino, or use another sketch to process the data.
These statements include the Wire library and define a number of constants you use in the sketch. Notice that you have an address for the first sample (the position in memory on the chip), the address for the chip, a pin for the pushbutton, the maximum size (for the 256 chip), and the number of bytes per sample.
Notice the process for communicating with the chip. First, you start a transmission with the chip, send the address that you intend to read, and then end the transmission. The address is a two-byte value, and the statements show you how to manipulate the bytes to form a word (two bytes). The next method, requestFrom(), tells the chip you want to read a single byte. If the chip is ready, you read the data. Finally, you return the value to the caller.
Notice that you have the same setup—you begin the transmission and set the value at the index specified. What differs is that you send the data (write it) before you end the transmission.
But how do you know what is written to which address (or index)? Rather than just write data willy-nilly or in some illogical order, let’s use the first byte at index 0 to store the number of data samples (or rows) and the second byte to store how many bytes each sample consumes (or columns). In this way, you make the data easier to read because it is uniform and easier to manage on a reboot.
You use the write_byte() method to write 0 for the number of bytes and the constant defined earlier for the number of bytes per sample. The method begins by writing 0xff to the first 10 bytes to ensure that you have no data stored; then the number of bytes is written to index 0 and the number of bytes per sample to index 1. You add some print statements for feedback.
Once again, the method is similar to the write_byte() method, but you use a loop to write the bytes for a sample. Notice that you include a debug statement to show the starting index used. You do this so that you can see the value increase if you run the sketch multiple times.
You may have noticed that I duplicated the code among the *_byte() and *_sample() methods. I did so for clarity of the code, but it isn’t strictly necessary. For example, you could consolidate the code if you changed the *_sample() methods to use an additional parameter indicating how many bytes to read/write. I leave this optimization to you as an exercise.
Notice how you keep track of the number of samples and the next index for the next sample. You use the variable you created earlier and increment it by the number of bytes in the sample. This way, you always know what the next address is without reading the number of samples first and calculating the index. The last method updates the number of samples value.
Storing and Retrieving Data on an External EEPROM
Notice that you include some additional statements for communicating the progress of the sketch via the serial monitor. Take some time to examine these so that you are familiar with what to expect when the sketch runs.
If you want to write-protect the chip, disconnect the WP pin. Doing so makes the chip read-only.
Testing the Sketch
To test the sketch, be sure the code compiles and you have your hardware set up correctly. When you have a sketch that compiles, upload it to your Arduino and launch a serial monitor.
Serial Monitor Output for EEPROM Example
Did you see something similar? If you run the sketch again (e.g., by pressing the Reset button), you should see the value for the start index (from the write_sample() method) increase. Go ahead and give it a try.
Once you’ve done it a few times Saving Data in Nonvolatile Memory, press the pushbutton and notice what happens. As you can see in Listing 7-2, the start index is reset, and the next samples are stored at the beginning of memory.
For More Fun
Add some visual aids for use in embedded projects (cases with no serial monitor capability). You can add an LED that illuminates when there is data on the chip. You can also add a set of seven-segment LEDs to display the number of data samples stored.
Improve the code for reuse. Begin by removing the redundancy described earlier in the read and write methods, and then move the code to a class to make it easier to use the EEPROM in other sketches.
Add a second EEPROM chip to expand the amount of storage available. Hint: You need to set each chip to a different address, but the methods used are the same.
Perhaps a bit easier and more inline with the hardware-hacking element of Arduino is moving the EEPROM to another Arduino and reading all the values stored. This demonstrates the nonvolatile nature of EEPROM chips.
Use appropriate grounding to avoid electrostatic discharge (ESD) damage to the chip.
Project: Writing Data to an SD Card
Aside from an EEPROM chip, you can also store data locally on an Arduino by writing the data to an SD drive. The SD drive is a good choice for storing data because the data is stored in files, which other devices can read (and write to).
For example, although writing data to an EEPROM chip is not difficult, reading that chip on a personal computer requires writing a sketch for the Arduino to transfer the data. However, the SD card can be removed from the Arduino (once files are closed) and inserted in an SD drive connected to a personal computer, allowing you to read the files directly. Thus, the SD card makes a better choice for sensor networks where your sensor nodes are not connected via a network or other wireless connections.
There are several choices for adding an SD card reader to an Arduino. Two of the most popular are the Arduino Ethernet shield and the microSD shield from SparkFun (www.sparkfun.com/categories/240). If you use the Arduino Ethernet shield, you can use the networking capabilities and the SD card together. A number of similar devices are available from a variety of vendors.
Adafruit also has a Data Logging shield for Arduino with an onboard SD drive (www.adafruit.com/product/1141). The Data Logging shield also includes an RTC, making it possible to store date and time along with the sample. I discuss using an RTC in the next project.
Both the microSD shield and the Data Logging shield offer a prototyping area that you can use to mount your sensor components or even an XBee module.
An SD drive allows you to create a hybrid node where you store data locally as well as transmit it to another node in the network. This redundancy is one of the ways you can build durability in to your sensor network. For example, if a node loses its connection to another node via the network, it can still record its data locally. Although it is a manual process to recover the data (you must go get the SD card), the fact that the data is recoverable at all means the network can survive network failures without losing data.
It is possible to use an EEPROM as a local storage backup option, but an EEPROM is harder to use, is not as durable as an SD card, does not have the same storage capacity, and is not as easy to use in other devices.
There is one other very important thing to consider concerning building a durable sensor node. Having a local backup of the data may not be helpful if you do not know when the data was stored. The Arduino does not have any time-keeping capability beyond a limited accuracy cycle time. Thus, if you store data locally without a timestamp of any kind that you can relate to other data, the samples taken may not be meaningful beyond the sequence itself (the order of the values).
To mitigate this, you can add an RTC module to the Arduino. The RTC allows you to store the date and time a sample was taken. This information may be critical if you are trying to plot values over time or want to know when a spurious or otherwise interesting event took place.
Hardware Setup
The hardware for this project uses the Arduino Ethernet shield, the microSD shield from SparkFun (with an SD card installed), or the Data Logging shield from Adafruit. For simplicity, I used the Arduino Ethernet shield and show the code changes necessary to use the microSD shield or the Data Logging shield (via #define statements).
DS1307 Real-Time Clock breakout board (courtesy of Adafruit)
SparkFun also has a product named Real-Time Clock module (www.sparkfun.com/products/99) that uses the same DS1307 chip and interface as the Adafruit offering. You can use either in this project.
The Adafruit RTC module requires assembly. The RTC module from SparkFun does not.
Arduino with an Ethernet shield and RTC module
Notice that the Ethernet shield is installed on the Arduino. Wiring connections would be the same if you were using the SparkFun microSD shield.
Adafruit Data Logging shield (courtesy of Adafruit)
If you do not have a Data Logging shield, you can use the RTC module as described earlier.
Software Setup
With the Data Logging shield in place, you are ready to start writing a sketch to write data to the SD card. Be sure to insert a formatted SD card before you power on the board. But first, you must download and install the RTC library from Adafruit (https://github.com/adafruit/RTClib).
Installing the Adafruit RTCLib in Library Manager
With the preliminaries complete, you need a method to save a sensor sample to the SD card. The method must read the date and time from the RTC module, accept the sample as a parameter, and store the data. In this example, you place the date and time first, followed by the sample value. Name this method record_sample().
Notice that also you have code to turn off the Ethernet W5100 SPI interface. This is only necessary for the Ethernet shield and then only if you do not plan to use the networking capabilities.
The loop() method is where you place calls to the record_sample() method. In this case, leave the loop() method empty for brevity. Feel free to add your own code to read sensors here and call the record_sample() method for each.
Storing Data on an SD Card
I added debug statements to the setup() method for illustration purposes and to make sure the sketch works. Placing these calls in the setup() method permits you to load the sketch (or reboot the Arduino) and check the contents of the SD card to see if the code worked. If you place the statements in the loop() method, then depending on when you turn off your Arduino (unplug it), you may not know how many lines were added or even if the file were closed properly. Placing the record_sample() statements in the setup() method means you have expected output to check.
If you get SD drive initialization errors, check the pin assignment used in the definition section to make sure you are using the correct pin for your SD drive/shield.
If you encounter file write or open errors, make sure the SD card is formatted as a FAT partition, the SD card is not write protected, and you can create and read files on the drive using your personal computer.
Testing the Sketch
The first time you run the sketch, you may see a message about initializing the SD card and creating the file. This is normal. Subsequent runs (restarts of the Arduino) may not show the messages.
If you examine the file and find more sets of entries than you expect, try deleting the data from the file, starting your Arduino, and then pressing the Reset button twice. When you look at the contents, you should see exactly three sets of entries (one for the initial start because the sketch was in memory to begin with and one for each time you restarted the Arduino).
If you see only partial sets (fewer than three rows for each set), check to ensure that you are allowing the Arduino to start before powering it off. It is best to use the serial monitor and wait until all three statements are echoed to the monitor before shutting down the Arduino.
Should the case arise that your sketch compiles and no errors are shown in the serial monitor but the data file is empty, check to make sure the card is usable and not corrupt. Try reformatting the card with the FAT file format.
MicroSD cards are very fragile. They can be damaged easily if handled improperly or subjected to ESD or magnetic fields. If your card does not work properly and you cannot reformat it, it is possible that it is damaged beyond use. You can try using a formatting program from sdcard.org, but if it fails, your card is no longer viable. So far, this has happened to me only once.
Now that you have examined two primary methods for storing data locally on an Arduino, let’s look at the options available for the Raspberry Pi.
Local Storage Options for the Raspberry Pi
Because the Raspberry Pi is a personal computer, it has the capability to create, read, and write files. Although it may be possible to use an EEPROM connected via the GPIO header, why would you do that? Given the ease of programming and the convenience of using files, there is very little need for another form of storage.
You also know the Raspberry Pi can be programmed in a number of ways and with one of the most popular languages, Python.3 Working with files in Python is very easy and is native to the default libraries. This means there is nothing that you need to add to use files.
The following project demonstrates the ease of working with files in Python. The online Python documentation explains reading and writing files in detail (https://docs.python.org/2/tutorial/inputoutput.html#reading-and-writing-files).
One thing you will notice is that it doesn’t matter where the file is located—on the SD card or an attached USB drive. You only need to know the path to the location (folder) where you want to store data and pass that to the open() method.
Savvy Python programmers4 know that the Python library contains additional libraries and classes for manipulating folders, navigating paths, and much more. For more information, examine the Python documentation for the OS and Sys libraries. For example, look for normpath() and the Path5 class.
Project: Writing Data to Files
This project demonstrates how easy it is to use files on the Raspberry Pi with Python. Because no additional hardware or software libraries are needed, I can skip those sections and jump directly into the code.
In this example, you first import the datetime library . You use the datetime to capture the current date and time. Next, you open the file (notice that you are using the Pi users’ home directory) using the newer with clause and write a row to the file (you do not need to close the file—that is done for you when execute leaves the scope of the with clause). If you feel better using the explicit open and close, feel free to do so.
Notice the open() method. It takes two parameters—the file path and name and a mode to open the file. You use "a+" to append to the file (a) and create the file if it does not exist (+). Other values include r for reading and w for writing. Some of these can be combined: for example, "rw+" creates the file if it does not exist and allows for both reading and writing data.
Using write mode truncates the file. For most cases in which you want to store sensor samples, you use append mode.
Did you get similar results? If not, correct any errors and try again until you do. As you can see from this simple example, it is very easy to write data to files using Python on the Raspberry Pi.
Remote Storage Options
Remote storage means the data is sent to another node or system for recording. This normally requires some form of communication or network connectivity to the remote system. Sensor networks by nature are connected and thus can take advantage of remote storage.
To give you an idea of what I am discussing, consider an Arduino sensor node with an XBee module connected to a Raspberry Pi–based node. Suppose also that you want to write your sample data to files. Rather than using an SD card on the Arduino node to store data, you could send that data to the Raspberry Pi–based node and store the data in a file there. The main motivation is that it is much easier to use files via Python on the Raspberry Pi. If you also factor in the possibility of having multiple Arduino sensor nodes with XBee modules, you can use the Raspberry Pi–based node as a data aggregate, storing all the data in a single file.
I sometimes get this question when discussing storing aggregate data. If your data is similar (e.g., temperature), you can consider storing data from like sensors to the same file. However, if the data differs (such as temperature from one node and humidity from another), you should consider using different files. This makes reading the files easier because you don’t have to write code (or use tools) to separate the data.
But are you really talking about only storing data in files? The answer is no. There are a number of mechanisms for storing data remotely. Although storing data in files is the easiest form, you can also store data in the cloud or even on a remote database server.
If you are experienced with using databases for storing and retrieving data, this method will appeal to you—especially if you plan to use other tools to process the data later. For example, you may want to perform statistical analyses or create charts that track the samples over time. Because working with databases is a complex topic, I examine this form of remote storage in the next couple of chapters.
You have already seen how easy it is to use files, but what about storing data in the cloud? What is that about? Simply stated, storing data in the cloud involves using a cloud-based data storage service to receive your data and host it in some way. The most popular form presents the data for others on the Internet to view or consume for their own use.
The following section discusses storing sample data in the cloud using a popular, easy to use, cloud-based IoT data-hosting service from MathWorks called ThingSpeak (www.thingspeak.com). You will see example projects for using ThingSpeak on both the Arduino and the Raspberry Pi.
Storing Data in the Cloud
Unless you live in a very isolated location, you have likely been bombarded with talk about the cloud and IoT. Perhaps you’ve seen advertisements in magazines and on television, or read about it in other books, or attended a seminar or conference. Unless you’ve spent time learning what cloud means, you are probably wondering what all the fuss is about.
Simply stated,6 the cloud is a name tagged to services available via the Internet. These can be servers you can access (running as a virtual machine on a larger server), systems that provide access to a specific software or environment, or resources such as disks or IP addresses that you can attach to other resources. The technologies behind the cloud include grid computing (distributed processing), virtualization, and networking. The correct scientific term is cloud computing. Although a deep dive into cloud computing is beyond the scope of this book, it is enough to understand that you can use cloud computing services to store your sensor data.
Oracle IoT: www.oracle.com/internet-of-things/
Microsoft Azure IoT Hub: https://azure.microsoft.com/en-us/product-categories/iot/
Google IoT Core: https://cloud.google.com/iot-core
IBM IoT: www.ibm.com/internet-of-things
Arduino IoT Cloud: www.arduino.cc/en/IoT/HomePage
MathWorks ThingSpeak: https://thingspeak.com/
Most of the vendors offer commercial products, but a few like Google, Azure, Arduino, and ThingSpeak offer limited free accounts. As you may surmise, some of the offerings are complex solutions with steep learning curve, but the Arduino and ThingSpeak offerings are simple and easy to use. Since we want a solution that supports Arduino and Raspberry Pi (and other platforms), we will use ThingSpeak in this chapter as an example of what is possible when storing data in the cloud.
If you want or need to use one of the other vendors, be sure to read all of the tutorials thoroughly before jumping into your code.
ThingSpeak offers a free account for non-commercial projects that generate fewer than 3 million messages (or data elements) per year or around 8000 messages per day. Free accounts are also limited to seven channels (a channel is equivalent to a project and can save up to eight data items). If you need to store or process more data than that, you can purchase a commercial license in one of four categories, each with specific products, features, and limitations: Standard, Academic, Student, and Home. See https://thingspeak.com/prices and click each of the license options to learn more about the features and pricing.
Unless you use a work or school account, you may need to pay to use some of the products such as MatLab.
ThingSpeak works by receiving messages from devices that contain the data you want to save or plot. There are libraries available that you can use for certain platforms or programming languages such as Arduino or Python. That is by far the easiest way to connect to ThingSpeak and transmit data.
However, you can also use a machine-to-machine (M2M) connectivity protocol (called MQTT7) or representational state transfer (REST8) API designed as a request-response model that communicates over HTTP to send data to or read data from ThingSpeak. Yes, you can even read your data from other devices.
See www.mathworks.com/help/thingspeak/channels-and-charts-api.html for more details about the ThingSpeak MQTT and REST API.
When you want to read or write a ThingSpeak channel, you can either publish MQTT messages, send requests via HTTP to the REST API, or use one of the platform-specific libraries that encapsulate these mechanisms for you. A channel can have up to eight data fields represented as a string or numeric data. You can also process the numeric data using several sophisticated procedures such as summing, average, rounding, and more.
We won’t get too far into the details of these protocols; rather, we will see how to use ThingSpeak as a quick-start guide. MathWorks provides a complete set of tutorials, documentation, and examples. So, if you need more information about how ThingSpeak works, check out the documentation at www.mathworks.com/help/thingspeak/.
Don’t believe all the hype or sales talk about any product that includes “cloud” in its name. Cloud computing services and resources should be accessible via the Internet from anywhere, available to you via subscription (fee or for free), and permit you to consume or produce and share the data involved. Also, consider the fact that you must have access to the cloud to get to your data. Thus, you have no alternative if the service is unreachable (or down).
Getting Started with ThingSpeak
To use ThingSpeak, you must first sign up for an account. Fortunately, they provide the option for a free account. In fact, you get a free account to start with and add (purchase) a license later. To create a free account, visit https://thingspeak.com/users/sign_up and fill in your email address, location (general geographic), and first and last name and then click Continue. You will then be sent a validation email. Open that and follow instructions to verify your email and complete your free account by choosing a password and completing a short questionnaire.
Creating a Channel
Creating a channel in ThingSpeak
New Channel form
At a minimum, you need only name the channel, enter a description (not strictly required but recommended), and then select (tick) one or more fields naming each. That’s it. Click Save Channel to complete the process.
Percentage complete: A calculated field based on the completion of the name, description, location, URL, video, and tags in your channel.
Channel Name: Unique name for the channel.
Description: Description of the channel.
Field#: Tick each box to enable the field.
Metadata: Additional data for the channel in JSON, XML, or CSV format.
Tags: A comma-separated list of keywords for searching.
Link to External Site: If you have a website about your project, you can provide the URL here to publish on the channel.
- Show Channel Location: Tick this box to include the following fields:
Latitude: Latitude of the sensor(s) for the project or source of the data
Longitude: Longitude of the sensor(s) for the project or source of the data
Elevation: Elevation in meters for use with projects affected by elevation
Video URL: If you have a video associated with your project, you can provide the URL here to be published on the channel.
Link to GitHub: If your project is hosted in GitHub, you can provide the URL to be published on the channel.
Wow, that’s a lot of stuff for free! As you will see, this isn’t a simple toy or severely limited product. You can accomplish quite a lot with these settings. Notice there are places to put links to video, website, and GitHub. This is because channels can be either private (only your login or API KEY as we will see can access) or public. Making a channel public allows you to share the data with anyone and thus those URL fields may be handy to document your project. Cool.
Once you create your channel, it is time to write some data. There are two pieces of information you will need for most projects: the API Key for the channel and for some libraries the channel number (the integer value shown on the channel page). There are libraries available for many platforms, and on some platforms, there may be several ways (libraries or techniques) to write data to a ThingSpeak channel.
API Keys for a ThingSpeak channel
Notice I masked out the keys. If you make your channel public, do not share the write key with anyone you don’t want to allow to write to your channel. You can create new keys by clicking the Generate New Write API Key or Add New Read API Key buttons. You can delete read keys by clicking the Delete API Key button.
We use the key in our code to allow the device to connect to and write data to the channel. So, we typically copy this string from the channel page and paste it into our code as a string. Recall, we may use a library that encapsulates the HTTP or MQTT mechanism or, in the case of the Raspberry Pi Python library, we use a Python library and the HTTP protocol. We will see both in the upcoming sample projects for Arduino and Raspberry Pi.
Now that you understand the basics of writing data to ThingSpeak, let’s take a look at how to do it in more detail for the Arduino. This is followed by an example for the Raspberry Pi.
Project: Writing Data to ThingSpeak with an Arduino
Set up a channel for the MKR1000 and TMP36 sensor
Click the Save Channel button to create the channel. Then, on the API Key tab, copy the write key and paste it a new file for later use.
Now that we have a channel created, let’s set up the hardware.
Hardware Setup
The hardware for this project is an Arduino MKR1000, a breadboard, breadboard wires, a TMP36 temperature sensor, and a 0.10uF capacitor. Wire the sensor and the capacitor as shown in Figure 7-12. Attach pin 1 of the sensor to the 5V pin on the Arduino, pin 2 of the sensor to the A1 pin on the MKR1000, and pin 3 to ground on the MKR1000. The capacitor is also attached to pins 1 and 3 of the sensor (orientation does not matter).
You can use the newer MKR models, provided they are of the Wi-Fi variety.
Wiring setup for the ThingSpeak temperature feed for the Arduino
To use connect to the Internet, you will also need a Wi-Fi access point or router to connect. You will need the SSID and password for the connection.
Now, let’s see how we set up the software and the sketch for the project.
Configuring the Arduino IDE
We will need to set up several things in our Arduino IDE in order to create the sketch. We will need to ensure the MKR boards are supported and the ThingSpeak and the Wi-Fi 101 libraries are installed. Let’s begin with the MKR board support.
Installing the SAMD Boards support
Click the Install button to install the board support module. On some PC platforms, the Arduino IDE may prompt you to install the board support when you start the IDE for the first time with the board connected to your PC.
Installing the ThingSpeak library
Installing the WiFi 101 library
Write the Sketch
Now that you have the necessary libraries and board module installed, open a new Arduino project and name it arduino_thingspeak.ino. Recall, we will use a TMP36 and read the values from pin A1 on the MKR1000. We have seen code to read the TMP36 in Chapter 6, so we will skip the explanation and dive into how to interact with ThingSpeak.
Add New Tab
Example channel data (MKR1000_TMP36)
Set up a channel for the Raspberry Pi and TMP36 sensor
12-bit ADC module (courtesy of Adafruit)
Wiring the TMP36 and ADC to the Raspberry Pi
Verifying the ADC module
Sample ThingSpeak feed for the Raspberry Pi
Notice we display the actual result if it does not return a code of 200. Notice also we add a sleep (delay()) at the end to sleep for 20 seconds. We do this because the ThingSpeak free account is limited to updates once every 15 seconds.
Arduino-Based ThingSpeak Channel Write
Be sure to substitute your API key and channel number in the secrets.h file. Failure to do so will result in compilation errors.
Take some time to make sure you have all the code entered correctly and that the sketch compiles without errors. Once you reach this stage, you can upload the sketch and try it out.
Testing the Sketch
Did you see similar output? If you did not, check the return code as displayed in the serial monitor. You should be seeing a return code of 200 (meaning success). If the return code was a single digit (1, 2, 3, and so on), you are likely encountering issues connecting to ThingSpeak. If this occurs, connect your laptop to the same network cable, and try to access ThingSpeak.
If the connection is very slow, you could encounter a situation in which you get an error code other than 200 every other or every N attempts. If this is the case, you can increase the timeout in the loop() method to delay processing further. This may help for some very slow connections, but it is not a cure for a bad or intermittent connection.
Let the sketch run for about 3 minutes before you visit ThingSpeak. Once the sketch has run for some time, navigate to ThingSpeak, log in, and click your channel page. You should see results similar to those shown in Figure 7-17.
Notice the peaks near the beginning and end of the graph. I simulated a spike in the data by pressing a warm object device on the TMP36 (my finger). If you try this, be careful not to touch any of the wires!
For More Fun
You can have a lot of fun with this script. Try connecting other sensors and creating other channels in ThingSpeak. You can also experiment with reading the data you saved in ThingSpeak.
Now that you know how to save data to ThingSpeak on the Arduino, let’s explore how to do the same on the Raspberry Pi.
Project: Writing Data to ThingSpeak with a Raspberry Pi
This project demonstrates the ease of using the ThingSpeak REST API via HTTP on the Raspberry Pi to write sensor data to a ThingSpeak channel. Recall, to read the analog temperature sensor (TMP36), you use an I2C module that provides 12-bit precision for reading values.
This example demonstrates how to use the HTTP interface to write data to ThingSpeak. However, this is also a ThingSpeak Python library you can use if you want. You can install it with the pip3 install thingspeak command. Documentation for the ThingSpeak Python library can be found at https://thingspeak.readthedocs.io/en/latest/.
If you have not yet created a ThingSpeak channel for the Raspberry Pi, do that now and record the channel ID and API key generated. Use the following data for the channel and name it RASPI_TMP36 as shown in Figure 7-18.
Click the Save Channel button to create the channel. Then, on the API Key tab, copy the write key and paste it a new file for later use.
Now that we have a channel created, let’s set up the hardware.
Hardware Setup
The hardware for this project consists of a Raspberry Pi, a Raspberry Pi Cobbler+ (optional), a breadboard, the TMP36 sensor, jumper wires, and an ADC module.
I mentioned the Raspberry Pi does not include any ADCs, so you cannot use an analog sensor. In this project, you explore how to use a multichannel ADC with the Raspberry Pi to enable the use of the TMP36 analog temperature sensor. Figure 7-19 shows the 12-bit ADC from Adafruit (www.adafruit.com/products/1083). This module supports up to four sensors (channels). In the figure, you can see pins A0–A3; these are the pins used for each of the channels supported.
You are exploring the use of the ADC module with a Raspberry Pi, because it supports the I2C protocol, but you can use the module with the Arduino too. See http://learn.adafruit.com/adafruit-4-channel-adc-breakouts for more details.
You also require connectivity to the Internet via a network connection on the Raspberry Pi. The Internet connection can be via a wired Ethernet connection or via a wireless connection. There are no specific requirements for connectivity as there are with an Arduino.
Figure 7-20 shows the connections you need to make. Most of these should be familiar to you if you have completed the projects in previous chapters. For the TMP36, connect pin 1 to the same 5V connection as the ADC module and pin 3 to the GND connection on the ADC module. Pin 2 on the sensor connects to the A0 pin on the ADC module.
Connect the TMP36 sensor as follows (again, see Figure 7-20).
Be sure to double-check your connections and compare them to Figure 7-20. Failure to connect things properly on the Raspberry Pi can lead to a damaged board.
You should see the ADC module appear as address 0x48 in the output, as shown in Figure 7-21.
Write the Code
Now that you have the libraries you need, it is time to write a script to read samples from a TMP36 sensor (via the ADC module) and save the data to your ThingSpeak channel. Since we have already written code to read the TMP36 sensor, we will concentrate on the code for writing data to ThingSpeak.
Begin by opening a new file on your Raspberry Pi named raspi_tmp36.py. You can use the Thonny IDE or a text editor or nano in a terminal to create the file.
Next is the core code for the script. We will use a try…except block to capture the keyboard interrupt (Ctrl+C). Inside that, we prepare a special URL to encode the field data for our channel and then open the URL.
Complete Code for the raspi_thingspeak.py Script
Be sure to substitute your API key in the location marked. Failure to do so will result in runtime errors.
Now that you have all the code entered, let’s test the script and see if it works.
Testing the Script
Python scripts are interpreted programs. Although there is a fair amount of syntax checking at the start of a script, logic errors are not discovered until the statement is executed. Thus, you may encounter errors or exceptions if the script was not entered correctly (e.g., if you misspelled a method or variable name). This may also happen if you failed to replace the placeholder for the API key and feed number.
Let the script run for 3 minutes or so, and then navigate to your Raspberry Pi channel on ThingSpeak. You should see your sensor data displayed, similar to that shown in Figure 7-22.
If you do not see similar data, go back and check the return codes as discussed in the last project. You should see return codes of 200 (success). Check and correct any errors in network connectivity or syntax or logic errors in your script until it runs successfully for several iterations (all samples stored return code 200).
If you see similar data, congratulations! You now know how to generate data and save it to the cloud using two different platforms.
For More Fun
You can have a lot of fun with this script. Try connecting other sensors and creating other channels for them in ThingSpeak. You can also experiment with reading the data you saved in ThingSpeak.
Storing Sensor Data in a Database
As you may have surmised, it is possible to store sensor data to a database on a Raspberry Pi. You can use MySQL as your database server and the Connector/Python library to write Python scripts that read sensor data and store the data in tables for later processing. Because there is a lot more involved than a few dozen lines of code (like setting up MySQL on the Raspberry Pi), you explore this topic in greater detail in Chapters 8 and 9.
Component Shopping List
Components Needed
Item | Vendors | Est. Cost USD | Qty Needed |
|---|---|---|---|
I2C EEPROM | $1.95 | 1 | |
Arduino Ethernet Shield | $24.95 | 1* | |
microSD Shield | $14.95 | * | |
Data Logging shield for Arduino | $19.95 | * | |
DS1307 Real-Time Clock breakout board | $7.50 | 1** | |
Real-Time Clock module | $14.95 | ** | |
12-bit ADC module | $9.95 | 1 |
Components Reused from Previous Chapters
Item | Vendors | Est. Cost USD | Qty Needed |
|---|---|---|---|
Pushbutton | $0.35 | 1 | |
Breadboard (not mini) | $4.95 | 1 | |
Breadboard jumper wires | $3.95 | 1 | |
TMP36 sensor | $1.50 | 1 | |
0.10uF capacitor | $0.25 | 1 | |
Raspberry Pi 3B, 3B+, or 4B | Most online and retail stores | $35 and up | 1 |
HDMI or HDMI to DVI cable | Most online and retail stores | Varies | 1 |
HDMI or DVI monitor | Most online and retail stores | Varies | 1 |
USB keyboard | Most online and retail stores | Varies | 1 |
USB power supply | Most online and retail stores | Varies | 1 |
USB Type A to USB micro male | Most online and retail stores | Varies | 1 |
SD card, 2GB or more | Most online and retail stores | Varies | 1 |
Cobbler+ | $7.95 | 1 | |
10K Ohm Resistor | Most online and retail stores | Varies | 1 |
4.7K Ohm Resistor | Most online and retail stores | Varies | 2 |
Summary
This chapter explored the local storage options for the Arduino and Raspberry Pi. You completed a number of small projects demonstrating each of the possible storage options. I also discussed storing sensor data in the cloud using the ThingSpeak IoT site from MathWorks. There, we learned how to create channels and send data to the channel.
In the next two chapters, I take a break from this exploration of sensor projects and begin discussing another form of remote storage: a database server. Chapter 8 focuses on setting up a MySQL server, and Chapter 9 focuses on using the MySQL server with the Arduino via a special database connector (library) written for the Arduino.