MQTT architecture

The MQTT protocol is a client-server architecture. A typical system will consist of a cloud-based server, which is called a Broker (Fig. 6.1). Each IoT device is a client of the Broker and can send and receive messages to and from the Broker.

MQTT uses a publish and subscribe communications model. Each client can connect to the Broker and subscribe to a particular message known as a Topic. Any client may connect to the Broker and publish new data to any Topic. When the data within a Topic is updated, a message with the new data will be broadcast to any and all clients that are subscribed to that Topic. In this case, clients may be other IoT devices. Additionally, a typical Broker will also provide an HTTPS socket interface for services and REST-based applications. MQTT Brokers are designed to support several thousand IoT devices allowing you to build a large scale system very easily. Most commercial IoT services will support MQTT with regional servers to provide a global network infrastructure.

Message topics

When an MQTT client connects to a Broker, it can publish a message Topic. The Broker does not need to be preconfigured with a list of available Topic messages. A new Topic will automatically be added to the Broker database when first published by a client (Fig. 6.2). A message Topic will also be created if any client subscribes to it and it is not currently in the Broker database.

A Topic consists of a message header followed by the message data. A Topic is a form of addressing that creates a message hierarchy similar to the folder structure within a file system. A typical Topic header is shown below:

• weather/uk/cities/london

It is also worth noting that adding a final forward slash creates a different Topic:

• weather/uk/cities/london/

Each Topic header is a UTF-8 string that consists of a number of user-defined levels delimited by Topic separators in the form of a forward slash (/). Our message data will then follow the Topic header. The message data can be in any arbitrary format, but it is common to use a standard data interchange format such as JAVA Script Object Notation (JSON) and more recently, Concise Binary Object Representation (CBOR).

Topic subscription

Any client can subscribe to any topic, whether a publishing node has created it or not. If you subscribe to a nonexisting Topic, the Broker will be forced to create it. When a client subscribes to a topic, it is possible to use a range of wildcards to receive groups of messages. The hash (#) character can be used to receive all messages below the topic level elements in the header.

• Weather/uk/#will receive all weather messages for UK locations.
• We can also use the plus character (+) to substitute for any name in a given topic level.
• Weather/+/cities/+ would subscribe to all countries and all cities.

Quality of service

Each message Topic will also be assigned Quality of Service (QoS). The QoS defines the delivery effort between the client and the Broker. The MQTT protocol defines three QoS levels. Level zero relies on the network transport layer to make a best-effort attempt to deliver the message but does not guarantee that it will be delivered. If you select QoS level one, the MQTT protocol does guarantee delivery, but the message may be received more than once, while QoS level two guarantees that a message will be delivered exactly once with no duplicates (Table 6.1).

When you are defining message QoS Levels, it is important to realize that the QoS Level is defined by both the publishing client and the subscribing client (Fig. 6.3).

Thus a client could publish a message at QoS Level 1 where a message may be delivered to the Broker with possible duplicates. A separate client could subscribe to the same Topic with a QoS Level 2 and expect a single update without duplicated messages.

Retained topics

It is also possible for a client to publish a message topic to the Broker and define it as a “retained message.” This will cause the Broker to store the message along with its QoS. When a new client subscribes to the message topic or an existing subscribed client reconnects to the Broker, the stored message will be immediately sent to the subscribing client. This is very useful when communicating with low power nodes that may frequently connect and disconnect to a network.

Heartbeat

When a client connects to a server, it defines a "keep-alive period" in seconds. Then during normal operation, the client must send at least one message during the keep-alive period. If the node is a subscriber-only, it must send a PINGREQ message and receive a PINGACK back from the Broker. If the Broker does not receive any communication from a client in one and a half times the keep-alive period, it must disconnect the client. Similarly, the client must disconnect from the Broker if it does not receive any messages and the keep-alive period is exceeded.

Last will and testament

When a client connects to a Broker it can publish a “last will and testament” message. This is a normal MQTT message that any other client can subscribe to. However, the published message is stored until the Broker determines that it has lost communication with the client. When this happens the last will and testament message will be sent to all the subscribed clients. The Broker will determine it has lost communication with the client if the keep-alive period is exceeded and it has not received a disconnect message.

Methods

The MQTT protocol is encapsulated within a network transport. This is normally a TCP message which provides for guaranteed delivery. Since the MQTT protocol is designed to support resource constrained devices, a relatively small number of methods support the connect/disconnect, publish, and subscribe model (Table 6.2).

Exercise: PC broker and client

In this section, we will install an MQTT Broker on a development PC along with a test client. We can then check both the Broker and client's operation before we move to design an embedded node. This also allows us to generate and test X.509 certificates before using them with an IoT device. We will use the popular Eclipse Mosquitto Broker and a widely used client called MQTT.fx for these examples.

Download both applications from the links in Table 6.3.

Installing the Mosquitto Broker

This tutorial assumes you have installed the Mosquitto Broker as a window service by following the default installation settings. Depending on the version of the Mosquitto server, you may also need to install OpenSSL for supporting dll files.

• In the pack installer copy Exercise 6.1.

This exercise contains an embedded MQTT project that we will use in the next section along with a directory called utilities, which contains some X.509 certificates and a configuration file for the Mosquitto broker.

• Copy the mosquitto.conf file to c:program files(x86)mosquitto directory.
• Copy the certificates from the utilities folder to a new folder c:certs or use the certificates created in Exercise 5.8

You must use the c:certs directory or change the paths in the Mosquitto configuration file.

• In the windows search box type services and open the services control app (Fig. 6.4).

  

• Locate the Mosquitto service and restart it to load the new configuration file.

If the Mosquitto Broker does not restart it will most likely be an error in the configuration file, probably failing to load the certificates (Fig. 6.5).

This configuration file will enable three ports on the server, which we can use for testing (Table 6.4).

Table 6.4

Mosquitto MQTT ports.

Port	Service
1883	MQTT unencrypted communication
8883	MQTT with TLS protocol. Broker authentication
8884	MQTT with TLS protocol. Broker and device authentication

Once the Mosquitto Broker is running with the new configuration file, we can connect to it with the MQTT.fx client.

• Start MQTT.fx
• Select and press the paper sheet icon (1) on the far left side (Fig. 6.6).

  

This will set the broker address as the local host (127.0.0.1) and the plain text port (1883).

The settings button (2) will allow you to define your own profile.

• Next press the Connect button (Fig. 6.7).

  

This will connect you to the Broker. When this happens, the gray circle (3) will turn green to show the connection and an open padlock will show that the connection is unencrypted.

• Next we can subscribe to a Topic. This can be “made up” as we do not need to do any configuration within the Broker (Fig. 6.8).

  

• Select the Subscribe menu (4).
• Enter a Topic (5) and press the second Subscribe button (6).

Here, I have used test/var/param1 but you could use anything, just avoid a slash at the start or end of the topic.

• Now select the Publish menu (7) (Fig. 6.9).

  

• Add the same Topic string (8).
• Enter a message in the payload box (9) and then press Publish (10).

This will publish the message to the Broker where it will be echoed to any client which has subscribed to the same Topic (Fig. 6.10).

Now that we have a working broker and client, we can add an MQTT client to our WiFi example and connect to the local broker. The Keil pack system provides a set of cloud connectors for commercial IoT cloud systems, including a plain MQTT broker.

The MQTT Paho client is part of the eclipse project and is designed to integrate to any network stack that provides a socket-type interface.

Exercise: Embedded MQTT client

In this example, we will connect an embedded client to our local Mosquitto client running on your PC and then to a cloud server. For the local connection, you will need to create a rule or exception within your PC firewall to allow a remote client to connect.

• Open the microVision project that was copied in Example 6.1

This is a multiproject workspace. The exercise project is our WiFi example and the second project is a working solution.

Now add the Paho MQTT client using the microVision RTE.

• Open the RTE and select the following components:
• IoT Client::MQTTClient-C
• IoT Utility::MQTTPacket
• SocketAPI::WiFi (Fig. 6.11)

  

• Click OK to add the files to the project.
• In socket_startup.c, update the code to set the SSID name for your network and its password.

   #define SSID""   #define PASSWORD ""   #define SECURITY_TYPE ARM_WIFI_SECURITY_WPA2

Once the device is connected to the WiFi network, it will invoke a function called MQTTEcho_Test(). This code is provided as a standard template.

• Select the source Group 1 folder, right click and select Add New.
• Select user code template/IoT Client::MQTT Echo (Fig. 6.12).

  

• Now click Add.

As its name implies, this will add example code to add a simple echo test similar to the one we just performed with the PC client.

• Open MQTT_Echo.c.
• Go to line 11 and change the server name to the IP address of your PC.

•           //#define SERVER_NAME "mqtt.eclipse.org"
•           #define SERVER_NAME "192.168.0.127"

The address mqtt.eclipse.org is the address of a public test server.

Once we have tested a local connection to the PC, we can set the server address to use the eclipse server and test against a public cloud server:

   #define SERVER_PORT 1883   MQTTClient client;   Network network;   MQTTPacket_connectData connectData = MQTTPacket_connectData_initializer;   NetworkInit(&network);   MQTTClientInit(&client, &network, 30000, sendbuf, sizeof(sendbuf), readbuf, sizeof(readbuf));

   NetworkConnect(&network, SERVER_NAME, SERVER_PORT);   connectData.MQTTVersion = 3;   connectData.clientID.cstring = "MDK_sample";   MQTTConnect(&client, &connectData);

Once a WiFi connection is established, the echo code will create a network and client instance and then connect to the Broker using the unencrypted port 1883.

Once connected, it will subscribe to a collection of topics using the # wildcard.

The subscription function defines a callback function void messageArrived (MessageData* data) which will be triggered when a matching message is received:

          MQTTSubscribe(&client, "MDK/sample/#", QOS2, messageArrived);

The messageArrived callback function is passed a pointer to a message data structure:

   typedef struct MessageData   {MQTTMessage* message;MQTTString* topicName;   } MessageData;

This in turn contains structures for the message topic and the message payload:

   typedef struct MQTTMessage{enum QoS qos;unsigned char retained;unsigned char dup;unsigned short id;void *payload;size_t payloadlen;   } MQTTMessage;   typedef struct   {   char* cstring;   MQTTLenString lenstring;   } MQTTString;

Once we have subscribed to the message topics, we can create a message payload:

MQTTMessage message;

   char payload[30];   message.qos = QOS1;   message.retained = 0;   message.payload = payload;   message.payloadlen = strlen(payload);

And then publish the message as the Topic “MDK/sample/a”

          MQTTPublish(&client, "MDK/sample/a", &message);

The example code publishes ten messages and then disconnects. We can provide a delay between each message by using a dedicated delay function provided by the MQTT client function:

          MQTTYield(&client, 1000);

Unlike an osDelay function this will suspend the thread for 1000 mSec while still allowing for MQTT messages to be received.

When all of the messages have been published, we will disconnect from the Broker and network:

          NetworkDisconnect(&network);

• Start the MQTT.fx client connect to the Broker as before.
• Subscribe to MDK/sample/#.
• Build the project and start the debugger.
• Start Tera Term and connect to the virtual serial port.
• Run the code.

You should see the arrival of each message in the debugger and the MQTT.fx client (Fig. 6.13).

If not check your PC firewall settings.

Retained messages

When we publish a message, it is possible to set the retained option. This will keep the last topic payload within the Broker. When a client subscribes to the topic, the retained payload will immediately be sent.

• Disconnect the MQTT.fx client from the Broker.
• Go to line 70 in mqtt_echo.c and set the retained value.

          message.retained = 1;

• Build the code.
• Start the debugger and run the code.
• Wait until the embedded client has sent the messages and disconnected.
• Now connect the MQTT.fx client and subscribe to MDK/sample/#.

You will be immediately sent the last message published to this Topic (Fig. 6.14).

Connection object

When we established a connection to the Broker the connection parameters were passed in the variable connectData which was initialized with its default settings.

          MQTTPacket_connectData connectData = MQTTPacket_connectData_initializer;

MQTTPacket_connectData is a structure that contains the elements shown in Table 6.5.

Keep alive interval

The connection object allows us to set a keep-alive interval in seconds. The client must send a message to the Broker within one and a half times this period, or the Broker will disconnect the client. If the client does not have a valid message to send, it can send a PINGREQ message to the Broker. In this case, the client should monitor the Broker's PINGRESP reply. If this does not arrive in a timely fashion, the client should disconnect from the network. This mechanism can be disabled if the keep-alive interval is set to zero. If the keep-alive interval is enabled, it can be used in conjunction with the client last will and testament.

Last will and testament

When we connect to a server it is possible to define the Last Will and Testament (LWT) of a client device.

This message will be sent to all clients that are subscribed to the LWT topic if the publishing client does not disconnect gracefully. For example, it may run out of battery power and shutdown without performing a network disconnect. This will leave the Broker connection open, but the keep-alive will fail. If this or a similar failure occurs, the Broker will send the LWT message.

We can define a LWT message in the connection object. The will value should be set to logic “1,” and the message can be defined in the “will” object (Table 6.6).

Enter the following code on line 12 of mqtt_echo.c after the definition of connectData.

• MQTTPacket_connectData connectData = MQTTPacket_connectData_initializer;

   connectData.keepAliveInterval = 10;   connectData.willFlag = 1;   MQTTString topic = MQTTString_initializer;   topic.cstring = "MDK/LWT";   connectData.will.topicName = topic;   MQTTString message = MQTTString_initializer;   message.cstring = "Bye";   connectData.will.message = message;

• Build the code and start the debugger.
• Set a breakpoint on the MQTTYeild() function line 78 of MQTTEcho.c (Fig. 6.15).

  

• Connect the MQTT.fx client and subscribe to MDK/LWT.
• Run the embedded code.

The client will connect to the Broker and define the LWT and a keep-alive interval of ten seconds. It will then subscribe and publish a message as before. When it hits the breakpoint, no further messages will be sent. When one and a half times the keep-alive interval (fifteen seconds) has passed, the LWT will be sent to the subscribed client.

• Remove the breakpoint.
• Reset the embedded client and run again.

This time the code runs to completion and the LWT message is sent when the networkDisconnect function is called. We have disconnected from the network without disconnecting from the Broker.

• Add the following line of code before the networkDisconnect function.

   MQTTDisconnect(&client);   NetworkDisconnect(&network);

• Rebuild the code and run again.

This time the embedded code will run to completion without sending the LWT.

In this example, we have added an MQTT client, connected to a Broker and been able to publish and subscribe to MQTT messages. We have also explored the keep-alive interval and the LWT message. However, all of this communication has been in plain text. In the next section, we will look at how to add the TLS protocol.

Exercise: TLS encryption

We are going to add the TLS security in two stages. First, we will set up a system that authenticates the server but let any client connect. Next, we will set up the TSL security to authenticate both the server and client.

In the Mosquitto program directory, locate and open the mosquitto.conf file.

The configuration file allows you to define multiple active MQTT ports. Each port is defined as a listener.

Our first encrypted port is defined in the Broker configuration file as port 8883, and this is the default setting in the mosquitto config file.

          listener 8883

In the security section we define the CA certificate, the Broker certificate and the Broker private key.

• cafile C:certsiot_ca.crt
• # Path to the PEM encoded server certificate.
• certfile C:certsiot_broker.crt
• # Path to the PEM encoded keyfile.
• keyfile C:certsiot_broker.pem

We can first test that the certificates are setup correctly with the MQTT.fx client.

• Start the MQTT.fx client.
• Make sure the connection list view is selected (1).
• Now press the settings icon (2).
• Now configure the TLS settings (Fig. 6.16).

  

• Set the profile name to localTLS.
• Enter your PC IP address (or use localhost).
• Set the Broker port to 8883.
• Next select the SSL/TLS menu.
• Select CA certificate file and select the IoT_CA.crt.
• Then press OK to accept the settings.
• In the main dialog select the localTLS profile and press connect.

This should now negotiate a secure connection to the Broker.

You can further test the connection by subscribing and publishing Topics as before.

Now that we have established that the secure port on the Broker is working and the CA certificate is correct we can add the mbedTLS library to the embedded client and establish a secure embedded connection.

• In the pack installer copy project 6.2.
• This is a multiproject workspace with a working project and a reference project.
• Set the working project as the active project.
• Open the RTE and add the Security/mbedTLS component (Fig. 6.17).

  

In the project directory there is a preconfigured version of the mbedTLS_Conf.h file.

This is setup with the necessary options for the TLS protocol.

• Copy this file to the <  project  >/RTE/security folder.

The mqtt_echo.c template code is designed to support a secure connection if the following define is declared at a project level.

          #define MQTT_MBEDTLS

• Open the option for target/C/C  ++ tab and enter the define (Fig. 6.18).

  

The define will cause the embedded client to connect to the encrypted port with a secure version of the network connect function:

   #define SERVER_PORT 8883   NetworkConnectTLS(&network, SERVER_NAME, SERVER_PORT, &tlscert)

Here, we need to provide the Certificate Authority certificate, the default is an array called tlscert[].

   #if (MQTT_MBEDTLS != 0)   #include "certificates.h"   #endif

A template header file is provided for you to add the Broker CA cert. In this example, a preconfigured version is located in the project directory:

• Open the certificates.h file in the project directory.

The Broker CA certificate is held in const char array and formatted as shown below:

   static const char CA_Cert[] =   "-----BEGIN CERTIFICATE-----
"   "MIIElDCCA3ygAwIBAgIBATANBgkqhkiG9w0BAQsFADCBkjELMAkGA1UEBhMCVUsx"    ……………   "1Do2MazGz  +  kKBHKdOYtsWa0xVhLOkOjS
"    "-----END CERTIFICATE-----
"    ;

• Build the project.
• Start the debugger and run the code.

The embedded client will now connect to the encrypted TLS port and send the same set of MQTT messages, but this time they will be sent over an encrypted channel.

In this example, we are authenticating the server and allowing any client to connect. The Broker has an additional listening channel on port 8884. This port has the same security options as 8883, but we have enabled the additional option “requires_certificate true.” The security settings in the Mosquitto configuration file are as follows:

   cafile C:certsiot_ca.crt   certfile C:certsiot_broker.crt   keyfile C:certsiot_broker.pem   require_certificate true

This will force the Broker to authenticate the client. As before we can test the certificates in mqtt.fx by configuring a new profile for client authentication:

• Start MQTT.fx and open the settings dialog.
• Create a new profile as before.
• This time use the port number to 8884.
• In the TLS menu, add the certificates and device private key in the self-signed certificates section (Fig. 6.19).

  

• Click OK to save the profile.
• In the main dialog click connect.

This will now establish a secure connection which authenticates both the client and Broker.

• Open the embedded client example.

The certificate.h is preconfigured with the client certificate and key.

These are defined in two additional arrays:

   static const char ClientCert[];   static const char ClientKey[];

• Open mqtt_echo.c

We need first switch the port to 8884 so we connect to the mutual authentication port:

          Line 14#define SERVER_PORT 8884

• And then provide the additional certificate and key.

          Line 30 TLScert tlscert = {(char *)CA_Cert,(char *) ClientCert,(char *) ClientKey};

• Rebuild the project.
• Start the debugger and run the code.

This will now establish mutually authenticated secure connection with the Broker and allow us to transfer data between the IoT device and the Broker.

JavaScript Object Notation

Put simply, JSON is a syntax for storing and exchanging data between different computing systems. There are three big advantages to using JSON. First, it is easy to adopt and use. Second, JSON is text-based, making it human-readable and intuitive to understand. Third, it is schema-less, in that the parser will format the message as a set of labels and data which the receiver can interpret without a detailed knowledge of the message structure. This allows you to develop reliable communication packets where the content is evolving over time. The JSON format has achieved widespread adoption, particularly in IoT networks and system configuration files.

JSON object

A JSON object is an ASCII string enclosed by curly braces and contains a set of key/value pairs. A key is a label for the following data. It is encoded as an ASCII string enclosed by double-quotes. There are various data types that can be used as values and we will look at these below but for example, a quote enclosed string is a valid value, an example is shown below:

{“key_0”: “value_0”}

We can add other key value pairs by using a comma (,) as a delimiter.

The following data types are available within a JSON Object (Table 6.7).

Exercise: JSON encoding

This example will encode and decode some sample data into a JSON string and send this as the payload of our MQTT message. As we are subscribed to the same message, we can receive and decode the JSON data back to usable variables.

• In the pack installer copy project 6.3.

• Open the RTE and select the Data exchange:JSON::cJSON component.

cJSON is a lightweight JSON parser that can be used to encode and decode JSON objects.

• Open the json_test.c file.

This module contains functions to serialize and deserialize some example application data.

The serialize function first creates some dummy data values for typical process values of temperature, pressure and humidity. We then create the JSON object and JSON variables for each data value. The cJSON type is a structure with elements to store each JSON type and a linked list to chain successive elements together. We also create a char pointer, which is used to store the resulting JSON serialized string:

   char *serialize() {   int temperature = 23, pressure = 1000, humidity = 40;   cJSON *Jobject, *Jtemperature, *Jpressure, *Jhumidity ;   static char *J_string = NULL;   .................

Next we can create the JSON object then create and initialize each JSON value:

   Jobject = cJSON_CreateObject();   Jtemperature = cJSON_CreateNumber(temperature);   Jpressure = cJSON_CreateNumber(pressure);   Jhumidity = cJSON_CreateNumber(humidity);

Then we can assemble the JSON object and serialize it to the string:

   cJSON_AddItemToObject(Jobject, "temperature", Jtemperature);   cJSON_AddItemToObject(Jobject, "pressure", Jpressure);   cJSON_AddItemToObject(Jobject, "humidity", Jhumidity);   J_string = cJSON_PrintUnformatted(Jobject);   cJSON_Delete(Jobject);

The function returns the formatted string, which is used as the payload for the MQTT message.

• Run the code so that it sends the formatted message.
• Observe the received message in MQTT.fx.

We can now deserialize the string back to usable data.

In the deserialize function, we create a JSON object and then parse the received string to a JSON object.

   int deserialize(const char * const Jstring){   const cJSON *jnum = NULL;   cJSON *Jobject = cJSON_Parse(Jstring);

We can then extract the value for each element into a cJSON number:

          jnum = cJSON_GetObjectItemCaseSensitive(Jobject, "temperature");

We can then test for the type of data and if a value is present. If all is good we can extract the value into a variable:

   if (cJSON_IsNumber(jnum) && (jnum->valueint != NULL))    printf("Temperature - %i
", jnum->valueint);

JSON arrays

JSON also allows us to declare arrays of JSON types. An array obeys the same rules as a key value pair while the value is a collection of comma delimited JSON types enclosed by square brackets:

          {"locations":[ "office", "Meeting_1", "Kitchen" ]}

We can extend our serialize() function to add an array.

• Uncomment the function JSON_ARRAY()

This function contains code to create and populate a JSON array:

   cJSON *Jarray = cJSON_CreateArray();   cJSON_AddItemToObject(Jobject, "Locations", Jarray);

Next we can create some JSON values:

   cJSON *location_0 = cJSON_CreateString("Office");   cJSON *location_1  = cJSON_CreateString("Meeting_1");

          cJSON *location_2 = cJSON_CreateString("Kitchen");

and then add them to the array:

   cJSON_AddItemToArray(Jarray, location_0);   cJSON_AddItemToArray(Jarray, location_1);   cJSON_AddItemToArray(Jarray, location_2);

In the deserialize function, we can create cJSON variables for the JSON array and the locations:

   cJSON *Jarray, *location[3];   int index;

We can now load read the JSON array within the JSON object and then loop through the stored values and extract the location strings:

   Jarray = cJSON_GetObjectItemCaseSensitive(Jobject, "Locations");   for(index = 0; index <  3; index  ++) {location[index] = cJSON_GetArrayItem(Jarray,index);if (!cJSON_IsString(location[index]) || location[index]->valuestring != NULL)printf("Location - %s
", location[index]->valuestring);   }

• Rebuild and rerun the code.

You can observe the received message in mqtt.fx and the decoded values in the debugger console window.

Nested JSON objects

It is also possible to have a JSON object within a JSON object. This follows the same rules as a key-value pair, but the value is a self-contained JSON object. While this may look a bit like an array, we are storing key: value pairs. This allows us to search within a secondary JSON object without having to know much about its actual structure. In a real-world system, this becomes a powerful technique.

          {"office":{ "temperature":23, "pressure":1000, "humidity":40}}

• In the code uncomment the function Nested_JSON_Object();

We create our JSON object as before and then add a further JSON object:

   Jsub_object = cJSON_CreateObject();   cJSON_AddItemToObject(Jobject, "office", Jsub_object);

We can then populate the nested object with key/value pairs:

   cJSON_AddItemToObject(Jsub_object, "temperature", Jtemperature);   cJSON_AddItemToObject(Jsub_object, "pressure", Jpressure);   cJSON_AddItemToObject(Jsub_object, "humidity", Jhumidity);

if we receive a nested object it is possible to search for the sub object and then extract values from it:

   Jsub_object = cJSON_GetObjectItemCaseSensitive(Jobject, "office");cJSON *Jtemp = cJSON_GetObjectItemCaseSensitive(Jsub_object,"temperature");if (cJSON_IsNumber(Jtemp) && (Jtemp->valueint != NULL)){    printf("Office Temperature - %i
", Jtemp->valueint);}

Rebuild and rerun the code.

You can observe the received message in mqtt.fx and the decoded values in the debugger console window.

Concise binary object representation

While JSON is widely used to encode IoT data, it is designed to transport values encoded as ASCII strings. In many cases, we want to transport large amounts of binary data such as encryption keys and even firmware updates. The only way to do this with JSON is to base 64 encode the data. This significantly increases the complexity and bulk of our data packets. To efficiently transfer data in a binary encoded form, we can use a different encoding format called Concise Binary Object Representation (CBOR). While there are other encoding schemes available such as ASIN.1, BSON and Message Pack, CBOR is becoming more widely used for IoT because a typical encoder/decoder requires minimal resources so that it will run on very limited devices. CBOR is also based on the JSON data model and like JSON, you can easily adapt and extend the message schema without risking breaking the whole communication scheme.

There are a number of open-source CBOR encoders available. The one currently used as part of the Platform Security Architecture firmware is QCBOR. This implementation is optimized for speed and code size.

The CBOR data format describes how to encode different data types as a serialized byte string. Each data item is preceded by a header byte which defines the encoded data type (Fig. 6.20).

The header byte is split into two fields the top three bits is the major type field while the lower five bits is the additional information field.

The major type field describes the data item following the header byte. The current CBOR standard defines seven data types which are encoded as follows (Table 6.8).

Each data type will use the additional information field to fully describe the data item while the data value is held in adjacent bytes (Table 6.9).

Integer encoding

To encode an unsigned integer the major type field will be set to zero. If the value in the integer is in the range 0–23, it will be directly encoded in the additional information field. If the integer value is greater than 23, the remaining values in the additional information field 24–27 specify the number of bytes (1–4 following the header byte which are used to hold the unsigned integer value.

Exercise: CBOR encoding

• In the pack installer copy Exercise 6.4.
• In microVision open the RTE.
• Add the QCBOR encoder/decoder (Fig. 6.21).

  

• Open main.c.

This file contains examples of how to encode/decode the CBOR major types.

We can use QBOR to encode three bytes as shown below:

• First include the QCBOR header:

   #include "qcbor.h"   int main(void) {

Next create a context for this CBOR serialization and a structure which contains a pointer to the serialized object and its length:

QCBOREncodeContext EC;  UsefulBufC Encoded;

Now create some input data items and a buffer to hold the final serialized string:

uint8_t data_A = 0x10, data_B = 0x55; pBuf[300];uint16_t data_C = 0xAA55;

In the code, we must first initialize the encoder with the context and output buffer:

QCBOREncode_Init(&EC, UsefulBuf_FROM_BYTE_ARRAY(pBuf));

Now we can encode each unsigned integer. Although the encoder function produces an optimized string for each word size up to uint64_t:

   QCBOREncode_AddUInt64(&EC, data_A);QCBOREncode_AddUInt64(&EC, data_B);QCBOREncode_AddUInt64(&EC, data_C);

Finally, the encoder can be exited. The encoded structure contains a pointer to the serialized pBUF array and the length of the CBOR string:

QCBOREncode_Finish(&EC, &Encoded));

This will generate the serialized string shown in Table 6.10.

The CBOR string can now be decoded.

First create our data items and the CBOR context:

   uint8_t decode_A;uint8_t decode_B;uint16_t decode_C;QCBORDecodeContext DC;

Then create a CBOR item which is used to hold the initial decoded value and initialize the decoder with the previously encoded byte string:

QCBORItem Item;  QCBORDecode_Init(&DC, Encoded, QCBOR_DECODE_MODE_NORMAL);

We can now read each item in turn:

QCBORDecode_GetNext(&DC, &Item);

Once an item has been read, we can query the data type and then read the data value:

if(Item.uDataType == QCBOR_TYPE_UINT64) {      decode_A = Item.val.uint64;}QCBORDecode_GetNext(&DC, &Item);if(Item.uDataType == QCBOR_TYPE_UINT64) {    decode_B = Item.val.uint64;   }QCBORDecode_GetNext(&DC, &Item);if(Item.uDataType == QCBOR_TYPE_UINT64) {    decode_C = Item.val.uint64;}

Once the full string has been parsed, we can exit the decoder and release the context:

QCBORDecode_Finish(&DC)

It is also possible to encode signed integer using a mix of major type 0 and 1 items:

   int8_t data_A = -0x10, data_B = 0x55;int16_t data_C = -0xAA55;   QCBOREncode_AddInt64(&EC, data_A);   QCBOREncode_AddInt64(&EC, data_B);   QCBOREncode_AddInt64(&EC, data_C);

Which encodes to (Table 6.11).

We can decode the serialized data the same way, but this time, we can test for a signed value and read the signed integer tag:

   if(Item.uDataType == QCBOR_TYPE_INT64) {    decode_C = Item.val.int64;}

It is also possible to store text strings byte strings and arrays in a similar fashion:

   QCBOREncode_OpenArray(&EC);QCBOREncode_AddUInt64(&EC, 451);QCBOREncode_AddUInt64(&EC, 331);QCBOREncode_CloseArray(&EC);

This encodes as a set bytes of plus an initial header that with the major type set to array and the number of elements is held in the additional data field (Table 6.12).

We can also encode text and byte strings:

   QCBOREncode_Init(&EC, UsefulBuf_FROM_BYTE_ARRAY(nBuf));QCBOREncode_AddText(&EC, UsefulBuf_FROM_SZ_LITERAL("bar bar foo bar"));QCBOREncode_AddSZString(&EC, "oof
");

When decoded the item structure will provide a pointer and length of the string data:

   QCBORDecode_GetNext(&DC, &Item);if(Item.uDataType == QCBOR_TYPE_TEXT_STRING ) {    bufPtr = (uint8_t *) Item.val.string.ptr;    size = Item.val.string.len;}

Byte and text strings

CBOR is also able to store strings of data and text. For fixed-length strings, the length of the string is stored as an integer at the beginning of the string using the Type 0 integer rules (Table 6.13).

Two types of string are defined: a byte string (Type 2), which contains binary data, and a text string, which holds a human readable ASCII string (Type 3).

Array of data items

We can also store arrays of integers using the same approach as byte arrays. However, the array size is defined as the number of objects, not the number of bytes.

Array of maps

The CBOR data format also supports the storage of maps. Here, maps are tables of key-value pairs which equate to objects in JSON. The first item is the key and the second is the value. Like a data item array, the array's size is the number of map pairs, not the overall number of bytes (Table 6.14).

Indefinite arrays, strings, and maps

CBOR can also handle arrays, strings and maps of variable length. In this case, an array is opened without defining the number of data items. Data items can be added as normal. When all of the current data has been written, the array is closed by writing a data item encoded as a major type 7 plus a break character as the data value (Table 6.15).

Semantic tags

In addition to providing the header for standard data types, CBOR provides support for semantic tabs. A tag is header type 6 and contains a single value. This is used as a hint to the decoder to describe a custom format for the following encoded bytes.

This allows us to use tabs to extend CBOR encoding types in a number of useful ways. To start with, we can use tags to create additional data items. The tag is placed immediately before the custom data to provide a self-described blob of binary data. The decoder may understand the tag, or it can be passed to a higher layer for decoding. The CBOR standard defines a number of tags and unassigned ranges for extension a small selection is shown in Table 6.16.

To encode a positive bignum value the encoder will first write the tag value 0x02 followed by a byte stream, which holds the value of the bignum (Table 6.17).

The decoder will then return the tab and an integer array, which can then be used to initialize an MPI value.

In addition to the tags defined in the CBOR specification (RFC 7049), the range of tags has been extended and maintained in an IANA registry. The draft list of tags is available from:

• https://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml

We can use these tags to improve an encoded CBOR stream's efficiency. For example, if we need to encode an array of 32-bit floating-point values. In a standard encoding, each float will be stored with a header followed by 4 bytes. By using a tag, we can define a much terser encoding. We can use a tag to declare a float array followed by the byte values of the IEEE754 floating-point number. In the IANA registry, a 32-bit float array is defined by tab 81 follower by a byte string. This allows us to store our array as a single tag followed by a binary blob where each float value is stored as four bytes.