MQTT protocol
The MQTT protocol is a lightweight network protocol used for publish/subscribe messaging, optimized for wireless remote devices that might have limited bandwidth and intermittent connectivity. This appendix describes the following concepts about the MQTT protocol:
•Quality of service (QoS) levels and flows
•QoS determination
•QoS effect on performance
•Client ID
•Durable and non-durable subscribers
•MQTT persistence
•MQTT header
•Keep alive
•Retry message delivery
•Last Will and Testament
•Retained flag on messages
•TCP/IP stack
Quality of service levels and flow
QoS levels determine how each MQTT message is delivered, and must be specified for every message sent through MQTT.
A useful comparison can be made with letters sent through a postal service. An informal letter to a friend can be dropped into a mailbox, and the sender might never think about it again. The sender expects that the letter will get to its destination, but there are no significant consequences if it does not arrive.
In contrast, a letter to which the recipient must respond is more important, so the sender might choose a higher level of delivery service that provides proof that the letter was received at its destination. In each situation, the choices made by the sender of the letter are comparable to choosing QoS levels in MQTT.
It is important to choose the appropriate QoS value for each message, because this value determines how the client and the server communicate to deliver
the message.
QoS 0: At most once delivery
It is termed as At most once message delivery because messages are delivered according to the best effort of the underlying network. A response is not expected, and no retry semantics are defined in the protocol. The message arrives at the MQTT server either once or not at all.
This is the lowest level of QoS. The MQTT client or the server attempts to send the message without waiting for any kind of receipt. There are no steps taken to ensure message delivery, other than the features provided by the TCP/IP layer. Also, there is no retry attempted by the MQTT layer if there is a failure to deliver the message. Therefore, if the client is sending a message, it can arrive at the MQTT server once or never.
A QoS 0 message can get lost if the client unexpectedly disconnects or if the server fails. From a performance perspective, this adds value, because it is the fastest way to send a message using MQTT.
The MQTT command message used is PUBLISH. No other command messages flow for the QoS 0 messages. Table A-1 shows the QoS level 0 protocol flow.
Table A-1 QoS level 0 protocol flow
Client
Message and direction
MQTT server
QoS = 0
PUBLISH
------------>
Action: Publish message to subscribers
QoS 1: At least once delivery
With QoS set to 1, the message is delivered at least once. The MQTT client or the server attempts to deliver the message at least once, but there can be a duplicate message. The receipt of a message by the MQTT server is acknowledged by a PUBACK message.
If there is an identified failure of either the communications link or the sending device, or the acknowledgment message is not received after a specified period of time, the sender resends the message with the DUP bit set in the message header. The message arrives at the MQTT server at least once. Both SUBSCRIBE and UNSUBSCRIBE messages use QoS level 1.
If the client does not receive a PUBACK message (either within a time period defined in the application, or if a failure is detected and the communications session is restarted), the client resends the PUBLISH message with the DUP flag set. When it receives a duplicate message from the client, the MQTT server republishes the message to the subscribers, and sends another PUBACK message.
At the client side, the implementations of the MQTT protocol also provides an additional feature known as MQTT persistence. The MQTT persistence layer is not described in the specification of MQTT, but is normally available with MQTT client implementations. When a QoS 1 message is published to the server, the client needs to wait for the acknowledgement to arrive.
There can be a program termination or a crash at the client device. When the client is started again, it will need to resume from the point it left before the crash. Therefore, the message is stored in a persistence layer, such as a disk, and retrieved soon after the reconnection to the MQTT server.
The MQTT command messages used are PUBLISH and PUBACK. When the publish happens, the message will be logged to the MQTT persistence layer, and removed when a PUBACK message is received. A message with QoS level 1 has a Message ID in the message header.
Table A-2 shows the QoS level 1 protocol flow.
Table A-2 QoS level 1 protocol flow
Client
Message and direction
MQTT server
QoS =1
DUP = 0
Message ID = x
PUBLISH
------------>
Actions:
•Store message in database
•Publish message to subscribers
Action:
Discard message
PUBACK
<-----------
N/A
QoS 2: Exactly once delivery
Exactly once is the highest level of QoS. Additional protocol flows higher than QoS level 1 ensure that duplicate messages are not delivered to the receiving application. The message is delivered once and only once when QoS 2 is used. The MQTT client or the server will ensure that the message is sent only once.
This QoS must be used only when duplicate messages are not wanted. From a performance perspective, there is a price to be paid in terms of network traffic and processing power.
The MQTT command messages used are PUBLISH, PUBREC, PUBREL, and PUBCOMP. The message is sent in the PUBLISH flow, and the client will store that message in the MQTT persistence layer (if it is used). The message will remain locked on the server. PUBREC is sent by the server in response to PUBLISH. PUBREL will be dispatched to the server from the client in response to PUBREC.
After PUBREL is received by the MQTT server, it can dispatch the messages to the subscribers and send back PUBCOMP to the PUBREL. A message with QoS level 2 has a Message ID in the message header.
If a failure is detected, or after a defined time period, each part of the protocol flow is tried again with the DUP bit set. The additional protocol flows ensure that the message is delivered to subscribers once only.
Because QoS1 and QoS2 indicate that messages must be delivered, the MQTT server stores messages in a database. If the MQTT server has problems accessing this data, messages might be lost.
Table A-3 shows the QoS level 2 protocol flow.
Table A-3 QoS level 2 protocol flow
Client
Message and direction
MQTT server
QoS = 2
DUP = 0
Message ID = x
PUBLISH
----------->
Action:
Store message in a database
 
PUBREC
<-----------
Message ID = x
Message ID = x
PUBREL
----------->
Actions:
•Update database
•Publish message to subscribers
Action:
Discard message
PUBCOMP
<-------------
Message DI = xb
Assumptions for QoS levels 1 and 2
In any network, it is possible for devices or communication links to fail. If this happens, one end of the link might not know what is happening at the other end. These are known as in doubt windows. In these scenarios, assumptions have to be made about the reliability of the devices and networks involved in message delivery.MQTT assumes that the client and server are generally reliable, and that the communications channel is more likely to be unreliable.
If the client device fails, it is typically a catastrophic failure, rather than a transient one. The possibility of recovering data from the device is low. Some devices have non-volatile storage (for example, flash ROM). The provision of more persistent storage on the client device protects the most critical data from some modes
of failure.
Beyond the basic failure of the communications link, the failure mode matrix becomes complex, resulting in more scenarios than the specification for MQTT can handle.
The time delay (retry interval) before resending a message that has not been acknowledged is specific to the application, and is not defined by the protocol specification.
QoS determination
When the client subscribes to the topic, it will specify a QoS at which it wants to receive messages from the server. Consider a scenario where publisher A is sending the messages to the topic at QoS 2. A subscriber can subscribe at QoS 0, and then the messages to the client will be delivered with QoS 0.
The QoS value is further used in SUBSCRIBE and UNSUBSCRIBE requests. A subscribe request from the client is a QoS 1 request, so the server will respond with SUBACK, which will ensure that the subscription has happened.
QoS effect on performance
There is a simple rule when considering the performance effect of QoS. It is: The higher the QoS, the lower the performance. Evaluate performance corresponding with a higher QoS. Suppose the time taken for sending a PUBLISH message is pt. If QoS is used, the total time taken to transfer n number of messages will be npt. Now in case of QoS 1, the PUBACK message (that is, the reply for the PUBLISH message) will flow from the server to the client.
This is a two-byte message, and can take much less time than pt. Therefore, call it mt. The time taken for transferring n messages will be n(pt + mt). Also, for QoS 2, the PUBREC, PUBREL, and PUBCOMP messages will be flowing. Therefore, the n number of messages will take approximately n(pt + 3mt).
So, if 10 messages need to be transferred from the client to the server, and pt is 1 second and mt is 0.4 seconds, a QoS 0 message takes 10(1) = 10 seconds. A QoS 1 message takes 10(1 + 0.4), which is 14 seconds, and a QoS 2 message takes 22 seconds.
MQTT client identifier
The MQTT protocol defines a client identifier (client ID) that uniquely identifies a client in a network. In simple terms, when connecting to a server, a client needs to specify a unique string that is not used currently and will not be used by any other client that will connect to the MQTT server. There are several ways of choosing a client identifier.
The following list includes a few examples:
• A Sensor installed in a particular location can use the location code as the client ID.
• A mobile device having network ability can choose the MAC address or a unique device ID as the client ID
MQTT restricts the client ID length to 23 characters, so some situations will require that the client ID be shortened. When shortening the client ID, you need to ensure that the client ID is not the same as any other client ID used in the network. To keep the identifier short and unique, introduce a reliable identifier generation mechanism.
For instance, you might create a client ID from the 48-bit device MAC address. If transmission size is not a critical issue, you might use the remaining 17 bytes to make the address easier to administer, such as including some human-readable text in the identifier.
If two clients were to have the same client identifier, one of the clients might receive a message and the other one might not. The MQTT server keeps track of the pending messages to be sent to a client based on the client ID. Therefore, if a client has been using QoS 1 or QoS 2, and subscribed to any topic and disconnected from the server, the server saves the messages that arrived for the client when it was disconnected.
After the client reconnects, the server will send those messages to the client. If some other MQTT device uses the same client ID and connects to the server, the server will send the messages it had saved to that device instead.
Another scenario related to the client IDs is duplicate connections. Suppose that a particular device using client ID DeviceA is connected to the MQTT server. If another client includes the same client ID (DeviceA), the server can allow the new client to connect and disconnect the existing client, or keep the old connection open and disallow the new client. This is an optional feature of an MQTT server.
Durable and non-durable subscribers with MQTT
A durable subscriber is any client that can receive all of the messages published on a topic, including messages published while the client was inactive. A client is said to be inactive in the following cases:
•A client has connected and subscribed to a topic, and later disconnected from the server without unsubscribing to that topic.
•A client has connected and subscribed to a topic, and later disappeared from the network due to some network issue or lost server connection.
A non-durable subscriber is any client that does not intend to receive messages published while it is inactive. A client is said to be non-durable in the following cases:
• A client always uses the clean session flag set to true when it connects to the server.
• A client always unsubscribes from all of the topics it subscribed for, before disconnecting.
In MQTT, QoS 0 messages are never persisted while a durable subscriber is inactive. Only QoS 1 and QoS 2 messages are persisted by the server and sent when the client becomes active. MQTT further provides a facility that enables MQTT clients to refrain from receiving messages while they were disconnected by setting the clean start flag to true. Here is an example flow for a typical durable MQTT subscriber:
1. MQTT ClientA connects to the server by specifying the clean session flag as false.
2. ClientA subscribes to the topic/test topic.
3. MQTT ClientB connects to the server and publishes messages from QoS 1 and QoS 2 to the topic/test topic.
4. ClientA receives the messages.
5. ClientA disconnects from the server (note that ClientA did not unsubscribe).
6. ClientB publishes more messages from QoS 1 and QoS 2.
7. ClientA connects to the server by specifying the clean session flag as false.
8. ClientA receives all of the messages that were published while it was inactive, and all of the future messages that are published to the topic/test topic.
Durable subscribers are useful when the subscribing client needs all of the messages that are published to the topic it subscribes to. Therefore, a client does not really need to worry about being in the network, because it is assured of receiving the messages while it was disconnected or even lost connection.
Although durable subscription is a nice feature to have, it adds additional responsibility to the server to hold the messages until the client connects back to the server.
MQTT persistence
The MQTT protocol is designed with the assumption that the client and server are generally reliable. What this means is that the client and the server will not fail, hang, or get into power failure issues. The assumption might prove to be costlier in some situations for the clients and therefore a feature known as MQTT client persistence is provided. The local persistence store is used to achieve
QoS 1 and QoS 2 level message flows:
•Before the client actually sends the PUBLISH command, the client stores the message data on a disk (or any other available storage).
•The client, when it receives an acknowledgement, deletes the message from the disk. Therefore, in case of a power failure and restart of the client application, the first action soon after the reconnect is to check the pending messages in the client persistence, and then send them. On the server side, this feature is generally managed by the messaging engine.
Message delivery from JMS to MQTT
When messages are sent from a Java Message Service (JMS) application to MQTT applications, the JMS reliability and persistence map to MQTT QoS levels. For the fastest but least reliable messaging between JMS and MQTT, non-persistent JMS messages can be sent with the IBM MessageSight DisableACK property set to true. This configuration provides QoS 0 for MQTT subscriber applications.
For the best reliability but slowest messaging rates, send persistent JMS messages and user per-message acknowledgements. This configuration provides up to QoS 2 for MQTT subscribers.
Table A-4 shows the possible combinations for message delivery from JMS to MQTT.
Table A-4 Mapping of JMS to MQTT messages
JMS message type sent
QoS of matching MQTT subscription
Reliability of delivered message
Persistence of delivered message
Either non-persistent or persistent
QoS 0
At most once
(QoS 0)
Non-persistent
Non-persistent, acknowledgements turned off
QoS 1
At most once
(QoS 0)
Non-persistent
Non-persistent
QoS 1
At least once
(QoS 1)
Non-persistent
Persistent
QoS 1
At least once
(QoS 1)
Persistent
Non-persistent, acknowledgements turned off
QoS 2
At most once
(QoS 0)
Non-persistent
Non-persistent
QoS 2
At least once
(QoS 1)
Non-persistent
Persistent
QoS 2
Exactly once
(QoS 2)
Persistent
MQTT header
The MQTT protocol only requires a two byte header. Consider a scenario where a client must send a message (for example, Hello) 10 times. Additionally, the client must tell the server that the destination is a/b for every message. We can calculate how many additional bytes of data will flow from client to server for one of the scenarios.
For sending the messages using MQTT, here is the computation of the total number of bytes:
1. CONNECT: Fixed (2 bytes) + Variable (12 bytes) = 14 bytes.
2. CONNACK: Fixed (2 bytes) + Variable (2 bytes) = 4 bytes.
3. PUBLISH: Fixed (2 bytes) + Variable: Length(a/b) = 3.
So, two bytes for representing the topic name, followed by the topic name, followed by the msg ID, followed by the payload, which is 2 + 3 + 2 + 5 = 12 bytes.
4. There are 10 publications happening... therefore (12) 10 = 120 bytes.
5. DISCONNECT: Fixed (2 bytes) = 2 bytes.
Therefore, this scenario needs 14 + 4 + 120 + 2 = 140 bytes.
As you can see from this example, the fixed portion of every message is two bytes. The total fixed amount is 140 - 80 = 60 bytes for this scenario.
MQTT keep alive
An MQTT server is able to determine if the MQTT client is still in the network (and vice versa) by using the keep alive timer. The TCP/IP time out and error handling mechanisms are at the network layer, but the MQTT client and server need not depend on that. The client says hello to the server and the server responds back acknowledging it. That is how simple keep alive works.
According to MQTT, the client needs only to tell the server that it is alive when there has not been any interaction between them for a period of time. The time period is a configurable option. This option can be set while connecting to the server by the client. It is the client that chooses the keep alive time. The server just keeps a record of the value in the client information table on the server side.
There are two messages that constitute the keep alive interaction:
•PINGREQ
The ping request is sent by the client to the server when the keep alive timer expires.
•PINGRESP
The ping response is the reply sent by the server for a ping request from the client.
Both these messages are short 2-byte messages. They are not associated with any QoS. The PINGREQ is sent only once, and the client waits for the PINGRESP.
Internally, there is a simple timer that triggers in the client. The timer checks if there has been any other recent activity between the client and the server. If not, a PINGREQ is sent. The client waits until a PINGRESP is received.
On the server side also, if there is no message received from the client within the keep alive time period, the server disconnects the client. But the server offers a grace period to the client, which is an additional 50% of the keep alive time period.
When choosing a keep alive time for a client, there are a few considerations. If the MQTT client is less active and sends one or two messages an hour, and if the keep alive is set to a low value (for example, 10 seconds), the network can be flooded with PINGREQ and PINGRES messages.
Alternatively, if the keep alive timer is set to a high value (for instance, one hour), and the client goes out of the network, the server will not know that the client has gone away for a long time. This can affect administrators who monitor the connected clients. In addition, the server keeps trying PUBLISH messages again. The keep alive timer setting ought to be chosen based on the message flow and amount of time that the client can be idle.
Retry message delivery
Consider a scenario when a message of QoS 1 is published from a client to a server, but the PUBACK is not received from the server. The MQTT client tries sending the message again after a specified time period. This time out is different from the keep alive. The retry time out is the maximum amount of time that the client waits for the server to send a response (or vice versa).
This retry happens until there is an error from the TCP/IP layer. That is, if there is any type of socket exception, retry processing ceases. In cases where QoS 2 message delivery is not used, there might be duplicate messages seen due to retry by the MQTT client or server. Therefore, when a retry happens, the message is marked with a duplicate flag. The application receiving the message knows if the received message is a duplicate.
It is possible that a duplicate message is received by an MQTT client while using QoS 1 due to MQTT retry. On some slow and fragile networks, one can observe a large number of duplicate messages due to the retry processing. If the retry time out is low, the network can be clogged with a large number of duplicates.
Last Will and Testament
When a client connects to the server, it can define a topic and a message that needs to be published automatically when it unexpectedly disconnects. This is called the Last Will and Testament (LWT).
If an unexpected disconnect happens wherein the client does not send a DISCONNECT command to the server, the publication is sent by the telemetry service when it detects that the connection to the client has broken without the client requesting a disconnect. The client might have experienced a loss in network connection, or an abrupt termination of the client program.
When a client connects to the server, the following “Will” parameters can be specified:
•The Will topic at which the Will message needs to be published.
•The Will message that needs to be published.
•The Will QoS of the message that will get published on the topic.
•The Will retain flag that signifies if the message needs to be retained.
When the client unexpectedly disconnects, the keep alive timer at the server side detects that the client has not sent any message or the keep alive PINGREQ. Therefore, the server immediately publishes the Will message on the Will topic specified by the client.
The LWT feature can be useful in some scenarios. For remote MQTT clients, this feature can be used to detect when the device goes out of the network. The LWT feature can be used to create notifications for an application that is monitoring the client activity.
Retained flag on messages
MQTT provides a feature for holding a message for a topic even after it is delivered to the connected subscribers. This is achieved through a retained flag on the publish message.
The publisher of the message sets this flag on the message while publishing. Here is an example flow to understand retained messages:
1. Client A connects to the server and subscribes to the a/b topic.
2. Client B connects to the server and publishes the message Hello with the retain flag set to a/b.
3. Client A receives the message without the retain flag set.
4. Client C connects to the server and subscribes to a/b.
5. Client C receives the message with the retain flag set.
Even if the server is restarted, the retained message will not be lost. It is also important to note that there is only one retained message that is held per topic. The retained publications are primarily used to maintain state information.
If a particular topic is used to publish a state message from a device, the messages can be retained messages. The advantage is that the new monitoring program that connects can subscribe to this topic, and gets to know the last published state messages from the device.
TCP/IP
When designing applications for mobile wireless networks, give consideration to TCP/IP. MQTT is just like any other protocol, such as HTTP or FTP, for the TCP/IP stack. TCP/IP is the communication protocol of the Internet. It defines how the client and server need to communicate with each other. All of the major operating systems on the server support TCP/IP.
The client devices also comes with the operating system that has a built-in capability of TCP/IP. Although MQTT adds only two bytes of additional header to a message, there is more data that flows in the network layers. The additional header data cannot be exactly computed on a per-MQTT message basis. The IP header, frame header, and other keep alive flows at the network layer add to additional data flows while using MQTT clients.
TCP/IP port 1883 is reserved with for use with MQTT. TCP/IP port 8883 is also registered for using MQTT over Secure Sockets Layer (SSL).
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.147.66.178