MQTT – The defacto IoT protocol – MQTT series #1

Welcome to the first of a series of MQTT-related articles. In this series, we’ll cover different aspects of the MQTT protocol, from its low-level protocol specification to the deployment of an MQTT broker, from simple to more advanced scenarios.

If you’re starting with MQTT or want to learn more about this protocol, you have come to the right place.

In this first article, we’ll cover the protocol basics and why it is so relevant for IoT communication.

Why is MQTT the IOT protocol

The Internet of Things (IoT) has revolutionized the way we interact with devices and the way devices communicate with the Internet. From smart homes to industrial automation and smart vehicles, IoT is everywhere.

Most of these IoT devices (think smartphones, smart sensors, embedded devices) have a low computational capacity, have slow and unreliable network connections and are often limited in energy consumption, as they can be battery-powered.
The communication between these devices needs to be efficient, and reliable while being secure. This is where MQTT (Message Queuing Telemetry Transport) comes into play.

So what is MQTT?

MQTT is a lightweight messaging protocol designed for constrained devices and low-bandwidth, high-latency, or unreliable networks. It uses a publish/subscribe model where devices (clients) can publish and subscribe to topics. A central server (a.k.a. the broker) dispatches all messages between the senders and the correct receivers.

Wait, isn’t HTTP enough?

You may be wondering “isn’t there a simple enough protocol, widely used that can be used for IoT device communication, called HTTP?”. If you are, that’s good – it’s a natural reaction.
HTTP is indeed a simple protocol, with a large support in most (if not all) stacks and operating systems.
However, MQTT still presents a few advantages, that depending on the use case and scenario may make a huge difference:

• It allows bi-directional communication, as a device can act both as a subscriber, consuming data from one or more topics; and as a publisher, sending data to one or more topics. Achieving a similar flow with HTTP would involve more complex setups with long polling, server-sent events or web sockets
• the overhead for each package sent is lower than HTTP (depending on the Quality of Service – the higher the QoS the higher the overhead). This is relevant for metered connections where data plans are expensive, especially when multiplied by hundreds or thousands of devices
• MQTT is faster than HTTP as several benchmarks have shown (e.g. HTTP vs MQTT performance tests (flespi.com))
• MQTT being a lighter protocol, requires less computational power, and therefore less energy, making it more suitable for power-restrained devices
• MQTT has built-in reliability features such as the usage of QoS2, and retained messages of the Last Will and Testament, which help deal with unreliable networks or devices that could be powered off without notice. These can help reduce data loss.
• it can be served via WebSockets, for direct browser usage
  
  Having said all this, HTTP is a simpler protocol to implement and it may make sense for some IoT use cases. Since HTTP/2, the issue with a new connection per request is reduced, as a single connection can now multiplex multiple requests.

So how is MQTT this good for IoT?

MQTT is designed to be lightweight and efficient
IoT devices often have limited processing power and memory. MQTT’s small footprint makes it ideal for such devices. The protocol minimizes network bandwidth, which is crucial for battery-operated devices.

Quality of Service
MQTT offers three levels of Quality of Service (QoS): “At most once”, “At least once”, and “Exactly once”. This allows developers to choose the right level of message delivery assurance depending on their needs. The different QoS are explained in more detail, below.

Last Will and Testament
MQTT has a feature called “Last Will and Testament”. If a device disconnects ungracefully (for instance a device battery is depleted), the broker can automatically publish a predefined message to a predefined topic. This allows other devices to know about the disconnection and act accordingly.

Easy to Implement
There are MQTT libraries available for almost every programming language, making it easy for developers to implement MQTT in their applications.

Scalability
MQTT can support a large number of devices. This makes it suitable for IoT applications where thousands or even millions of devices need to communicate.

Security
MQTT supports secure connections using TLS/SSL. It also supports username/password authentication. This allows developers to build secure IoT applications.

Connecting to a broker

Before diving into the different Quality of Service levels, it’s important to understand that all MQTT communication needs to be initialized by the MQTT client. This initialization happens as follows:
1. The client sends a CONNECT packet to the broker.
2. The broker responds with a CONNACK packet.
3. The client can now publish or subscribe to topics (more detail below)
4. To close the connection (gracefully), the client sends a DISCONNECT packet.

Quality of Service (QoS)

Once a connection has been established, the client can subscribe to one or more topics, or publish data to one or more topics. The CONNECT packet differs, depending on the selected QoS level.

MQTT has three levels of Quality of Service (QoS):

QoS 0 (At most once a.k.a. fire-and-forget)

This is the lowest level where the message is delivered at most once. It could be lost if the client is disconnected. This is also the lightest form of MQTT communication with the simpler handshake: none. For QoS 0, the payload of the CONNECT packet contains:

• Client Identifier: A string that identifies the client to the broker.
• Will Topic and Will Message (optional): If the client disconnects ungracefully, the broker will publish the Will Message to the Will Topic.
• Username and Password (optional): Credentials for connecting to the broker.

QoS 1 (At least once)

The message is assured to arrive but duplicates can occur. The application needs to deal with messages in an idempotent manner, to ensure duplication does not corrupt the application state (e.g. by validating if a certain message hasn’t already been processed)

For QoS 1, the payload of the CONNECT packet contains the same fields as QoS 0. In addition, it includes:

• Packet Identifier: A unique identifier for the PUBLISH packet. The broker will send a PUBACK packet with the same Packet Identifier to acknowledge receipt.

Note that if a network split happens during the sending of the PUBACK, the broker will consider the message as not received and will send it again, causing a duplicate.

QoS 2 (Exactly once)

This is the highest level, where message delivery is assured to arrive exactly once, due to a more network-intensive handshake.

For QoS 2, the payload of the CONNECT packet contains the same fields as QoS 1. However, the broker will send a PUBREC packet instead of a PUBACK packet. The client then sends a PUBREL packet, and finally, the broker sends a PUBCOMP packet to complete the four-way handshake.
This is not only slower (as it involves more roundtrips) but it also involves more intensive data transfer. It is, however, useful in some cases where guaranteed delivery of a message exactly once is needed.

Retained Messages

MQTT brokers can retain a message for a topic, even after the message has been delivered to all current subscribers. If a new subscription is made, the last retained message for that topic is sent to the subscriber. This feature is useful for late joiners to receive the most recent value immediately.

An example use case would be a sensor that publishes temperature readings every minute. A set of other controlling devices connect and subscribe to these readings to make adjustments to the HVAC. If a new device subscribes to the temperature topic immediately after the temperature has been published, it will miss the message and will need to wait for another minute before a new temperature arrives.

With retained messages, the message is retained in the broker and sent to new subscribers. The sensor can then publish a new temperature, which will replace the previous retained temperature.

It’s the publisher’s choice to set the retained flag for a message and only one message can be retained in a topic.

Last Will and Testament

The Last Will and Testament feature allows a broker to publish a message automatically when a client disconnects ungracefully. The client sets up the “will” when it connects to the broker. If the client disconnects without sending a DISCONNECT packet, the broker publishes the “will” message to the specified topic.

Shared Sessions

Shared sessions allow multiple clients to consume data from the same topic, in a worker pattern fashion. This is useful in scenarios where multiple instances of a client are running for redundancy or load balancing. The broker ensures that each message is delivered to only one of the clients in the shared session.
Note that fan-out approaches can still be implemented, using different shared session IDs.

MQTT Versions

Most of this article refers to MQTT v5, but some brokers (e.g. RabbitMQ) only support the previous version – 3.1.1 (there’s no V4).
While the protocol core remains the same between v3.1.1 and v5, V5 has some interesting additions:

1. Extended Functionality of Topic Filters: MQTT v5 extends the functionality of topic filters, allowing the use of attribute matching and more complex logical operators for subscriptions.

2. Explicit Message Delivery Confirmation: MQTT v5 introduces an explicit message delivery confirmation mechanism, allowing the client to specify whether a confirmation is needed when publishing a message.

3. Error Codes and Reason Strings: MQTT v5 introduces the concept of error codes and reason strings for more detailed error descriptions. This makes it easier to identify and handle errors, improving system reliability and debugging.

4. Large-Scale Data Transmission: MQTT v5 can better handle large-scale data transmission, improving communication efficiency and performance.

5. Batch Publish and Message Prefetch: MQTT v5 supports features like Batch Publish and Message Prefetch, which can significantly reduce the overhead of MQTT communication.

Wrap-up

MQTT is a powerful protocol designed for IoT applications. Its features like QoS, retained messages, last will and testament, and shared sessions make it a great choice for real-time, reliable communication between low-powered devices with unreliable connectivity. With the availability of free MQTT brokers, it’s easier than ever to get started with MQTT.
Here are a few free MQTT brokers to get started with IoT development:

1. HiveMQ: A public MQTT broker that supports both TCP and WebSocket connections – Free for the community edition and paid for the commercial edition which supports clustering.

2. MQTTHQ: A free, high availability, public MQTT broker that supports both TCP and WebSocket connections.

3. Mosquitto: An open-source MQTT broker that’s part of the Eclipse Foundation.

4. Mosca: A multi-transport MQTT broker for node.js.

5. RabbitMQ: An open-source multi-protocol messaging broker with support for MQTT (only mqtt v3.1.1 support up to RabbitMQ 3.12)

Stay tuned for the next articles, where we’ll have hands-on samples.