Hello MQTT Version 5.0!

On August 9, 2017, the OASIS MQTT Technical Committee announced that MQTT Version 5.0 is now available for public review and comment until September 8th. And the release of the next version of Message Queue Telemetry Transport (MQTT) is expected by end of this year.

Time to have a closer look at what’s new.

MQTT v5.0 is the successor of MQTT 3.1.1 (find out why it’s not MQTT 4 ).

Most likely important: MQTT v5.0 is not backward compatible (like v3.1.1). Obviously too many new things are introduced so existing implementations have to be revisited.

According to the specification, MQTT v5.0 adds a significant number of new features to MQTT while keeping much of the core in place.

The major functional objectives are:

• Enhancements for scalability and large scale systems in respect to setups with 1000s and millions of devices.
• Improved error reporting (Reason Code & Reason String)
• Formalize common patterns including capability discovery and request response
• Extensibility mechanisms including user properties, payload format and content type
• Performance improvements and improved support for small clients

My MQTT v5.0 Highlights

User Properties

User properties (UTF-8 encoded Strings) can be part of most MQTT packets: PUBLISH and CONNECT, and to all packets with a Reason Code.
User properties on PUBLISH are forwarded with the message and are defined by the client applications. They are forwarded by the server to the receiver of the message.
User properties on CONNECT and ACKs are defined by the sender, are unique to the sender implementation, and are not defined by MQTT.
An unlimited number of user properties can be added!

Payload Format Indicator & Content Type 

Another identifier/value pair is available for use when sending a PUBLISH message. This is the Payload Format indicator. If present and set to 1, this indicates that the PUBLISH payload is UTF-8 encoded data. If set to 0, or if the indicator is not present, then the payload is an unspecified byte format, exactly as with MQTT v3.1.1.

• Optional part of PUBLISH message
• Reason Codes for ACK messages available if payload format is invalid
• Receiver may validate the format indicator
• „Content Type“ optional header can carry a MIME type
• Payload Format Indicator“ can be binary or UTF-8

Shared Subscriptions

With support of shared subscriptions, Client Load Balancing is now included in MQTT. The message load of a single topic is distributed amongst all subscribers (this was already supported by HiveMQ for MQTT 3.1 & MQTT 3.1.1).

A Shared Subscription is identified using a special style of Topic Filter.

The format of this filter is:

$share/{ShareName}/{filter} 

$share – is a literal string that marks the Topic Filter as being a Shared Subscription Topic
{ShareName} – is a character string that does not include “/”, “+” or “#”
{filter} – is the remainder of the string has the same syntax and semantics as a Topic Filter in a non-shared subscription.

Reason Codes & Reason Strings

With MQTT v5.0, Reason Codes and Reason Strings are introduced at application level. Now clients are allowed to figure out why they were disconnected.
Almost all control packets like CONNACK, PUBACK, PUBREC, PUBREL, UNSUBACK, DISCONNECT, SUBACK and AUTH can carry Reason Codes in their variable header.
But: Reason Codes are optional, the server/broker can still decide to just disconnect or reject the clients like in MQTT v3.1.1, e.g. for security reasons.
In addition to the Reason Code a Reason String can be associated with the response to provide a more human readable message.

Session management: Session Expiry & Message Expiry

Support of offline/persistent sessions is a major feature of MQTT to handle connection interrupts. The specification of MQTT v3.1.1 does not define a mechanism to control the expiry of a persistent session. Thus it will never expire and it will never be deleted (aside from some brokers like HiveMQ which already supports session expiry with MQTT v3.1.1).
In MQTT v3.1.1 and earlier, a client can control how the server handles the client session (a session means the subscriptions of a client and any queued messages) via the „clean session“ flag. If set to 1, the server would delete any existing session for that client and would not persist the session after disconnecting. If set to 0, the server would restore any existing session for a client when it reconnected, and persist the session when the client disconnected.

“Clean Session” is now split into „Clean Start“. If Clean Start is set to 1 it indicates that the session should start without using an existing session (otherwise session information is kept), and a Session Expiry interval which says how long to retain the session after a disconnect.

• Session Expiry is an optional part of the CONNECT message (3.1.2.11.2 Session Expiry Interval) and the DISCONNECT control packet  (3.14.2.2.2 Session Expiry Interval). If the the Session Expiry Interval is absent in the DISCONNECT message, the Session Expiry Interval in the CONNECT packet is used. It defines the session expiry Interval in seconds. If set the Broker expires the session after the given interval as soon as the client disconnects. Setting of Clean Start to 1 and Session Expiry Interval to 0 is equivalent in MQTT v3.1.1 of setting Clean Session to 1.
• Message expiry is an optional part of the PUBLISH control packet (3.3.2.3.3 Publication Expiry Interval). The Publish Expiry Interval applies to online and queued messages and is the lifetime of the publication in seconds.

Another issue to be addressed under this banner is called Simplified State Management.
This has at least two major advantages.

1. As an application, I only want my session state to be discarded when I’ve completed all my work, not when my network connection fails. This was inconveniently hard in all previous versions of MQTT – not in version 5.
2. The ability for session state to have an expiry time. If the client does not connect within a certain amount of time, the session state can be deleted by the server. This obviates the need for a client to reconnect just to clean up session state.

Repeated topics when publishing

When publishing data to a single topic, a new feature will help reduce bandwidth use. A client or server can set the topic in a PUBLISH message to be a zero length string. This tells the client/server being published to, to use the previous topic instead. This goes some way to reducing the current overhead associated with publishing – a shame it isn’t quite as good as the registered topics available in MQTT-SN.

Publication Expiry interval

This is an identifier/value pair for use when publishing. If present, this value is a 4 byte integer which gives the number of seconds for which the server will attempt to deliver this message to a subscriber. This means that an offline client with messages being queued may not receive all of the messages when it reconnects, due to some of them expiring. Interestingly, when the server does deliver a message that had a Publication Expiry set, it sets the Publication Expiry on the outgoing message to the client but with the amount of time that there is left until the message expires. This means that the true time to expiry will propagate through bridges or similar.

Publish Reason Codes

The PUBACK and PUBREC packets have a new entry in their variable header which is the Publish Reason Code. This can be used to tell the client a message has been:

• refused for various reasons
• accepted, or
• accepted with no matching subscribers.

For the PUBREC packet, if the message is refused or accepted with no matching subscribers then there is no expectation for the PUBREL/PUBCOMP messages to be sent for that message.
The PUBCOMP packet also has a similar entry which has the same set of Reason Codes and an additional one for the case when a message had expired.
This is for the case when a client reconnects with clean start set to 0 and it has a QoS 2 message part way through its handshake, but the server has already expired the message.
There is still no way to tell a client that its QoS 0 message was refused but for a good reason: QoS 0 Messages do not get an acknowledgement -> no piggybacking of Reason Codes!

Disconnect notification

In MQTT v3.1.1 and before, only the client sends a DISCONNECT packet.
In MQTT v5.0, either the client or the server can send DISCONNECT, and it is used to indicate a reason for disconnection.
Examples for the new disconnect reason codes:

Client implementations

Currently there is no ready-to-use MQTT v5.0 client implementation available. I think a production-ready version will not be available until mid-2018.

Anyway as soon as client implementations are available I am looking forward to adding support for the new Version 5.0 features into MQTT.fx (www.mqttfx.org) in time.

Links

OASIS Message Queuing Telemetry Transport (MQTT) TC

https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=mqtt

Latest specification version

http://docs.oasis-open.org/mqtt/mqtt/v5.0/

MQTT.fx

http://www.mqttfx.org