In a mosquitto cluster, clients can subscribe to every node, and can also publish to every other node. The cluster will make sure that published messages are forwarded as needed.
The cluster is full decentralized, autonomy system without any leader or key node, to make the system with a high availablity.
E.g., each node has a fault rate with 1%, then a decentralized cluster with N nodes has a service availability which is 1-1%^N.
Install mosquitto on all of the nodes and write the addresses into mosquitto.conf, e.g.,
node_name node1
node_address 192.168.1.1:1883
node_name node2
node_address 192.168.1.2:1883
Then config the loadbalancer, take above adresses as real server address. It is strongly recommend to terminate TLS on the loadbalancer, and use plain TCP inside the cluster.
> git clone https://github.com/hui6075/mosquitto-cluster.git
> cd mosquitto-cluster && vi config.mk
# WITH_BRIDGE:=yes
WITH_CLUSTER:=yes
> make && make install
In order to avoid infinite PUB/SUB forwarding, the publishes from other brokers will only send to client, the subscription and unsubscription from other brokers will not be forward.
Save local clients' subscriptions and unsubscriptions by a reference counter, only forward the subscription which is fresh for local broker, and only forward the unsubscription which is no longger been subscribed by local client.
Some private messages was introduced in order to support cluster session and retain message, i.e.,
PRIVATE SUBSCRIBE
Fix header|PacketID|Topic Filter|QoS|ClientID|SubID
PRIVATE RETAIN
Fix header|Topic|QoS|[PacketID]|ClientID|SubID|OriginalRecvTime|Payload
SESSION REQ
Fix header|ClientID|Clean Session
SESSION RESP
Fix header|ClientID|PacketID|NRsubs|sub1(topic|qos)|...|subN|NRpubs|pub1(topic|state|dir|dup|qos|mid|payload)|...|pubN|
For cluster session, SESSION REQ would be broadcast for each client CONNECT which has no previous context found in local broker, remote broker which has this client's context will kick-off this client and if clean session set to false, the remote broker would return this client's session include subscription, incomplete publishes with QoS=0/1 inside SESSION RESP. This feature could be disable in mosquitto.conf, in order to endure instantaneous large number of concurrent connections at system startup phase.
For retain message, PRIVATE SUBSCRIBE would be broadcast for each client subscription that
is fresh for local broker, and if there exists a retain message, remote broker would
return the retain message inside PRIVATE RETAIN. Client will receive the most recent retain message inside the cluster after a frozen window which can be configure inside mosquitto.conf.
For QoS, all the publishes set with it's original QoS inside the cluster, and process with QoS=0, to saves the inside traffic. The actual QoS send to client decide by the broker which client connected with.
All validation, utf-8, ACL checking has disabled for the messages inside the cluster, in order to improve cluster efficiency.
Once CONNECT success with other broker, local broker will send all the local clients' subscription to remote broker, to avoid subscription loss while some brokers down and up.
The cluster current also support node/subscription recover, crash detection.
Mosquitto cluster is a distributed implementation of Mosquitto brokers with following goals:
Scalable in a horizontal fashion, which means you can expand cluster capability by increase number of brokers.
Provide continuous service under single point of failure.
Take on the role of one logical MQTT broker for millions of MQTT clients.
Pic1. Mosquitto Cluster Overview
While receive client subscriptin: Notification other brokers that the topic has been subscribed by me,While receive client publish: route the message to the correct brokers which subscribed this topic.
Pic2. Mosquitto Cluster Private Messages
There's only one logic channel between 2 brokers, in order to reuse this channel for multi clients, some private messages has been introduced, include client id, subscription id, raw timestamp and etc., which help the broker to make a correct route decision.
Pic3. Mosquitto Cluster Internal Message Flow
Forwarding rules:Forward local client subscriptions to the broker which is a fresh guy for the cluster or just recovered from a fault.
DO NOT forward internal messages to any broker.
Only broadcast local fresh subscription to other brokers.
Broadcast unsubscription until a topic is no longer subscribed by any local client.
Session messages broadcasting can be disable by configuration.
DO NOT forward PUB/SUBs under $SYS.
Using krylovsk/mqtt-benchmark as the benchmark tool to give a simply testing for the cluster. The bandwidth stop increase with 3 or more brokers due to client machine's bottleneck.
Pic4. Mean bandwidth of Mosquitto cluster
n=10k means 10000 messages to send per client, c=100 means 100 clients to start. MsgSize=1000bytes, QoS=2.A more detailed test report is available under: https://github.com/hui6075/mosquitto/tree/develop/benchmark
Pic5. Mosquitto code analyze(benchmark/mosquitto_code_analyze.jpg, w/o cluster)
- MQTT v5.0 Chinese Translation: https://github.com/hui6075/mqtt_v5
- MQTT broker forward latency measurement tool: https://github.com/hui6075/mqtt-bm-latency