Tags: MQTT
Getting started with Mosquitto is deceptively easy. We show you how to avoid the pitfalls.
The Docker Hub page for the Mosquitto server makes it look trivially easy to deploy the server. You just run the server, and immediately you can set up listeners and post some messages and it's all so easy. But, that's not what happens. There's a configuration file, data, and logs directories to mount into the container, and there is the matter of permissions to access the server anonymously. Once those are properly squared away, it is very easy to install Mosquitto and kick the tires.
Mosquitto is a message broker for the MQTT protocol.
https://mosquitto.org/ It is described as being light weight, and suitable for use on a range of devices from embeddable single board computers (e.g. Raspberry Pi) up to large servers. The Mosquitto project also includes a C library for MQTT, as well as the
mosquitto_pub and mosquitto_sub clients.
MQTT is a protocol for distributing messages using the publish-subscribe (pubsub) model. In MQTT parlance, the broker is the service that distributes messages. MQTT is widely used and the protocol is so lean that it's easy to implement an MQTT client on tiny embedded microcontrollers. A typical usage is to attach sensors to microcontrollers like the ESP32, then deliver data over the Internet to an MQTT service. The protocol is documented at:
https://mqtt.org/
Fortunately we are normally not required to implement MQTT ourselves, since there are many existing client libraries and broker services. An example is the Mosquitto project we're using in this tutorial.
We'll look at deploying Mosquitto using Docker, and run a few commands. Along the way we'll discuss a couple common deployment failures. We're not going to dive into using MQTT or Mosquitto for anything significant and instead stay with deployment and light configuration.
Mosquitto deployment on a Linux host
Before getting to Docker, let's quickly run over how to deploy Mosquitto on a Linux host.
On a Debian-based system like Raspbian (Raspberry Pi OS), Ubuntu, Debian, etc, these are useful commands:
# Install server and client packages
sudo apt install mosquitto mosquitto-clients
# Start the server
sudo systemctl enable mosquitto
# Check its status
sudo systemctl status mosquitto
# Restart the server
sudo service mosquitto restart
This is easy to do and this is the traditional way of running servers on a Linux system. For other Linux distros, you'll use similar commands (e.g. yum) and perhaps with different package names.
In the Docker age this is not recommended.
The rationale is that Docker containers limit the potential destruction if someone breaks in through a bug in the Mosquitto service. Any intruder won't be able to damage anything outside the container. Since the container is easily destroyed and recreated if there are problems, recovery is straight-forward.
The Docker container for Mosquitto contains the exact same server and client programs that we discuss in the rest of this article. That means the commands and configuration files we discuss are directly applicable if you prefer to install Mosquitto on the host system rather than inside Docker.
A simple Docker Compose file for Mosquitto
On the Docker Hub page for the Mosquitto server, we're given this docker command:
$ docker run -it -p 1883:1883 -p 9001:9001 \
-v mosquitto.conf:/mosquitto/config/mosquitto.conf \
eclipse-mosquitto
This serves to acquaint us with running the Mosquitto container. It mounts a mosquitto.conf file into the container, and it exposes a pair of ports. Looks simple, doesn't it?
For the configuration file, we're suggested to use this:
persistence true
persistence_location /mosquitto/data/
log_dest file /mosquitto/log/mosquitto.log
An issue with this is the two directories, /mosquitto/data/ and /mosquitto/log/ are not persisted outside the container. There's another issue that we'll talk about later, but this is a good starting point for configuring Mosquitto.
When using Docker, I prefer Docker Compose because of its flexibility. The website https://www.composerize.com/ can help you with converting a docker run command into a docker-compose.yml. After a bit of tweaking, we get this:
services:
mosquitto:
image: eclipse-mosquitto
ports:
- 1883:1883
- 9001:9001
volumes:
- ./mosquitto.conf:/mosquitto/config/mosquitto.conf
- ./data:/mosquitto/data/
- ./log:/mosquitto/log/
This is the same as the docker command, but it also specifies volume mounts for the data and log directories.
To launch this you run:
$ docker compose up
Note that I have used docker compose rather than docker-compose. The old docker-compose command has been replaced by the new docker compose command. Let's wish docker-compose a peaceful retirement. The implementation is complete enough that for most purposes you can replace the - with a space character for the same effect.
The up command will download the eclipse-mosquitto image if required, set up containers, etc, then launches the stack described in docker-compose.yml. By default the service will be launched in the foreground so you can see any logging output from the server.
$ docker compose up -d
Run this way, it runs the services detached from the terminal, and they will run in the background.
The up -d command is almost suitable for a long-term background server. One issue is whether the service will automatically restart if (when) it crashes. In the cloud computing era we're admonished to always plan for failure of system components. As it stands, if the Mosquitto server crashes it will not restart automatically.
To fix this, make the following change to docker-compose.yml:
services:
mosquitto:
image: eclipse-mosquitto
restart: unless-stopped # Add this
...
The restart tag controls the conditions under which to restart a service. The unless-stopped will ensure the service automatically restarts unless you explicitly stop it.
If you forget to create the mosquitto.conf you'll get this error:
Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/home/david/Projects/nodejs/mqtt/mosquitto.conf" to rootfs at "/mosquitto/config/mosquitto.conf": mount /home/david/Projects/nodejs/mqtt/mosquitto.conf:/mosquitto/config/mosquitto.conf (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
I mention this because I made this mistake several times and maybe others will do the same. Simply create the file, however Docker will have created a directory that you must remove first.
Stopping, restarting, and updating the container
Your deployed service can be easily managed using the docker compose. To do so your terminal session must be in the containing the docker-compose.yml file. The commands for this are:
# Start the service
$ docker compose up -d
# Shut down the service
$ docker compose down
# Restart the service
$ docker compose restart
# Update the service
$ docker compose stop
$ docker compose pull
$ docker compose up -d
The last will update the service if the container image on Docker Hub has been updated.
Running Mosquitto commands inside the container
If you run docker compose ps or docker ps -a you'll see that the container has started.
To see it in action let's listen to an MQTT topic, look at the log file, and send some messages.
In one terminal window you can run this command:
$ sudo tail -f log/mosquitto.log
[sudo] password for david:
1665640941: Saving in-memory database to /mosquitto/data//mosquitto.db.
1665641912: mosquitto version 2.0.15 terminating
1665641912: Saving in-memory database to /mosquitto/data//mosquitto.db.
1665642425: mosquitto version 2.0.15 starting
1665642425: Config loaded from /mosquitto/config/mosquitto.conf.
1665642425: Opening ipv4 listen socket on port 1883.
1665642425: Opening ipv6 listen socket on port 1883.
1665642425: mosquitto version 2.0.15 running
1665642425: New connection from 172.29.0.1:37076 on port 1883.
1665642425: Client auto-7F81B465-FE74-A7CC-7A16-854BC3AA23E0 disconnected, not authorised.
This watches the log. It must be run with sudo because this directory is protected.
Next, if we try to enter the container to run commands we'll get this message:
$ docker exec -it mqtt-mosquitto-1 bash
OCI runtime exec failed: exec failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown
This is because the eclipse-mosquitto container is built on Alpine Linux which doesn't have bash. Replace bash with sh and you'll get a command prompt inside the container.
This command should start listening to messages on a specific topic:
$ mosquitto_sub -v -t test/message
Connection error: Connection Refused: not authorised.
But, as you can see, there is a lack of authorization. It's good to know that Mosquitto has an authorization system. In production use it's necessary that the MQTT service is not wide open to the public.
A quick fix is to change the configuration file to this:
allow_anonymous true
persistence true
persistence_location /mosquitto/data/
log_dest file /mosquitto/log/mosquitto.log
listener 1883
What we're doing is allowing anyone in the world to use this MQTT broker. Clearly this is not something to do in a production deployment. It lets us proceed with this quick tutorial.
We must restart the service (docker compose restart) and then rerun the command. A simpler way to run commands inside the container is this:
$ docker exec -it mqtt-mosquitto-1 \
mosquitto_sub -v -t test/message
This sets up a subscriber to the test/message topic. Any messages posted to this topic will be printed.
To send a message we can do this:
$ docker exec -it mqtt-mosquitto-1 \
mosquitto_pub -t test/message -m 'Hello World!'
This publishes a message to the named topic. In the other command window, the subscriber process will print this:
test/message Hello World!
Message delivered, hello to the world.
Using the Mosquitto client programs outside the Docker container
We've used the two Mosquitto client programs, mosquitto_sub and mosquitto_pub, running them inside the container. The docker exec command lets us enter the running container, and execute commands. While useful for a simple demonstration, this is not at all how a production system would operate. Instead, our applications would connect over the network to the service. To mimic that, let's run the Mosquitto clients outside the container.
Often we can install the clients on our laptop. For example, my laptop runs Ubuntu, and I ran this command:
$ sudo apt install mosquitto-clients
For macOS there are packages in the MacPorts or HomeBrew repositories, and on Windows it is recommended to look at the Chocolatey repository.
This means I can run Mosquitto commands from the laptop, such as these:
mosquitto_sub -h localhost -p 1883 -v -t test/message
mosquitto_pub -h localhost -p 1883 -t test/message -m 'Hello World!'
The -h option specifies a host name or IP address for the Mosquitto server. The -p option specifies the port number. You might have deployed Mosquitto to a distant server, and this is how you get there.
Implementing authentication for Mosquitto
The mosquitto server supports several kinds of authentication. As we said earlier, this is necessary for a production deployment. The configuration file shown above allows anonymous access, which throws any sense of security out the window.
What we'll do is set up a simple password file.
In mosquitto.conf add this line:
password_file /mosquitto/etc/passwd
This says to use the named file to store username/password pairs. The format is username:password, and the contents are managed using the mosquitto_passwd program.
To support this we need to change docker-compose.yml to mount a new directory:
services:
mosquitto:
image: eclipse-mosquitto
...
volumes:
...
- ./etc:/mosquitto/etc/
We mount a directory into the container, and therefore must create that directory:
$ mkdir etc
$ touch etc/passwd
Next, you can start or restart the Mosquitto container. There are no users defined in the passwd file, so let's create one.
$ docker exec -it mqtt-mosquitto-1 mosquitto_passwd /mosquitto/etc/passwd henry
Password:
Reenter password:
This works similarly to the Unix/Linux/etc passwd command, of course.
It is necessary at this point to restart the Mosquitto container. In testing, it seemed that the server would not recognize new users or changed passwords until it is restarted.
At this point you can subscribe to a topic by adding the user name and password:
$ mosquitto_sub -u henry --pw passw0rd -h localhost -p 1883 -v -t test/message
The new options are -u, specifying the user name, and --pw, specifying the password.
To send messages requires a similar change:
$ mosquitto_pub -u henry -P passw0rd -h localhost -p 1883 -t test/message -m 'Hello World!'
The -P option is an alternate way of specifying --pw. As before, running this command will cause the message to be printed by mosquitto_sub.
Summary
The Mosquitto server supports many more capabilities than we've touched on in this article. For example, we can authenticate users using SSL certificates, and each user can have an ACL limiting their capabilities. Another important feature is the ability to persist data.
Setting up the Mosquitto server in Docker is very simple. Doing so simply requires a few configuration files and settings.
