Tags: Docker
Podman, the Docker alternative for running Docker containers, runs great on MacOS. But, it is a little tricky to get going, such as mounting directories into a container.
I'm developing a server using Node.js/TypeScript which persists its data to MongoDB. The simplest path is to run MongoDB in Docker, and to run Docker on my MacOS laptop I prefer to run Docker inside of Multipass. For setup of Multipass on MacOS to run Docker, see Replacing Docker Desktop with Multipass, to avoid Docker Desktop fees
However, for this project I want to have better control over things like memory allotment. The Multipass approach wasn't cutting it for me, and while I have the official Docker Desktop installed it doesn't work very well. That meant turning to Podman which I'd been avoiding because it doesn't support docker compose.
Supposedly running Podman instead of Docker is simple. You substitute the word podman in place of docker in commands, and you're good to go. Unfortunately it wasn't that simple.
To take a step back slightly -
Podman is an open source project for building/executing Docker containers, that is "Kubernetes Ready". It offers some interesting features, such as root-less execution. The project was started by Redhat.
Installing Podman on MacOS
The official instructions say to use Homebrew to install Podman. Myself, I'm a MacPorts man, and prefer using it for installing open source tools. Fortunately the MacPorts project also has a Podman package.
Installation:
#### Homebrew
$ brew install podman
#### MacPorts
$ sudo port install podman
Either method results in the same software.
To initialize Podman, run:
$ podman machine init
$ podman machine start
Starting machine "podman-machine-default"
Waiting for VM ...
Mounting volume... /Users:/Users
Mounting volume... /private:/private
Mounting volume... /var/folders:/var/folders
This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command:
podman machine set --rootful
API forwarding listening on: /var/run/docker.sock
Docker API clients default to this address. You do not need to set DOCKER_HOST.
Machine "podman-machine-default" started successfully
This initializes a QEMU-based virtual machine within which a small Linux system is executing where a Podman service is available to run Docker containers. The machine is named podman-machine-default and notice that it mounts a few directories into the machine.
The machine can be tested:
$ podman run -it docker.io/library/busybox
Trying to pull docker.io/library/busybox:latest...
Getting image source signatures
Copying blob sha256:3f4d90098f5b5a6f6a76e9d217da85aa39b2081e30fa1f7d287138d6e7bf0ad7
Copying config sha256:a416a98b71e224a31ee99cff8e16063554498227d2b696152a9c3e0aa65e5824
Writing manifest to image destination
Storing signatures
/ #
This drops you into a shell in a Busybox container.
So.. it should be easy to run MongoDB, right?
Successfully running a MongoDB container on MacOS using Podman
For any database, like MongoDB, it is 100% necessary to persist the data directory on the host machine. Otherwise your database disappears in a puff of smoke as soon as you recreate the container.
This made running MongoDB on Podman surprisingly difficult.
But, first, there's the issue of docker compose support. The Compose specification is a relatively recent development of the Docker community. As a result the old docker-compose command is deprecated, and the official new way to run Compose files is with docker compose. Notice the lack of a hyphen.
Unfortunately the Podman community has not caught up with this. They suggest using a docker-compose-compatible tool with Podman. But, I am in disagreement and instead converted my Compose file into a podman run command.
podman run \
--restart always \
--expose 27017 \
--publish 27017:27017 \
--env MONGO_INITDB_ROOT_USERNAME=root \
--env MONGO_INITDB_ROOT_PASSWORD=example \
--volume /Users/david/Project/db/data:/data:rw \
--detach \
mongo
This was the final result after going through issues we're about to discuss.
No such file or directory
My first attempt at mounting the data directory used a different --volume option:
--volume /Volumes/Project/db/data/db:/data/db:rw
My MacOS laptop has two drives in it, with the second drive containing several partitions. It's not important that I have it set up this way, but the key here is that the host machine mount point was not within /Users.
With that --volume option, I got this error:
Error: statfs /Volumes/Project/db/data/db: no such file or directory
I very carefully compared the path to the --volume command and to host system directories. Everything matched up correctly. This took many 10s of minutes until a
Stackoverflow posting gave an answer.
Read back to the podman machine start command and you see a few lines talking about mounting directories.
Mounting volume... /Users:/Users
Mounting volume... /private:/private
Mounting volume... /var/folders:/var/folders
For some reason, this limits the directory hierarchies from which directories can be mounted into a container running under Podman. But, I did not know that at the time.
Specifically, it seems that with Podman, the --volume option is looking for directories inside the virtual machine where the Podman containers actually execute. This is different from how Docker Desktop runs where there is a seamless mounting of host directories into the container.
The solution is, when running podman machine init, to mount host directories into the machine. Those host directories can then be mounted into containers.
$ podman machine stop
$ podman machine rm podman-machine-default
$ podman machine init \
--volume /Users/david:/Users/david \
--volume /Volumes/Project/:/Volumes/Project
$ podman machine start
The first two commands get one back to a clean slate, recreating the Podman machine from scratch. The --volume option is not available with the podman machine start command, only with podman machine init.
You can inspect the directories mounted into the Podman virtual machine like so:
$ podman machine ssh
Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
Fedora CoreOS 38.20231002.2.2
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos
[root@localhost ~]# ls /
Users Volumes bin boot dev etc
home lib ...
Explore the directory further and you'll see that it's the host machine mounted into the Podman virtual machine. The --volume mount on the podman run command now works. But, with MongoDB there is another problem.
MongoDB: chown: changing ownership of '/data/db': Operation not permitted
The podman run command for running MongoDB got closer to running, except that podman ps showed that the container was not running, and that it had stayed up for less than one second.
Running podman logs _container name_ showed a zillion copies of this error:
chown: changing ownership of '/data/db': Operation not permitted
chown: changing ownership of '/data/db': Operation not permitted
chown: changing ownership of '/data/db': Operation not permitted
Thinking this had to do with directory permissions, I ran chmod 777 on the host directory. No change.
Another
Stackoverflow posting gave a solution. Instead of mounting the volume to
/data/db, mount it to /data instead.
Hence, instead of this option:
--volume /Volumes/Project/db/data/db:/data/db:rw
The solution is to use:
--volume /Volumes/Project/db/data:/data:rw
Or, specifically:
--volume /Users/david/Project/db/data:/data:rw
This way, MongoDB is not executing chown on the mount-point.
One way to verify MongoDB is running is:
$ mongosh --host localhost:27017 \
--username root \
--password example
While MongoDB is now running, and we can load it up with data, the host machine directory does not have the database.
$ ls ~/Project/db/data
configdb db
$ ls ~/Project/db/data/db
Exploring Podman volume mounts
As with Docker, you can inspect the details of a running container image.
$ podman inspect _container name_
This gives a very long JSON blob. The Mounts field is where information about volume mounts is stored.
You can inspect the volumes with:
$ podman volume ls
$ podman volume inspect _volume ID_
I found that Podman mounted a private volume at /data/db which exists as a directory inside the virtual machine, but does not map to the host directory. Inside the MongoDB Dockerfile, there is a directive VOLUME /data/db causing this directory to be a VOLUME.
Mounting a host directory to /data did not change the fact that /data/db is a volume, and Podman allocated a volume inside its private virtual machine.
To make an experiment, change the --volume option to this:
--volume /Users/david/Evoke/db/data:/david-data
The point is for the mount-point inside the container to be different from the VOLUME specified in the Dockerfile.
After restarting the container, start a shell inside the container like so:
$ docker exec -it mongodb bash
You can then explore the directories
root@b2c2743b9af2:/# ls /david-data/
configdb db
root@b2c2743b9af2:/# ls /data/db
WiredTiger WiredTigerHS.wt collection-4--1401331427186642920.wt index-1--1401331427186642920.wt index-8--1401331427186642920.wt sizeStorer.wt
WiredTiger.lock _mdb_catalog.wt collection-7--1401331427186642920.wt index-3--1401331427186642920.wt index-9--1401331427186642920.wt storage.bson
WiredTiger.turtle collection-0--1401331427186642920.wt diagnostic.data index-5--1401331427186642920.wt journal
WiredTiger.wt collection-2--1401331427186642920.wt docker-initdb.log index-6--1401331427186642920.wt mongod.lock
The /david-data directory exists and has the contents from the host machine. The /data/db directory also exists, with MongoDB database files that you'll find in the private volume directory inside the Podman virtual machine.
Inside the MongoDB container, run this:
root@b2c2743b9af2:/# touch /david-data/db/foo
root@b2c2743b9af2:/# exit
And then on your host machine run this:
$ ls ~/Project/db/data/db/
foo
Hence, the /data-david directory is mounted from the host machine. We can create a file inside the container, and it exists in the host machine.
Therefore, the solution is to tell MongoDB to use a different data directory.
Changing the MongoDB data directory
It's relatively simple to do this.
podman run \
--name mongodb \
--user mongodb \
--userns=keep-id:uid=999,gid=999 \
--restart always \
--expose 27017 \
--publish 27017:27017 \
--env MONGO_INITDB_ROOT_USERNAME=root \
--env MONGO_INITDB_ROOT_PASSWORD=example \
--volume /Users/david/Project/db/data:/project-data:rw \
--detach \
mongo --dbpath /project-data/db
One can pass mongod command-line options simply by adding them after the mongo container name.
Unfortunately there were a large number of problems which ensued having to do with mapping user IDs between the host user ID and the container user ID.
The --user and --userns options were an attempt to handle that mapping. The --user option changes the user ID used within the container to be mongodb. That user name has the ID of 999.
Setting a memory limit with Podman
Another desired feature was to set a higher memory limit:
$ podman machine stop
$ podman machine set --memory 4096
$ podman machine start
This changes the memory limit without having to completely rebuild the podman machine.
Conclusion
It seemed like it would be an easy task to bring up MongoDB under Podman on MacOS.
Unfortunately there are a large number of problems. In retrospect, it seems Podman is probably not as polished as Docker Desktop. With Docker Desktop issues like mapping container UIDs to the MacOS host UIDs is handled transparently, as are mounting host directories into the container. Both of those tasks were much more difficult using Podman.
As a result, I'm now installing MongoDB using MacPorts.
