Tags: Docker
Podman, the Docker alternative for running Docker containers, runs great on MacOS. But, some scenarios can be tricky, such as mounting macOS host directories into a container.
I'm developing a server using Node.js/TypeScript which persists its data to MongoDB. Normally, I run MongoDB in Docker, mounting the data directory into a host directory. With Docker Desktop for macOS mounting the database directory into the container is simple. You transparently use volume mounts giving the host directory and container directory, and Docker Desktop takes care of it all.
But, my laptop is old enough that Docker Desktop is not supported on its old version of macOS. For awhile I've run Docker inside of Multipass on macOS, as I described in an earlier article Replacing Docker Desktop with Multipass, to avoid Docker Desktop fees
However, for this project I want to have better control over things like memory allotment. The Multipass made it difficult enough that I wanted a different approach. That meant turning to Podman which I'd been avoiding because it doesn't support docker compose.
Supposedly running Podman instead of Docker is simple. You substitute the command name podman in place of docker in commands, and you're good to go. Unfortunately it wasn't that simple, for this case.
To take a step back slightly -
Podman is an open source project for building/executing Docker containers, that is "Kubernetes Ready". It offers some interesting features, such as root-less execution. The project was started by Redhat.
Unfortunately, getting Podman to support MongoDB on macOS brought up enough problems that I gave up. I'm now doing that project on my Linux/Ubuntu laptop where everything just works.
Installing Podman on MacOS
The official instructions say to use Homebrew to install Podman. Myself, I'm a MacPorts man, and prefer using it for installing open source tools. Fortunately the MacPorts project also has a Podman package.
Installation:
#### Homebrew
$ brew install podman
#### MacPorts
$ sudo port install podman
Either method results in the same software.
To initialize Podman, run:
$ podman machine init
$ podman machine start
Starting machine "podman-machine-default"
Waiting for VM ...
Mounting volume... /Users:/Users
Mounting volume... /private:/private
Mounting volume... /var/folders:/var/folders
This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command:
podman machine set --rootful
API forwarding listening on: /var/run/docker.sock
Docker API clients default to this address. You do not need to set DOCKER_HOST.
Machine "podman-machine-default" started successfully
This initializes a QEMU-based virtual machine within which a small Linux system is executing where a Podman service is available to run Docker containers. The machine is named podman-machine-default and notice that it mounts a few directories into the machine.
You can test Podman use as so:
$ podman run -it docker.io/library/busybox
Trying to pull docker.io/library/busybox:latest...
Getting image source signatures
Copying blob sha256:3f4d90098f5b5a6f6a76e9d217da85aa39b2081e30fa1f7d287138d6e7bf0ad7
Copying config sha256:a416a98b71e224a31ee99cff8e16063554498227d2b696152a9c3e0aa65e5824
Writing manifest to image destination
Storing signatures
/ #
This drops you into a shell in a Busybox container.
So.. we can run a Docker container, so it should be easy to run MongoDB, right?
(Un-)Successfully running a MongoDB container on MacOS using Podman
For any database, like MongoDB, it is 100% necessary to persist the data directory on the host machine. Otherwise your database disappears in a puff of smoke as soon as you recreate the container.
For Docker with docker compose, this Compose file snippet does the trick:
version: '3.8'
services:
mongo:
image: mongo
restart: always
ports:
- 27017:27017
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: example
volumes:
- /home/david/Projects/docker/mongodb/data:/data/db:rw
deploy:
resources:
limits:
cpus: '4'
memory: 10G
This is from the Compose file I use on my Ubuntu laptop. The volumes tag describes how to mount a host directory into the container.
Unfortunately Podman doesn't properly support docker compose. Note, I did not write docker-compose since that has been deprecated for years. The proper docker command supports a superset of Compose functionality using the docker compose command.
There is a podman compose command but its
documentation describes it as a thin wrapper around the old (deprecated)
docker-compose command. No thanks.
Hence, the Podman community has not caught up with the Compose specification, nor the new docker compose command with all its advanced features available.
Instead of trying that stuff, I converted my Compose file into a podman run command.
podman run \
--restart always \
--expose 27017 \
--publish 27017:27017 \
--env MONGO_INITDB_ROOT_USERNAME=root \
--env MONGO_INITDB_ROOT_PASSWORD=example \
--volume /Users/david/Project/db/data:/data:rw \
--detach \
mongo
This was the final result after going through issues we're about to discuss.
No such file or directory
My first attempt at mounting the data directory used a different --volume option:
--volume /Volumes/Project/db/data/db:/data/db:rw
My MacOS laptop has two drives in it, with the second drive containing several partitions. My /Users drive is small enough I use the second drive for everything. The key point is that the host machine mount point was not within /Users.
With that --volume option, I got this error:
Error: statfs /Volumes/Project/db/data/db: no such file or directory
I very carefully compared the path to the --volume command and to host system directories. Everything matched up correctly. This took many 10s of minutes until a
Stackoverflow posting gave an answer.
Read back to the podman machine start command and you see a few lines talking about mounting directories.
Mounting volume... /Users:/Users
Mounting volume... /private:/private
Mounting volume... /var/folders:/var/folders
For some reason, this limits the directory hierarchies from which directories can be mounted into a container running under Podman. But, I did not know that at the time.
Specifically, it seems that with Podman, the --volume option is looking for directories inside the virtual machine where the Podman containers actually execute. This is different from how Docker Desktop runs where there is a seamless mounting of host directories into the container.
The solution is, when running podman machine init, to mount host directories into the intermediary virtual machine. Those host directories can then be mounted into containers. Supposedly.
$ podman machine stop
$ podman machine rm podman-machine-default
$ podman machine init \
--volume /Users/david:/Users/david \
--volume /Volumes/Project/:/Volumes/Project
$ podman machine start
The first two commands get one back to a clean slate, recreating the Podman machine from scratch. The --volume option is not available with the podman machine start command, only with podman machine init.
You can inspect the directories mounted into the Podman virtual machine like so:
$ podman machine ssh
Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
Fedora CoreOS 38.20231002.2.2
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos
[root@localhost ~]# ls /
Users Volumes bin boot dev etc
home lib ...
Explore the directory further and you'll see that it's the host machine directories mounted into the Podman virtual machine. The --volume mount on the podman run command now works. But, with MongoDB there is another problem.
MongoDB: chown: changing ownership of '/data/db': Operation not permitted
The podman run command for running MongoDB got closer to running, except that podman ps showed that the container was not running, and that it had stayed up for less than one second.
That behavior happens when a container fails to start, and when Docker restarts the container which then fails to start, after which Docker tries to restart the container, leading to a long semi-infinite loop.
Running podman logs _container name_ showed a zillion copies of this error:
chown: changing ownership of '/data/db': Operation not permitted
chown: changing ownership of '/data/db': Operation not permitted
chown: changing ownership of '/data/db': Operation not permitted
Thinking this had to do with directory permissions, I ran chmod 777 on the macOS host directory. No change.
Another
Stackoverflow posting gave a solution. Instead of mounting the volume to
/data/db, mount it to /data instead. That way, the MongoDB service would not attempt to chown the mount point, which is what was failing.
Hence, instead of this option:
--volume /Volumes/Project/db/data/db:/data/db:rw
The solution is to use:
--volume /Volumes/Project/db/data:/data:rw
Or, specifically:
--volume /Users/david/Project/db/data:/data:rw
One way to verify MongoDB is running is:
$ mongosh --host localhost:27017 \
--username root \
--password example
While MongoDB is now running, and we can load it up with data, the host machine directory does not have the database.
$ ls ~/Project/db/data
configdb db
$ ls ~/Project/db/data/db
Data had been added to the database, but nothing showed up in the host directory.
Exploring Podman volume mounts
As with Docker, you can inspect the details of a running container image.
$ podman inspect _container name_
This gives a very long JSON blob. The Mounts field is where information about volume mounts is stored.
You can inspect the volumes with:
$ podman volume ls
$ podman volume inspect _volume ID_
I found that Podman mounted a private volume at /data/db which exists as a directory inside the virtual machine, but does not map to the host directory. Inside the MongoDB Dockerfile, there is a directive VOLUME /data/db causing this directory to be a VOLUME.
Mounting a host directory to /data did not change the fact that /data/db is a volume, and Podman allocated a volume inside its private virtual machine.
To make an experiment, change the --volume option to this:
--volume /Users/david/ProjectName/db/data:/david-data
The point is for the mount-point inside the container to be different from the VOLUME specified in the Dockerfile.
After restarting the container, start a shell inside the container like so:
$ docker exec -it mongodb bash
You can then explore the directories
root@b2c2743b9af2:/# ls /david-data/
configdb db
root@b2c2743b9af2:/# ls /data/db
WiredTiger WiredTigerHS.wt collection-4--1401331427186642920.wt index-1--1401331427186642920.wt index-8--1401331427186642920.wt sizeStorer.wt
WiredTiger.lock _mdb_catalog.wt collection-7--1401331427186642920.wt index-3--1401331427186642920.wt index-9--1401331427186642920.wt storage.bson
WiredTiger.turtle collection-0--1401331427186642920.wt diagnostic.data index-5--1401331427186642920.wt journal
WiredTiger.wt collection-2--1401331427186642920.wt docker-initdb.log index-6--1401331427186642920.wt mongod.lock
The /david-data directory exists and has the contents from the host machine. The /data/db directory also exists, with MongoDB database files that you'll find in the private volume directory inside the Podman virtual machine.
Inside the MongoDB container, run this:
root@b2c2743b9af2:/# touch /david-data/db/foo
root@b2c2743b9af2:/# exit
And then on your host machine run this:
$ ls ~/Project/db/data/db/
foo
Hence, the /data-david directory is mounted from the host machine. We can create a file inside the container, and it exists in the host machine.
Therefore, the solution is to tell MongoDB to use a different data directory.
Changing the MongoDB data directory
It's relatively simple to do this.
podman run \
--name mongodb \
--user mongodb \
--userns=keep-id:uid=999,gid=999 \
--restart always \
--expose 27017 \
--publish 27017:27017 \
--env MONGO_INITDB_ROOT_USERNAME=root \
--env MONGO_INITDB_ROOT_PASSWORD=example \
--volume /Users/david/Project/db/data:/project-data:rw \
--detach \
mongo --dbpath /project-data/db
The --dbpath option is for the mongod command. Its options can be passed on the command line as shown here, after the mongo container name.
Unfortunately there were a large number of problems which ensued having to do with mapping user IDs between the host user ID and the container user ID.
The --user and --userns options were an attempt to handle that mapping. The --user option changes the user ID used within the container to be mongodb. That user name has the ID of 999.
Setting a memory limit with Podman
Another desired feature was to set a higher memory limit:
$ podman machine stop
$ podman machine set --memory 4096
$ podman machine start
This changes the memory limit without having to completely rebuild the podman machine.
Conclusion
It seemed like it would be an easy task to bring up MongoDB under Podman on MacOS. With regular Docker Desktop it's very easy to do so. But, Docker Desktop is only available for macOS versions currently supported by Apple, and my macOS version fell out of support long ago.
Generally speaking, I ran into a large number of problems. In retrospect, it seems Podman is probably not as polished as Docker Desktop. With Docker Desktop issues like mapping container UIDs to the MacOS host UIDs is handled transparently, as are mounting host directories into the container. Both of those tasks were much more difficult using Podman.
A more serious problem is that Podman does not directly support modern Compose functionality. Really? The docker-compose command has been deprecated for YEARS. Advanced Compose functionality is available, and Podman can't be bothered to use it?
One option is to use the MacPorts version of MongoDB. Since I don't want MongoDB running full time, I would need to shut it down and restart it when needed.
Another option is to upgrade the operating system on my macOS laptop, using the Open Core Legacy Patcher thingy. That's a disruptive enough option that I have not done this.
Another option is to turn to Multipass, and set up a Multipass instance where MongoDB is natively installed, and the data directory is mounted into the Multipass instance.
I didn't do any of that, and instead switched to using my Ubuntu laptop. On Ubuntu, Docker runs natively, and none of the issues described here exist at all.
Ideally, Docker support on macOS must provide the same level of transparency. Docker Desktop does so, and Podman promises to do so. Docker Desktop delivers on their promise, while Podman does not.
