Multi-stage Docker image builds, minimizing final image size, supporting multi-architecture containers

; Date: Sun Oct 20 2024

Tags: Docker

Docker supports creating images/containers for (almost) every CPU architecture. But, the image build workflow differs based on the architecture. We discuss one way to handle building Docker images across platforms while minimizing the resulting image size.

You might have a project with a perfectly fine Dockerfile like this:

FROM python:3.9-alpine
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY . /usr/src/app
RUN pip3 install -r requirements.txt
EXPOSE 8080
ENTRYPOINT [ "python3", "-m", "server" ]

This is a normal every-day Dockerfile that would surely work everywhere, right? But, what if you're using an M2 MacBookPro, and everyone else uses an x86 machine, and this Dockerfile fails to build on your laptop but not on everyone elses laptop?

This is a real life situation in a team I'm working in right now. The error is:

$ docker buildx build -t server  -f Dockerfile .
... lots of successful build steps
337.9       creating build/temp.linux-aarch64-cpython-39
337.9       Running '(cd  "/tmp/pip-install-ed4meyzc/gevent_d76cda038ad9489889b7f55a9f529594/deps/libev"  && sh ./configure -C > configure-output.txt )' in /tmp/pip-install-ed4meyzc/gevent_d76cda038ad9489889b7f55a9f529594
337.9       configure: error: in `/tmp/pip-install-ed4meyzc/gevent_d76cda038ad9489889b7f55a9f529594/deps/libev':
337.9       configure: error: no acceptable C compiler found in $PATH
... long stack trace and other additional information

The same Dockerfile builds correctly on other machines, but not on the M2 Mac. The failure discusses no C compiler being available and, therefore, it is unable to install CPython.

The solution is fairly straight-forward. Rewrite the Dockerfile to install compiler tools:

FROM python:3.9-alpine
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY . /usr/src/app
# RUN apk update
RUN apk add -u gcc libc-dev linux-headers libffi-dev python3-dev musl-dev make
RUN pip3 install --upgrade pip
RUN pip3 install -r requirements.txt
EXPOSE 8080
ENTRYPOINT [ "python3", "-m", "server" ]

Once that's done, the Dockerfile can be used on everyones laptop to build the image. Problem solved, right? We can move on to other tasks? Nothing further to consider here?

Well, no.

The rationale for using an Alpine-based image is to minimize the resulting image size. But, this resulting image includes development tools like GCC, increasing image size considerably. Further, might the compiler tools introduce security risks?

Also, we haven't understood the problem. Why did the build failed on the M2 Mac and not on other systems? Why is the compiler required on M2 Mac's?

Cross-architecture Docker Image builds

One can go for years using Docker for building and deploying complex services without thinking about a basic thing - the CPU architecture. Traditionally the developer laptop runs an Intel or AMD CPU, and deployment is to a server with an Intel or AMD CPU. In such a case we're not exposed to architecture differences.

But, an M1/M2/M(n) Mac uses an ARM CPU, as does SBC's like the Raspberry Pi, as do an increasing number of deployment hosts. We need to be considering ARM support. Further, there are other existing platforms such as IBM mainframes. Our cozy world of x86 Docker images ignored the reality of modern application development. There are many possible CPU architectures to which we can deploy Docker containers and therefore we need to know about Docker's multi-architecture features.

To begin, let's examine the architecture of an image. We do this with the docker inspect command, giving an image name, which gives us a big JSON blob. To digest this, let's use jq to pull out the fields of interest:

$ docker buildx build -t server-orig  -f Dockerfile .
... build output
$ docker inspect server-orig \
        | jq '.[0].Architecture, .[0].Os, .[0].Size'
"amd64"
"linux"
147842107

This image is for a Linux operating system, on amd64 architecture, and has a size of 147 MB.

When building an image, Docker defaults to building for the CPU architecture of the host machine. My laptop has an Intel Core i7 CPU, resulting in the amd64 CPU architecture. The M2 Mac used by my coworker has an Apple-designed ARM CPU, which differs from amd64.

To build images for another operating system, Docker has a flag --platform. Did you know about that flag? I didn't until now.

At the Docker Hub page for the image in question, https://hub.docker.com/_/python/tags?name=alpine, we can view the supported CPU architectures for the Python/Alpine image.

This means we can run this:

$ docker buildx build -t server-orig-arm64  \
        -f Dockerfile --platform linux/arm64 .

This is the same build command, but specifying the linux/arm64 architecture. When executed on my x86 laptop, it means Docker needs to somehow interpolate the ARM binaries on x86.

But, on my laptop, which runs Ubuntu Linux, that failed with an error like this:

 > [2/5] RUN mkdir -p /usr/src/app:
0.235 exec /bin/sh: exec format error
------
 1 warning found (use docker --debug to expand):
 - InvalidBaseImagePlatform: Base image python:3.9-alpine was pulled with platform "linux/arm64/v8", expected "linux/arm64/v8" for current build (line 1)
Dockerfile:2
--------------------
   1 |     FROM python:3.9-alpine
   2 | >>> RUN mkdir -p /usr/src/app
   3 |     WORKDIR /usr/src/app
   4 |     COPY . /usr/src/app
--------------------

The short story here is that an ARM image was executed on an AMD64 CPU. Running the mkdir command failed with the error exec format error, which means that the exec(2) system call was used, but the binary being executed was for an unsupported CPU architecture. Namely, the ARM binaries in the ARM image do not execute on my x86 laptop.

The Docker documentation includes a treatise on multi-platform builds: https://docs.docker.com/build/building/multi-platform/

The article describes how to run Docker containers for non-native CPU architectures, as well as to build such containers. In this case we want to build/run ARM containers on x86, or x86 containers on ARM.

For systems with Docker Desktop, it appears from the article that one enables a few features in the Docker Desktop application.

For users, like myself, using Docker Engine on Linux, it describes enabling the use of QEMU to support cross-architecture Docker image execution/building. Doing so requires this command:

$ docker run --privileged --rm \
        tonistiigi/binfmt --install all

The impact is configuring Docker to transparently use QEMU to support building or running images built for non-native CPU architectures.

After running that command, we can rerun the build, but now we get the same error our co-worker saw on his M2 Mac. It means we've replicated the error, and proved that it has nothing to do with macOS on ARM. Instead it has to do with ARM-architecture Docker images.

Our coworker supplied this Dockerfile which worked on his laptop:

FROM python:3.9-alpine
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY . /usr/src/app
# RUN apk update
RUN apk add -u gcc libc-dev linux-headers \
        libffi-dev python3-dev musl-dev make
RUN pip3 install --upgrade pip
RUN pip3 install -r requirements.txt
EXPOSE 8080
ENTRYPOINT [ "python3", "-m", "server" ]

The substantive change here is to install compiler tools. Obviously since the step of installing CPython failed due to lack of a C compiler, one must install a C compiler.

On my x86 Ubuntu laptop, I can now run this build command:

$ docker buildx build -t server-mac-arm64  \
        -f Dockerfile-mac --platform linux/arm64 .

This successfully builds the image, but takes a very long time to finish. Emulating ARM CPU instructions using QEMU is clearly inefficient.

Inspecting the image:

$ docker inspect server-mac-arm64 \
        | jq '.[0].Architecture, .[0].Os, .[0].Size'
"arm64"
"linux"
475916098

The newly built image is for the linux/arm64 CPU architecture, and the image size is now 475 MB.

We have a successful build, but the image is much bigger.

The Dockerfile-mac build was created to support building on the M2 Mac hardware, where ARM is the native CPU architecture. It can also be used to build on x86 systems.

$ docker buildx build -t server-mac-amd64 \
    -f Dockerfile-mac \
    --platform linux/amd64 .
... lots of build output
$ docker inspect server-mac-amd64 \
    | jq '.[0].Architecture, .[0].Os, .[0].Size'
"amd64"
"linux"
196077571

This means that Dockerfile-mac supports building this image for any CPU architecture supported by QEMU. There are other choices discussed in the article, such as deploying BuildKit Builders onto target machine hardware.

Multi-architecture Docker Image builds

Docker supports multi-platform images, where the same image supports multiple CPU architectures. For example, the Python/Alpine image supports several architectures, as we saw above.

Reread the documentation for advice: https://docs.docker.com/build/building/multi-platform/ We learn there that the --platform parameter can take a comma-separated list of CPU architectures.

This fails:

$ docker buildx build -t server-mac-multi-arch  \
        -f Dockerfile-mac \
        --platform linux/arm64,linux/amd64 .
[+] Building 0.0s (0/0)     docker:default
ERROR: Multi-platform build is not supported for the docker driver.
Switch to a different driver, or turn on the containerd image store, and try again.
Learn more at https://docs.docker.com/go/build-multi-platform/

The failure here is that this containerd image store is not configured. Because my laptop runs Docker Engine on Linux (Ubuntu), I must follow these instructions: https://docs.docker.com/engine/storage/containerd/

For those with Docker Desktop, see: https://docs.docker.com/desktop/containerd/

Once the containerd image store is enabled, we can successfully rerun the above command. The output shows that both architectures are built in parallel, as one might expect.

We can now rerun the build, specifying multiple platforms.

$ docker buildx build -t server-mac-multi-arch  \
    -f Dockerfile-mac \
    --platform linux/arm64,linux/amd64 .
... lots of build output
$ docker inspect server-mac-multi-arch \
    | jq '.[0].Architecture, .[0].Os, .[0].Size'
"amd64"
"linux"
196079181

The final image size, 196 MB is very large. But there's a confusion here, since docker inspect does not show information for both architectures. The inspect output only displays data for the architecture shown here, even though the build did go for both architectures.

From a Docker team blog post explains a few things about multi-architecture builds. https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/

The first thing we learn is about the Manifest. For example we can inspect the Manifest of an image on Docker Hub this way:

$ docker manifest inspect -v python:alpine
... JSON manifest

But inspecting the manifest of our locally-built image gives this error:

$ docker manifest inspect -v --insecure server-mac-multi-arch 
errors:
denied: requested access to the resource is denied
unauthorized: authentication required

The blog post discusses using a --push flag when building a multi-architecture Docker image. Adding that flag results in this error:

$ docker buildx build --push \
    -t server-mac-multi-arch  \
    -f Dockerfile-mac \
    --platform linux/arm64,linux/amd64 .
... lots of build output
ERROR: failed to solve: failed to push docker.io/library/server-mac-multi-arch:latest: push access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

It appears the blog post in question assumes there is a Docker registry being used. The --push option will be sending each portion to the registry. In my case there is no registry.

Likewise, the error with docker manifest inspect seems to be that manifest inspect needs to pull information from a registry, which doesn't exist in my case.

I was able to verify that the image supports multiple architectures this way:

$ docker run --rm \
    --platform linux/arm64 \
    server-mac-multi-arch
... output from starting the service
$ docker run --rm \
    --platform linux/amd64 \
    server-mac-multi-arch
... output from starting the service
$ docker run --rm \
    --platform linux/386 \
    server-mac-multi-arch
Unable to find image 'server-mac-multi-arch:latest' locally

Since this image was not built for linux/386, it could not find the requested image, while it could find the image for the other two architectures.

In other words, the server-mac-multi-arch image does contain image support for both architectures, but for some reason the image inspect command does not show that information. The solution is probably to run a local repository.

Multi-stage Docker Image builds

The image builds we've shown above ballooned from 147 MB to 475 MB. Somewhere the quest for a small image size got lost.

Docker supports what are called multi-stage image builds, which will allow us to reduce the image size.

A multi-stage build is defined by using multiple FROM commands in the Dockerfile. You can selectively copy artifacts from one build stage to another.

In our case we'll use a build stage containing the C compiler and other tools, then copy to an execution stage just the code required to execute the application. The final image contains only what is copied into the execution stage, and the compiler tools get left behind.

FROM python:3.9-alpine AS builder
RUN apk add -u gcc libc-dev linux-headers \
        libffi-dev python3-dev musl-dev make
RUN pip3 install --upgrade pip
COPY requirements.txt .
RUN pip3 install --user -r requirements.txt
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY . /usr/src/app

FROM python:3.9-alpine AS execution

WORKDIR /usr/src/app

COPY --from=builder /root/.local /root/.local
COPY --from=builder /usr/src/app .

EXPOSE 8080
ENV PATH=/root/.local/bin:$PATH
ENTRYPOINT [ "python3", "-m", "server" ]

This is derived from the Dockerfile we saw earlier, and uses two stages to minimize the resulting image.

The first stage, labeled builder, contains the native-code build tools, and builds the application in that stage.

The second stage, labeled execution, copies from builder the built application.

An important detail for building Python applications with PIP is the --user option. To copy artifacts to the execution stage, the build stage must contain them in a known location.

The --user option says to place the PIP artifacts into the .local directory in the user home directory. Therefore, the root user directory is /root/.local. This gets copied to the execution stage in the same location.

This is a very simple multi-stage build. There are many options available that aren't utilized here.

We can now build it three ways:

$ docker buildx build -t server-multi-amd64 \
    -f Dockerfile-multi-stage \
    --platform linux/amd64 .
... lots of build output
$ docker buildx build -t server-multi-arm  \
    -f Dockerfile-multi-stage \
    --platform linux/arm64 .
... lots of build output
$ docker buildx build -t server-multi-multi-arch  \
    -f Dockerfile-multi-stage \
    --platform linux/amd64,linux/arm64 .
... lots of build output

Each of these build successfully.

And, let's check the characteristics:

$ docker inspect server-multi-amd64 \
        | jq '.[0].Architecture, .[0].Os, .[0].Size'
"amd64"
"linux"
39735422
$ docker inspect server-multi-arm \
        | jq '.[0].Architecture, .[0].Os, .[0].Size'
"arm64"
"linux"
39652492
$ docker inspect server-multi-multi-arch \
        | jq '.[0].Architecture, .[0].Os, .[0].Size'
"amd64"
"linux"
18999255

Testing a Docker image built for a non-native CPU architecture

We've built our Docker image for multiple architectures. Just because the image builds does not mean it will function correctly. Obviously you must test the image for every architecture you build.

In this case a simple test is to see that the service launches. It may give errors due to lack of configuration, but if it launches we have greater confidence the build is successful.

$ docker run --platform selected/platform \
    --other-options \
    image-name \
    image arguments

That is, the --platform parameter is available on docker run. You can use docker run as usual, specifying the CPU architecture choice.

In Docker Compose files, the platform option serves the same role as the --platform parameter. See: https://github.com/compose-spec/compose-spec/blob/main/05-services.md

Conclusion

We might not think about it too often, but Docker does support building images for multiple CPU architectures, running images built for non-native CPU architectures, and creating multi-architecture Docker images. Most of develop Docker images on an x86 laptop for deployment to an x86 server. But, as ARM gains in popularity multi-architecture Docker issues will be more common.

The learning we gained from this article is:

• To learn which CPU architectures are supported by a given Docker image. - docker inspect or docker image inspect or docker manifest inspect
• To build an image for a non-native CPU architecture: docker buildx build --platform linux/xyzzy ... That might require some configuration.
• To build an image for multiple CPU architectures: docker buildx build --platform linux/amd64,linux/arm64 ...
• To optimize the image size: Use multi-stage builds, and do not distribute build tools in the image.
• To run an image for a specific CPU architecture: docker run --platform linux/xyzzy ...

The image size reduction by using a multi-stage build we gained was significant:

Image name	Size
server-orig	147842107
server-orig-arm64	FAIL
server-mac-amd64	196077571
server-mac-arm64	475916098
server-mac-multi-arch	196079181
server-multi-amd64	39735422
server-multi-arm64	39652492
server-multi-multi-arch	39736175

Distributing the build tools (GCC, etc) in the image bloated its considerably, and created extra possible security issues. A nefarious person breaking into our container might have used build tools for their nefarious goals. By implementing a simple multi-stage build, the image remained small, while disabling any possible attacks relying on the presence of a C compiler.

We're now better prepared for deployment to other architectures such as ARM.