Tags: Docker
Docker images contain instructions for building a filesystem one layer at a time. Sometimes we need to explore the layers to ensure the image is built correctly. Since it's non-obvious how to do that, let's explore several tools.
We normally create a Docker image by writing a Dockerfile, then building the image from that file. The file contains instructions on copying files around, compiling software, and so forth. We may have read that each command creates a new layer in the image, without really knowing what that means.
The other day I was trying to figure out why the image I'd built didn't have the contents I expected. I realized the only way I knew how to examine the container was to start it running, but in this case the container exited as soon as it started giving me no opportunity to get into the container to see what it contained.
I expected the docker image inspect command to let me see the contents of each layer. Nope.
But, I knew from having written another blog post (Deploying Docker images to a server without using a Docker Registry) about how to export a Docker image to a tar file. That turned into a method for exploring Docker image layers, and I was able to solve my problems.
On exploring that command, I came across a few other commands for exploring Docker container content as well.
Exporting a Docker image to explore its contents
What do you think these commands would do:
$ docker build -t group-name/image-name .
$ docker image save -o image-name.tar group-name/image-name
The first is familiar to anyone who creates Docker images. It builds an image from a Dockerfile in the current directory. The image is saved into some virtual space with the name group-name/image-name. Most of us run that command without thinking too much about the details.
The second command saves (exports) an image as a tar file. THe purpose for that command in my other article was to be able to deploy the image to a server without saving it to a Docker registry. In this case it gives us a way to explore the image layers.
Let's try this on a real image, the nginx image from hub.docker.com:
$ docker image save -o nginx.tar nginx
$ ls -l nginx.tar
-rw------- 1 david david 168737280 Jul 16 22:26 nginx.tar
$ (mkdir t && cd t && tar xvf ../nginx.tar )
blobs/
blobs/sha256/
blobs/sha256/0139d4090a4ccd1b50d591de26436f70f0fdca1891d6771cb09e9690d920f5c2
blobs/sha256/09be960dcde4138561c482b1688f3486db8fd50c8ef3e660e6ec8343644b0ba2
blobs/sha256/1178c02f07cd3eef09fdf4e7a540e5cc2fc0ad6e603bc7ba6e6629e7953e8dd9
blobs/sha256/18be1897f9402f27c8c0eba7c86e0c00f5656db62961b5949d55f9565c42c6e7
blobs/sha256/1de3ade92dae9fe29f100e488e9c67f38cfe0a8b7d85663f43465fdcf8e79180
blobs/sha256/231bb7482ad95f986611220eef7fc87cfa6e94a8cdcf4e51d7b636758f316865
blobs/sha256/243270682edd0b83547380e1b3b650e4a966ead5b9fd307da5474f6ed64b33b5
blobs/sha256/258d235edd1020a35ecf5ea21db89c885787700eb97681f3ae80373b38d75f89
blobs/sha256/27ea236fd951182cea444fdb7bb16e5f9cccbb8a6068a1439345cfe155cae470
blobs/sha256/2ddb6ca9e8a09833c6c7bb65d137718b1827c51fb4ed0c949b2b3777d63d9069
blobs/sha256/3e207b409db364b595ba862cdc12be96dcdad8e36c59a03b7b3b61c946a5741a
blobs/sha256/44d72874678dccc8f953fb5a0229a587ad7e40f52d709db86c2a405e9a0dd8ae
blobs/sha256/5385450e076ac9279408416130f701405aaf007cb071b00b5172f5afe6035737
blobs/sha256/570fc47f255858a1ea028ee56a4472c23a37f8103d10066028a534147661d943
blobs/sha256/5d17421f15719347504420acd3af66f55803fc2f82de17d8be90c432db47c297
blobs/sha256/7bb2a9d373374aee9939eff4c7a7883e74c22fd4ad9499497a1a03fa987befd4
blobs/sha256/7d0cdcc60a96a5124763fddf5d534d058ad7d0d8d4c3b8be2aefedf4267d0270
blobs/sha256/8c796ea234695ea4f62c566e8d695702f79a21130146b5afbaac8be0363f43a5
blobs/sha256/96e0ca1d8e5e08181324774483d8044e2972250bb432f164717e2f1b4cbfad1d
blobs/sha256/a059c9abe376b8511be8cafbfe23b18631c161af8cabd4910183860ed7595dda
blobs/sha256/a181cbf898a0262fa8f9dfcf769014445ca54f25c0440de8c40692392b57de03
blobs/sha256/d253f69cb991b5e14ba7c21bb8ae6c130de83c39fb46dc48f3ee840152c965a3
blobs/sha256/de2543b9436b7b0e2f15919c0ad4eab06e421cecc730c9c20660c430d4e5bc47
blobs/sha256/dfe7577521f0d1ad9f82862f3550e12a615fcb07319265a3d23e96f2f21f62ec
blobs/sha256/f4289fe31d858c4d6f7fdf0c8a85cd845ace2202abcafdcd2eabd986ea98e0e6
blobs/sha256/fd95118eade99a75b949f634a0994e0f0732ff18c2573fabdfc8d4f95b092f0e
index.json
manifest.json
oci-layout
repositories
That gave us a bunch of anonymous files named with SHA codes, and it looks like manifest.json records more information about each file. We can reason that the files in blobs/sha256 are the layers of the container. Also, since the file names are SHA256 hashes the alphabetical order won't be related to the layer order. Instead it's likely manifest.json contains the ordering.
In my case I found this command to be most useful:
$ file blobs/sha256/*
blobs/sha256/0139d4090a4ccd1b50d591de26436f70f0fdca1891d6771cb09e9690d920f5c2: JSON data
blobs/sha256/09be960dcde4138561c482b1688f3486db8fd50c8ef3e660e6ec8343644b0ba2: POSIX tar archive
blobs/sha256/1178c02f07cd3eef09fdf4e7a540e5cc2fc0ad6e603bc7ba6e6629e7953e8dd9: JSON data
blobs/sha256/18be1897f9402f27c8c0eba7c86e0c00f5656db62961b5949d55f9565c42c6e7: POSIX tar archive
blobs/sha256/1de3ade92dae9fe29f100e488e9c67f38cfe0a8b7d85663f43465fdcf8e79180: JSON data
blobs/sha256/231bb7482ad95f986611220eef7fc87cfa6e94a8cdcf4e51d7b636758f316865: JSON data
blobs/sha256/243270682edd0b83547380e1b3b650e4a966ead5b9fd307da5474f6ed64b33b5: JSON data
blobs/sha256/258d235edd1020a35ecf5ea21db89c885787700eb97681f3ae80373b38d75f89: JSON data
blobs/sha256/27ea236fd951182cea444fdb7bb16e5f9cccbb8a6068a1439345cfe155cae470: JSON data
blobs/sha256/2ddb6ca9e8a09833c6c7bb65d137718b1827c51fb4ed0c949b2b3777d63d9069: JSON data
blobs/sha256/3e207b409db364b595ba862cdc12be96dcdad8e36c59a03b7b3b61c946a5741a: POSIX tar archive
blobs/sha256/44d72874678dccc8f953fb5a0229a587ad7e40f52d709db86c2a405e9a0dd8ae: JSON data
blobs/sha256/5385450e076ac9279408416130f701405aaf007cb071b00b5172f5afe6035737: JSON data
blobs/sha256/570fc47f255858a1ea028ee56a4472c23a37f8103d10066028a534147661d943: POSIX tar archive
blobs/sha256/5d17421f15719347504420acd3af66f55803fc2f82de17d8be90c432db47c297: POSIX tar archive
blobs/sha256/7bb2a9d373374aee9939eff4c7a7883e74c22fd4ad9499497a1a03fa987befd4: POSIX tar archive
blobs/sha256/7d0cdcc60a96a5124763fddf5d534d058ad7d0d8d4c3b8be2aefedf4267d0270: JSON data
blobs/sha256/8c796ea234695ea4f62c566e8d695702f79a21130146b5afbaac8be0363f43a5: JSON data
blobs/sha256/96e0ca1d8e5e08181324774483d8044e2972250bb432f164717e2f1b4cbfad1d: JSON data
blobs/sha256/a059c9abe376b8511be8cafbfe23b18631c161af8cabd4910183860ed7595dda: POSIX tar archive
blobs/sha256/a181cbf898a0262fa8f9dfcf769014445ca54f25c0440de8c40692392b57de03: POSIX tar archive
blobs/sha256/d253f69cb991b5e14ba7c21bb8ae6c130de83c39fb46dc48f3ee840152c965a3: POSIX tar archive
blobs/sha256/de2543b9436b7b0e2f15919c0ad4eab06e421cecc730c9c20660c430d4e5bc47: JSON data
blobs/sha256/dfe7577521f0d1ad9f82862f3550e12a615fcb07319265a3d23e96f2f21f62ec: POSIX tar archive
blobs/sha256/f4289fe31d858c4d6f7fdf0c8a85cd845ace2202abcafdcd2eabd986ea98e0e6: JSON data
blobs/sha256/fd95118eade99a75b949f634a0994e0f0732ff18c2573fabdfc8d4f95b092f0e: POSIX tar archive
This tells us some files are JSON and others are tar archives.
Focusing on the tar files we learn they have snippets of file systems:
$ tar tvf blobs/sha256/dfe7577521f0d1ad9f82862f3550e12a615fcb07319265a3d23e96f2f21f62ec
-rwxrwxr-x 0/0 1202 2022-05-18 01:35 docker-entrypoint.sh
$ tar tvf blobs/sha256/a181cbf898a0262fa8f9dfcf769014445ca54f25c0440de8c40692392b57de03
drwxr-xr-x 0/0 0 2020-06-02 19:23 docker-entrypoint.d/
-rwxrwxr-x 0/0 1043 2020-06-02 19:23 docker-entrypoint.d/20-envsubst-on-templates.sh
$ tar tvf blobs/sha256/570fc47f255858a1ea028ee56a4472c23a37f8103d10066028a534147661d943
drwxr-xr-x 0/0 0 2020-06-02 19:23 docker-entrypoint.d/
-rwxrwxr-x 0/0 1963 2020-06-02 19:23 docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
Many of the tar files contain a full filesystem dump, but these are small enough to view here.
There's a lot more to learn from this. In my case looking at the tar files gave me information about the files in the image layers to debug the build.
Creating a container without running the container
We've all run docker run --rm --it nginx bash (or the like) to execute/run a container. This particular command gives us a bash shell inside the container, and then the container is deleted after we exit out.
This is the common way to inspect the insides of a container, by using regular shell commands in a command shell inside the running container. An alternate to this for a container that's already running is: docker exec -it nginx bash
But, what if we're uncertain if the image has malware? Or if the container crashes before we can get the running shell, as I described earlier. In cases like that we cannot run the container.
Instead, we can do this:
$ docker create --name inspect-nginx nginx
d8804e15d1680f69941a6184e07b319f3344e975fb239da71747d2e81a059ebc
This creates, but does not run, the container from the image. But since it's not a running container we cannot start a shell.
But, we can run this:
$ docker export inspect-nginx >inspect-nginx.tar
This exports the filesystem from the container into a tar file. We can then untar the file and explore its contents.
This does not give us a layer-by-layer view of the container, instead it gives us the resulting filesystem.
When you're done remember to run this:
$ docker container rm inspect-nginx
inspect-nginx
Otherwise the container will sit around unused taking up space on your computer.
Inspecting the commands from which the image was built
Another view into the image contents is to review how it was built.
$ docker image history nginx
IMAGE CREATED CREATED BY SIZE COMMENT
de2543b9436b 2 years ago /bin/sh -c #(nop) CMD ["nginx" "-g" "daemon… 0B
<missing> 2 years ago /bin/sh -c #(nop) STOPSIGNAL SIGQUIT 0B
<missing> 2 years ago /bin/sh -c #(nop) EXPOSE 80 0B
<missing> 2 years ago /bin/sh -c #(nop) ENTRYPOINT ["/docker-entr… 0B
<missing> 2 years ago /bin/sh -c #(nop) COPY file:09a214a3e07c919a… 4.61kB
<missing> 2 years ago /bin/sh -c #(nop) COPY file:0fd5fca330dcd6a7… 1.04kB
<missing> 2 years ago /bin/sh -c #(nop) COPY file:0b866ff3fc1ef5b0… 1.96kB
<missing> 2 years ago /bin/sh -c #(nop) COPY file:65504f71f5855ca0… 1.2kB
<missing> 2 years ago /bin/sh -c set -x && addgroup --system -… 61.1MB
<missing> 2 years ago /bin/sh -c #(nop) ENV PKG_RELEASE=1~bullseye 0B
<missing> 2 years ago /bin/sh -c #(nop) ENV NJS_VERSION=0.7.3 0B
<missing> 2 years ago /bin/sh -c #(nop) ENV NGINX_VERSION=1.21.6 0B
<missing> 2 years ago /bin/sh -c #(nop) LABEL maintainer=NGINX Do… 0B
<missing> 2 years ago /bin/sh -c #(nop) CMD ["bash"] 0B
<missing> 2 years ago /bin/sh -c #(nop) ADD file:4a0bb88956083aa56… 80.4MB
This is again the standard nginx container. We should be able to compare this with the Dockerfile.
Diving into image internals
An open source tool,
dive gives you a terminal GUI for exploring Docker images layer-by-layer.
The GitHub repository has installation instructions, and usage instructions. Unfortunately I wasn't able to determine how to move from one layer to the next.
Summary
We have learned about inspecting the contents of a Docker image. As software engineers we should be thinking how hard is it to read the JSON files in this image, and write a tool for examining the content. I sure am thinking along those lines, but lack the interest in following through on that idea.
For example, shouldn't a tool like Portainer allow us to inspect the internals of an image? While it shows us a list of images available on the local system, and it allows us to export an image, which we can inspect as shown earlier, its GUI does not offer a way to explore the contents.
For 90% or more of us, the commands shown above are enough. With them we can inspect the filesystem without instantiating the container. We can then inspect the files, look for malware, make sure the files were built correctly, etc.
