Tags: Docker »»»» Docker Swarm »»»» Docker on AWS »»»» AWS
Launching a Docker Swarm on EC2 instances is relatively easy, but of course there are pitfalls. One involves deploying a service to the swarm, but getting an error message about being unable to access the container image from the ECR registry.
In my case, I have a mix of my container definitions and 3rd party containers in a Stack file (a.k.a. Compose file) that I deploy using docker stack deploy --compose-file. The 3rd party containers deploy correctly. But for my containers, the output from this includes an error message like this:
image ${REGISTRY}/${IMAGE}:latest could not be accessed on a registry to record
its digest. Each node will access ${REGISTRY}/${IMAGE}:latest independently,
possibly leading to different nodes running different
versions of the image.
{serviceName}
Since --detach=false was not specified, tasks will be updated in the background.
In a future release, --detach=false will become the default.
Further, if I watch the deployment using docker service ps <container-name>, I see the swarm attempt to deploy on EC2 instance on the private subnet, which fails, and as soon as the swarm attempts to deploy to the EC2 on the public subnet the deployment works.
In other words, the error only occurs when the swarm tries to deploy my containers on a private subnet. The images for my containers are all hosted in ECR repositories that require authentication.
The key thing is that access to the ECR repository requires having run docker login. The solution is to pass the --with-registry-auth option to the swarm.
Of course I had run docker login on my laptop. The swarm, however, is on the EC2 instances. I access the swarm remotely using a Docker Context, in case that makes a difference. The point is that docker login had not been executed on the EC2 instances.
If you think this through, it's a situation where the Docker Registry credentials are required on the server. In other words, when we log-in to a Docker Registry some credentials of some sort are downloaded. Indeed ~/.docker/config.json does get an entry listing the Docker Registry URL for which we've logged-in. But, of course, the remote Docker servers also must have those Registry credentials.
There is an option used with several Docker commands, --with-registry-auth, that passes along such Registry login credentials. It is used as so:
docker stack deploy --with-registry-auth \
--compose-file docker-compose.yml \
<stack name>
Once you've done this, the swarm records the Registry credentials in its own data store, and then passes the credentials to any node in the swarm which needs them. More importantly every EC2 instance in your cluster will be able to install the images, and the error message will go away.
Reference:
https://github.com/moby/moby/issues/34153
