How to share a Docker network between Stacks deployed to a Docker Swarm

; Date: Sun Sep 20 2020

Tags: Docker »»»» Docker Swarm

How do you handle a system deployed to Docker Swarm, with multiple Stacks, where a container in one Stack must communicate with a container in another Stack? For example, you may have a database Stack, and an application layer Stack, where the application needs to communicate with the database. It's simplest to put both into the same Stack. But it's a best practice for the database to stay running, and to separately bring the application up and down to deploy updates. Therefore it's best to use separate Stacks for each layer, raising the question of how will the application containers find the database containers.

For the sake of an example, the system I'm deploying is:

• Application layer: NGINX to implement HTTPS, and Apache/PHP to host Drupal and WordPress sites
• Database layer: MySQL plus PHPMyAdmin

My first go-around of deploying this to my server was in one docker-compose.yml. If you've used Docker Swarm, you know that deploying a Docker Compose file results in what Swarm calls a Stack. The containers in each Stack are somewhat isolated from containers in other Stacks deployed to the same Swarm.

I first deployed that system as one Stack. Then, while thinking about how to better manage this system, it seemed best to split the system into at least two Stacks. At the minimum the two Stacks are the layers mentioned above, but perhaps the NGINX instance could be its own Stack. For any configuration change, like adding another Drupal site, or tweaking a setting, would mean redeploying the whole Stack. As a one-Stack system, such an update means rebooting the MySQL instance, causing a long time delay to restart the database server, resulting in a longer downtime for the website. Since it's not necessary to reboot the database each time, and it'd be best to split the system into two (or more) Stacks.

But then how do the containers in one Stack find containers in other Stacks? Specifically, how will the Drupal and WordPress containers find the database?

This question, finding the database, is a form of what's called Service Discovery. Generally that means a mechanism for learning the location of a Service on a network. Docker uses the domain name system (DNS) to support service discovery. Docker supports using the DNS name (a.k.a. host name) in both bridge and overlay networks.

Using the application layers described above, I have two Stacks, defined in two Compose files, as so:

• The application layer Compose file uses these networks:
  • servernet for the application layer containers to communicate with each other
  • dbnet to communicate with the database
• The database layer Compose file uses this network:
  • the aforementioned dbnet is used between MySQL and PHPMyAdmin

We need to share dbnet between the two Stacks.

The naive approach is in each Compose file declare the corresponding network as an overlay network like so:

networks:
    dbnet:    # or servernet
        driver: overlay

In a simple Swarm scenario, with everything declared in one Stack, the overlay network driver is most excellent. It automatically handles communication between services in the swarm even when the individual containers are on different Swarm nodes. But this approach makes a network that's only accessible to other containers declared in the Swarm Stack. The network is not reachable from other Stacks.

For example with the dbnet network declared that way in the database Compose file, I tried this declaration in the application layer Compose file:

networks:
    servernet:
        driver: overlay
    dbnet:
        external: true

I have done this with simple (non-Swarm) Compose files and it worked. But in this case, with two Stacks deployed to a Swarm, I got this error:

network "dbnet" is declared as external, but could not be found. You need to create a swarm-scoped network before the stack is deployed

That's an obtuse message, and nowhere in the Docker documentation is this properly explained. The phrase "swarm-scoped" sort of makes sense, that the network needs a Swarm-wide presence. But is there a way to do this via a declaration in one of the Compose files?

After some reading I tried this:

networks:
    dbnet:
        driver: overlay
        scope: swarm

That is, on docker network create there is a --scope attribute, and that attribute should surface this way in the Compose file. But, that simply gave this message:

scope Additional property scope is not allowed

Oh well, one theory down the drain.

What I eventually determined is this is necessary:

$ docker network create dbnet --scope swarm --driver overlay
$ docker network create servernet --scope swarm --driver overlay

This declares the two networks. It seems impossible to specify the scope inside the Compose file. That requires us to to declare the networks using the command-line tools instead.

Leaving off --driver overlay seemed to work in that the system would launch. But the application layer container was unable to resolve the host name for the database.

After inspecting the networks, I could see the IP address of the database container, and try to ping it from the Apache container as so:

root@933bd924ca1f:/var/www# ping 172.20.0.2
PING 172.20.0.2 (172.20.0.2): 56 data bytes
64 bytes from 172.20.0.2: icmp_seq=0 ttl=64 time=0.142 ms
64 bytes from 172.20.0.2: icmp_seq=1 ttl=64 time=0.141 ms
64 bytes from 172.20.0.2: icmp_seq=2 ttl=64 time=0.137 ms
64 bytes from 172.20.0.2: icmp_seq=3 ttl=64 time=0.130 ms
...

Normally, the Ping application will resolve the IP address and start showing the domain name. But this time it did not, and further trying all kinds of variations of the container name did not result in connecting to the database. The Docker overlay network advertises each container name as a DNS entry letting you use the container name as a host name to connect with.

Adding --driver overlay to each network create statement caused the container names to be advertised as DNS names as expected.

Then, in the database Compose file we have this:

networks:
    dbnet:
        external: true

And in the application layer Compose file we have this:

networks:
    servernet:
        external:
            name: servernet
    dbnet:
        external:
            name: dbnet

In other words, for Swarm Stacks, the networks must be declared completely separately from the Compose files. In each Compose file, the required networks are referenced using external: true.

That stands in contrast to what we do in normal Compose files, that are deployed using docker-compose. In that case one can declare a network inside one Compose file, and reference it using external: true from other Compose files.

Finally, where the rubber hits the road, meaning in the Drupal settings.php file, we have this:

$databases['default']['default'] = array(
    'driver' => 'mysql',
    'database' => 'DATABASE-NAME',
    'username' => 'DATABASE-USER',
    'password' => 'PASSWORD',
    'host' => 'db',   // <--- container name as host name
    'port' => 3306,
    'prefix' => '',
    'collation' => 'utf8_general_ci',
 );

For any application connecting to a database, there will be an object or connection string describing how to connect with the database. That will include a host name. For it to work the host name must resolve to an IP address. The same will hold for any other service, such as a REDIS server.

Docker helpfully uses the domain name system to support Service Discovery. That way one service finds other services via the host name (a.k.a. domain name). But as we've seen here, it sometimes requires careful configuration to make it work right.