; Date: Sun Jul 05 2020

Tags: Docker »»»» Letsencrypt »»»» HTTPS »»»» Nginx

NGINX is a hugely flexible webserver with which it's very easy to manage HTTPS with Lets Encrypt. With a simple configuration file it is easy to add HTTPS support to a back-end service, using NGINX's reverse proxy. It's possible to use this for deployments both small and large. As with any website Internet-visible nowadays, it is necessary to use HTTPS. Which leads to the task of using Lets Encrypt to provision SSL certificates for several domains, using NGINX to terminate the HTTPS connections, while proxying from NGINX to the actual services. This blog post discusses a Docker container designed explicitly for that purpose.

This article is about a Docker container I've developed, called Cronginx, that makes it easy to manage HTTPS service using NGINX and Lets Encrypt. The container runs both NGINX and Cron, the latter there to support a daily background task that automatically renews the SSL certificates provisioned from Lets Encrypt. The user of this container provides one or more NGINX configuration files for domains this server is supposed to handle.

The features are:

• NGINX as the webserver with its world class capabilities
• It is easy to host multiple domains, because NGINX is configured to load one or more config files
• Cron is present for running background tasks, including a certificate renewal script
• Lets Encrypt tools (Certbot) is installed, along with a couple useful shell scripts
• The shell scripts make it easy to register domains with Lets Encrypt, and to manage SSL certificate renewal
• Light-weight and easy to configure

Cronginx can be used in any circumstance where there is one or more domains requiring HTTPS service. Simply install and instantiate the container (we have a sample Docker Compose file later), do a little configuration, and run a provided shell script that helps with registering the domains with Lets Encrypt. As a bonus is the low cost, wince Lets Encrypt is a free service for SSL certificates. The only cost is the hosting fee required to run the container.

An example of using Cronginx is the scenario for which I developed it. I am self-hosting a development environment at home using Gogs/Gitea, Jenkins, NextCloud, and other services. The services are also visible on the public Internet by using Cronginx. On my desk is an Intel NUC running Ubuntu on which I've installed Docker, in which the named services are running. This serves as a Git repository server, and build machine, for any software project where the Git repository cannot be on a public server. For example, this machine is where TechSparx and several other websites are rendered to HTML. The public domains (git.DOMAIN.com, build.DOMAIN.com, cloud.DOMAIN.com, etc) all point at my home network so these services can be accessed not only from my home network, but from anywhere on the Internet. Each domain has HTTPS thanks to Cronginx.

There are many other ways of using Cronginx, since NGINX is so flexible. The key is whether your use case involves provisioning SSL certificates using Lets Encrypt. If so, Cronginx is a convenient way of proceeding.

Why involve Cron? That's because, while Lets Encrypt supplies free SSL certificates, the certificates expire after 90 days. In order to avoid an administrative hassle -- how easy will it be to forget to renew the certificates if you had to do it manually -- Cron lets us automate the renewal process. A cron job is supplied which runs certbot renew followed by telling NGINX to reload its configuration files.

Why bring these elements together in one container? It seemed best to combine NGINX, Cron and Lets Encrypt in one Docker container. For the cron job to tell NGINX to reload config files requires sending a SIGHUP signal. That means the cron job must be executing on the same container where NGINX is sitting. Which then means Cron must be executing in the same container.

By combining these things, Cronginx can handle SSL certificate provisioning for one or more domain names, with automatic certificate renewal, while requiring a simple non-complicated NGINX configuration for each domain.

The Cronginx implementation is described in Manage Letsenrypt HTTPS/SSL certificates with a Docker container using Cron, Nginx, and Certbot

In this article we'll go over its practical use, and show how to handle several domains. I've been using this technique for over 2 years. For an earlier implementation, see HTTPS with nginx, using Lets Encrypt, proxying to Gogs and Jenkins back-end services, and Scheduling background tasks using cron in a Docker container

Getting starting with Cronginx

As if to prove I'm not a marketeer, Cronginx is a name I concocted for this Docker container where the primary goal is integrating Lets Encrypt with NGINX.

• Docker Repo: https://hub.docker.com/r/robogeek/cronginx
• Github Repo: https://github.com/robogeek/cronginx

This is not exactly a container where you just run it. That's because there is a small amount of required configuration, so let's talk about that for a second.

• Lets Encrypt CLI tool: This container integrates the Certbot CLI tool for Lets Encrypt for you to use for provisioning and renewing SSL certificates. The tool requires two directories:
  • /etc/letsencrypt - where it stores data to track domains that have been registered and the status of the SSL certificates
  • /webroots/DOMAIN/.well-known - where it interfaces with the web server (NGINX) to handle the HTTP-01 challenge
• NGINX Configuration files: For NGINX to serve domains it needs configuration files. The directory for this in the parent nginx container is /etc/nginx/conf.d. We will store one config file in that directory per hosted domain.

The earlier posting has details of how Cronginx works and the purpose of those directories.

For that purpose consider this docker-compose.yml:

version: '3'

services:
  nginx:
    image: robogeek/cronginx
    container_name: cronginx
    networks:
      - buildnet
      - cloudnet
      - gitea
    volumes:
        - ./etc-letsencrypt:/etc/letsencrypt
        - ./webroots:/webroots
        - ./nginx-conf-d:/etc/nginx/conf.d
        - ./home.DOMAIN:/www/home.DOMAIN
    ports:
      - '80:80'
      - '443:443'

networks:
  buildnet:
    external: true
  cloudnet:
    external: true
  gitea:
    external: true

My suggestion is to use a Docker Compose file like this. This references the Docker image stored on Docker Hub, and gives the resulting container a nice name.

For the networks entries, those correspond to the networks created by the various services being run on the server. In my case these are:

• buildnet is for the existing Gogs installation, and gitea is for the Gitea installation that will replace Gogs
• cloudnet is for the NextCloud container

Each of the networks are declared in the corresponding docker-compose.yml that's used for each individual service. In this container we can declare those as external networks and gain access to them.

Since this handles both HTTP and HTTPS ports, it is exporting ports 80 and 443.

The volumes are mostly what was just discussed. There is an additional volume that is to serve as a simple HTML website for the master domain. In the next section let me explain what I mean by that.

Pointing domains to your home (or office) network

I created Cronginx for self-hosting several services on my home network. Clearly Cronginx could be used for any circumstance where you have multiple domains to set up HTTPS reverse proxies using NGINX. But let's consider the circumstance I have.

I have a stereotypical DSL connection to my home, with the only uniqueness being that this Internet Service Provider gives fixed IP address. That means I don't have to play games with dynamic DNS services.

For services on my home network, I've designated one of my domains to use subdomains for each service. These are:

• home.DOMAIN.com - The primary domain for my home network. No services are hosted here.
• git.DOMAIN.com - The Gogs instance
• build.DOMAIN.com - The Jenkins build system instance
• gitea.DOMAIN.com - The Gitea instance
• cloud.DOMAIN.com - The NextCloud instance

The master domain name I referred to earlier is home.DOMAIN.com.

In the DNS configuration for these subdomains, each has an A record containing the IP address of my home network. If instead you're using a Dynamic DNS setup, you'll need something more. For example I earlier wrote about DuckDNS, a free dynamic DNS system: Remotely access your home network using free dynamic DNS service from DuckDNS

For the home.DOMAIN.com domain, I use this NGINX configuration file:

server {
    listen       80;
    server_name  home.DOMAIN.com www.home.DOMAIN.com;
    access_log   /var/log/home.DOMAIN.com.access.log  main;
    error_log   /var/log/home.DOMAIN.com.error.log  debug;
    location / {
        root /www/home.DOMAIN.com/;
    }
}

Store it in nginx-conf-d as home.DOMAIN.com.conf.

Refer back to the docker-compose.yml and you'll see that a directory is mounted on /www/home.DOMAIN.com/ in the container. Whether to put a website on that domain is up to you, and this directory is where those HTML files would go. This configuration file listens to HTTP on the domain, and uses the contents of /www/home.DOMAIN.com/ to satisfy requests on that domain.

For the remainder of this, I'll assume you've implemented something like this setup.

Initial NGINX configuration file for domains that aren't registered with Lets Encrypt

There is a significant difference in the NGINX configuration for a domain that's been registered with Lets Encrypt, for which we have SSL certificates, and for a domain that's not been registered. At the moment none of the domains have been registered with Lets Encrypt.

For the sake of explanation let's take one domain, git.DOMAIN.com. This will be for an existing service that could be Gogs, Gitea, or Gitlab. You should already have that service running in its own Docker container, and connected to one of the networks named in the docker-compose.yml.

In the nginx-conf-d directory add a file named git.DOMAIN.com.conf containing:

# HTTP — redirect all traffic to HTTPS
server {
    listen 80;
    # listen [::]:80 default_server ipv6only=on;

    # Here put the domain name the server is to be known as.
    server_name  git.DOMAIN.com www.git.DOMAIN.com;
    access_log   /var/log/git.DOMAIN.com.access.log  main;
    error_log   /var/log/git.DOMAIN.com.error.log  debug;

    # This is for handling the ACME challenge.  Substitute the
    # domain name here.
    location /.well-known/ {
       root /webroots/git.DOMAIN.com/;
    }

    # Use this to proxy for another service
    location / {
       proxy_pass      http://gogs:3000/;
    }
}

This listens to HTTP (port 80) on the domain you've chosen, and has Access and Error logs for the domain.

The /.well-known URL will be handled from a directory that is injected into the container. Lets Encrypt uses this URL in validating each domain that's registered with the service. When we ask Certbot to register a domain, it contacts Lets Encrypt, and Lets Encrypt responds with a coded URL in the /.well-known space. Certbot creates the file to correspond with the URL, and then Lets Encrypt tries to retrieve that file. If successful, Lets Encrypt knows that you own the domain which you're trying to register.

The last entry here is a simple Proxy so that any other request on git.DOMAIN.com will be sent to the back-end server. Notice that the URL for that server uses a simple domain name. This is assuming that the service is also hosted on Docker. Docker will make sure the simple domain name works this way.

Launching Cronginx, and registering a domain

We've discussed creating two NGINX configuration files in the nginx-conf-d directory. The parent NGINX container has a /etc/nginx/nginx.conf file that loads any file in /etc/nginx/conf.d, so therefore that directory is convenient for us. All we need to do is create one or more file in nginx-conf-d for NGINX to use the configuration we specify.

It's helpful to put one file in nfinx-conf-d for each hosted domain. The two we've discussed so far are:

• home.DOMAIN.com.conf -- The master domain for the network.
• git.DOMAIN.com.conf -- The Gogs/Gitea/Gitlab service.

We have the minimum necessary to get started with Cronginx.

Run this:

$ docker-compose up

This brings up the Docker services described in docker-compose.yml. The service will execute in the foreground so that you can see logging output.

The important thing to do next is to register git.DOMAIN.com with Lets Encrypt. To do that we must get a root shell inside the Cronginx container.

To do so, run this:

$ docker ps
...  verify the container exists, and its name
$ docker exec -it cronginx bash

First we run docker ps to verify that the container exists, and the name it was assigned. With the second command we'll start a root shell inside the container.

Once inside the container, and we can poke around, run commands, etc. In particular, we can run this:

root@f6fa12d79613:/scripts# ./register.sh git.DOMAIN.com

The register.sh script is provided with Cronginx to aid with registering domains with Lets Encrypt. It first creates a directory in /webroots/git.DOMAIN.com and then runs this: certbot certonly --webroot -w /webroots/$1 -d $1 (where $1 is the domain name you put on the command line)

The -w option says to use /webroots/git.DOMAIN.com for the /.well-known directory, and the -d option tells Certbot the domain that's being registered.

It will then run the HTTP-01 challenge, meaning it tries to retrieve this:

http://your-domain.com/.well-known/acme-challenge/CODED-FILE-NAME

If successful you'll get output like this:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/YOUR.DOMAIN/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/YOUR.DOMAIN/privkey.pem
   Your cert will expire on 2020-09-19. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
...

If you've followed the instructions correctly this will work correctly. But if it doesn't work, the error messages may be useful. There may be useful messages in the error log. You can also use Curl to retrieve a file from the URL to see if that helps you understand what might have gone wrong.

In any case, let's assume it worked correctly. The next task is to rewrite the configuration file to handle HTTPS. The output gives two file names for the SSL certificates, and NGINX can directly use those files.

You'll see those files are in the /etc/letsencrypt directory. Take a look there and you'll see all kinds of files now available. And if you exit out of the container, you'll find the same files in the etc-letsencrypt directory.

Reconfiguring Cronginx for HTTPS support using SSL certificates from Lets Encrypt

Change the git.DOMAIN.com.conf file to this:

# HTTP — redirect all traffic to HTTPS
server {
    listen 80;
    # listen [::]:80 default_server ipv6only=on;

    # Here put the domain name the server is to be known as.
    server_name  git.DOMAIN.com www.git.DOMAIN.com;
    access_log   /var/log/git.DOMAIN.com.access.log  main;
    error_log   /var/log/git.DOMAIN.com.error.log  debug;

    # This is for handling the ACME challenge.  Substitute the
    # domain name here.
    location /.well-known/ {
       root /webroots/git.DOMAIN.com/;
    }

    # Use this to force a redirect to the SSL/HTTPS site
    return 301 https://$host$request_uri;
}

# HTTPS service
server { # simple reverse-proxy
    # Enable HTTP/2
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    # Substitute here the domain name for the site
    server_name  git.DOMAIN.com www.git.DOMAIN.com;
    access_log   /var/log/git.DOMAIN.com.access.log  main;
    error_log   /var/log/git.DOMAIN.com.error.log  debug;

    # Replication of the ACME challenge handler.  Substitute
    # the domain name.
    location /.well-known/ {
       root /webroots/git.DOMAIN.com/;
    }

    # Use the Let’s Encrypt certificates
    # Substitute in the domain name
    ssl_certificate /etc/letsencrypt/live/git.DOMAIN.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/git.DOMAIN.com/privkey.pem;

    # Use this to proxy to Socket.IO service on the back-end service
    # The reason is that Socket.IO requires upgrading the
    # HTTP/1.1 connection to WebSocket.
    # See:
    # https://stackoverflow.com/questions/29043879/socket-io-with-nginx
#    location ^~ /socket.io/ {
#
#        proxy_set_header X-Real-IP $remote_addr;
#        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#        proxy_set_header Host $http_host;
#        proxy_set_header X-NginX-Proxy false;
#
#        proxy_pass http://gogs:3000;
#        proxy_redirect off;
#
#        proxy_http_version 1.1;
#        proxy_set_header Upgrade $http_upgrade;
#        proxy_set_header Connection "upgrade";
#    }

    # Use this for proxying to a backend service
    # The HTTPS session is terminated at this Proxy.
    # The back end service will see a simple HTTP service.
    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-NginX-Proxy true;
        proxy_pass http://gogs:3000/;
        proxy_ssl_session_reuse off;
        proxy_set_header Host $http_host;
        proxy_cache_bypass $http_upgrade;
        proxy_redirect off;
    }

}

The first part, that listens to HTTPS, is largely the same. Instead of the proxy to the back-end service, it instead sends a permanent redirect to the HTTPS version of the domain.

The second part is what handles the HTTPS version of the domain. This is similar to what we've done, so lets discuss the differences.

The ssl_certificate and ssl_certificate_key lines are where we specify the SSL certificates. The file names here should match what was in the output when registering the domain.

If you are hosting a service that uses Socket.IO, the next section will be useful. Socket.IO requires an Upgrade from HTTP/1.1 to WebSocket, which is what this does.

The last section is a more complete reverse proxy. This handles the HTTPS inbound connection, proxying it to the HTTP back-end service.

Having replaced git.DOMAIN.com.conf with this, you need to let NGINX know to reload the configuration. One way is to restart the Cronginx container like so:

$ docker restart cronginx

That's one way to handle it, simply restart the container. Another way is this:

$ docker exec -it cronginx bash
root@f6fa12d79613:/scripts# kill -HUP `cat /var/run/nginx.pid`

This sends a SIGHUP signal to the NGINX process. When NGINX receives that signal it reloads its configuration files. This way you can reload the configuration without having to restart the container.

Once you do so, visiting your website on the HTTP URL will automatically redirect over to the HTTPS URL.

Handling HTTPS for multiple domains in Cronginx

To handle multiple domains, simply repeat the above steps multiple times. Namely, for each domain:

• Create a DOMAIN.conf file in nginx-conf-d using the HTTP-only pattern for the first configuration file
• Restart the Cronginx container, or send SIGHUP to nginx process to cause configuration reload
• Shell into the Cronginx container, and run register.sh
• Rewrite DOMAIN.conf using the HTTP/HTTPS pattern of the second configuration file
• Restart the Cronginx container, or send SIGHUP to nginx process to cause configuration reload

Automating SSL certificate renewal in Cronginx

One advantage of Cronginx is that certificate renewal automation is built in. Included inside Cronginx is a Cron instance. Included is a preconfigured Cron job that runs certbot renew every couple days.

When this command is run, for each configured domain, a message like this is printed:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/DOMAIN.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not yet due for renewal

This is what's printed, obviously, when the certificate is not due for renewal. I haven't seen the message for when the certificate is due for renewal, but one presumes that it says so, and then proceeds to initiate the renewal process. That process should require revalidation of the domain, hence the necessity to leave the /.well-known URL configured.