; Date: Sun Mar 15 2020

Tags: Docker »»»» Docker Development »»»» Self Hosting

Self-hosting is about hosting Internet services in an easy-to-manage environment. Docker is an excellent tool for managing these kind of services since it is easy to reuse software packages, and it has a fairly strong security barrier around those packages. In many cases we'll just need to install and configure the Docker container. But to ease the task we need a well organized system for managing configuration files, data directories, and other stuff associated with running several Docker containers.

With Docker it's easy to bring up a service and keep it running in the background. You might develop a need to run a local database on your laptop, that can be easily brought up and down as needed. You make a directory somewhere, containing a docker-compose.yml, and you're up and running. You type docker-compose up any time you want the database running, and docker-compose down to shut it down. But what happens as you add more services? How do you organize the configuration files over the long haul? How do you keep track of exactly how everything works?

As is so often the case a little bit of organization can save headaches in the future. The idea is to develop a system to keep track of the files and setup instructions required to run the services you are self-hosting. What follows is close to the system I'm using, but I'll also talk about the general principles. You may or may not want to use this system, but in any case it should inform you on how to best proceed.

Successfully self-hosting several services requires the ability to keep track of the configuration, and setup, of the Docker containers for those services, and to be easily able to upgrade those services. Your data must be kept separate from the container so that the data doesn't vaporize when containers are destroyed and recreated.

I'm making an assumption that Docker is the preferred tool for managing self-hosted services. That's because Docker neatly packages everything required to launch a service, and can keep a service robustly running, while imposing a small overhead on the host computer.

How many services will you be self-hosting?

I started my self-hosting journey thinking about keeping private Git repositories. I have some projects which should not be visible to the whole world, yet I want to use Git to for keep track of changes etc.

The modern approach is to use a cloud hosting system - Github, Gitlab, etc - but to keep a private repository means spending $$'s every month. I did some math and realized that, at a $7/month cost to keep private repositories on Github, it doesn't take that many months to pay for an Intel NUC. It's not just possible Github costs. How much does a premium account on Dropbox or Box or Google Drive cost per month? It might cost $300 to outfit a NUC. If owning that NUC can help you avoid paying for several premium services, then how quickly does the NUC pay for itself?

The Intel NUC plus Gogs and Jenkins gave me a nice system for managing not only the private source repositories, but for managing software builds and other tasks.

But - being a slippery slope - those two services begat others and now I have

Service	Why
Nextcloud	For file management
Nginx	To provide HTTPS and also proxying the services into separate domains
Portainer	To better manage the Docker services
Plex (switching to Jellyfin)	For streaming of video files and other multimedia
Gogs (switching to Gitea)	Github-like user experience of Git hosting
Jenkins	Software builds
Docker Registry	Local registry for Docker containers
MySQL (occasionally)	Databases
Wordpress or Drupal	I use both for public websites on a VPS, and could use a local environment for test deployments of those sites

There's a whole world of software out there. What will you want to self-host?

That Intel NUC is still handling all of that, and I believe it can handle quite a bit more. But, then, I don't put that much of a burden on it.

Making the host computer visible to the public Internet

My Intel NUC is sitting on my desk, connected to the DSL router. While it is useful at home, I also use it when away from home. I use a subdomain like git.example.com and voila, have access to the Gitea server on that NUC even though it is sitting at home.

Any business could do the same. The business has a router connecting to the Internet. A simple computer like a NUC could sit next to that router, and then also be made visible to the Internet.

In my case my DSL provider gives a static IP address as part of the service. I then configured git.example.com, build.example.com, cloud.example.com and so forth to connect to that static IP address. This means, when away from home, using one of those services sends traffic to my DSL router.

For someone with a non-static IP address at home, a dynamic DNS service is required. See Remotely access your home network using free dynamic DNS service from DuckDNS

The DSL router or cable modem normally stops traffic from going into your home network. But a pinhole lets traffic into the home network.

I have configured my DSL router with pinholes for HTTP (port 80), HTTPS (port 443) and other ports, where all such incoming traffic is directed to the Intel NUC.

The Intel NUC is then configured with an NGinx server that recognizes the incoming domain name of any request, and gateways it to the appropriate service.

Installing Docker

Docker is fairly easy to install. I have instructions for this on every operating system:

See Getting started with Docker: Installation, first steps

A directory hierarchy for managing multiple Docker containers

The issue is how to manage an ever-growing set of services to self-host.

What I do is to set aside one directory to be the parent of several directories containing files for managing the containers. Namely:

david@nuc2:~$ ls /home/docker
buildnet  gogs      nc.tar         nginx      registry     start-mariadb-nextcloud.txt
gitea     jellyfin  nextcloud      plex       root         start-nextcloud.txt
gitlab    jenkins   nextcloud-old  portainer  root-backup  start-plex.txt

In /home/docker the intention is to have one directory per project. A project might require multiple containers. What's actually there is a little bit of a mess because of .. er .. history that hasn't been fully cleaned up. The plan is to have this structure:

Service	Directory
Nginx	/home/docker/nginx
Gitea	/home/docker/gitea
Jenkins	/home/docker/jenkins
Nextcloud	/home/docker/nextcloud
Jellyfin	/home/docker/jellyfin
Registry	/home/docker/registry
Portainer	/home/docker/portainer
MySQL	/home/docker/mysql

That's an example of what one might do, and is more or less what actually exists on my computer.

Within each directory we have these things to manage:

• Scripts related to building and upgrading the container(s)
• Configuration files used inside the container(s)
• Data directories used by the container(s)

As an example consider:

david@nuc2:~$ ls /home/docker/gitea
app.ini  data  docker-compose.yml  gitea-data  README.md
david@nuc2:~$ ls /home/docker/portainer/
data  docker-compose.yml

The Portainer directory was the most recent package I've set up. The data directory is used for the data maintained by Portainer, while the docker-compose.yml documents how Portainer is setup.

For Gitea, it is almost as clean, but there are two data directories. Right now I don't remember which is actually in use. The other is a left-over from an experiment which hasn't been cleaned up. The README.md contains some notes-to-self.

Therefore we can define a general architecture:

• /home/docker/SERVICE-NAME contains everything related to the service
• Use docker-compose.yml to document the configuration
• For any configuration files, store them outside the container and use Docker to mount the config file inside the container
• For any data directory, do the same
• Use a README.md to record any notes or instructions
• Use shell scripts or build automation tools to record any shell commands that are required

With Portainer we can avoid docker-compose.yml

With Portainer we could skip creating a docker-compose.yml file and simply point-and-click our way to setting up a container. Some of us may want to do it this way because it simplifies the process. Not all of us have the patience to learn how to create a Docker Compose file, after all.

The general architecture given in the previous section works for someone who can handle docker-compose.yml. Therefore the general architecture becomes:

• /home/docker/SERVICE-NAME contains everything related to the service
• For any configuration files, store them outside the container and use Docker to mount the config file inside the container
• For any data directory, do the same
• Use a README.md to record any notes or instructions

This will work for

Starting and stopping services

The advantage of installing Portainer is it gives you a GUI with which to start/stop services, inspect their status, view the logs, connect to a shell inside the container, and more.

Upgrading services

Using Git to track revisions