Tags: Docker »»»» Docker Development »»»» Self Hosting
Self-hosting is about hosting Internet services in an easy-to-manage environment. Docker is an excellent tool for managing these kind of services since it is easy to reuse software packages, and it has a fairly strong security barrier around those packages. In many cases we'll just need to install and configure the Docker container. But to ease the task we need a well organized system for managing configuration files, data directories, and other stuff associated with running several Docker containers.
With Docker it's easy to bring up a service and keep it running in the background. You might develop a need to run a local database on your laptop, that can be easily brought up and down as needed. You make a directory somewhere, containing a docker-compose.yml, and you're up and running. You type docker-compose up any time you want the database running, and docker-compose down to shut it down. But what happens as you add more services? How do you organize the configuration files over the long haul? How do you keep track of exactly how everything works?
As is so often the case a little bit of organization can save headaches in the future. The idea is to develop a system to keep track of the files and setup instructions required to run the services you are self-hosting. What follows is close to the system I'm using, but I'll also talk about the general principles. You may or may not want to use this system, but in any case it should inform you on how to best proceed.
Successfully self-hosting several services requires the ability to keep track of the configuration, and setup, of the Docker containers for those services, and to be easily able to upgrade those services. Your data must be kept separate from the container so that the data doesn't vaporize when containers are destroyed and recreated.
I'm making an assumption that Docker is the preferred tool for managing self-hosted services. That's because Docker neatly packages everything required to launch a service, and can keep a service robustly running, while imposing a small overhead on the host computer.
How many services will you be self-hosting?
I started my self-hosting journey thinking about keeping private Git repositories. I have some projects which should not be visible to the whole world, yet I want to use Git to for keep track of changes etc.
The modern approach is to use a cloud hosting system - Github, Gitlab, etc - but to keep a private repository means spending $$'s every month. I did some math and realized that, at a $7/month cost to keep private repositories on Github, it doesn't take that many months to pay for an Intel NUC. It's not just possible Github costs. How much does a premium account on Dropbox or Box or Google Drive cost per month? It might cost $300 to outfit a NUC. If owning that NUC can help you avoid paying for several premium services, then how quickly does the NUC pay for itself?
The Intel NUC plus Gogs and Jenkins gave me a nice system for managing not only the private source repositories, but for managing software builds and other tasks.
But - being a slippery slope - those two services begat others and now I have
| Service | Why |
|---|---|
| Nextcloud | For file management |
| Nginx | To provide HTTPS and also proxying the services into separate domains |
| Portainer | To better manage the Docker services |
| Plex (switching to Jellyfin) | For streaming of video files and other multimedia |
| Gogs (switching to Gitea) | Github-like user experience of Git hosting |
| Jenkins | Software builds |
| Docker Registry | Local registry for Docker containers |
| MySQL (occasionally) | Databases |
| Wordpress or Drupal | I use both for public websites on a VPS, and could use a local environment for test deployments of those sites |
There's a whole world of software out there. What will you want to self-host?
That Intel NUC is still handling all of that, and I believe it can handle quite a bit more. But, then, I don't put that much of a burden on it.
Making the host computer visible to the public Internet
My Intel NUC is sitting on my desk, connected to the DSL router. While it is useful at home, I also use it when away from home. I use a subdomain like git.example.com and voila, have access to the Gitea server on that NUC even though it is sitting at home.
Any business could do the same. The business has a router connecting to the Internet. A simple computer like a NUC could sit next to that router, and then also be made visible to the Internet.
In my case my DSL provider gives a static IP address as part of the service. I then configured git.example.com, build.example.com, cloud.example.com and so forth to connect to that static IP address. This means, when away from home, using one of those services sends traffic to my DSL router.
For someone with a non-static IP address at home, a dynamic DNS service is required. See Remotely access your home network using free dynamic DNS service from DuckDNS
The DSL router or cable modem normally stops traffic from going into your home network. But a pinhole lets traffic into the home network.
I have configured my DSL router with pinholes for HTTP (port 80), HTTPS (port 443) and other ports, where all such incoming traffic is directed to the Intel NUC.
The Intel NUC is then configured with an NGinx server that recognizes the incoming domain name of any request, and gateways it to the appropriate service.
Installing Docker
Docker is fairly easy to install. I have instructions for this on every operating system:
See Getting started with Docker: Installation, first steps
A directory hierarchy for managing multiple Docker containers
The issue is how to manage an ever-growing set of services to self-host.
What I do is to set aside one directory to be the parent of several directories containing files for managing the containers. Namely:
david@nuc2:~$ ls /home/docker
buildnet gogs nc.tar nginx registry start-mariadb-nextcloud.txt
gitea jellyfin nextcloud plex root start-nextcloud.txt
gitlab jenkins nextcloud-old portainer root-backup start-plex.txt
In /home/docker the intention is to have one directory per project. A project might require multiple containers. What's actually there is a little bit of a mess because of .. er .. history that hasn't been fully cleaned up. The plan is to have this structure:
| Service | Directory |
|---|---|
| Nginx | /home/docker/nginx |
| Gitea | /home/docker/gitea |
| Jenkins | /home/docker/jenkins |
| Nextcloud | /home/docker/nextcloud |
| Jellyfin | /home/docker/jellyfin |
| Registry | /home/docker/registry |
| Portainer | /home/docker/portainer |
| MySQL | /home/docker/mysql |
That's an example of what one might do, and is more or less what actually exists on my computer.
Within each directory we have these things to manage:
- Scripts related to building and upgrading the container(s)
- Configuration files used inside the container(s)
- Data directories used by the container(s)
As an example consider:
david@nuc2:~$ ls /home/docker/gitea
app.ini data docker-compose.yml gitea-data README.md
david@nuc2:~$ ls /home/docker/portainer/
data docker-compose.yml
The Portainer directory was the most recent package I've set up. The data directory is used for the data maintained by Portainer, while the docker-compose.yml documents how Portainer is setup.
For Gitea, it is almost as clean, but there are two data directories. Right now I don't remember which is actually in use. The other is a left-over from an experiment which hasn't been cleaned up. The README.md contains some notes-to-self.
Therefore we can define a general architecture:
/home/docker/SERVICE-NAMEcontains everything related to the service- Use
docker-compose.ymlto document the configuration - For any configuration files, store them outside the container and use Docker to mount the config file inside the container
- For any data directory, do the same
- Use a
README.mdto record any notes or instructions - Use shell scripts or build automation tools to record any shell commands that are required
With Portainer we can avoid docker-compose.yml
With Portainer we could skip creating a docker-compose.yml file and simply point-and-click our way to setting up a container. Some of us may want to do it this way because it simplifies the process. Not all of us have the patience to learn how to create a Docker Compose file, after all.
The general architecture given in the previous section works for someone who can handle docker-compose.yml. Therefore the general architecture becomes:
/home/docker/SERVICE-NAMEcontains everything related to the service- For any configuration files, store them outside the container and use Docker to mount the config file inside the container
- For any data directory, do the same
- Use a
README.mdto record any notes or instructions
This will work for
Starting and stopping services
The advantage of installing Portainer is it gives you a GUI with which to start/stop services, inspect their status, view the logs, connect to a shell inside the container, and more.
Upgrading services
Using Git to track revisions
