Tags: Docker »»»» git »»»» Software Development »»»» Web Developer Resources »»»» Node Web Development
The Raspberry Pi is an amazing little computer that, while it's targeted at the DIY Hardware Maker, it is a full-fledged Linux computer that can be used to run services that used to require much bigger and more expensive computers. How long ago were office servers required to be $4000 systems from the likes of Dell Computers? It seems that the Raspberry Pi (and other tiny computers) can perform the same tasks at a low cost with minuscule energy requirements. To this end I'm setting up Gogs (a github-like server for Git repositories) on a Raspberry Pi. As I worked on the project it seemed most straightforward to use Docker to manage the Gogs process, and therefore the project became setting up Docker on Raspberry Pi to run other services.
While there have been experimental Docker implementations on Raspbian or other Raspberry Pi-focused distro's,
in August the Raspberry Pi Foundation announced with the Docker organization official support for Docker on Raspbian. This is cool because it's so easy to build a small network of Raspberry Pi's, and Docker Swarm makes it easy to set up a slew of systems with an array of services. Even if you wouldn't use Raspberry Pi's in production (using an SD-drive based system for production use sounds silly) it's feasible to use them for low cost experimentation or in situations where downtime (e.g. dead SD card) is not a problem.
Because the Raspberry Pi is an ARM system, and the vast majority of Docker images are for Intel x86 systems, we'll have to carefully pick the Docker images for ARM. Fortunately there is an ARM image for Gogs (gogs/gogs-rpi).
Setting up Docker on Raspbian
You should have first installed Raspbian on your Raspberry Pi - I assume you know how to do this, but if not the instructions are on the raspberrypi.org website.
Installing Docker is now as easy as the following:
$ curl -sSL https://get.docker.com | sh
A
suggested first thing to kick the tires of Docker is this:
$ docker run -ti resin/rpi-raspbian:jessie-20160831 /bin/bash
root@0c68918c47d3:/# cat /etc/issue
Raspbian GNU/Linux 8 \n \l
root@0c68918c47d3:/# exit
Making sure Docker runs when the system reboots
That didn't make sure Docker began when the system rebooted.
Follow the instructions here:
https://docs.docker.com/engine/admin/systemd/
$ sudo systemctl start docker
$ sudo systemctl enable docker
Installing Gogs
The documentation for running Gogs in Docker is here:
https://github.com/gogits/gogs/tree/master/docker
However, it is written assuming an Intel x86 based Linux, but we have an ARM system. Therefore we must modify the instructions a little, and we also want this container to automatically start on system reboot.
$ docker pull gogs/gogs-rpi
This is the Raspberry Pi Gogs image. This is our first deviation from the official instructions.
$ mkdir -p /var/gogs
Make the suggested directory for holding Gogs data.
$ docker run --name=gogs --restart always -p 10022:22 -p 80:3000 -v /var/gogs:/data gogs/gogs-rpi
The --restart always flag configures the restart policy so it will, as implied, restart the service automatically. The next argument configures port mapping for the SSH service so it'll appear at port 10022, which will be important when using an SSH URL to access a Git repository. The next makes the service appear on port 80. The next tells Gogs to use the data directory we made. The last references the correct image.
This will start Gogs interactively showing you an activity log. You'll be able to access the service at http://localhost while running a browser ON the Raspberry Pi, or else specifying the IP address of the Raspberry Pi while on another computer.
You go through the normal setup of Gogs at this point. I configured it to use SQLITE3 as the database because there isn't a MySQL available. Otherwise the options are left as default. After you reach the Gogs login page you'll need to create a user account - and note this will be the first user account on that server, and therefore it will have administrator privileges.
You can if you like go through creating a Git repository on this server, import another repository, etc.
$ docker start gogs
You can kill the interactive Gogs container, and then restart it this way non-interactively.
You can also reboot the computer and make sure that both the Docker service and the Gogs container restarted automatically.
Extra Credit -- Adding storage to the Raspberry Pi
It's not a good idea to trust an SD-card based computer to be reliable. Therefore it's useful to set up a plan to add more reliable storage to the Raspberry Pi and/or automatically backup the repositories to a reliable server.
I have a stack of 2.5" laptop-style hard drives that had formerly been in MacBook Pro's but were left over. They're installed in portable bus-powered USB drive enclosures making it easy to use them as a portable drive. They're probably low-powered enough to run off a Raspberry Pi USB port -- I've now set up Gogs to use the drive, and it seems to be handling itself perfectly well.
The first step is plugging the disk into your desktop computer and verifying it is working well. I did so and selected one of the drives.
Second, plug it into the Raspberry Pi -- for a Mac drive it won't be recognized, but a Windows drive probably will be. However, seeing that this is Linux we want to format the drive as EXT4.
The method I chose is to run parted -- a particularly geek-friendly-user-hostile application that happens to get the job done.
# parted
GNU Parted 3.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) help
align-check TYPE N check partition N for TYPE(min|opt) alignment
help [COMMAND] print general help, or help on COMMAND
mklabel,mktable LABEL-TYPE create a new disklabel (partition table)
mkpart PART-TYPE [FS-TYPE] START END make a partition
name NUMBER NAME name partition NUMBER as NAME
print [devices|free|list,all|NUMBER] display the partition table, available devices, free space, all found partitions, or a particular partition
quit exit program
rescue START END rescue a lost partition near START and END
resizepart NUMBER END resize partition NUMBER
rm NUMBER delete partition NUMBER
select DEVICE choose the device to edit
disk_set FLAG STATE change the FLAG on selected device
disk_toggle [FLAG] toggle the state of FLAG on selected device
set NUMBER FLAG STATE change the FLAG on partition NUMBER
toggle [NUMBER [FLAG]] toggle the state of FLAG on partition NUMBER
unit UNIT set the default unit to UNIT
version display the version number and copyright information of GNU Parted
(parted) print devices
/dev/sda (250GB)
/dev/mmcblk0 (31.9GB)
(parted) print
Model: Hitachi HTS545025B9SA02 (scsi)
Disk /dev/sda: 250GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
CURRENT PARTITION MAP
This gives you a taste of how the application works. You have a command prompt (parted) and some obscure commands. There are two things to do:
- Use the `rm` command to delete the existing partitions. I typed "`rm 1`" and "`rm 2`" because my drive had two partitions.
- Create a new partition (`mkpart`) with type `ext4` starting at 0 and going to the end of the disk, or address "-1s". When I did this, I got a warning this wasn't the optimum settings, but this program is so obtuse it doesn't help me understand what the optimum setting are. Bleah.
Once you've done that exit parted, and at the command line run:
$ sudo mkdir /mnt/usbdrive
$ sudo mkfs.ext4 /dev/sda1
$ sudo mount /dev/sda1 /mnt/usbdrive
This creates an EXT4 file system on the drive, and then mounts the drive to the named directory.
Once you've done this you can verify as so:
$ df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 30545276 4366592 24877268 15% /
devtmpfs 469536 0 469536 0% /dev
tmpfs 473868 0 473868 0% /dev/shm
tmpfs 473868 6492 467376 2% /run
tmpfs 5120 4 5116 1% /run/lock
tmpfs 473868 0 473868 0% /sys/fs/cgroup
/dev/mmcblk0p1 64456 21224 43232 33% /boot
/dev/sda1 240234132 924664 227083160 1% /mnt/usbdrive
tmpfs 94776 0 94776 0% /run/user/1000
overlay 30545276 4366592 24877268 15% /var/lib/docker/overlay/40be435ac259593d8232099d1ecf9058e925f115900a463dc74467fdd516aca0/merged
shm 65536 0 65536 0% /var/lib/docker/containers/4cd93d7c29ea62653bc914bdbfc105579bd265e53fe12119249f74c215825c58/shm
The important bit of this is the line for /dev/sda1. The last three lines are interesting because they're used behind the scenes by Docker.
Earlier we did this to create a place for Gogs to put its data:
$ mkdir -p /var/gogs
The Docker container is still pointing to this directory and perhaps we want to leave it that way. But we want to get the Gogs data to reside on this drive. Rather than change the Docker container to mount a different directory I did a symlink.
$ sudo su -
# docker stop gogs
# cd /var
# tar cf - gogs | (cd /mnt/usbdrive; tar xvf -)
# rm -rf gogs
# ln -s /mnt/usbdrive/gogs gogs
# ls -l gogs
lrwxrwxrwx 1 root root 18 Oct 9 14:06 gogs -> /mnt/usbdrive/gogs
# docker start gogs
This copies the data directory over to the USB drive, and creates a symlink.
You should be able to visit the Gogs web user interface and see that everything still functions correctly.
Memory considerations with Gogs on Raspberry Pi - WORKAROUND, direct SSH access to repository
In using the Raspberry Pi, I've found it easily runs out of memory. Hey, Bill Gates, we're way past the 640 kilobytes of memory that you claimed would be sufficient for everyone. The Raspberry Pi has about 1 gigabyte of memory, or about 1500x what you said would be sufficient, and it's not sufficient. With the Gogs server running, an X server running, a terminal session displaying on the X server, a Chromium browser, and a few tabs open, the system ran out of memory, running excruciatingly slow, and telling me the tabs were unresponsive, etc. Therefore I don't think Raspberry Pi's are ready as a serious desktop computer replacement. To make that work, you have to be really patient and cognizant of memory limits.
In light of that I've configured the Raspberry Pi to login directly to the command line, no X server, to save memory. The Docker instance is already configured to start automatically, and start Gogs automatically. That's all which is necessary.
The next issue is accessing the GIT repository. I have a large (about 1 GB) repository to upload -- it's managed by another Gogs instance running on a MacBook Pro. In theory all I'd need to do is switch the origin and do a push --mirror origin and the repository would be transferred to the new server.
$ git remote add origin http://192.168.1.97/david/hmp.git
$ git push --mirror origin
Username for 'http://192.168.1.97': david
Password for 'http://david@192.168.1.97':
Counting objects: 3809, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (1978/1978), done.
error: unable to rewind rpc post data - try increasing http.postBuffer
error: RPC failed; curl 56 Recv failure: Connection reset by peer
fatal: The remote end hung up unexpectedly
Writing objects: 100% (3809/3809), 842.42 MiB | 34.01 MiB/s, done.
Total 3809 (delta 1781), reused 3801 (delta 1776)
fatal: The remote end hung up unexpectedly
Everything up-to-date
This was the result - an obtuse error message. I tried cloning that repository to check, but was told this:
$ git clone http://192.168.1.97/david/hmp.git hmp2
Cloning into 'hmp2'...
warning: You appear to have cloned an empty repository.
Checking connectivity... done.
In other words, the "push --mirror" did not work.
After some thinking, and discussing with my girlfriend, it came to mind that this entailed more memory consumption since the Gogs server was in the middle of the transaction. If instead I were to use an SSH URL it would directly access the repository and consume less memory and would possibly work.
Unfortunately Gogs tells me this is the SSH URL: git@localhost:david/hmp.git
I know that isn't true since a) the Gogs server is on the RPI, while I'm on one of my laptops, b) even that wouldn't work if I were at the command line of that RPI since the SSH port for the Gogs server is actually at port# 10022 -- because of the docker command way up there.
After some experimenting I found the following to work.
First, I had to upload my SSH key from this laptop to the Gogs server on the RPI. You do this in the Gogs web interface - in the upper right corner where your Avatar appears, click to use the dropdown menu, go to "Your Settings" and there will be a section for SSH keys. Add the public SSH key (in ~/.ssh/id_dsa.pub - or some such) as an SSH key.
Next you do this:
$ git remote add origin ssh://git@192.168.1.97:10022/david/hmp.git
$ git push --mirror origin
Counting objects: 3809, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (1978/1978), done.
Writing objects: 100% (3809/3809), 842.42 MiB | 1022.00 KiB/s, done.
Total 3809 (delta 1781), reused 3801 (delta 1776)
To ssh://git@192.168.1.97:10022/david/hmp.git
* [new branch] master -> master
To decode this ... the SSH server we need to access is the one in the Gogs container on the Raspberry Pi. In the docker run command we defined that to appear at port# 10022 on the host. Hence, "git@IP-ADDRESS:10022" is the correct address in the SSH URL. The tail part of the SSH URL has to match whatever Gogs tells you in its web-based user interface. In this case it is /david/hmp.git -- you would plug in your user name and repository name.
You'll see it pushed 850 MB of data across the WiFi at about 1 MB/second throughput. Not too shabby? The important thing is this worked decently well.
Conclusion
I got to a successful conclusion - but I'm left wondering whether this will work for the long run. By contrast, I've found some used older generation Intel Nuc's on eBay for $99, that with under $100 in parts could be an 8GB Linux server and give better performance without the memory limits of the Raspberry Pi. Hurm.
