Make your own Raspberry Pi git repository server with Gogs and Docker

By: ; Date: 2016-10-09 13:38

Tags: Docker » git » Software Development » Web Developer Resources » Node Web Development

The Raspberry Pi is an amazing little computer that, while it's targeted at the DIY Hardware Maker, it is a full-fledged Linux computer that can be used to run services that used to require much bigger and more expensive computers. How long ago were office servers required to be $4000 systems from the likes of Dell Computers? It seems that the Raspberry Pi (and other tiny computers) can perform the same tasks at a low cost with minuscule energy requirements. To this end I'm setting up Gogs (a github-like server for Git repositories) on a Raspberry Pi. As I worked on the project it seemed most straightforward to use Docker to manage the Gogs process, and therefore the project became setting up Docker on Raspberry Pi to run other services.

While there have been experimental Docker implementations on Raspbian or other Raspberry Pi-focused distro's, in August the Raspberry Pi Foundation announced with the Docker organization official support for Docker on Raspbian. This is cool because it's so easy to build a small network of Raspberry Pi's, and Docker Swarm makes it easy to set up a slew of systems with an array of services. Even if you wouldn't use Raspberry Pi's in production (using an SD-drive based system for production use sounds silly) it's feasible to use them for low cost experimentation or in situations where downtime (e.g. dead SD card) is not a problem.

Because the Raspberry Pi is an ARM system, and the vast majority of Docker images are for Intel x86 systems, we'll have to carefully pick the Docker images for ARM. Fortunately there is an ARM image for Gogs (gogs/gogs-rpi).

Setting up Docker on Raspbian

You should have first installed Raspbian on your Raspberry Pi - I assume you know how to do this, but if not the instructions are on the raspberrypi.org website.

Installing Docker is now as easy as the following:

$ curl -sSL https://get.docker.com | sh

A suggested first thing to kick the tires of Docker is this:

$ docker run -ti resin/rpi-raspbian:jessie-20160831 /bin/bash

root@0c68918c47d3:/# cat /etc/issue  
Raspbian GNU/Linux 8 \n \l

root@0c68918c47d3:/# exit  

Making sure Docker runs when the system reboots

That didn't make sure Docker began when the system rebooted.

Follow the instructions here: https://docs.docker.com/engine/admin/systemd/

$ sudo systemctl start docker
$ sudo systemctl enable docker

Installing Gogs

The documentation for running Gogs in Docker is here: https://github.com/gogits/gogs/tree/master/docker

However, it is written assuming an Intel x86 based Linux, but we have an ARM system. Therefore we must modify the instructions a little, and we also want this container to automatically start on system reboot.

$ docker pull gogs/gogs-rpi

This is the Raspberry Pi Gogs image. This is our first deviation from the official instructions.

$ mkdir -p /var/gogs

Make the suggested directory for holding Gogs data.

$ docker run --name=gogs --restart always -p 10022:22 -p 80:3000 -v /var/gogs:/data gogs/gogs-rpi

The --restart always flag configures the restart policy so it will, as implied, restart the service automatically. The next argument configures port mapping for the SSH service so it'll appear at port 10022, which will be important when using an SSH URL to access a Git repository. The next makes the service appear on port 80. The next tells Gogs to use the data directory we made. The last references the correct image.

This will start Gogs interactively showing you an activity log. You'll be able to access the service at http://localhost while running a browser ON the Raspberry Pi, or else specifying the IP address of the Raspberry Pi while on another computer.

You go through the normal setup of Gogs at this point. I configured it to use SQLITE3 as the database because there isn't a MySQL available. Otherwise the options are left as default. After you reach the Gogs login page you'll need to create a user account - and note this will be the first user account on that server, and therefore it will have administrator privileges.

You can if you like go through creating a Git repository on this server, import another repository, etc.

$ docker start gogs

You can kill the interactive Gogs container, and then restart it this way non-interactively.

You can also reboot the computer and make sure that both the Docker service and the Gogs container restarted automatically.

Extra Credit -- Adding storage to the Raspberry Pi

It's not a good idea to trust an SD-card based computer to be reliable. Therefore it's useful to set up a plan to add more reliable storage to the Raspberry Pi and/or automatically backup the repositories to a reliable server.

I have a stack of 2.5" laptop-style hard drives that had formerly been in MacBook Pro's but were left over. They're installed in portable bus-powered USB drive enclosures making it easy to use them as a portable drive. They're probably low-powered enough to run off a Raspberry Pi USB port -- I've now set up Gogs to use the drive, and it seems to be handling itself perfectly well.

The first step is plugging the disk into your desktop computer and verifying it is working well. I did so and selected one of the drives.

Second, plug it into the Raspberry Pi -- for a Mac drive it won't be recognized, but a Windows drive probably will be. However, seeing that this is Linux we want to format the drive as EXT4.

The method I chose is to run parted -- a particularly geek-friendly-user-hostile application that happens to get the job done.

# parted
GNU Parted 3.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) help                                                             
  align-check TYPE N                        check partition N for TYPE(min|opt) alignment
  help [COMMAND]                           print general help, or help on COMMAND
  mklabel,mktable LABEL-TYPE               create a new disklabel (partition table)
  mkpart PART-TYPE [FS-TYPE] START END     make a partition
  name NUMBER NAME                         name partition NUMBER as NAME
  print [devices|free|list,all|NUMBER]     display the partition table, available devices, free space, all found partitions, or a particular partition
  quit                                     exit program
  rescue START END                         rescue a lost partition near START and END
  resizepart NUMBER END                    resize partition NUMBER
  rm NUMBER                                delete partition NUMBER
  select DEVICE                            choose the device to edit
  disk_set FLAG STATE                      change the FLAG on selected device
  disk_toggle [FLAG]                       toggle the state of FLAG on selected device
  set NUMBER FLAG STATE                    change the FLAG on partition NUMBER
  toggle [NUMBER [FLAG]]                   toggle the state of FLAG on partition NUMBER
  unit UNIT                                set the default unit to UNIT
  version                                  display the version number and copyright information of GNU Parted
(parted) print devices                                                    
/dev/sda (250GB)
/dev/mmcblk0 (31.9GB)
(parted) print                                                            
Model: Hitachi HTS545025B9SA02 (scsi)
Disk /dev/sda: 250GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End    Size   File system  Name  Flags

     CURRENT PARTITION MAP

This gives you a taste of how the application works. You have a command prompt (parted) and some obscure commands. There are two things to do:

  • Use the `rm` command to delete the existing partitions. I typed "`rm 1`" and "`rm 2`" because my drive had two partitions.
  • Create a new partition (`mkpart`) with type `ext4` starting at 0 and going to the end of the disk, or address "-1s". When I did this, I got a warning this wasn't the optimum settings, but this program is so obtuse it doesn't help me understand what the optimum setting are. Bleah.

Once you've done that exit parted, and at the command line run:

$ sudo mkdir /mnt/usbdrive
$ sudo mkfs.ext4 /dev/sda1
$ sudo mount /dev/sda1 /mnt/usbdrive

This creates an EXT4 file system on the drive, and then mounts the drive to the named directory.

Once you've done this you can verify as so:

$ df -k
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root       30545276 4366592  24877268  15% /
devtmpfs          469536       0    469536   0% /dev
tmpfs             473868       0    473868   0% /dev/shm
tmpfs             473868    6492    467376   2% /run
tmpfs               5120       4      5116   1% /run/lock
tmpfs             473868       0    473868   0% /sys/fs/cgroup
/dev/mmcblk0p1     64456   21224     43232  33% /boot
/dev/sda1      240234132  924664 227083160   1% /mnt/usbdrive
tmpfs              94776       0     94776   0% /run/user/1000
overlay         30545276 4366592  24877268  15% /var/lib/docker/overlay/40be435ac259593d8232099d1ecf9058e925f115900a463dc74467fdd516aca0/merged
shm                65536       0     65536   0% /var/lib/docker/containers/4cd93d7c29ea62653bc914bdbfc105579bd265e53fe12119249f74c215825c58/shm

The important bit of this is the line for /dev/sda1. The last three lines are interesting because they're used behind the scenes by Docker.

Earlier we did this to create a place for Gogs to put its data:

$ mkdir -p /var/gogs

The Docker container is still pointing to this directory and perhaps we want to leave it that way. But we want to get the Gogs data to reside on this drive. Rather than change the Docker container to mount a different directory I did a symlink.

$ sudo su -
# docker stop gogs
# cd /var
# tar cf - gogs | (cd /mnt/usbdrive; tar xvf -)
# rm -rf gogs
# ln -s /mnt/usbdrive/gogs gogs
# ls -l gogs
lrwxrwxrwx 1 root root 18 Oct  9 14:06 gogs -> /mnt/usbdrive/gogs
# docker start gogs

This copies the data directory over to the USB drive, and creates a symlink.

You should be able to visit the Gogs web user interface and see that everything still functions correctly.

Memory considerations with Gogs on Raspberry Pi - WORKAROUND, direct SSH access to repository

In using the Raspberry Pi, I've found it easily runs out of memory. Hey, Bill Gates, we're way past the 640 kilobytes of memory that you claimed would be sufficient for everyone. The Raspberry Pi has about 1 gigabyte of memory, or about 1500x what you said would be sufficient, and it's not sufficient. With the Gogs server running, an X server running, a terminal session displaying on the X server, a Chromium browser, and a few tabs open, the system ran out of memory, running excruciatingly slow, and telling me the tabs were unresponsive, etc. Therefore I don't think Raspberry Pi's are ready as a serious desktop computer replacement. To make that work, you have to be really patient and cognizant of memory limits.

In light of that I've configured the Raspberry Pi to login directly to the command line, no X server, to save memory. The Docker instance is already configured to start automatically, and start Gogs automatically. That's all which is necessary.

The next issue is accessing the GIT repository. I have a large (about 1 GB) repository to upload -- it's managed by another Gogs instance running on a MacBook Pro. In theory all I'd need to do is switch the origin and do a push --mirror origin and the repository would be transferred to the new server.

$ git remote add origin http://192.168.1.97/david/hmp.git
$ git push --mirror origin
Username for 'http://192.168.1.97': david
Password for 'http://david@192.168.1.97':
Counting objects: 3809, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (1978/1978), done.
error: unable to rewind rpc post data - try increasing http.postBuffer
error: RPC failed; curl 56 Recv failure: Connection reset by peer
fatal: The remote end hung up unexpectedly
Writing objects: 100% (3809/3809), 842.42 MiB | 34.01 MiB/s, done.
Total 3809 (delta 1781), reused 3801 (delta 1776)
fatal: The remote end hung up unexpectedly
Everything up-to-date

This was the result - an obtuse error message. I tried cloning that repository to check, but was told this:

$ git clone http://192.168.1.97/david/hmp.git hmp2
Cloning into 'hmp2'...
warning: You appear to have cloned an empty repository.
Checking connectivity... done.

In other words, the "push --mirror" did not work.

After some thinking, and discussing with my girlfriend, it came to mind that this entailed more memory consumption since the Gogs server was in the middle of the transaction. If instead I were to use an SSH URL it would directly access the repository and consume less memory and would possibly work.

Unfortunately Gogs tells me this is the SSH URL: git@localhost:david/hmp.git

I know that isn't true since a) the Gogs server is on the RPI, while I'm on one of my laptops, b) even that wouldn't work if I were at the command line of that RPI since the SSH port for the Gogs server is actually at port# 10022 -- because of the docker command way up there.

After some experimenting I found the following to work.

First, I had to upload my SSH key from this laptop to the Gogs server on the RPI. You do this in the Gogs web interface - in the upper right corner where your Avatar appears, click to use the dropdown menu, go to "Your Settings" and there will be a section for SSH keys. Add the public SSH key (in ~/.ssh/id_dsa.pub - or some such) as an SSH key.

Next you do this:

$ git remote add origin ssh://git@192.168.1.97:10022/david/hmp.git
$ git push --mirror origin
Counting objects: 3809, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (1978/1978), done.
Writing objects: 100% (3809/3809), 842.42 MiB | 1022.00 KiB/s, done.
Total 3809 (delta 1781), reused 3801 (delta 1776)
To ssh://git@192.168.1.97:10022/david/hmp.git
 * [new branch]      master -> master

To decode this ... the SSH server we need to access is the one in the Gogs container on the Raspberry Pi. In the docker run command we defined that to appear at port# 10022 on the host. Hence, "git@IP-ADDRESS:10022" is the correct address in the SSH URL. The tail part of the SSH URL has to match whatever Gogs tells you in its web-based user interface. In this case it is /david/hmp.git -- you would plug in your user name and repository name.

You'll see it pushed 850 MB of data across the WiFi at about 1 MB/second throughput. Not too shabby? The important thing is this worked decently well.

Conclusion

I got to a successful conclusion - but I'm left wondering whether this will work for the long run. By contrast, I've found some used older generation Intel Nuc's on eBay for $99, that with under $100 in parts could be an 8GB Linux server and give better performance without the memory limits of the Raspberry Pi. Hurm.

« Peter Capaldi et al appears on Canadian Television OpenMediaVault on Raspberry PI 3 - Plex Media Server Plugin »
2016 Election Acer C720 Android Apple Hardware History Apple iPhone Hardware April 1st ARM Compilation Authoritarianism Big Brother Botnets Cassette Tapes Cellphones Christopher Eccleston Chrome Apps Chromebook Chromebooks ChromeOS CIA CitiCards Civil Liberties Clinton Cluster Computing Computer Hardware Computer Repair Cross Compilation Crouton Cybermen Daleks Darth Vader Data backup Data Storage David Tenant DIY Repair Docker Doctor Who Drobo DVD Emdebian ESP8266 Eurovision Facebook Fake News FireFly Fraud Freedom of Speech Gallifrey git Google Google Chrome Google Gnome Great Britain Home Automation HTTPS Internet Internet of Things Internet Privacy iPad iPhone iPhone hacking Iron Man Iternet of Things John Simms Lets Encrypt Linux Single Board Computers Mac OS MEADS Anti-Missile Mercurial Michele Gomez Military Hardware Missy Mobile Applications Mondas Monty Python MQTT Music Player Music Streaming MySQL NanoPi Node Web Development Online Fraud Open Media Vault Open Source Software OpenVPN Personal Flight Peter Capaldi Plex Media Server Political Protest Power Control Privacy Public Violence Raspberry Pi Raspberry Pi 3 Raspberry Pi Zero Recycling Retro-Technology Right to Repair Rocket Ships Russia Season 1 Season 10 Season 11 Security Security Cameras Silence Simsimi Skype Social Media Warfare Software Development Space Flight Space Ship Reuse Space Ships SpaceX SQLite3 SSD Drives SSD upgrade SSL Terrorism The Cybermen The Daleks The Master Trump Trump Administration Ubuntu Virtual Private Networks VOIP Web Developer Resources Weeping Angels WhatsApp