Easily have quality Git server on your laptop with Gogs and Docker, and enable auto-push to remote repository

By: (plus.google.com) +David Herron; Date: 2016-10-01 20:47

Tags: git » Gitlab » Software Development » Web Development Tools

Github doesn't have to be the only game for git servers - while they run an excellent service, you can't install it on your own server, making it little better (in "Freedom" regards) than a closed source proprietary software package. Yes it's expedient to host your repositories on Github, but do we all have to cede this functionality to them? There happen to be several alternatives to github that offer a competent web-based git repository service. I've looked at Gitlab and now Gogs (Go Git Server) and while both are competent systems, I think I'll be running Gogs (in Docker) full time on my laptop. While both Gitlab and Gogs can run in Docker containers making it easy to run them full time in the background, Gogs is lighter weight.

It's useful first to explore WHY - why spend ones precious time on this planet doing such a task? You can easily sign up for a Github or Gitlab or Bitbucket account and just use their hosted service. Why, then, do it on your laptop?

In my case I'm setting up a git-based workflow for website and electronic book development and deployment. Meaning, I have software tools ( (akashacms.com) AkashaCMS) running in Node.js that build static HTML websites or electronic books (EPUB3 format). It's turning out to be nice to have a build system where I edit the files on my laptop, then commit edited files to a git repository somewhere, and the website is rebuilt/redeployed automagically for me.

I've set this up with Gitlab using the free Gitlab CI service. It's nice - you edit, push the changes, automatically Gitlab starts a build job, my software runs, and a couple minutes later the website is updated. I can do it all on my laptop, of course, but I've also convinced my girlfriend to use the software and she's not very technical. Meaning going to a command line and typing npm run build && npm run deploy is foreign. It's a lot simpler to tell her to use the website, edit the files through the repository web interface, then configure the system so it builds/deploys automatically in the background.

As the name ("Go Git Server") implies, (gogs.io) Gogs is a git server written in Go. While I'm sure it was nice for them to write the thing in a modern language, it makes it difficult to deploy to my VPS. Dreamhost in its infinite wisdom no longer allows VPS "owners" to perform commands as root nor to run background processes of our own. Sigh. It means I cannot simply run Gogs on my VPS and be done with it, at least not without renting another VPS from another service.

But with Docker on ones laptop it's easy to run Gogs on the laptop. I've also ordered a Raspberry Pi that I intend to use for running Gogs. The steps shown below should work for either. My laptop is of course a Mac OS X based - 2011 Macbook Pro - system. I have installed the Docker for Mac application - which is the best way to run Docker on your laptop, either Windows or Mac.

Finding the documentation for running Gogs under Docker is not intuitively obvious - go here:- (github.com) https://github.com/gogits/gogs/tree/master/docker

$ docker pull gogs/gogs
$ mkdir ~/gogs
$  docker run --name=gogs -p 10022:22 -p 10080:3000 -v $HOME/gogs:/data gogs/gogs

This is a little different than they recommend - namely, I'm putting the Gogs data volume in my home directory. The first command pulls down the Gogs image, and the second command makes a directory where Gogs will store its data. The third command has a lot going on, so let's examine it part by part.

With docker run you initialize an image as a container and execute that container. For an already initialized container, you use docker start. The --name argument gives it a nicer name than gogs/gogs. The two -p options map ports inside the running container to ports you access on the laptop's environment. The first maps the standard SSH port to port 10022, so any SSH interaction (such as interacting with a git repository) is done via that port. The second maps the Gogs web service to port 10080. The -v is what maps the local directory, $HOME/gogs, into the Gogs container as a volume, and it will be available as /data inside the container. The last argument specifies that we're running the gogs/gogs image.

When started you can visit (localhost) http://localhost:10080 and then go through an installation process. Then within a couple minutes you've got a web service running on your laptop that looks astonishingly like Github. (fewer features, though)

Once you've got it set up, it's easy to create a repository, to push repositories into the Gogs server and so forth. You do have to be careful of one thing, the port numbers Gogs tells you to access. Because Gogs does not know the web port number has been mapped to port 10080, it will tell you to access things at (localhost) http://localhost:3000 ... you'll have to edit that URL to use port 10080. Likewise it might give you an SSH URL for accessing a repository, and you'll have to insert port 10022 into that URL.

By the way, in the terminal window where you started Gogs a running log of your activity is being printed. While interesting, it's not how we want this to be over the long-term. First, kill the running Docker instance - by typing Control-C once or twice. Then run this:

$ docker start gogs

You can now pick up where you left off and your terminal is now free for doing other things.

SSH access from inside the Docker container - for autopushing the repository elsewhere

You probably do not want the repository to solely live on your laptop, right? Why not automatically push the repository to one on a server that has routine backups? Oh, and you should probably have backups happening for your laptop. Mac OS X makes this painless using the Time Capsule feature. I have a Drobo sitting on my home network, and Drobo has built-in capability to offer itself as a Time Capsule. I don't even have to think about it, the laptop takes care of backing itself up to the Drobo.

In my case the repository needs to be pushed to my VPS, and the build will occur there.

I was unable to find documentation for the following - but it does work fine.

$ docker exec -i -t gogs bash

The "docker exec" command runs a process inside a Docker container, while the -i -t flags assign a terminal (-t) and makes it run interactively (-i), and since we're executing bash that gives us a bash shell with which to run commands inside the container. That lets us explore the container and see what's what.

bash-4.3# ps auxf
PID   USER     TIME   COMMAND
    1 root       0:00 /bin/s6-svscan /app/gogs/docker/s6/
   20 root       0:00 s6-supervise crond
   21 root       0:00 s6-supervise openssh
   22 root       0:00 s6-supervise syslogd
   23 root       0:00 s6-supervise gogs
   24 root       0:00 /usr/sbin/sshd -D -f /app/gogs/docker/sshd_config
   25 root       0:00 /sbin/syslogd -nS -O-
   27 git        0:13 /app/gogs/gogs web
  168 root       0:00 bash
  173 root       0:00 bash
  179 root       0:00 ps auxf

That's the apps running, and notice that openssh is running. It means we can do the following:

# cd /data/git
bash-4.3# ls -a
.                  ..                 .gitconfig         .ssh               git                gogs-repositories
bash-4.3# ls -l .ssh
total 4
-rw-------    1 git      git             23 Oct  1 22:39 environment

The Gogs server executes as the "git" user ID, and the home directory for that user is /data/git. As you recall we mounted $HOME/gogs as /data inside the container, and it's possible to inspect the file system from inside the container and outside the container (on your laptop) to see it's the same.

bash-4.3# su - git                                                                           

Become the "git" user

2fa56ab52226:~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/data/git/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /data/git/.ssh/id_rsa.
Your public key has been saved in /data/git/.ssh/id_rsa.pub.

Then we can run ssh-keygen to generate an SSH key. You can then add the generated id_rsa.pub to the authorized_keys file of other places, such as the SSH login to an account on a VPS that you're renting.

With the SSH key added to a server, you can then login without password:

2fa56ab52226:~$ ssh -l example-user example.org
The authenticity of host 'example.org (###.###.###.###)' can't be established.
RSA key fingerprint is SHA256:..........
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'example.org,###.###.###.###' (RSA) to the list of known hosts.
Welcome to Ubuntu ...

 * Documentation:  https://help.ubuntu.com/
Last login: Sat Oct  1 14:46:58 2016 from 108.213.68.139

Automatically pushing to a remote server

So far I've verified that the "git" user inside the Gogs Docker container can freely ssh over to the server. It's time to exploit this by setting up a git hook to push the repository to a remote.

The official way to mirror a git repository to another location is with this command:

$ git push --mirror git@example.com/new-location.git

Let's simply put that in the post-receive hook of the repository. Gogs makes this easy.

A "git hook" is a shell script that runs at certain moments of Git's actions. The post-receive hook is run after a commit has been received and has been fully processed.

You can go to this directory on your laptop:

$ ls ~/gogs/git/gogs-repositories/gogs-user/repository-name.git/hooks/
applypatch-msg.sample		post-update.sample		pre-push.sample			update
commit-msg.sample		pre-applypatch.sample		pre-rebase.sample		update.sample
post-receive			pre-commit.sample		prepare-commit-msg.sample

Or this directory inside the Docker container:

2fa56ab52226:~$ ls gogs-repositories/gogs-user/repository-name.git/hooks/
applypatch-msg.sample      post-receive               pre-applypatch.sample      pre-push.sample            prepare-commit-msg.sample  update.sample
commit-msg.sample          post-update.sample         pre-commit.sample          pre-rebase.sample          update

And see that the files are the same.

In the Gogs web user interface, go to the home page of your project, then click on Settings, then click on Git Hooks. You'll see pre-receive and post-receive. Click on the pencil next to post-receive, and you'll get into a little text edit window.

For starters enter this:

#!/bin/bash
(
env
) >> $HOME/log.txt 2>&1

This prints the environment, sending it to a log file for your perusal. That log file will be visible both inside the Docker container, and outside, similarly to the commands above. Also as you edit the file you'll see the contents of post-receive change.

Make a change to your repository and check it in. Immediately this script should run and you'll be able to see output in log.txt.

Now change the post-receive script to this:

#!/bin/bash
git push --mirror ssh://remote-user@example.org/home/remote-user/git/repository-name-mirror.git

This gets right to the business of pushing the repository to your remote server. The URL used here is a correct SSH URL specifying the user name and host name for the remote repository.

If your git commit is run at the command line you'll see it execute as follows:

$ git push
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 311 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: To ssh://remote-user@visforvoltage.org/home/remote-user/git/repository-name-mirror.git
remote:    7eda04a..7297b40  master -> master
To http://localhost:10080/gogs-user/repository-name.git
   7eda04a..7297b40  master -> master

Notice the two lines starting with "remote". That's the output of this post-receive hook, being copied back to your terminal.

Performing a build step on the server

My use case is to build the code on the server, then deploy that to another server. I won't go over the steps, but obviously we can put a post-receive hook script into the ``/home/remote-user/git/repository-name-mirror.git/hooks` directory. Make that script perform our build, and voila the job is done.

« Swagger and Open API Spec @codecampnyc 2016 OpenMediaVault on Raspberry PI 3 - Plex Media Server Plugin »
2016 Election Acer C720 Ad block AkashaCMS Amazon Amazon Kindle Amiga Android Anti-Fascism AntiVirus Software Apple Apple Hardware History Apple iPhone Apple iPhone Hardware April 1st Arduino ARM Compilation Astronomy Asynchronous Programming Authoritarianism Automated Social Posting Ayo.JS Bells Law Big Brother Big Finish Bitcoin Mining Black Holes Blade Runner Blogger Blogging Books Botnet Botnets Cassette Tapes Cellphones Christopher Eccleston Chrome Chrome Apps Chromebook Chromebooks Chromebox ChromeOS CIA CitiCards Citizen Journalism Civil Liberties Clinton Cluster Computing Command Line Tools Computer Hardware Computer Repair Computers Cross Compilation Crouton Cryptocurrency Curiosity Rover Cyber Security Cybermen Daleks Darth Vader Data backup Data Storage Database Database Backup Databases David Tenant DDoS Botnet Detect Adblocker Developers Editors Digital Photography Diskless Booting DIY DIY Repair DNP3 Do it yourself Docker Docker Swarm Doctor Who Doctor Who Paradox Drobo Drupal Drupal Themes DVD E-Books E-Readers Early Computers Election Hacks Electric Bicycles Electric Vehicles Electron Emdebian Encabulators Energy Efficiency Enterprise Node EPUB ESP8266 Ethical Curation Eurovision Event Driven Asynchronous Express Facebook Fake News Fedora VirtualBox File transfer without iTunes FireFly Fraud Freedom of Speech Gallifrey git Gitlab GMAIL Google Google Chrome Google Gnome Google+ Government Spying Great Britain Heat Loss Hibernate Hoax Science Home Automation HTTPS I2C Protocol Image Analysis Image Conversion Image Processing ImageMagick InfluxDB Infrared Thermometers Insulation Internet Internet Advertising Internet Law Internet of Things Internet Policy Internet Privacy iOS Devices iPad iPhone iPhone hacking Iron Man Iternet of Things iTunes Java JavaScript JavaScript Injection JDBC John Simms Journalism Joyent Kaspersky Labs Kindle Marketplace Lets Encrypt LibreOffice Linux Linux Hints Linux Single Board Computers Logging Mac OS Mac OS X MacOS X setup Make Money Online MariaDB Mars Matt Lucas MEADS Anti-Missile Mercurial Michele Gomez Micro Apartments Military Hardware Minification Minimized CSS Minimized HTML Minimized JavaScript Missy Mobile Applications MODBUS Mondas MongoDB Mongoose Monty Python MQTT Music Player Music Streaming MySQL NanoPi Nardole NASA Net Neutrality Node Web Development Node.js Node.js Database Node.js Testing Node.JS Web Development Node.x North Korea Online advertising Online Fraud Online Journalism Online Video Open Media Vault Open Source Governance Open Source Licenses Open Source Software OpenAPI OpenVPN Personal Flight Peter Capaldi Photography PHP Plex Plex Media Server Political Protest Postal Service Power Control Privacy Production use Public Violence Raspberry Pi Raspberry Pi 3 Raspberry Pi Zero Recycling Remote Desktop Republicans Retro-Technology Reviews Right to Repair River Song Robotics Rocket Ships RSS News Readers rsync Russia Russia Troll Factory Russian Hacking SCADA Scheme Science Fiction Search Engine Ranking Season 1 Season 10 Season 11 Security Security Cameras Server-side JavaScript Shell Scripts Silence Simsimi Skype Social Media Social Media Warfare Social Networks Software Development Space Flight Space Ship Reuse Space Ships SpaceX Spear Phishing Spring Spring Boot SQLite3 SSD Drives SSD upgrade SSH SSH Key SSL Swagger Synchronizing Files Telescopes Terrorism The Cybermen The Daleks The Master Time-Series Database Torchwood Total Information Awareness Trump Trump Administration Trump Campaign Ubuntu UDOO Virtual Private Networks VirtualBox VLC VNC VOIP Web Applications Web Developer Resources Web Development Web Development Tools Web Marketing Website Advertising Weeping Angels WhatsApp Window Insulation Wordpress YouTube YouTube Monetization