Software Development

Self-hosting services using Docker

() Self-hosting is the opposite of using a 3rd party cloud service like Dropbox or Google Drive, and instead to host an equivalent service, like Nextcloud, on their own hardware. For every popular cloud service there is probably an independently developed open source equivalent one can host on their own hardware. Self-hosting gives one control, and peace of mind that their private information is not being sold to others, and it can save a lot of money from service fees that aren't paid. The personal security implications are huge, since it is known that governments around the world are pressuring internet services for data on user activity. While there is a legitimate need to know about criminal activities, do we really need to risk our data being snooped upon?

Self-hosted Docker infrastructure in home or office using low-cost computers like Intel NUC

(March 26, 2021) Using Docker, and a simple small computer, you can build a powerful computing "cloud" in your home, on your desktop, at low cost, giving you control over your data. If you need more power, adding another computer or three to the mix quickly adds more capabilities. For almost any popular 3rd party service like Github, Dropbox, and Trello, there is an open source package that might even be better. Those cloud services are temptingly easy to sign up and get going, but there are tradeoffs. What happens when a 3rd party service suddenly shuts down? That has happened on multiple occasions, sometimes deleting customer data in the wake. Or what about miscreants breaking in and stealing data? With open source operating systems, open source packages like Gitea, Nextcloud, and Kanboard, you gain control over your destiny.

Scheduling background tasks using cron in a Docker container

(March 25, 2021) The cron service has, for time immemorial in Unix-like systems, long helped us schedule the occasional background process that keeps our systems ticking. Typically it is used to gather up or summarize log files, or collect and process data from external systems. In the old days when e-mail exchange, and the all-too-important Usenet news exchange, happened using UUCP over modem lines, a cron job scheduled regular UUCP calls to neighboring servers. Having cron running in the background is part of normal Unix/Linux/etc system admin practices. Even though the crontab format is kind of hokey, we all learn it and set up automated background tasks to keep the world functioning. Let's see how to set this up in a Docker container.

Using multiple databases and PHPMyAdmin in a single MySQL Docker instance

(March 20, 2021) Sometimes our applications need multiple database, but with a single database server instance. The typical recommendation is for each Docker container to serve a single application. For databases, that means one database server for each application. That's the result of following a theoretical best practice, but isn't there another best practice to avoid excess processes and overhead? MySQL, like most other database servers, can support multiple databases, multiple user identities, and has an extensive system for controlling access on IP address, user names, and passwords. The MySQL container for Docker is easy to use, but supports only a single database instance per server container. In this article let's learn how to go ahead and configure the MySQL container for Docker to support multiple databases each with their own user ID.

Connect with SSL to MySQL in Docker container

(March 17, 2021) MySQL might throw an error when an application connects without using SSL to encrypt the connection. Therefore the MySQL team clearly hints it is best to use SSL connections to MySQL servers. Clearly a database connection has critical data that you don't want to leak to 3rd parties, and encryption is a great counter-measure. The official MySQL Docker container automatically generates a set of SSL certificates to use for connections, so let's see how to put those certificates to use.

Using SSH to remotely control a Docker Engine or Docker Swarm in two easy steps

(March 15, 2021) To remotely manage a Docker instance, we can SSH into that host to run Docker commands there. But a not-well-documented Docker feature lets us remotely access, and manage, Docker instances from the comfort of our laptop, via SSH. It's fast, easy and very powerful.

Set up MySQL using Docker on your laptop

(March 10, 2021) Setting up MySQL on Docker is fairly simple, and the MySQL team has created a flexible Docker image that can be used in many ways. Databases, like MySQL, serve as the data persistence and retrieval layer of many applications we might deploy.

Running MongoDB with Docker and Docker Compose

(March 10, 2021) MongoDB, as one of the popular NoSQL databases, is part of many software projects. Hence, one must sometimes know how to configure and setup MongoDB in a Docker environment.

Understanding MySQL the Access Denied error in or outside a Docker container

(March 10, 2021) I once wasted more than a full workday fighting the MySQL error message Access denied for user (using password: YES). After finding the solution I want to help others avoid this problem. Along the way to fixing the issue, I learned a lot about how MySQL authenticates user ID's. I've been using MySQL for years and had glossed over this topic, but it turns out to both not be terribly difficult, and very crucial to the solution.

Wordpress local development environment with Docker and Docker Compose on your laptop

(March 9, 2021) We normally install Wordpress on a public Internet server, and installation on your local machine is normally limited to developing or testing a Wordpress feature or theme. The development loop is a lot shorter when you can directly edit Wordpress files on your machine, as opposed to working out a method to edit remote files. While it's convenient, it's not always clear how to create a website hosting environment on your laptop. In the past we would use MAMP (or the Windows equivalent), which is a macOS-native environment for running Apache/MySQL/PHP stacks. But today we have a different tool, Docker, that is easily able to run that stack, as well as any other stack. Docker easily runs on a laptop, so let's take a look at using it to run a local Wordpress development environment.

Deploying MySQL to get started using a Docker Compose file

(March 8, 2021) While the docker run CLI command is powerful, Docker Compose files let us succinctly describe a whole system of Docker containers, virtual networks, and file systems, in one easy to read file. This is a powerful tool with which you can easily describe deployment of a full application stack. Compose files support not only deployment to the local Docker host, but to Docker Swarm clusters, and now to AWS ECS and Azure ACI cloud platforms.

Set up PHPMyAdmin and Wordpress Docker containers using existing MySQL

(March 5, 2021) There are several ways to launch a PHP service, backed by a MySQL database, on Docker. That combination is literally the LAMP Stack (Linux, Apache, MySQL, PHP/Python/etc), and we can use Docker to easily set up this combination on our laptop or deploy it to a production server. Elsewhere we went over setting up MySQL on Docker, so lets now demonstrate the LAMP Stack, on Docker, by deploying PHPMyAdmin and Wordpress on our laptop.

Familiarizing ourselves with Docker

(March 4, 2021) Getting to know Docker by running a few commands.

Implementing HTTPS with Docker services on AWS ECS using Docker Compose

(February 5, 2021) Thanks to a new Docker CLI feature we can directly deploy to AWS Elastic Container Service (ECS) using a Docker Compose file. It's very cool, and greatly simplifies deploying applications to AWS, but it lacks an important feature which is implementing HTTPS support. The solution offered by the Docker team involves rewriting the Cloud Formation template they generate for us, so let's learn how it works.

Getting started with Docker: Installation, first steps

(October 8, 2020)

Let's start this journey into Docker by learning how to install it on popular systems, namely macOS, Windows and Linux. Installation is very simple thanks to the hard work of the Docker team. You youngsters don't know how easy you have it now that Docker for Mac and Docker for Windows exist. Long gone are the days when we had to install VirtualBox along with a specialized virtual machine to use Docker.

Self-hosting your own web services in place of using 3rd party services

(September 27, 2020) Self-hosting is about hosting Internet services on your computers, controlled by you. This is different from just to just Use the Cloud by signing up for services like Dropbox, Github, Travis, or other 3rd party controlled services. Many people just use the cloud that way, because it's easy to just sign up, provide a credit card number, and get on with business. For each of those commercial services there are equivalent open source alternatives you can host on your own hardware. Self-hosting purposely takes a different route, and in this article we'll go over some reasons why.

Wordpress production deployment with Docker Compose

(September 25, 2020) Hosting a Wordpress site in Docker is relatively easy, and is largely a matter of properly configuring three standard containers. It requires a MySQL database, NGINX, and the Wordpress PHP-FPM container, with simple configuration. Starting from a freshly provisioned virtual server, the process takes less than two hours to set up hosting with HTTPS support.

How to share a Docker network between Stacks deployed to a Docker Swarm

(September 20, 2020) How do you handle a system deployed to Docker Swarm, with multiple Stacks, where a container in one Stack must communicate with a container in another Stack? For example, you may have a database Stack, and an application layer Stack, where the application needs to communicate with the database. It's simplest to put both into the same Stack. But it's a best practice for the database to stay running, and to separately bring the application up and down to deploy updates. Therefore it's best to use separate Stacks for each layer, raising the question of how will the application containers find the database containers.

Configuring Terraform to use for deploying AWS Infrastructure

(August 13, 2020) Terraform is a powerful tool for configuring infrastructure on a long list of cloud-based systems. Since AWS is such a complex system we look for simpler ways to do anything on AWS. Get started means installing and configuring the AWS CLI before using Terraform to simplify AWS deployment.

Fixing could not read Username for 'https://github.com': No such device or address in Jenkins

(August 12, 2020) You might have a successfully executing build task in Jenkins, then suddenly need that build task to push to a Github repository. For example, it might render a statically generated website to push to the gh-pages branch for display using Github Pages. But this inscrutable error message comes up. There is a fairly simple fix, especially if you're using an HTTPS URL to access the Github repository.

Setting up the AWS Command-Line-Interface (CLI) tool on your laptop

(August 10, 2020) The AWS platform is an incredibly diverse and mature cloud hosting platform, and the AWS CLI is your gateway to interacting with AWS from the comfort of your laptop. Instead of going click-click-click on the AWS website, you type commands from the command line, and you can even use those commands in scripts for semi-automated control of AWS resources. In this guide we'll go over how to install the AWS CLI tool on your laptop.

Using NGINX, Lets Encrypt and Cron in Docker to manage HTTPS/SSL certificates for multiple domains

(July 5, 2020) NGINX is a hugely flexible webserver with which it's very easy to manage HTTPS with Lets Encrypt. With a simple configuration file it is easy to add HTTPS support to a back-end service, using NGINX's reverse proxy. It's possible to use this for deployments both small and large. As with any website Internet-visible nowadays, it is necessary to use HTTPS. Which leads to the task of using Lets Encrypt to provision SSL certificates for several domains, using NGINX to terminate the HTTPS connections, while proxying from NGINX to the actual services. This blog post discusses a Docker container designed explicitly for that purpose.

Manage Letsenrypt HTTPS/SSL certificates with a Docker container using Cron, Nginx, and Certbot

(June 29, 2020) Modern websites must have HTTPS support for security reasons. As a result web browsers and search engines have begun downgrading sites that do not support HTTPS. That means we all must have a simple, low cost, way to set up HTTPS support on our websites. The Letsencrypt project offers free SSL certificates for HTTPS. In this project we will create a Docker container for handling HTTPS via Nginx, and automated SSL certificate renewal using the Letsencrypt command-line tools (Certbot).

Avoid 'could not be accessed' error when deploying a Service to a Docker Swarm on AWS

(May 30, 2020) Launching a Docker Swarm on EC2 instances is relatively easy, but of course there are pitfalls. One involves deploying a service to the swarm, but getting an error message about being unable to access the container image from the ECR registry.

Creating a Docker Swarm using Multipass and Ubuntu 20.04 on your laptop

(May 21, 2020) Docker is a cool system for deploying applications as reusable containers, and Docker Swarm is a Docker Orchestrator that let's us scale the number of containers across multiple machines. Multipass is a very light weight virtual machine manager application running on Windows, Linux and macOS, that let's us easily set up multiple Ubuntu instances on our laptop at low performance impact. Therefore Multipass can serve as a means to easily experiment with Docker Swarm on your laptop, learning how it works, setting up networks, etc.

How to correctly log in to AWS Elastic Container Registry on Windows command line

(May 10, 2020) The Elastic Container Registry (ECR) is a Docker container registry for use while deploying containers to AWS ECS. It is convenient and relatively easy to use. AWS helpfully supplies command strings to log in to ECR from the command line so you can publish container images with docker push. But the supplied command line for Windows doesn't work right, and needlessly relies on a tool that seems to not be very useful.

Correctly launch MySQL on Docker for Windows, avoiding 'Bind on unix socket' error

(May 6, 2020) Docker is an excellent tool for launching Linux-containerized applications, and it even runs on Windows. But running Docker containers on Windows has a few unexpected rough edges. One will come if you try the default way to launch MySQL doesn't work on Windows. Instead of the expected successful launch you might instead be told Can't start server : Bind on unix socket and asked whether there is another MySQL server running. That misleading error can send you on a tangential wild goose chase.

Solve 'Drive has not been shared' error with Docker on Windows

(May 5, 2020) We often mount folders into a Docker container to ensure data is persistent while letting us freely destroy and recreate the container. But on Windows you might get a head-scratching error message saying "Unhandled exception: Drive has not been shared". The most common advice that might come up when searching the Internet is not about this situation, but about general file sharing in Windows. In this case the error refers to configuration settings in Docker.

Easily launch RDS and EC2 instance, with SSH access, using Terraform

(April 6, 2020) We commonly must deploy on AWS both running code and a database service. AWS offers a cloud based database system, RDS, and in this tutorial we will look at launching an AWS EC2 instance along with an RDS database inside a custom-built AWS Virtual Private Cluster (VPC). The RDS database will not be reachable from the public Internet, and only from within the VPC we will create. Our tool for this will be Terraform, a powerful tool for configuring infrastructure on a long list of cloud-based systems

Easily launch EC2 instance, with SSH access, using Terraform

(April 4, 2020) Terraform is a powerful tool for configuring infrastructure on a long list of cloud-based systems. Since AWS is such a complex system we look for simpler ways to do anything on AWS. Terraform promises to make AWS deployments much simpler. To test that let's learn how to deploy some EC2 instances on AWS using Terraform.

How to fix AWS CLI hangs on AWS EC2 instance

(March 18, 2020) It seems that setting up an AWS EC2 instance in the default VPC stands a high chance of being unable to use the AWS CLI from inside the instance. That was my experience, anyway, and the solution is extremely non-obvious, non-intuitive, and requires ensuring that the instance can do outbound HTTPS traffic, and uses the correct public DNS servers.

Installing a self-hosted Docker Registry to aid Docker image development

(March 15, 2020) When developing Docker images it's useful to store them in a Docker registry. While using Docker Hub is free, it's bad form to fill up that shared resource with images built for personal use. It's better to only publish the Docker images that are truly useful to everyone, and that have documentation. That leaves us with the problem of a location to host our own Docker images. Do we pay for a private repository somewhere? Or, as a self-hoster, do we host a local Docker Registry to store our personal Docker images? In this post let's explore the latter idea.

Installing Gitea for self-hosted Git service, replacing Gogs

(March 11, 2020) While Gogs is an excellent tool to have a self-hosted Git service (like Github), I recently found out the project is semi-abandoned. A group of Gogs users launched Gitea as a replacement, and in any case it looks like a better server. The goal here is to install Gitea, evaluate it, and see how to convert Gogs-based repositories over to Gitea. The result will be managed in the Docker self-hosting machine I have at home.

Easily manage Docker containers on both local and remote Docker hosts with Portainer

(March 9, 2020) Docker is a wonderful advancement for software engineers and system administrators. It simplifies launching and maintaining background processes, while adding a layer of much-needed encapsulation and security. But the default command-line administrative tools are less than pleasant to use, and we instead want a good GUI with which to manage our Docker hosts. That's where Portainer comes in. It manages the Docker containers we have running on our local host, or on remote hosts.

Directly deploying a Docker service to AWS ECS using Docker Compose

(February 6, 2020) Deploying Docker services to AWS ECS is becoming much simpler thanks to new feature collaboration between the Docker and AWS teams. It is now possible to take a normal Docker Compose file, using modern Compose features, and deploy directly to AWS. Nothing complex or unusual is required, with all the complexity handled under the covers with a generated CloudFormation file. In this article we'll take a first look at using these features, which are in early release right now.

Run Linux/X11 apps in Docker and display on a Mac OS X desktop

(December 10, 2019) While the MacPorts and Homebrew projects bring many Linux apps to the Mac environment, they don't support every app we'd want to run. Since the X11 environment is not native to macOS it's not a simple recompile, since you have to rewrite the GUI system. Thankfully there is an X11 display server for Mac OS X that can be used to run an application in a Linux environment and display it on the macOS desktop. In this article we'll look at one way to get this all connected up and running.

Understanding how Python classes work

(September 26, 2019) Python classes allow programmers to define new data types to use in their programs. They act like a blueprint with which a program creates objects. Objects can contain both methods and attributes, giving us a useful higher-level abstraction over the built-in Python data types.

Google's Accelerated Mobile Pages (AMP) considered harmful to web freedom - change my mind

(August 23, 2019) Google first told us about AMP five years ago, spinning it as a way to - as the name suggests - speed up the web experience on mobile devices. Mobile devices are clearly extremely important, since most people are using such devices to read websites. Since mobile devices have constrained abilities, we should develop websites with technology to match, and AMP was a way to force us to do so. While laudable, the implementation puts Google squarely in control of web technology, and that's not right.

Managing versioned machine learning datasets in DVC, and easily share ML projects with colleagues

(July 23, 2019) DVC is a powerful set of tools for managing data files associated with data science or machine learning projects. It works hand-in-hand with a Git repository to track both the code and the datasets in an ML project. A core feature is for versioning datasets, meaning that it correlates the dataset to exactly match what existed at each Git commit. By using a DVC "remote cache" it is very easy to share a project with colleagues, or to copy the dataset to a remote machine.

Introduction to using DVC to manage machine learning project datasets

(July 7, 2019) DVC is a powerful set of tools for managing data files associated with data science or machine learning projects. The code for such a project is committed to a Git repository, and DVC manages the data files in parallel to that repository.

Principled Data Processing

(May 22, 2019) Data Science, and Machine Learning, looks like regular software engineering. But software engineering tools do not work well in either field. Instead other kinds of tools are required. This talk goes over some project management problems in Data Science, that also apply to Machine Learning, discusses some principles on solving the problems, and presents a methodology that the presenter found useful in managing his data science projects.

How to build scrollable dropdown menu in Bootstrap v4 when it must hold dozens of choices

(April 10, 2019) The Bootstrap dropdown component is very easy to use, but doesn't work so well when it must contain a lot of content. In my case the dropdown contained the Table of Contents for a book with lots of ToC entries, and it didn't even fit well on a large screen. On a small screen for a mobile device, it shrank into unusableness, and Google recently lodged a usability complaint against the site. After researching a ton of alternative designs, a simple solution appeared.

How to read array of values from browser DOM using jQuery, or in Node.js using Cheerio

(March 12, 2019) Typically with jQuery or Cheerio we read or write strings and numbers to/from the DOM, but what if we must use objects or arrays of data? The jQuery documentation mentions data-* attributes, but the documentation is not entirely clear how to use this for reading/writing objects or arrays to/from the DOM.

With ARM servers becoming a thing, developers may need ARM PC's

(February 23, 2019) ARM-based CPU's are going into not just mobile devices. ARM is popular in cell phones and the like because it is low power. That, and heat dissipation, is proving popular in server computers as the world moves towards cloud computing. Linus Torvalds recently suggested that software developers need ARM-based PC's to go along with ARM-based servers.

Nuxt.js overview, running Vue.js on the server to pre-render on the server

(September 4, 2018) Vue.js is an application framework for the front end running inside browsers. It supports building single-page-applications (SPA) or simply enhancing a web page with fancy interactivity. However it isn't always useful to have a big application running in the client. With Nuxt.js the Vue.js code is pre-rendered on the server so that the initial load of the page has already-rendered HTML. Subsequent interactivity on the page is still managed by client-side Vue.js code running in the browser.

Showing a tree structure, like a file-system, in Vue.js

(July 26, 2018) Many kinds of applications show a tree structure of data items, such as files in a file system. Over the last couple weeks I implemented (blog.sourcerer.io) a simple Markdown editor in Vue.js/Electron - and while the sample application did not show a file-system tree a full version of the application would do so. We use this sort of widget all the time in programs and web applications, but do you know how to implement such a widget? Speaking for myself I could probably work it out - it's nested UL and LI elements with some custom styling and whatnot - but I'd rather use a pre-baked tree component for Vue.js instead. The application shown here demonstrates using one such tree component.

Add/remove list items in Vue.js built with Bulma/Buefy

(July 21, 2018) Most applications with item lists you want to add or remove individual items. With Vue.js it's easy and obvious how to render an item list. What's not so obvious is how to manipulate the list afterwards. The Vue.js documentation doesn't say how to remove items or change items.

Rudimentary form validation in Vue.js with Buefy/Bulma

(July 21, 2018) As a follow-up to Software Development we want to study form validation. That example includes a little form in which we enter some information, and we want a little bit of validation to make sure we don't add bad entries.

Getting started with Vue.js application development

(July 20, 2018) Vue.js is an exciting new framework for developing front-end JavaScript applications. It promises to be a "Progressive" web framework, and I don't think they mean in a Political sense but means that Vue.js can be plugged into the part of an application which requires a richer experience. It is also supposed to be Approachable, works directly from HTML, CSS and JavaScript, while being Versatile and Performant. In this example we'll build a starter application, and embed it into an HTML web page. Specifically, this web page.

Compiling Rust for Raspberry Pi

(June 28, 2018) Instructions for cross-compiling Rust code to ARMv7, such as the Raspberry Pi.

Setup Google Recaptcha v2 in PHP scripts

(June 9, 2018)

Recaptcha has been around a long time, fighting the good fight against spammers. One of my sites has been using Recaptcha for years, long enough to require updating to new API implementations a couple times. Google has updated to Recaptcha v2 and soon v3, and somehow my site stopped enforcing Recaptcha's with the subsequent Spammer intrusions. Fixing the implementation proved to be difficult, especially as the available tutorials missed some details.

Global and Local Secondary indexes in DynamoDB

(June 8, 2018) Years ago, Amazon engineers studying SQL queries learned complex queries with JOINs and whatnot were not widely used, and from that observation developed DynamoDB. DynamoDB is a NoSQL database available in the Amazon cloud service offering. It has some interesting characteristics, one of which is the method to use secondary indexes.

How to get AWS SDK for Node.js to return Promises to use with async/await functions

(June 8, 2018) The AWS SDK for JavaScript/Node.js documentation describes receiving results the old way, using a callback function. In the modern era of JavaScript we know what to do -- we wrap such functions in await new Promse, hold our nose, and embed the callback within the Promise. I published an article on the Sourcerer blog describing how to interact with DynamoDB from a Node.js Lambda function, and then a reader asked why it used "await new Promise". A bit of searching revealed that indeed Amazon had updated the SDK to support Promises, but it seems the SDK documentation has not been updated. What follows is a quick example of an AWS Lambda function calling DynamoDB methods using Promise objects in an async/await, and deployed using the Serverless Framework.

Getting started with using Node.js and the Serverless framework on AWS Lambda

(June 3, 2018) While Amazon's AWS platform is excellent, there are other similar platforms from Google, IBM, Microsoft and others. Why should our code be written solely to support the AWS infrastructure? The Serverless framework makes it easy to reuse the same application code on any of these functions-as-a-service platforms. Perhaps more importantly, it simplifies coding against these platforms. The AWS platform is incredibly complex, and using the Serverless framework is simplifies developing applications for the AWS platform.

Introduction to Node.js with the Serverless framework on AWS Lambda

(May 19, 2018) AWS Lambda has excellent support for Node.js code. As of this writing Lambda supports Node.js 8.10, meaning we have excellent async function support and other modern ES2015+ language features.

Introduction to Node.js with the Serverless framework on AWS Lambda

(May 19, 2018) AWS Lambda lets programmers focus on their code, with AWS taking care of all the deployment drama issues.

Getting started with AWS Lambda, Amazons Function as a Service platform

(May 7, 2018) AWS Lambda lets developers write code, execute the code at any scale in the AWS cloud, without worrying about implementing the scaling algorithm. Lambda focuses on very small modules - the name implies Functions, and that is the preferred granularity level. "Function as a Service" platforms are available from several providers, AWS Lambda having been joined by Microsoft Azure Functions, Google Cloud Functions, and IBM Cloud Functions. All are focusing on the same idea, Functions as the granularity level, and automatic deployment and scaling, letting developers focus on their code and the cloud provider focuses on the rest. AWS Lambda is the most developed of these, so lets take a look.

Simple Node.js AWS Lambda function

(May 7, 2018) AWS Lambda has excellent support for Node.js code. As of this writing Lambda supports Node.js 8.10, meaning we have excellent async function support and other modern ES2015+ language features.

Bare-metal Rust for AVR/Arduino and ARM microcontrollers coming soon

(April 28, 2018) In contemplating a project for an embedded microcontroller product, the customer is suggesting C++ and I'm having bad memories of memory allocations gone haywire. I've heard good things about Rust, being a modern language with the same mindset of C/C++ in terms of doing low level operations. Would Rust make a good choice for embedded device programming? Seems that it's still early days for that dream, as the tools aren't all lined up. Important thing is that folks are thinking in that direction and working towards that result. For some specific embedded CPU's it is possible to configure a Rust/LLVM toolchain that works fine.

Learn to use Docker for application development and deployment

(April 3, 2018)

Docker is a wonderful tool that abstracts away all kinds of details about configuring and maintaining Linux Containers. The power to simply type "docker run image-name" and have a bunch of complexity automatically handled is great. However Docker is one of those tools with lots of moving parts behind the scenes, and some training is needed to use it well.

HTTPS with nginx, using Lets Encrypt, proxying to Gogs and Jenkins back-end services

(March 31, 2018)

Modern development environments require a continuous integration system, along with a reasonable git-based repository hosting service. It's possible to rent these services, Github and Gitlab are both excellent hosted git repository services for example, and there are several hosted continuous integration systems. Gitlab in particular is a one-stop-shop offering both Git hosting and continuous integration in one service. But, you can easily host Git and Continuous Integration services on your own hardware. And with a little work the services can be HTTPS-protected using Lets Encrypt.

Moving Docker's files to a custom location

(January 30, 2018)

Docker is a wonderful tool that abstracts away all kinds of details about configuring and maintaining Linux Containers. The power to simply type "docker run image-name" and have a bunch of complexity automatically handled is great. But you may want to change Docker's defaults, and just how do you do so? In my case "/var/lib/docker" would be on an SSD drive, and to lengthen its lifetime I want to minimize the number of writes to that drive. Moving this directory to the SSD should help with that goal.

Installing Composer on a Dreamhost VPS, or perhaps other VPS's

(January 12, 2018)

Modern PHP software development has been modernized by Composer. My exposure to Composer is because the Drupal team adopted the Symfony framework, which in turn is based on Composer. Installing Drupal any longer seemingly requires to first install Composer. Which isn't installed by default on Dreamhost VPS's, but it's not too hard to get it going.

These github alternatives give you powerful git hosting, on your own terms

(January 8, 2018)

Git is a powerful source code revision control system, letting teams large and small track changes between team members. Github is an excellent service for hosting Git repositories. The Github corporation offers powerful free features, and even more powerful paid features, and software teams around the world are using Github to collaborate. But what if you want to, or must, host your repositories yourself? Some corporate security teams require no use of 3rd party services. Are you stuck using bare git?

Fortunately several Bitbucket and Github alternatives exist letting you create source repositories similar to Github, with most of the capabilities of Github. These open source Github/Bitbucket alternatives can be installed on your own hardware.

Accessing the Spring H2 database console when X-Frame-Options says deny

(January 5, 2018)

You're coding a Spring Boot application, need a database, and quickly configure in the H2 in-memory database thinking it'll be easy. Rerun the application, do a few things, then try to get into the H2 console, and it's a blank screen. The JavaScript console says "Refused to display ... in a frame because it set 'X-Frame-Options' to 'deny'." and there's a sinking feeling, it's another one of those obtuse error messages from the bowels of somewhere you've never heard of. Fortunately the solution is easy.

What is In-memory Computing?

(December 5, 2017)

Traditional computer architecture has a performance bottleneck due to the slow speed of persistent data storage on disk. We've developed complex systems like databases, and even distributed databases, to manage the data used in computations. The key limitation is the latency between requesting data, and getting the result. With in-memory computing the data is right there in quickly accessed memory, which greatly amplifies system performance.

Solving Http Message NotWritable Exception, Could not write JSON: Could not set field value in Spring Boot

(October 16, 2017) My experience of programming on the Spring platform is that you'll be coding along, everything is going great, until an inscrutable complex error message emerges from the depths of who knows where. This message seemed to involve some module or another having created a round peg to fit into a square hole. That is, I'd sent a GET request to retrieve a large object tree, and it seemed that while filling data into a field of a subsidiary object Spring/Hibernate/etc chose the wrong data type, and it threw up an error. The specific type of error was receiving no discussion anywhere, so it was impossible to see how others had solved the problem. The documentation was of no use in explaining anything about this. But, after a couple days of trials and errors and more errors, a solution was found. And, indeed, Spring/Hibernate had chosen the wrong data type to fill into one of the fields, and the solution is fairly simple.

Persisting complex Embeddable/Embedded objects in Spring/Hibernate

(October 8, 2017) Hibernate and Spring and the Java Persistence packages make for a powerful combination. You can define a complex tree of objects, and the system automagically creates the database schema, and automagically saves and retrieves that object tree from the database. It means you can innocently code away, until one day you try to include an Embedded object but it silently fails to be saved to the database, Hibernate silently fails to create database columns for that object, and when retrieving the object for which the Embedded object is a field, the object is null. No error messages are printed in the log. Instead you're left wondering, why didn't Hibernate persist this object? What's presented here is a solution using EntityListener methods to convert values in and out of shadow values stored in the Entity.

A fix for "Persistent Object Exception - detached entity passed to persist" in Spring Boot

(September 6, 2017) Error messages coming out of Spring can be more than inscrutable. As discussed in my solution to transient unsaved properties, I'm defining a REST API using Swagger/OpenAPI targeting a Spring Boot implementation. The object models are complex, with lots of nested objects. Today's inscrutable message, "detached entity passed to persist", doesn't make much sense as a phrase does it? Detached? Detached Entity? Say what? Turns out it's not that hard a problem to fix.

A fix for org hibernate TransientPropertyValueException in Spring Boot

(September 1, 2017)

Error messages coming out of Spring can be more than inscrutable. In this case I'm defining a REST API using Swagger/OpenAPI targeting a Spring Boot implementation. The object models are complex, with lots of nested objects. It turns out you can POST a big blob of JSON with all kinds of structure, and Spring/JPA/Hibernate will swallow the whole thing and automatically parcel everything out to the corresponding database table. If you define the JPA annotations correctly, that is.

Get an annotation incorrect, and you'll be left scratching your head over what the heck you're being told by the error message. The Spring Team promises on a huge stack of bibles that Spring Boot automatically takes care of zillions of things for you, and it's so easy to write your code. In my experience it's instead a matter of papering complexity over complexity over a system containing about a thousand kitchen sinks worth of excess capabilities. There's no amount of papering-over that kind of complexity that will make the system simpler. But, the boss-man says to use Spring Boot, so here goes.

Deploying Docker images to a server without using a Docker Registry

(August 30, 2017)

We formerly deployed server applications to a Linux server using manual processes. An advanced team might use shell scripts to automate deployment. Over time tools like Chef or Ansible and more grew to handle ever-more-complex server application deployment scenarios. A few years ago, Docker came onto the scene with a whole new approach involving building a "Container" housing a complete operating system image that runs your application. Having built the Container, it's easy to ship that container to a server or run it on your laptop. The compelling gain is having the exact same development environment on your laptop as is deployed to your servers. Using the EXACT same environment streamlines your work by removing a ton of potentially destabilizing variables.

The preferred method is to build a Docker container image on your laptop, or on a build server, and upload the image to a Docker Registry. The image can then be downloaded from the Registry onto any number of systems.

What if you don't want to, or cannot, use a Registry? You could instead deploy the source code to the server, and build the container image on the server. That's a very unwise move, and it's better to ship the container image to the server. Turns out that is easy to do.

CSS Grid Changes Everything

(August 9, 2017)

Getting web content to layout as you want usually results in messy HTML and messy CSS. Limitations of both usually means you're nesting DIV's to get a structure where you can use various arcane CSS paradigms like absolute or relative positioning. CSS Grids changes all that. You can use simple HTML markup accompanied by straight-forward CSS rules to lay it out. It's a breath of fresh air. It's supported in ALL the modern web browsers.

Why not flexbox? Isn't that the thing that's supposed to be our page layout savior? The problem with flexbox is it runs in only one direction. For more complex layouts you need to nest flex containers. CSS Grid is a two-dimensional layout system and can handle more complex layouts.

Traditional CSS methodology, as well as flexbox, works from the content outward. CSS Grid works from the layout inward.

Throwing away local changes in Git workspace, allowing 'git pull' to proceed

(August 8, 2017)

Git is a powerful source code revision control system, letting teams large and small track changes between team members. It is widely used and very flexible. However, it's easy to get into an inscrutable state where it's not clear how to proceed.

Just now I found myself wishing to update my local repository with changes from co-workers. The repository contains UML models designed with Enterprise Architect, hence the primary thing in the repository is a large binary file. I'd been running Enterprise Architect on my local file, and there must have been some changes made to that file. As such Git told me there were local changes which I must merge before I could fetch the upstream changes. (specifically: Your local changes to the following files would be overwritten by merge) But I knew any change that had been made was unimportant, unneeded, and should be tossed away. Leaving me wondering how to throw away changes to a local Git repository.

Create a Spring Boot REST API using Swagger, OpenAPI, to "Generate Swagger OpenAPI REST API documentation for Spring Boot application"

(July 31, 2017)

The Swagger tools, and the OpenAPI format, are an excellent way to document REST API's and even to generate client or server stub libraries to ease implementation. The technology serves two purposes -- a) standardized documentation for REST API's, b) generating code from API documentation in several programming languages. An OpenAPI file is fairly simple to write, you declare REST endpoints, describe the parameters and the request type, and then describe responses. It allows you to define complex object models that can be used either as input to a service, or its output.

Unfortunately the Swagger website doesn't have adequate documentation of using the tools. And it proved difficult to find clear straight-forward tutorials showing how to get started. Even the most powerful tool can be hampered if folks are unable to use it.

The following tutorial is a complete demonstration of, starting from scratch, developing a small Spring Boot service using OpenAPI and the Swagger tools. We show how to go from an OpenAPI spec to generated Spring Boot code, and also how to generate an OpenAPI spec from running Spring Boot code. There are several issues with the workflow of generating code from the OpenAPI spec. It's more effective to instead write the service code, and add in the annotations required for the Swagger tools to generate the OpenAPI spec for you.

With the OpenAPI spec it's easy to produce interactive API documentation that programmers can try out directly in their web browser.

Correct character encoding with DOMDocument implementing a Wordpress content filter

(July 6, 2017) Using DOMDocument in a Wordpress content filter lets you correctly manipulate the content as HTML. It might be that Wordpress filters are commonly using regular expressions or text search/replace functions. While that can be fast and powerful, correctly changing HTML elements requires an HTML-oriented API. The specifics of HTML elements are such that regular expressions and text search/replace functions just don't cut it due to the many pitfalls from highly specific details of HTML. With DOMDocument you simply load the HTML into the library, then you use DOM functions to manipulate the HTML, then you serialize the DOM to HTML text, and voila powerful HTML manipulations easily performed. Unfortunately that method comes with its own pitfalls you must be careful of.

Best implementation of the Swagger UI tool for your OpenAPI based Spring Boot REST server

(June 15, 2017)

Among the Swagger Tools is Swagger UI, an excellent tool for browsing an OpenAPI specification, the methods defined in the specification, the data models used in the specification. Swagger UI even lets you interact with the service from the documentation. It's a most useful take on "documentation", and to large measure is the whole reason for adopting OpenAPI/Swagger.

Unfortunately implementing the springfox-swagger-ui plugin in a Spring Boot Swagger application gives you a Swagger UI implementatation that's nowhere near as nice/capable as the demo app you see at petstore.swagger.io. There is zero documentation on how to implement the improved Swagger UI via the springfox-swagger-ui plugin. Instead, after quite a bit of experimentation, it seems one must turn to the Swagger UI Docker container.

How to easily edit a Swagger/OpenAPI API specification using free tools

(June 15, 2017) The official way to edit a Swagger/OpenAPI document is using the Swagger Editor available through the swagger.io website, or to use the SwaggerHub website. The Editor can be run on your laptop inside a web browser which allows local JavaScript execution. As browsers tighten the screws on security the ability to do that may cease, and I see in the issue queue a request to make Electron versions of the Swagger tools. The SwaggerHub website is another option, especially as it offers lots of interesting features (at a fee). But, since OpenAPI is a free and open specification it's possible for others to develop tools. In this post we'll go over using the Swagger Editor, and using Atom to edit an OpenAPI specification.

Introduction to Time Series and InfluxDB

(May 30, 2017) Influx DB is an easy-to-use time-series database, that uses a familiar query syntax, allows for regular and irregular time series, and is part of a broad stack of platform components. This video goes over what Time Series Data is, a comparison of different Time Series Databases, and more.

Time-series data with InfluxDB, overview

(May 17, 2017) Time-series databases are used for scenarios involving collecting time-stamped data. You probably want to generate summaries (rollups and aggregations), data retention policies, and so forth. That every datum is time-stamped makes it immediately different from regular SQL databases. InfluxDB is an open source time-series database.

Easily use Let's Encrypt to HTTPS-protect your own server, for free

(2017-04-11 14:47 PDT) The search engines and browser makers are telling us to encrypt all websites. A driving factor is to protect everyone from not only miscreants wanting to hijack the web for nefarious goals, but the government security agencies who are snooping into everything. If everything on the Web is encrypted, then we'll all be better off. Until Let's Encrypt came along, the requirement to encrypt carried with it a high cost of paying for SSL certificates, and therefore many website owners would be unable to keep going. The free Lets Encrypt service opens HTTPS up to regular folk, allowing all website owners to encrypt their web traffic irregardless of how deep their pockets are. With that in mind, let's look into what it takes to set up HTTPS using Let's Encrypt.

SwaggerHub 101 An Introduction to Getting Started with SwaggerHub

(March 3, 2017) As the title suggest, this shows how to use SwaggerHub in a software development team. SwaggerHub covers a full life-cycle of API development in a team. Each project or "Organization" in SwaggerHub can have members and various settings. It can automatically generate a "mock API server" to try out the API directly via SwaggerHub.

Using Docker to host ARM toolchain to cross-compile C code

(2017-01-19 16:27) I'm starting up a project that will see me doing custom software development for an ARM single-board-computer running Linux. The recommendation isn't to do compiles ON the board, but instead to cross compile from a Linux workstation (Debian). But, I use a Mac laptop, as do most software engineers these days. While I could run VirtualBox to set up a Debian cross-compiling environment, Docker is much lighter weight. While Docker was originally targeted for deploying server applications, it is useful for packaging anything. In this case there's a ready-made set of Docker containers for cross-compilation including for ARM CPU's.

Swagger tutorial -- How to add Swagger OpenAPI to your REST API documentation

(Jan 9, 2017) An introduction to Swagger/OpenAPI. A core idea given is documentation that's interactive, and unlike typical documentation as it appears in a slide deck. The power of having an API described in an industry-standard format is the wide variety of tools available.

Building an API with Swagger

(Jan 1, 2017) Tutorial about creating an API using Swagger.

Make your own Raspberry Pi git repository server with Gogs and Docker

(2016-10-09 13:38) The Raspberry Pi is an amazing little computer that, while it's targeted at the DIY Hardware Maker, it is a full-fledged Linux computer that can be used to run services that used to require much bigger and more expensive computers. How long ago were office servers required to be $4000 systems from the likes of Dell Computers? It seems that the Raspberry Pi (and other tiny computers) can perform the same tasks at a low cost with minuscule energy requirements. To this end I'm setting up Gogs (a github-like server for Git repositories) on a Raspberry Pi. As I worked on the project it seemed most straightforward to use Docker to manage the Gogs process, and therefore the project became setting up Docker on Raspberry Pi to run other services.

Swagger and Open API Spec @codecampnyc 2016

(Oct 9, 2016)

Moving from "cloud-first" to "cloud-only" and therefore "API's" are more important than ever. Concept of the API Economy. Hence necessity for a better method to design REST-ful API's, and help others to use those API's. Another problem is the "Distributed Monolith" which is interlocking dependencies between microservices. Swagger is the solution.

Language agnostic. Helps to generate excellent documentation. No guesswork. Clearly documented REST API's.

Make a bash script detect the directory it's stored in, to access data there

(2016-09-19 15:52)

You may need to write a bash shell script that accesses data stored alongside the script, while your current working directory might be elsewhere. In my case the shell script needed to use Node.js scripts stored next to the shell scripts -- the shell script acting to simplify running the Node.js scripts. The "data" to be accessed in this case is the Node.js scripts, plus the support modules required to run them. You may have other data like a list of hostnames or who-knows-what.

InfluxDB CLI and Configuration Options

(Jul 27, 2016) Influx DB is an easy-to-use time-series database, that uses a familiar query syntax, allows for regular and irregular time series, and is part of a broad stack of platform components. The TICK stack is a set of four components that together make it quick and easy to collect time series data and graph it for users. One uses the CLI because it's always there, and is much easier than using CURL on the REST API.

Fixing 'Enter passphrase for /dev/fd/63' in a Gitlab CI job

(2016-06-30 16:53) If you're a Gitlab user you're probably hoping to use Gitlab CI to automate builds and deployments. You probably want to deploy something using rsync, using an SSH key for security. Unfortunately (in my opinion) the official Gitlab documentation is confusing. While the Gitlab team does provide example .gitlab-ci.yml files that are supposed to work, the actual specifics of what to do are sketchy, and I found myself puzzling over a curious error message: "Enter passphrase for /dev/fd/63" ... WTF?

Create hierarchical navigation for custom post types in Wordpress with PODS

(2015-06-20 23:27) On (longtailpipe.com) my Wordpress site I want to create "documentation" areas with a cluster of pages organized as a hierarchy, with an index to those pages showing as a sidebar. Many think of doing this for "Product Documentation", but I simply want to organize notes and additional material related to some books that I'm writing. Just as a book is organized by chapters and subsections, I want to hierarchically organize these pages.

How to customize the per-post authoring byline in Wordpress

(Mon Apr 06 2015 17:00:00 GMT-0700 (Pacific Daylight Time))

CodeAnywhere - fast and flexible in-browser developers code editor

(2014-04-17 12:30:58 -0700) Codeanywhere is a code editor in a browser with an integrated ftp client, and all popular web formats are supported (HTML, PHP, JavaScript, CSS, and XML).

Converting a MySQL enum for use in SQLite3

(2012-01-06 21:11) I've got a database & website I want to move from using MySQL to using SQLite3.  Well, I think I want to use SQLite3.  Their document saying what sorts of uses make sense for SQLite3 are directly in line with my website, and I do want to remove some of the load off of my MySQL server so that it can have  cycles free for more important purposes.

However I've run into a couple troubles converting the schema so that it fits within SQLite3's limited SQL support.  Turns out that it doesn't support some column types and indexes.  And that the SQL produced by mysqldump contains some MySQLisms which SQLite3 just doesn't understand.

Implementing MySQL style AUTOINCREMENT in SQLite3

(Fri Jan 06 2012 16:00:00 GMT-0800 (Pacific Standard Time))

Multiple headed Mercurial problems

(2008-11-20 20:37) Supposedly people who imbibe mercury become crazy as, well, a madhatter (so named because old-style hat-making practices involve the use of mercury). So maybe it sounds crazy to talk about multiple heads but in this case the mercury is the Mercurial source code management system. In Mercurial a 'head' is the endpoint of a chain of changes and it becomes downright inconvenient when there are multiple endpoints in the chain of changes in your source repository.

Using Mercurial patch queues

(Wed Nov 19 2008 16:00:00 GMT-0800 (Pacific Standard Time))

Forbidding multiple heads in a shared mercurial repository

(Thu Oct 09 2008 17:00:00 GMT-0700 (Pacific Daylight Time))