Technical on Golder.org

How This Site is Built: A Modern DevOps Pipeline

Wed, 02 Jul 2025 00:00:00 +0000

Architecture Overview

The site follows a GitOps approach, with every component self-hosted across multiple geographic regions:

Source Control: Self-hosted Gitea Git repository
Static Site Generator: Hugo with custom theme
Build Pipeline: Argo Workflows
Container Registry: Harbor
Orchestration: Kubernetes clusters across multiple regions
Deployment: Flux GitOps operator
Networking: Home networks with VPN interconnects

Static Site Generation with Hugo

The foundation is Hugo, a fast static site generator written in Go. Hugo was chosen for several key advantages:

Packer node builder

Thu, 28 Nov 2024 00:00:00 +0000

Introduction

In this article I will discuss my k8s-node-packer folder, which contains an environment I use to build fresh VM images which I then use to deploy VMs on various bare metal hosts running libvirtd.

Getting started

You will need a recent version of Packer. To summarise the installation for Ubuntu…

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install packer

Ensure some tools that will be needed are installed:

Enable Kubernetes auth on Hashicorp Vault

Wed, 24 Jan 2024 00:00:00 +0000

Introduction

In this post I describe the steps taken to enable a Kubernetes authentication backend for Hashicorp Vault and configure a PKI role for cert-manager to use to issue certificate signing requests.

Process

Start by enabling the auth method.

vault auth enable --path=k8s-yourdomain-production kubernetes

You should see:

Success! Enabled kubernetes auth method at: k8s-yourdomain-production/

The cluster needs a ’token reviewer’ service account that Vault can use to validate tokens presented by clients that claim to have been authenticated by the cluster.

Granting TLS user access to a Kubernetes Cluster

Tue, 23 Jan 2024 00:00:00 +0000

Introduction

I run up clusters with kubeadm. Once a cluster is up and running, it can be enticing to just copy the /etc/kubernetes/admin.conf credentials down to your local .kube/config file and crack on. However, as you can guess, that’s not best practice, especially when working in teams.

In order for audit logs to contain more specific details about which user performed which actions, and for general access control purposes, it helps a lot if users authenticate to clusters using their own credentials, not using shared credentials.

Granting an OIDC user access to a Kubernetes Cluster

Sun, 21 Jan 2024 00:00:00 +0000

Introduction

Adding new user to Redis Cluster

Thu, 04 Jan 2024 00:00:00 +0000

Introduction

I run Redis Cluster in K8S. I need to manage client access for service accounts.

Currently I do this manually using the following process.

Connect to Redis Cluster

Ensure your kubectl is configured to use the appropriate cluster.

export IP=$(kubectl -n redis get svc redis-redis-cluster -ojsonpath='{.spec.clusterIP}')
export REDISCLI_AUTH=$(kubectl -n redis get secret --namespace "redis" redis -o jsonpath="{.data.redis-password}" | base64 -d)
redis-cli -c -h $IP

This should give you a redis-cli prompt like the following:

Renumber cluster IP addresses

Sun, 14 May 2023 00:00:00 +0000

So, about a year after I redeployed my in-house staging cluster, I’ve reached a point where I’m not happy about the addressing scheme I’ve used and how it affects the routing on my internal network. So this article is going to try to cover the steps I took to re-number the whole cluster, and put all the nodes into their own LAN segment too to physically segregate/isolate it’s traffic for performance and security reasons.

AWS SSO with Keycloak via SAML

Tue, 12 Jul 2022 00:00:00 +0000

Introduction

Keycloak is a popular Java-based SAML/OpenID identity service provider application that can be self-hosted. It’s a very comprehensive package and widely supported, so even if it is a bit of a ‘heavyweight’ (resource and time-wise) to set up, host and support, it’s a more sensible and mature solution to adopt for now, and there doesn’t seem to be anything that fits our needs yet in the wider open source world.

Migration from WordPress to Hugo

Fri, 08 Oct 2021 00:00:00 +0000

I have three public-facing sites which I’d rather not have to manage with WordPress any more. None of them do anything fancy. They’re just a collection of pages and posts. I span them up and started adding content in the days where WordPress was all the rage. Recently, I’ve been looking at ditching the WordPress hosting burden and converting them to simple, static websites. This will save me time and money in the long run.

Development Snapshot Download Service

Sat, 06 Jun 2020 00:00:00 +0000

Suppose the database or state data for your production application is being backed up in full on a regular basis, encrypted and stored on an object storage service, such as S3. Suppose the backup tool you are using keeps track of successful backups in some kind of lookup service, such as an ElasticSearch index, database, etcd cluster or whatever. Suppose one of your developers needs a recent database snapshot to seed their local development environment to develop a feature or reproduce a bug.

Azure CLI can't find antlr4 python library

Sat, 14 Mar 2020 00:00:00 +0000

I’m trying to run up a VM on Azure using CLI commands on my Ubuntu 20.04 (Focal) workstation. I’m following the official instructions. On running the az vm create command, I get the following error:

The command failed with an unexpected error. Here is the traceback:

No module named 'antlr4'
Traceback (most recent call last):
 File "/usr/lib/python3/dist-packages/knack/cli.py", line 206, in invoke
 cmd_result = self.invocation.execute(args)
 File "/usr/lib/python3/dist-packages/azure/cli/core/commands/__init__.py", line 528, in execute
 self.commands_loader.load_arguments(command)
 File "/usr/lib/python3/dist-packages/azure/cli/core/__init__.py", line 300, in load_arguments
 loader.load_arguments(command) # this adds entries to the argument registries
 File "/usr/lib/python3/dist-packages/azure/cli/command_modules/vm/__init__.py", line 31, in load_arguments
 from azure.cli.command_modules.vm._params import load_arguments
 File "/usr/lib/python3/dist-packages/azure/cli/command_modules/vm/_params.py", line 31, in <module>
 from azure.cli.command_modules.monitor.actions import get_period_type
 File "/usr/lib/python3/dist-packages/azure/cli/command_modules/monitor/actions.py", line 7, in <module>
 import antlr4
ModuleNotFoundError: No module named 'antlr4'

Looks like the Azure CLI packages for Ubuntu have a dependency on ‘antlr4’ python library, and that isn’t being packaged or distributed upstream for whatever reason. In my case I’m using the Ubuntu Focal (20.04) series.

Dynamic DNS with external-dns

Sat, 14 Mar 2020 00:00:00 +0000

This came about from deciding to switch from Route53 to Cloudflare for DNS management. I’d already prepared the destination zones on Cloudflare but needed to update the mechanism I had in place that manages the DNS entry for the dynamic IP address. The mechanism was a NodeRED flow that made an API call directly to Route53. I’d need to do some research on how to do the same on Cloudflare.

Deploying OPNSense to no-name WAN router hardware

Fri, 10 Jan 2020 00:00:00 +0000

I live and work in a remote area, which does not have the most reliable power and internet connections. I’ve addressed the intermittent power outages by installing a battery back-up system which covers the server cabinet and various other circuits in the house. I’ve addressed the intermittent network issues by deploying a multi-WAN router, which is what this article is about.

The WAN router also provides me with an extra NAT layer between my home/work network and the Internet, as you never know if you can trust ISP-provided routers. Additionally, it provides me and members of my team with OpenVPN access to resources on the network when we’re out and about, and much more besides.

Grant read-only rights to backup user for PostgreSQL

Mon, 29 Jul 2019 00:00:00 +0000

This one comes up quite regularly, so worthy of note. Whenever an application using Postgres changes or creates tables (i.e. Odoo module updates for example), those tables don’t seem to be accessible by the backups user on the next backup run. This is the SQL I use to work around it:

GRANT SELECT ON ALL TABLES IN SCHEMA public to backups;
GRANT SELECT ON ALL SEQUENCES IN SCHEMA public to backups;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO backups;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON SEQUENCES TO backups;

It’d be nice to find a way for the backups user to automatically get read permissions, but I’ve yet to figure that out. I understood that the above ALTER DEFAULT PRIVILEGES would cover it, but I seem to need to run it myself each time there is a change.

Managing Kubernetes certificates with Python

Wed, 15 May 2019 00:00:00 +0000

I run into a small stumbling block the other evening while working on my ‘site domain manager’ project (for want of a better name). This is essentially a REST API running in a daemon service that manages the mappings of domains to websites, and uses ‘agents’ to automate the configuration via API calls to the various services involved (domain registrars, DNS servers, WAF providers, SSL certs etc.)

The problem arose because the manager class I was writing to interact with kubernetes needed to manage certificates. The kubernetes python client library has a whole bunch of useful higher-level API and model classes for listing, creating, updating the main models I needed to manage, such as the ConfigMap, DaemonSet, Service and Ingress, but because the Certificate is part of the cert-manager package, it doesn’t have the equivalent higher-level methods I needed.

Testing SMTP creds with Docker

Sun, 24 Mar 2019 00:00:00 +0000

One of our sites stopped sending it’s mail a few days ago. Unfortunately, the SMTP plugin used does not provide any debug logs of the SMTP connection, and it’s ’test’ tool just says that it sent the mail successfully. The logs for the SMTP service provider suggest they haven’t seen the connection. I issue a new password and reconfigure, still the same symptoms.

So, I just want to do a simple check of the new SMTP creds. I also wanted to use a simple tool to avoid having to recall (Google) the magic SMTP protocol incantations and perform them via ’telnet’ or whatever. The ‘ssmtp’ tool sprung to mind for some reason, so I figured I’d see how simple that would be to use in this case.

Monitoring Windows processes from Nagios

Tue, 12 Mar 2019 00:00:00 +0000

So, something I had to do recently was to set up monitoring for a couple of specific Windows processes, so that we get notification via a Discord channel if those processes are not running on various hosts.

Typically, you’d do this with something like NSClient++ but this was proving to be too problematic and time-consuming to get the client side configured and working correctly.

So, I decided to try a different approach to keep things quick and simple. However, things turned out to be neither quick nor simple. I’ll save the rant, needless to say I generally leave managing our Windows hosts to our Windows people. I haven’t used Windows since ‘98. I typically steer clear, and I’m not even sure why I took this on, or why I’m writing this up. I hope it helps someone else, but I personally hope never to have to come back here!

Nginx Ingress access logs to ElasticSearch via syslog

Mon, 25 Feb 2019 00:00:00 +0000

There are several ways to extract the logs from a Kubernetes Nginx Ingress deployment into an ElasticSearch instance. One way I found was to use Elastic Filebeats, but I couldn’t find any really good examples of how to apply that to our cluster, and I felt it would clutter up the proxy servers with more containers they may not necessarily need.

Instead, I chose to use nginx’s syslog facility, which is a little more lightweight, and serves our purposes for now.

Simple LDAP proxy container

Tue, 10 Apr 2018 00:00:00 +0000

So, you have an LDAP server running happily on port 636 but one of your client applications doesn’t seem to be happy with the SSL connection for whatever reason. You need an intermediary container to handle the SSL connection to the LDAP server on port 636, presenting it to the local application on port 389.

First, we write a Dockerfile that will describe a container that runs up an haproxy daemon.

Merging Confluence users

Tue, 30 Sep 2014 00:00:00 +0000

One of my clients is running Confluence. Somewhere along the line, two user accounts had been created for one user, and content had been added using both users.

So muggins here to the rescue. Unfortunately, not much help to be found Googling, so I roll up my sleeves and dig into the Confluence DB schema. Oh what fun.

The main users table appears to be the ‘user_mapping’ table, where each user has a record. The user appears to have a long hash-like ID that represents them in any other records. I chose the ID of the account I would be merging from, and attempted to find the other tables involved in the database that would link to the record I am about to delete.

Building a Resilient Rural Network Infrastructure

Fri, 15 Aug 2014 00:00:00 +0000

Designing and implementing a robust dual-WAN network with wireless mesh connectivity to serve multiple households in rural Thailand, featuring automatic failover and centralized monitoring.

GStreamer pipeline for RTSP stream

Tue, 21 May 2013 00:00:00 +0000

A simple gstreamer pipeline to display at RTSP stream (from an Aircam)…

gst-launch -m rtspsrc location=rtsp://172.16.2.251/live/ch00_0 ! rtph264depay ! ffdec_h264 ! ffmpegcolorspace ! autovideosink [/code]

Harvest e-mail addresses from stdin

Sat, 18 May 2013 00:00:00 +0000

A little python script to do this:

#!/usr/bin/env python

import sys import re

bulkemails = sys.stdin.read()

# regex = whoEver@wHerever.xxx r = re.compile("\[-a-zA-Z0-9.\_\]+@\[-a-zA-Z0-9\_\]+.\[a-zA-Z0-9\_.\]+") results = r.findall(bulkemails)

emails = "" for x in results: print str(x)

Magento API problem setting additional_attributes

Sat, 18 May 2013 00:00:00 +0000

For some reason, it’s damn hard to get the additional_attributes set via Magento’s v2 API. Even the example code in their API docs doesn’t cover it. After trying many permutations, I finally managed to get it working with the following snippet of code:

<?php

$soapopts = array('trace' => 1, 'exceptions' => 1, 'features' => SOAP_SINGLE_ELEMENT_ARRAYS); $client = new SoapClient ( 'http://www.yourmagentosite.com/api/v2_soap/?wsdl', $soapopts); $session = $client->login ( 'apiuser', 'apipassword' );

$productData = (object)array( 'additional_attributes' => (object)array( 'single_data' => array( (object)array( 'key' => 'custom_image_url', 'value' => 'http://www.yourmagentosite.com/nicepic.jpg', ), ), ), ); $result = $client->catalogProductUpdate($session, 'abjb91', $productData); print $client->__getLastRequest(); var_dump($result);