The server powering practicalgobook.net

I write my blog posts in Markdown format, and then use Hugo static site generator to generate HTML files. The topic of this blog post is how those HTML files are served to you.

If you happened to come across my GopherCon 2021 lightning talk, this blog post describes the topic in all the gory detail!

Background

When you type in https://practicalgobook.net in your browser (or click it from somewhere), a translation, commonly known as DNS resolution happens. As a result of this translation, the browser gets back an IP address:

# Ran this command on a Linux system
$ dig +short practicalgobook.net # this is a command to perform DNS resolution
172.105.175.12

This IP address refers to a virtual machine running in the cloud. As of a few months back, I am running this virtual machine on a cloud provider. When the browser’s request reaches this virtual machine, a program then sends back the HTML files that were generated by Hugo. This program is written using the Go programming language. I use the standard libraries and two third party packages for implementing advanced features.

Let’s dive in!

Content delivery - Proof of Concept

The first working iteration of the server looked as follows:

package main

import (
	"embed"
	"log"
	"net/http"
	"os"
)

//go:embed posts code
//go:embed index.html index.xml sitemap.xml
//go:embed categories css images
var siteData embed.FS

func main() {
	listenAddr := ":80"
	if len(os.Getenv("LISTEN_ADDR")) != 0 {
		listenAddr = os.Getenv("LISTEN_ADDR")

	}
	mux := http.NewServeMux()
	staticFileServer := http.FileServer(http.FS(siteData))
	mux.Handle("/", staticFileServer)

	log.Fatal(http.ListenAndServe(listenAddr, mux))
}

The key standard libraries used in this iteration were embed and net/http.

The embed package allowed me build an executable Go application containing all the blog content. As you can see in the //go:embed directives, I include all the directories that hugo generated in the default, public directory inside the application as a variable siteData of embed.FS type.

Once this was done, I use the http.FileServer handler to serve the files that were embedded and available to the application via the siteData variable.

http.FileServer expects an argument of a type which implements the http.FileSystem interface.

siteData which is of type embed.FS implements the fs.FS interface and hence, we use the http.FS function to convert siteData to a value of type http.FileSystem and thus, we have the following code snippet above:

mux := http.NewServeMux()
staticFileServer := http.FileServer(http.FS(siteData))
mux.Handle("/", staticFileServer)

Then, we call the ListenAndServe() function to start the HTTP server on the address specified in listenAddr.

To summarize, at this stage, I had:

Next, I set out to implement support for making the website acessible over HTTPS and without paying for the very expensive load balancer.

Content Delivery - Caddy as a reverse proxy

During this stage, I made a couple of changes:

Hence, I would need a way to manage the TLS certificates on the virtual machine myself.

I knew that I am going to use https://letsencrypt.org/ for TLS certificates. I considered managing TLS certificates in the Go server itself using one of the options listed here. However, quickly realizing it may not be a battle for the day, I gave Caddy a go.

I installed Caddy on my CentOS VM, and wrote the following Caddyfile:

practicalgobook.net {
    reverse_proxy localhost:8080
        handle_errors {
	     respond "I will be back soon!"
        }
}

And that’s it! I started caddy using the systemd service, started my Go application using a systemd service and I had HTTPS working again without the expense of running a cloud managed load balancer.

To summarize, at this stage, I had:

I was happy with the progress at this stage, but then I found an issue.

When I wanted to update my blog, I had to stop and start my application manually and which meant there was downtime.

And considering that this is a very important website, I couldn’t have that. So I set out to fix that.

Automating restart on updates

My first plan was, “yeah, I will create a new executable and simply overwrite the existing one in place and then find a way to reload it”. You can’t, you will get a “Text file busy” error as explained here. Now, of course, even if I could copy it in place, reloading itself would need some work, notable have a way to indicate to the program and “please reload yourself”. And that exploration led me to slayer/autorestart package and I integrated it as follows:

package main

import (
	"embed"
	"log"
	"net/http"
	"os"

	"github.com/slayer/autorestart"
)

//go:embed buy categories code css images posts support tags toc
//go:embed book_cover.jpg go.mod index.html index.xml sitemap.xml
var siteData embed.FS

func main() {
	listenAddr := ":8080"
	if len(os.Getenv("LISTEN_ADDR")) != 0 {
		listenAddr = os.Getenv("LISTEN_ADDR")

	}
	mux := http.NewServeMux()
	staticFileServer := http.FileServer(http.FS(siteData))
	mux.Handle("/", staticFileServer)

	// Notifier
	restart := autorestart.GetNotifier()
	go func() {
		<-restart
		log.Printf("Detected change in binary. Restarting.")
	}()

	autorestart.StartWatcher()
	log.Fatal(http.ListenAndServe(listenAddr, mux))
}

At this stage, my blog updates looked like:

$ GOOS=linux GOARCH=amd64 go build
$ scp server root@<ip-address>:/usr/local/bin/practicalgo-website-1
$ ssh root@ip mv /usr/local/bin/practicalgo-website-1 /usr/local/bin/practicalgo-website

I get around the “Text file busy” error by creating a temporary file and then using the mv command which uses the atomic rename() system call.

And thanks to slayer/autorestart, my application reloads itself and my new content becomes available to the world wide web!

To summarize, at this stage, I had:

However, for the automatic update to be running, the old process was being terminated and a new process being created, which meant, connections were being dropped and that is simply not an option.

So, time to fix that.

Zero downtime updates

Now, I “knew” that on Linux, network sockets are file descriptors and that you child processes inherit parent’s opened file descriptors. But I didn’t know enough to implement it myself. From the README of slayer/autorestart, I saw a reference to an unmaintained project facebookgo/grace which seemed very relevant to what I was looking for. So, I integrated my server with the grace/gracehttp package:

package main

import (
	"embed"
	"log"
	"net/http"
	"os"

	"github.com/facebookgo/grace/gracehttp"
	"github.com/slayer/autorestart"
)

//go:embed buy categories code css images posts support tags toc
//go:embed book_cover.jpg go.mod index.html index.xml sitemap.xml
var siteData embed.FS

func main() {
	logger := log.New(os.Stdout, "INFO: ", log.Lshortfile)
	listenAddr := ":8080"
	if len(os.Getenv("LISTEN_ADDR")) != 0 {
		listenAddr = os.Getenv("LISTEN_ADDR")

	}
	mux := http.NewServeMux()
	staticFileServer := http.FileServer(http.FS(siteData))
	mux.Handle("/", staticFileServer)

	srv := http.Server{
		Addr:    listenAddr,
		Handler: mux,
	}

	autorestart.RestartFunc = autorestart.SendSIGUSR2
	restart := autorestart.GetNotifier()
	go func() {
		<-restart
		logger.Printf("Detected change in binary. Restarting.")
	}()

	autorestart.StartWatcher()

	gracehttp.SetLogger(logger)
	logger.Fatalf("Server terminating: %v", gracehttp.Serve(&srv))
}

I alluded to running the Go server as a systemd service above, here’s the systemd service file:

[Unit]
Description=Practical Go Website

[Service]
Environment="LISTEN_ADDR=:8080"
ExecStart=/usr/local/bin/practicalgo-website
User=nobody

[Install]
WantedBy=multi-user.target

I also added in systemd socket activation for my server using the following .socket file:

[Socket]
ListenStream = 8080
BindIPv6Only = both

[Install]
WantedBy = sockets.target

The summary of the above changes results in the following behavior:

Thus, as it stands today:

Summary

In the book, we make heavy use of net/http standard library package and I allude to how useful the embed package can be. I hope you found this post useful and showed you a practical way to use them together. As I mentioned in my GopherCon 2021 lightning talk, there was no problem to solve here, but it proved to be a very fulfilling learning exercise for me. While writing this post, especially, the last section, I sensed a disconnect in my understanding between why I needed the socket activation, other than the initial automatic startup of the server. Was it also needed for the graceful restart? I will need to dig in a bit more again.

You can find the source code for the server in the file, server.go and other resources that are needed to deploy here.

If you are curious to learn more systemd socket activation and Go applications, please refer this post: https://mgdm.net/weblog/systemd-socket-activation/.

Finally, I got very curious about how facebook/grace/gracehttp worked and was able to implement a proof of concept myself only for TCP network sockets from scratch. Here’s some code, without any documentation here.

I have a plan to replace my use of both the third party libraries with my code for this server.