Backup K3s with Litestream

This blog post will give you a tutorial on how to back up a K3s single node control plane backed by sqlite3 with Litestream.

K3s is a lightweight Kubernetes distribution targeting Edge workloads and small-scale Kubernetes setups. It bundles many core components like the kubelet, all control plane components, a container runtime, and flannel as container network implementation into a single binary. The binary can be used for running both, a control plane node and a worker node. Another core feature of K3s is the support of sqlite3 or other SQL databases instead of etcd as storage for the Kubernetes API state. Using a single master node with a sqlite3 instead of etcd has many advantages since it reduces the overall complexity of the system. The setup has fewer components one has to master. Especially in Edge workloads, you do not need a load balancer to distribute the traffic to multiple Kubernetes API server nodes. What is more, sqlite3 is a very well-tested and well-known data store with a very low resource footprint. The cost of this setup is reduced resilience since a single master system cannot automatically mitigate hardware failures, or failover to another node during planned maintenance. Kubernetes is designed as an eventually consistent system, so during a control plane node downtime, the existing workload runs without interruption, but no new workload can be scheduled or auto-scaling cannot happen. This is often a valid trade-off to make for small-scale Kubernetes setups.

Litestream can replicate a sqlite3 database to an object store, another file, or SFTP. Supported object store protocols are S3, Azure Blob Storage and Google Cloud Storage. Ben Johnson is the author of Litestream, he also wrote the key-value store boltdb, initially used by etcd. Litestream uses the write-ahead log (WAL) feature of sqlite3 to offer point-in-time restores. Therefore, Litestream upon start copies a snapshot of the sqlite3 database to the replication target and after that subsequently copies the WAL files into the replication target as well. The default synchronization interval for WAL files is 1 second. A more detailed introduction to the motivation behind Litestream and the inner mechanism can be found on the Litestream blog.

In the next few paragraphs, I want to show you an example setup of how to use sqlite3 to back up and restore a sqlite3 backed K3s control plane. To restore a K3s control plane of a running cluster, we only need the sqlite3 database and the K3S_TOKEN used for that cluster.

Step by Step Tutorial for a Backup with Litestream

This tutorial was written for Ubuntu 22.04. This should work for other Linux distributions as well, but you may need to use a different Litestream installation process. In this tutorial, we want to sync to a local minio installation. If you want to replicate this with another object store, skip the minio install and setup process. We start minio, via docker:

Install docker if not already installed:

Install jq and pwgen if not previously installed, we use these helper tools during the tutorial:

Install a minio instance and create a bucket:

Install Litestream:

After we have installed litestream, we can create a litestream configuration file:

Install K3s via the official install script:

Now, after we have installed K3s, we can start the Litestream replication:

With the Litestream tool, we can check the snapshots stored in our minio storage:

We can see the data structure in minio bucket as well:

After we have verified that our backup stores data at the target location, we can test the restore process. The most crucial step to verify a backup solution is to test the restoring process. To have some example data in our backup, we create a Kubernetes deployment:

Now, let us uninstall K3s:

Eventually, we have successfully uninstalled K3s and purged all data. The next step is to use Litestream to restore K3s sqlite database:

The next step is to reinstall K3s with the same K3S_TOKEN as before.

After the installation and restore, let’s check if our deployment is still there:

It is.
If you want to automate the Litestream restore process, the litestream restore command has two helpful flags:

The first flag, “-if-db-not-exists“ will only restore the database if there is no database currently present. If there is already a database, the file is not overwritten. “-if-replica-exists“ will only restore, if there is a replica in the S3 bucket. Otherwise, it will return with code 0. This is helpful on first deployments. This allows restore the database before you install K3s:

This allows you in, e.g., Edge Setups or setups where you handle the control plane as cattle to not make a distinction between the first install and subsequent starts/reinstallations.

Cleanup

Finally, let’s clean up our the K3s, litestream and minio installation:

That’s it. We have set up a database-level backup for our K3s clusters. This can be handy in lab setups, allowing you to revert your cluster to 3h ago, or in smaller installations where you only have a single control plane machine running. Be aware that if you restore a database-level backup of a Kubernetes cluster used by multiple tenants, there is a chance that you do a rollback of their applications if the backup is too old. Therefore, you should prefer a GitOps style workflow for your disaster recovery workflows, or ensure you continuously backup your cluster’s sqlite database.