How to Set Up a K3s Cluster on WireGuard

Lesezeit
10 ​​min

This blog post is a tutorial on how to set up a WireGuard network using Raspberry Pis for a K3s cluster, with each device connected to a central cloud virtual machine (VM). 

At the core of inovex are the values innovation and excellence. If we want to be and remain innovative and excellent in a dynamic environment, we have to take care of it. Over the years we have developed a series of measures with which we ensure our high knowledge and quality standards. One of these are our Tech Days, where colleagues from a technology department meet and train together with and on cool technologies.

Our last Tech Day was held remotely due to the coronavirus. As a socialising event, we had a hackathon with every participant getting a Raspberry Pi sent home beforehand. My group had the idea to build a K3s cluster spanning across the desks of all team members. Since we were at home and most of our home networks are behind at least one NAT device, we had to build a VPN connecting our Raspberry Pis. In this blog post I want to show our initial prototype setup with a step by step tutorial to reproduce the setup.

Initial Setup

In our prototype we decided to build the WireGuard network as a star topology: every Raspberry Pi is connected to a central cloud VM. We did this since the VM has a stable public IP and thus every Raspberry Pi is able to initiate a WireGuard tunnel to the VM. The VM will also host our K3s master.

The Raspberry Pis are hosting the K3s agents. (After checking the network ranges of the participants’ home setups to avoid ip range collisions) we chose the network 10.222.0.0/24 for our nodes in the WireGuard network.
The next to section will talk you through the setup of the cloud VM and the Raspberry Pis.

Cloud VM

This section talks you through the installation and configuration of the cloud VM. All shell commands must be executed as the root user. As a convention we start the start of a shell command with a $ in our listings. You do not have to type that into your shell.

Wireguard

We choose Ubuntu 20.04 as the operating system for our cloud VM. The first step was to install the WireGuard package:

$ apt-get update && apt-get install wireguard

Since we want to route traffic between the Raspberry Pis, we had to enable IP forwarding on the cloud VM

$ sysctl -w net.ipv4.ip_forward=1

The next step is to generate the public and the private keys for our WireGuard tunnel:

$ wg genkey | tee privatekey | wg pubkey > publickey

With these two keys we are able to set up the WireGuard wg0 interface. Place the following config in /etc/wireguard/wg0.conf:

# /etc/wireguard/wg0.conf
[Interface]

# The IP address of this host in the wireguard tunnels
Address = 10.222.0.1

# Every Raspberry Pi connects via UDP to this port. Your Cloud VM must be reachable on this port via UDP from the internet.
ListenPort = 51871

# Set the private key to the value of the privatekey file generated by the previous command
PrivateKey = yBo18fnFVjKrRS0dfH0DDehGrVBH1aDaZValIwdEW1I=

# Set the MTU according to the internet connections of your clients
# In our case the autodetection assumed 8920 since the cloud network supported jumbo frames.
MTU = 1420

With this configuration in place we can bring up the wg0 interface:

$ wg-quick up wg0

After this step, your cloud VM has a network interface wg0 with the IP address 10.222.0.1 assigned. We set a static route for the network 10.222.0.0/24 to that interface:

$ ip -4 route add 10.222.0.0/24 dev wg0

This allows us to dynamically add new WireGuard clients to our network.

K3s

The next step is to install and configure the K3s master. We did this via the following oneliner:

$ curl -sfL https://get.k3s.io | sh -

We recommend replacing this with a proper installation routine in a production setup since executing foreign scripts from the internet with root privileges is a security nightmare.
After the installation has finished we have to reconfigure the flannel CNI in K3s to use the wg0 interface. Place this systemd drop in /etc/systemd/system/k3s.service.d/network.conf

# /etc/systemd/system/k3s.service.d/network.conf 
[Service]
ExecStart=
ExecStart=/usr/local/bin/k3s server --advertise-address 10.222.0.1 --flannel-iface=wg0

and restart the k3s systemd service:

$ systemctl daemon-reload && systemctl restart k3s.service

To check your K3s installation, run:

$ kubectl get nodes
NAME STATUS ROLES AGE VERSION

master Ready master 5m v1.18.9+k3s1


Raspberry Pis

The steps described in this section must be repeated for every Raspberry Pi you want to connect to your cluster.

In our prototype, we use the Ubuntu 20.04 image from the rasperry pi imager utility on a Raspberry Pi 4 with 2 GB RAM. Since we want to install K3s on the nodes, we had to modify the kernel command line to enable cgroup memory management. After installing the Ubuntu 20.04 image to a SD card, mount the SD card on your PC and append

cgroup_memory=1 cgroup_enable=memory

to the boot/cmdline.txt file. After the first boot you should change the hostname of the Raspberry Pi to something unique, since K3s uses the hostname as kubernetes node name by default.

“(On raspbian) this can be done with the „sudo raspi-config“ command or just editing the „/etc/hostname“ file and rebooting”

Install the WireGuard package via:

$ apt-get update && apt-get install wireguard

Generate a public and a private key for the Raspberry Pi:

$ wg genkey | tee privatekey | wg pubkey > publickey

Connecting the Raspberry Pi to the cluster

To connect the Raspberry Pi to the WireGuard network and the K3s cluster we have to exchange some information.

From the Raspberry Pi we need:

  • The IP address of the Raspberry Pi in the WireGuard network
  • The WireGuard public key generated on the Raspberry Pi

From the cloud VM we need:

  • The WireGuard public key generated on the cloud VM
  • The K3s token located in /var/lib/rancher/k3s/server/token
  • The public IP of the VM

Let’s assume the following information:

  • IP address of the Raspberry Pi in the network: 10.222.0.106
  • WireGuard public key of the Raspberry Pi: csQQ8c7waCFksyIQyCOIu/eqxaGUxueu8h02qr1f81Q=
  • WireGuard public key of the cloud VM: rcflbneYW/3wVQy8H/jDi/oGlLgyrC4vmJvt4YJOVmw=
  • K3s token: K10a5fe90072692d561a72e6f793ec8392748c4e9b604e81d644c755f9dd6207::server:b730afa6f2572b914be9f226fc774
  • Public IP of the cloud VM: 192.0.2.20

The first step is to add the Raspberry Pi as a peer to our WireGuard network on the cloud VM:

root@cloud-vm$ wg set wg0 peer csQQ8c7waCFksyIQyCOIu/eqxaGUxueu8h02qr1f81Q= allowed-ips=10.222.0.106/32


After we have allowed the Raspberry Pi to join our WireGuard network we configure the wg0 interface on the Raspberry Pi. Place the following configuration in /etc/wireguard/wg0.conf on the Raspberry Pi:

# /etc/wireguard/wg0.conf
[Interface]
# The IP address of the Raspberry Pi in the wireguard network
Address = 10.222.0.106/32

# Private key of the Raspberry Pi
PrivateKey = IG10ERcQaBIH0MA/thDn5Yo1XM9tJZXwJNVwpQuCFn4=

[Peer]
# Public Key of the cloud VM
PublicKey = rcflbneYW/3wVQy8H/jDi/oGlLgyrC4vmJvt4YJOVmw=

# Public IP of the cloud VM
Endpoint = 192.0.2.20:51871

# All traffic for the wireguard network should be routed to our cloud VM
AllowedIPs = 10.222.0.0/24

# Since our Raspberry Pis are located behind NAT devices, send keep alives to our cloud VM to keep the connection in the NAT tables.
PersistentKeepalive = 29

After creating this configuration run:

root@pi$ wg-quick up wg0

Now you should be able to ping the cloud VM via the WireGuard network:

root@pi$ ping -c 4 10.222.0.1
PING 10.222.0.1 (10.222.0.1) 56(84) bytes of data.
64 bytes from 10.222.0.1: icmp_seq=1 ttl=64 time=16.0 ms
64 bytes from 10.222.0.1: icmp_seq=2 ttl=64 time=15.9 ms
64 bytes from 10.222.0.1: icmp_seq=3 ttl=64 time=16.5 ms
64 bytes from 10.222.0.1: icmp_seq=4 ttl=64 time=15.7 ms

--- 10.222.0.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 15.726/16.035/16.473/0.275 ms

The next step is to install the K3s agent. We used this oneliner on our prototype:

root@pi$: curl -sfL https://get.k3s.io | K3S_URL=https://10.222.0.1:6443 K3S_TOKEN=K10a5fe90072692d561a72e6f793ec8392748c4e9b604e81d644c755f9dd6207::server:b730afa6f2572b914be9f226fc774  sh -


Replace the K3s_TOKEN with the token generated on your cloud VM. The token is in the file /var/lib/rancher/k3s/server/token on the cloud VM.

As already mentioned, please replace this step with a proper installation routine in a production setup to avoid the security nightmare of running unknown scripts from the internet with root permissions.

After the installation we have to configure the flannel overlay. Create the following systemd drop in:

# /etc/systemd/system/k3s-agent.service.d/node-ip.conf
[Service]
ExecStart=
ExecStart=/usr/local/bin/k3s agent --node-ip 10.222.0.106 --flannel-iface=wg0

Then run:

root@pi$ systemctl daemon-reload && systemctl restart k3s-agent.service

Final Step

The last step is to check if the Raspberry Pi joined your K3s cluster. Check via kubectl on the cloud VM:

root@cloud-vm$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 15m v1.18.9+k3s1
pi Ready <none> 1m v1.18.9+k3s1

With this setup we managed to build a K3s cluster in half a day with 22 nodes in many cities across Germany including Hamburg, Cologne and Karlsruhe.

Pitfalls

If you use raspbian, you have to build the kernel module for WireGuard by yourself. Make sure that the correct kernel headers for the correct running kernel are used when building, especially if you have previously made an “rpi-update”.

Hat dir der Beitrag gefallen?

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert.

Ähnliche Artikel