Cluster, Part 4: Weaving Wormholes | Peer-to-Peer VPN with WireGuard
Hey - welcome back! Last week, we set Unbound up as our primary DNS server for our network. We also configured cluster member devices to use it for DNS resolution. As a recap, here are links to the all the posts in this series so far:
- Cluster, Part 1: Answers only lead to more questions
- Cluster, Part 2: Grand Designs
- Cluster, Part 3: Laying groundwork with Unbound as a DNS server
In this part, we're going to setup a WireGuard) peer-to-peer VPN. This is a good idea for several reasons:
- It provides defence-in-depth
- It protects non-encrypted / unprotected private services from the rest of the network
The latter point here is particularly important - especially if you've got other device on your network like I have. If you're somehow following along with this series with devices fancy enough to have multiple network interfaces, you can connect the 2nd network interface of every server to a separate switch, that doesn't connect to anywhere else. Don't forget that you'll need to setup a DHCP server on this new mini-network (or configure static IPs manually on each device, which I don't recommend) - but this is out-of-scope of this article.
In the absence of such an opportunity, a peer-to-peer VPN should do the trick just as well. We're going to be using WireGuard), which I discovered recently. It's very cool indeed - and it's apparently special enough to be merged directly into the Linux Kernel v5.6! It's been given high praise from security experts too.
What I like most is it's simplicity. It follows the UNIX Philosophy, and as such while it's very simple in the way it works, it can be used and applied in so many different ways to solve so many different problems.
With this in mind, let's get to installing it! If you're on an Ubuntu or Debian-based machine, then you should just be able to install it directly:
sudo apt install wireguard
Note that you might have to have your kernel's development headers installed if you experience issues. For Raspbian users (like me), installation is slightly more complicated. We'll need to setup the
debian-backports apt repository to pull it in, since the Debian developers have backported it to the latest stable version of Debian (e.g. like a hotfix) - but Raspbian hasn't yet pulled it in. This is done in 2 steps:
# Add the debian-backports GPG key sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 04EE7237B7D453EC 648ACFD622F3D138 # Add the debian-backports apt repo echo 'deb http://httpredir.debian.org/debian buster-backports main contrib non-free' | sudo tee /etc/apt/sources.list.d/debian-backports.list
Now, we should be able to update to download the new repository metadata:
sudo apt update
Next, we need to install the Raspberry Pi Linux kernel headers. Unlike other distributions (which use the
linux-headers-generic package), this is done in slightly different way:
sudo apt install raspberrypi-kernel-headers
This might take a while. Once done, we can now install WireGuard itself:
sudo apt install wireguard
Again, this will take a while. Don't forget to pay close attention to the output - I've noticed that it's fond of throwing error messages, but not actually counting it as an error and ultimately claiming it completed successfully.
With this installed, we can now setup our WireGuard peer-to-peer VPN. WireGuard itself works on a public-private keypair per-device setup. A device first generates a keypair, and then the public key thereof needs copying to all other devices it wants to connect to. In this fashion, both a peer-to-peer setup (like we're after), and a client-server setup (like a more traditional VPN such as IPSec or OpenVPN) can be configured.
The overhead of configuring such a peer-to-peer WireGuard VPN is considerable though, since every time a device is added to the VPN every existing device needs updating to add the public key thereof to establish communications between the 2.
While researching an easier solution to the problem, I came across wesher, which does much of the heavy-lifting for you. It does of course come at the cost of slightly reduced security (since the entire VPN network is protected by a single pre-shared key) and reduced configurability, but from my experiences so far it works like a charm for my purposes - and it eases management too.
It is distributed as a single Go binary, that uses the Raft Consensus Algorithm (the same library that Nomad and Consul use actually) to manage cluster membership and provide self-healing properties. This makes is very easy for my to package into an apt package, which I've added to my apt repository.
The instructions to add my apt repository can be found on it's page, but here they are directly:
# Add the repository echo "deb https://apt.starbeamrainbowlabs.com/ ./ # apt.starbeamrainbowlabs.com" | sudo tee /etc/apt/sources.list.d/sbrl.list # Import the signing key wget -q https://apt.starbeamrainbowlabs.com/aptosaurus.asc -O- | sudo apt-key add - # Alternatively, import the signing key from a keyserver: sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys D48D801C6A66A5D8 # Update apt's cache sudo apt update
Don't forget that if you're using a caching server like
apt-cacher-ng (which we'll be setting up in the next post in this series), you'll probably want to change the
https in the first command there to regular
http in order for the caching to take effect. Note that this doesn't affect the security of the downloaded packages, since apt will verify the integrity and GPG signature of all packages downloaded. It only affects the privacy of the connection used to download the packages themselves (more on this in a future post).
Once setup, install wesher like so:
sudo apt install wesher
If you've got a systemd-based system (as I have sadly with Raspbian), I provide a systemd service file in a separate package:
sudo apt install wesher-systemd
Don't forget to perform these steps on all the machines you want to enter the cluster. Next, we need to configure our firewall. I'm using UFW - I hope you set up something similar when you first configured your servers you're clustering (I recommend UFW. Note also that this tutorial series will not cover the basics like this - instead, I'll link to other tutorials and such as I think of them). For UFW users, do this:
sudo ufw allow 7946 comment wesher-gossip sudo ufw allow 51820/udp comment wesher-wireguard
Wesher requires 2 ports: 1 for the clustering traffic for managing cluster membership, and another for the WireGuard traffic itself. Now, we can initialise the cluster. This has to be done on an interactive terminal for the first time, to avoid leaking the cluster's pre-shared key to log files. Do it like this in a terminal:
It should tell you that it's generated a new cluster key - it will save it in a private configuration directory. Save this somewhere safe, such as in your password manager. Now, you can press Ctrl + C to quit, and start the systemd service:
sudo systemctl enable --now wesher.service
It's perhaps good practice to check that the service has started successfully:
sudo systemctl status wesher.service
Having 1 node setup is nice, but not very useful. Adding additional nodes to the cluster is a bit different. Follow this tutorial up to and including the installation of the
wesher-systemd packages, and then instead of just doing
sudo wesher, do this instead:
sudo wesher --cluster-key CLUSTER_KEY_HERE --join IP_OF_ANOTHER_NODE --overlay-net 172.31.250.0/16 --log-level info
CLUSTER_KEY_HERE with your cluster key (don't forget to prefix the entire command with a space to avoid it from entering your shell history file), and
IP_OF_ANOTHER_NODE with the IP address of another node in the cluster on the external untrusted network. Note that the
--overlay-net there is required because of the way I wrote the systemd service file in the
wesher-systemd package. I did this:
[Unit] Description=wesher - wireguard mesh builder After=network-online.target After=syslog.target After=rsyslog.service [Service] EnvironmentFile=-/etc/default/wesher ExecStart=/usr/local/sbin/wesher --overlay-net 172.31.250.0/16 --log-level info Restart=on-failure Type=simple StandardOutput=syslog StandardError=syslog SyslogIdentifier=wesher [Install] WantedBy = multi-user.target
I explicitly specify the subnet of the VPN network here to avoid clashes with other networks in the 10.0.0.0/8 range. I don't have a network in that range, but I know that others do. Unfortunately there's currently a known bug that means that IP address collisions may occur, and the cluster can't currently sort them out. So you need to use a pretty large subnet for now to avoid such collisions (here's hoping this one is patched soon).
Note that if you want to set additional configuration options, you can do so in
/etc/default/wesher in the format
VAR_NAME=VALUE - 1 per line. A full reference of the supported environment variables can be found here.
Anyway, once wesher has joined the cluster on the new node, press Ctrl + C to exit and then start and enable the systemd service as before (note that wesher saves all the configuration details to a configuration directory, so the cluster key doesn't need to be provided more than once):
sudo systemctl enable --now wesher.service sudo systemctl status wesher.service
With this, you should have a fully-functional Wireguard peer-to-peer VPN setup with wesher. You can ask a node as to what IP address it has on the VPN by using the following command:
The IP address should be shown next to the
wgoverlay network interface.
The last thing to do here is to configure our Firewall. In my case, I'm using UFW, so the instructions I include here will be specific to UFW (if you use another firewall, translate these commands for your firewall, and command below!).
In my specific case, I want to take the unusual step of allowing all traffic in on the VPN. The reason for this will become apparent in a future post, but in short Nomad dynamically allocates outward-facing ports for services. There's a good reason for this I'll get into at the time, but for now, this is how you'd do that:
sudo ufw allow in on wgoverlay
Of course, we can add override rules here that block traffic if we need to. Note that this only allows in all traffic on the
wgoverlay network interface and not all network interfaces as my previous blog about UFW would have done - i.e. like this:
sudo ufw allow 1234/tcp
In the next part, we'll take a look at setting up an apt caching server to improve performance and reduce Internet data usage when downloading system updates.
Found this useful? Spotted a mistake? Confused about something? Comment below! It really helps motivate me to write these posts.