Cluster, Part 11: Lock and Key | Let's Encrypt DNS-01 for wildcard TLS certificates
Welcome one and all to another cluster blog post! Cluster blog posts always take a while to write, so sorry for the delay. As is customary, let's start this post off with a list of all the parts in the series so far:
- Cluster, Part 1: Answers only lead to more questions
- Cluster, Part 2: Grand Designs
- Cluster, Part 3: Laying groundwork with Unbound as a DNS server
- Cluster, Part 4: Weaving Wormholes | Peer-to-Peer VPN with WireGuard
- Cluster, Part 5: Staying current | Automating apt updates and using apt-cacher-ng
- Cluster, Part 6: Superglue Service Discovery | Setting up Consul
- Cluster, Part 7: Wrangling... boxes? | Expanding the Hashicorp stack with Docker and Nomad
- Cluster, Part 8: The Shoulders of Giants | NFS, Nomad, Docker Registry
- Cluster, Part 9: The Border Between | Load Balancing with Fabio
- Cluster, Part 10: Dockerisification | Writing Dockerfiles
With that out of the way, in this post we're going to look at obtaining a wildcard TLS certificate using the Let's Encrypt DNS-01 challenge. We want this because you need a TLS certificate to serve HTTPS without lighting everyone's browsers up with warnings like a Christmas tree.
The DNS-01 challenge is an alternate challenge to the default HTTP-01 challenge you may already me familiar with.
Unlike the HTTP-01 challenge which proves you have access to single domain by automatically placing a file on your web server, the DNS-01 challenge proves you have control over an entire domain - thus allowing you to obtain a wildcard certificate - which is valid for not only your domain, but all possible subdomains! This should save a lot of hassle - but it's important we keep it secure too.
As with regular Let's Encrypt certificates, we'll also need to ensure that our wildcard certificate we obtain will be auto-renewed, so we'll be setting up a periodic task on our Nomad cluster to do this for us.
If you don't have a Nomad cluster, don't worry. It's not required, and I'll be showing you how to do it without one too. But if you'd like to set one up, I recommend part 7 of this series.
In order to complete the DNS-01 challenge successfully, we need to automatically place a DNS record in our domain. This can be done via an API, if your DNS provider has one and it's supported. Personally, I have the domain name I'm using for my cluster (mooncarrot.space.) with Gandi. We'll be using certbot to perform the DNS-01 challenge, which has a plugin system for different DNS API providers.
We'll be installing the challenge provider we need with pip3 (a Python 3 package manager, as certbot is written in Python), so you can find an up-to-date list of challenge providers over on PyPi here: https://pypi.org/search/?q=certbot-dns
If you don't see a plugin for your provider, don't worry. I couldn't find one for Gandi, so I added my domain name to Cloudflare and followed the setup to change the name servers for my domain name to point at them. After doing this, I can now use the Cloudflare API through the certbot-dns-cloudflare plugin.
With that sorted, we can look at obtaining that TLS certificate. I opt to put certbot in a Docker container here so that I can run it through a Nomad periodic task. This proved to be a useful tool to test the process out though, as I hit a number of snags with the process that made things interesting.
The first order of business is to install certbot and the associate plugins. You'd think that simply doing an sudo apt install certbot certbot-dns-cloudflare would do the job, but you'd be wrong.
As it turns out, it does install that way, but it installs an older version of the certbot-dns-cloudflare plugin that requires you give it your Global API Key from your Cloudflare account, which has permission to do anything on your account!
That's no good at all, because if the key gets compromised an attacker could edit any of the domain names on our account they like, which would quickly turn into a disaster!
Instead, we want to install the latest version of certbot and the associated Cloudflare DNS plugin, which support regular Cloudflare API Tokens, upon which we can set restrictive permissions to only allow it to edit the one domain name we want to obtain a TLS certificate for.
I tried multiple different ways of installing certbot in order to get a version recent enough to get it to take an API token. The way that worked for me was a script called certbot-auto, which you can download from here: https://dl.eff.org/certbot-auto.
Now we have a way to install certbot, we also need the Cloudflare DNS plugin. As I mentioned above, we can do this using pip3, a Python package manager. In our case, the pip3 package we want is certbot-dns-cloudflare - incidentally it has the same name as the outdated apt package that would have made life so much simpler if it had supported API tokens.
Now we have a plan, let's start to draft out the commands we'll need to execute to get certbot up and running. If you're planning on following this tutorial on bare metal (i.e. without Docker), go ahead and execute these directly on your target machine. If you're following along with Docker though, hang on because we'll be wrapping these up into a Dockerfile shortly.
First, let's install certbot:
sudo apt install curl ca-certificates
cd some_permanent_directory;
curl -sS https://dl.eff.org/certbot-auto -o certbot-auto
chmod +x certbot-auto
sudo certbot-auto --debug --noninteractive --install-only
Installation with certbot-auto comprises downloading a script and executing it. with a bunch of flags. Next up, we need to shoe-horn our certbot-dns-cloudflare plugin into the certbot-auto installation. This requires some interesting trickery here, because certbot-auto uses something called virtualenv to install itself and all its dependencies locally into a single directory.
sudo apt install python3-pip
cd /opt/eff.org/certbot/venv
source bin/activate
pip install certbot-dns-cloudflare
deactivate
In short, we cd into the certbot-auto installation, activate the virtualenv local environment, install our dns plugin package, and then exit out of the virtual environment again.
With that done, we can finally add a convenience synlink so that the certbot command is in our PATH:
ln -s /opt/eff.org/certbot/venv/bin/certbot /usr/bin/certbot
That completes the certbot installation process. Then, to use certbot to create the TLS certificate, we'll need an API as mentioned earlier. Navigate to the API Tokens part of your profile and create one, and then create an INI file in the following format:
# Cloudflare API token used by Certbot
dns_cloudflare_api_token = "YOUR_API_TOKEN_HERE"
...replacing YOUR_API_TOKEN_HERE with your API token of course.
Finally, with all that in place, we can create our wildcard certificate! Do that like this:
sudo certbot certonly --dns-cloudflare --dns-cloudflare-credentials path/to/credentials.ini -d 'bobsrockets.io,*.bobsrockets.io' --preferred-challenges dns-01
It'll ask you a bunch of interactive questions the first time you do this, but follow it through and it should issue you a TLS certificate (and tell you where it stored it). Actually utilising it is beyond the scope of this post - we'll be tackling that in a future post in this series.
For those following along on bare metal, this is where you'll want to skip to the end of the post. Before you do, I'll leave you with a quick note about auto-renewing your TLS certificates. Do this:
sudo letsencrypt renew
sudo systemctl reload nginx postfix
....on a regular basis, replacing nginx postfix with a space-separated list of services that need reloading after you've renewed your certificates. A great way to do this is to setup a cron job.
Sweeping things under the carpet
For the Docker users here, we aren't quite finished yet: We need to package this mess up into a nice neat Docker container where we can forget about it :P
Some things we need to be aware of:
certbot has a number of data directories it interacts with that we need to ensure don't get wiped when the Docker ends instances of our container.- Since I'm serving the shared storage of my cluster over NFS, we can't have
certbot running as root as it'll get a permission denied error when it tries to access the disk.
- While
curl and ca-certificates are needed to download certbot-auto, they aren't needed by certbot itself - so we can avoid installing them in the resulting Docker container by using a multi-stage Dockerfile.
To save you the trouble, I've already gone to the trouble of developing just such a Dockerfile that takes all of this into account. Here it is:
ARG REPO_LOCATION
# ARG BASE_VERSION
FROM ${REPO_LOCATION}minideb AS builder
RUN install_packages curl ca-certificates \
&& curl -sS https://dl.eff.org/certbot-auto -o /srv/certbot-auto \
&& chmod +x /srv/certbot-auto
FROM ${REPO_LOCATION}minideb
COPY --from=builder /srv/certbot-auto /srv/certbot-auto
RUN /srv/certbot-auto --debug --noninteractive --install-only && \
install_packages python3-pip
WORKDIR /opt/eff.org/certbot/venv
RUN . bin/activate \
&& pip install certbot-dns-cloudflare \
&& deactivate \
&& ln -s /opt/eff.org/certbot/venv/bin/certbot /usr/bin/certbot
VOLUME /srv/configdir /srv/workdir /srv/logsdir
USER 999:994
ENTRYPOINT [ "/usr/bin/certbot", \
"--config-dir", "/srv/configdir", \
"--work-dir", "/srv/workdir", \
"--logs-dir", "/srv/logsdir" ]
A few things to note here:
- We use a multi-stage dockerfile here to avoid installing
curl and ca-certificates in the resulting docker image.
- I'm using
minideb as a base image that resides on my private Docker registry (see part 8). For the curious, the script I use to do this located on my personal git server here: https://git.starbeamrainbowlabs.com/sbrl/docker-images/src/branch/master/images/minideb.
- If you don't have minideb pushed to a private Docker registry, replace
minideb with bitnami/minideb in the above.
- We set the user and group certbot runs as to
999:994 to avoid the NFS permissions issue.
- We define 3 Docker volumes
/srv/configdir, /srv/workdir, and /srv/logsdir to contain all of certbot's data that needs to be persisted and use an elaborate ENTRYPOINT to ensure that we tell certbot about them.
Save this in a new directory with the name Dockerfile and build it:
sudo docker build --no-cache --pull --tag "certbot" .;
...if you have a private Docker registry with a local minideb image you'd like to use as a base, do this instead:
sudo docker build --no-cache --pull --tag "myregistry.seanssatellites.io:5000/certbot" --build-arg "REPO_LOCATION=myregistry.seanssatellites.io:5000/" .;
In my case, I do this on my CI server:
laminarc queue docker-rebuild IMAGE=certbot
The hows of how I set that up will be the subject of a future post. Part of the answer is located in my docker-images Git repository, but the other part is in my private continuous integration Git repo (but rest assured I'll be talking about it and sharing it here).
Anyway, with the Docker container built we can now obtain our certificates with this monster of a one-liner:
sudo docker run -it --rm -v /mnt/shared/services/certbot/workdir:/srv/workdir -v /mnt/shared/services/certbot/configdir:/srv/configdir -v /mnt/shared/services/certbot/logsdir:/srv/logsdir certbot certonly --dns-cloudflare --dns-cloudflare-credentials path/to/credentials.ini -d 'bobsrockets.io,*.bobsrockets.io' --preferred-challenges dns-01
The reason this is so long is that we need to mount the 3 different volumes into the container that contain certbot's data files. If you're running a private registry, don't forget to prefix certbot there with registry.bobsrockets.com:5000/.
Don't forget also to update the Docker volume locations on the host here to point a empty directories owned by 999:994.
Even if you want to run this on Nomad, I still advise that you execute this manually. This is because the first time you do so it'll ask you a bunch of questions interactively (which it doesn't do on subsequent times).
If you're not using Nomad, this is the point you'll want to skip to the end. As before with the bare-metal users, you'll want to add a cron job that runs certbot renew - just in your case inside your Docker container.
Nomad
For the truly intrepid Nomad users, we still have one last task to complete before our work is done: Auto-renewing our certificate(s) with a Nomad periodic task.
This isn't really that complicated I found. Here's what I came up with:
job "certbot" {
datacenters = ["dc1"]
priority = 100
type = "batch"
periodic {
cron = "@weekly"
prohibit_overlap = true
}
task "certbot" {
driver = "docker"
config {
image = "registry.service.mooncarrot.space:5000/certbot"
labels { group = "maintenance" }
entrypoint = [ "/usr/bin/certbot" ]
command = "renew"
args = [
"--config-dir", "/srv/configdir/",
"--work-dir", "/srv/workdir/",
"--logs-dir", "/srv/logsdir/"
]
# To generate a new cert:
# /usr/bin/certbot --work-dir /srv/workdir/ --config-dir /srv/configdir/ --logs-dir /srv/logsdir/ certonly --dns-cloudflare --dns-cloudflare-credentials /srv/configdir/__cloudflare_credentials.ini -d 'mooncarrot.space,*.mooncarrot.space' --preferred-challenges dns-01
volumes = [
"/mnt/shared/services/certbot/workdir:/srv/workdir",
"/mnt/shared/services/certbot/configdir:/srv/configdir",
"/mnt/shared/services/certbot/logsdir:/srv/logsdir"
]
}
}
}
If you want to use it yourself, replace the various references to things like the private Docker registry and the Docker volumes (which require "docker.volumes.enabled" = "True" in client → options in your Nomad agent configuration) with values that make sense in your context.
I have some confidence that this is working as intended by inspecting logs and watching TLS certificate expiry times. Save it to a file called certbot.nomad and then run it:
nomad job run certbot.nomad
Conclusion
If you've made it this far, congratulations! We've installed certbot and used the Cloudflare DNS plugin to obtain a DNS wildcard certificate. For the more adventurous, we've packaged it all into a Docker container. Finally for the truly intrepid we implemented a Nomad periodic job to auto-renew our TLS certificates.
Even if you don't use Docker or Nomad, I hope this has been a helpful read. If you're interested in the rest of my cluster build I've done, why not go back and start reading from part 1? All the posts in my cluster series are tagged with "cluster" to make them easier to find.
Unfortunately, I haven't managed to determine a way to import TLS certificates into Hashicorp Vault automatically, as I've stalled a bit on the Vault front (permissions and policies are wildly complicated), so in future posts it's unlikely I'll be touching Vault any time soon (if anyone has an alternative that is simpler and easier to understand / configure, please comment below).
Despite this, in future posts I've got a number of topics lined up I'd like to talk about:
- Configuring Fabio (see part 9) to serve HTTPS and force-redirect from HTTP to HTTPS (status: implemented)
- Implementing HAProxy to terminate port forwarding (status: initial research)
- Password protecting the private docker registry, Consul, and Nomad (status: on the todo list)
- Semi-automatic docker image rebuilding with Laminar CI (status: implemented)
In the meantime, please comment below if you liked this post, are having issues, or have any suggestions. I'd love to hear if this helped you out!
Sources and Further Reading
Mailing List
Articles Atom Feed
Comments Atom Feed
Twitter
Reddit
Facebook
