Deep dive: Email, Trust, DKIM, SPF, and more
(Above: Lots of parcels. Hopefully you won't get this many through the door at once..... Source)
Now that I'm on holiday, I've got some time to write a few blog posts! As I've promised a few people a post on the email system, that's what I'll look at this this post. I'm going to take you on a deep dive through the email system and trust. We'll be journeying though the fields of DKIM signatures, and climb the SPF mountain. We'll also investigate why the internet needs to take this journey in the first place, and look at some of the challenges one faces when setting up their own mail server.
Hang on to your hats, ladies and gentlemen! If you get to the end, give yourself a virtual cookie :D
Before we start though, I'd like to mention that I'll be coming at this from the perspective of my own email server that I set up myself. Let me introduce to you the cast: Postfix (the SMTP MTA), Dovecot (the IMAP MDA), rspamd (the spam filter), and OpenDKIM (the thing that deals with DKIM signatures).
With that out of the way, let's begin! We'll start of our journey by mapping out the journey a typical email undertakes.
Let's say Bob Kerman wants to send Bill an email. Here's what happens:
- Bill writes the email and hits send. His email client connects to his email server, logs in, and asks the server to deliver a message for him.
- The server takes the email and reads the
Fromheader (in this case it's
firstname.lastname@example.org), figures out where the mail server is located, connects to it, and asks it to deliver Bob's message to Bill.
mail.billsboosters.comtakes the email and files it in Bill's inbox.
- Bill connects to his mail server and retrieves Bob's message.
Of course, this is simplified in several places.
mail.bobsrockets.com will obviously need to do a few DNS lookups to find
billsboosters.com's mail server and fiddle with the headers of Bob's message a bit (such as adding a
Received header etc.), and
smtp.billsboosters.com won't just accept the message for delivery without checking out the server it came from first. How does it check though? What's preventing
seanssatellites.net pretending to be
bobsrockets.com and sending an imposter?
Until relatively recently, the answer was, well, nothing really. Anyone could send an email to anyone else without having to prove that they could indeed send email in the name of a domain. Try it out for yourself by telnetting to a mail server on port 25 (unencrypted SMTP) and trying in something like this:
HELO mail.bobsrockets.com MAIL From: <email@example.com> RCPT TO <firstname.lastname@example.org> DATA From: email@example.com To: firstname.lastname@example.org Hello! This is a email to remind you..... . QUIT
Oh, my! Frank at
franksfuel.io can connect to any mail server and pretend that
email@example.com is sending a message to
firstname.lastname@example.org! Mail servers that allow this are called open relays, and today they usually find themselves on several blacklists within minutes. Ploys like these are easy to foil, thankfully (by only accepting mail for your own domains), but it still leaves the problem of what to do about random people connecting to your mail server delivering spam to your inbox that claims to be from someone they aren't supposed to be sending mail for.
In response, some mail servers demanded things like the IP that connects to send an email must reverse to the domain name that they want to send email from. Clever, but when you remember that anyone can change their own PTR records, you realise that it's just a minor annoyance to the determined spammer, and another hurdle to the legitimate person in setting up their own mail server!
Clearly, a better solution is needed. Time to introduce our first destination: SPF. SPF stands for sender policy framework, and defines a mechanism by which a mail server can determine which IP addresses a domain allows mail to be sent from in it's name. It's a
TXT record that sites at the root of a domain. It looks something like this:
v=spf1 a mx ptr ip4:18.104.22.168 ip6:2001:41d0:e:74b::1 a:starbeamrainbowlabs.com a:mail.starbeamrainbowlabs.com -all
The above is my SPF
TXT record for
starbeamrainbowlabs.com. It's quite simple, really - let's break it down.
This just defines the version of the SPF standard. There's only one version so far, so we include this to state that this record is an SPF version 1 record.
a mx ptr
This says that the domain that the sender claims to be from must have an
a and an
mx record that matches the IP address that's sending the email. It also says that the
ptr record associated with the sender's IP must resolve to the domain the sender claims to be sending from, as described above (it does help with dealing with infected machines and such).
This bit says that the IP addresses
2001:41d0:e:74d::1 are explicitly allowed to send mail in the name of
After all of the above, this bit isn't strictly necessary, but it says that all the IP addresses found in the
a records for
mail.starbeamrainbowlabs.com are allowed to send mail in the name of
Lastly, this says that if you're not on the list, then your message should be rejected! Other variants on this include
~all (which says "put it in the spam box instead"), and
+all (which says "accept it anyway", though I can't see how that's useful :P).
As you can see, SPF allows a mail server to verify if a given client is indeed allowed to send an email in the name of any particular domain name. For a while, this worked a treat - until a new problem arose.
Many of the mail servers on the internet don't (and probably still don't!) support encryption when connecting to and delivering mail, as certificates were expensive and difficult to get hold of (nowadays we've got LetsEncrypt who give out certificates for free!). The encryption used when mail servers connect to one another is practically identical to that used in HTTPS - so if done correctly, the identity of the remote server can be verified and the emails exchanged encrypted, if the world's certification authorities aren't corrupted, of course.
Since most emails weren't encrypted when in transit, a new problem arose: man-in-the-middle attacks, whereby an email is altered by one or more servers in the delivery chain. Thinking about it - this could still happen today even with encryption, if any one server along an email's route is compromised. To this end, another mechanism was desperately needed - one that would allow the receiving mail server to verify that an email's content / headers hadn't been surreptitiously altered since it left the origin mail server - potentially preventing awkward misunderstandings.
Enter stage left: DKIM! DKIM stands for Domain Keys Identified Mail - which, in short, means that it provides a method by which a receiving mail server can cryptographically prove that a message hasn't been altered during transit.
It works by having a public-private keypair, in which the public key can only decrypt things, but the private key is capable of encrypting things. A hash of the email's headers / content is computed and encrypted with the private key. Then the encrypted hash is attached to the email in the
The receiving mail server does a DNS lookup to find the public key, and decrypts the hash. It then computes it's own hash of the email headers / content, and compares it against the decrypted hash. If it matches, then the email hasn't been fiddled with along the way!
Of course, not all the headers in the email are hashed - only a specific subset are included in the hash, since some headers (like
X-Spam-Result) are added and altered during transit. If you're interested in implementing DKIM yourself - DigitalOcean have a smashing tutorial on the subject, which should adapt easily to whatever system you're running yourself.
With both of those in place,
billsboosters.com's mail server can now verify that
mail.bobsrockets.com is allowed to send the email on behalf of
bobsrockets.com, and that the message content hasn't been tampered with since it left
mail.billsboosters.com can also catch
franksfuel.io in the act of trying to deliver spam from
There is, however, one last piece of the puzzle left to reveal. With all this in place, how do you know if your mail was actually delivered? Is it possible to roll SPF and DKIM out gradually so that you can be sure you've done it correctly? This can be a particular issue for businesses and larger email server setups.
This is where DMARC comes in. It's a standard that lets you specify an email address you'd like to receive DMARC reports at, which contain statistics as to how many messages receiving mail servers got that claimed to be from you, and what they did with them. It also lets you specify what percentage of messages should be subject to DMARC filtering, so you can roll everything out slowly. Finally, it lets you specify what should happen to messages that fail either SPF, DKIM, or both - whether they should be allowed anyway (for testing purposes), quarantined, or rejected.
DMARC policies get specified (yep, you guessed it!) in a DNS record. unlike SPF though, they go in
_dmarc.megsmicroprocessors.org as a
TXT record, substituting
megsmicroprocessors.org for your domain name. Here's an example:
v=DMARC1; p=none; rua=mailto:email@example.com
This is just a simple example - you can get much more complex ones than this! Let's go through it step by step.
Nothing to see here - just a version number as in SPF.
This is the policy of what should happen to messages that fail. In this example we've used
none, so messages that fail will still pass right on through. You can set it to
quarantine or even
reject as you gain confidence in your setup.
This specifies where you want DMARC reports to be sent. Each mail server that receives mail from your mail server will bundle up statistics and send them once a day to this address. The format is in XML (which won't be particularly easy to read), but there are free DMARC record parsers out there on the internet that you can use to decode the reports, like dmarcian.
That completes the puzzle. If you're still reading, then congratulations! Post in the comments and say hi :D We've climbed the SPF mountain and discovered how email servers validate who is allowed to send mail in the name of another domain. We've visited the DKIM signature fields and seen how the content of email can be checked to see if it's been altered during transit. Lastly, we took a stroll down DMARC lane to see how it's possible to be sure what other servers are doing with your mail, and how a large email server setup can implement DMARC, DKIM, and SPF more easily.
Of course, I'm not perfect - if there's something I've missed or got wrong, please let me know! I'll try to correct it as soon as possible.
Lastly, this is, as always, a starting point - not an ending point. An introduction if you will - it's up to you to research each technology more thoroughly - especially if you're thinking of implementing them yourself. I'll leave my sources at the bottom of this post if you'd like somewhere to start looking :-)
Sources and Further Reading
- The basics of SMTP by Dave Marshall
- Sender Policy Framework (SPF) - Explained by Christopher Knight
SPF Introduction on the Sender Policy Framework Project Website(Link dead - see below - thanks Chris!) SPF Record Syntax on the Sender Policy Framework Project Website(Link dead - see below - thanks Chris!)
- SPF Record Testing Tools by Kitterman Technical Services
- SPF Record Generator (also, a syntax cheat-sheet)
- The DKIM Website
- The DMARC Website
- DMARC Overview on the DKIM Website
- RFC 7489: Domain-based Message Authentication, Reporting, and Conformance (DMARC)
- Build Your Email Server on Ubuntu Part 3: Create DMARC Record
- How To Install and Configure DKIM with Postfix on Debian Wheezy by P. Sebastian
- Mail tester - test your SPF / DKIM / DMARC implementation