We run most of our own network services – inbound and outbound email, DNS and web presence. We run separate services for inbound and outbound email to give us more flexibility in how we set things up.
Our current smarthost was configured in 2018 and hasn’t really been touched much since. We’re moving it to a different hosting location, in the EU rather than the US, mostly so we can make stronger privacy statements about all customer data being handled on EU located servers.
So we’re starting from scratch. I thought I’d document the process.
Basics
It’s a nice little VPS from Mythic Beasts. It’s just being used for outbound email, so I don’t need much storage. It’ll only have a few users – some human, some apps – so it’s not going to need much CPU either. It’s mostly just going to be running postfix and some helper services.
I paid the extra £20 to get an IPv4 address as well as the default IPv6 range. Email is not ready for IPv6-only yet.
Mythic Beasts have an excellent reputation for not tolerating bad actors on their network. That’s part of the reason I chose them for this – IP reputation isn’t as important for delivery as it used to be, but you still don’t want to be on a network provider known as a source of spam. They actually had a member of staff manually review the order before spinning up the new VPS, which is a level of due diligence I’ve not seen in years.
The normal new server setup path: sudo, ssh keys. I add it to our tailscale VPN and set up ssh to only listen on the VPN. Then I do the tailscale-wait-for-ip dance that’s needed to make services that bind to addresses at startup happy. If I’d done that in a different order I wouldn’t have locked myself out and had to use a virtual serial console to get back in. Oops. Lock down the packet filters so only port 587 can come in from the public Internet.
The hostname is preconfigured to be mail.turscar.ie – that’s what it will HELO as, and what it’ll stamp into trace headers, so it’s important to get right. Mythic beasts set up reverse DNS for both its IPv4 and IPv6 addresses when the VM was created, and I’ve added A and AAAA records for it to the turscar.ie nameserver.
TLS Certificates
We’re going to be submitting mail over the ESMTP SUBMIT port, 587. That requires authentication, so we need to support STARTTLS. That means that we need to acquire a TLS certificate, and it’s 2025 so we want a real, publicly trusted certificate, not a self-signed one.
There are several certificate authorities that’ll issue free, domain authenticated certificates using ACME clients, but I’m going to stick with Lets Encrypt.
Lets Encrypt certificates are only valid for a few months, so need to be renewed every couple of months, which means that requesting and deploying them needs to be automated. The most commonly used infrastructure for that automation is aimed at certificates for web servers, and requires a webserver to be running at the hostname that a certificate is requested for.
We don’t have a webserver for mail.turscar.ie. Fortunately the ACME protocol supports domain validation in other ways. The client I use, LEGO, has great support for authentication via DNS, supporting the APIs of a 150 or so different providers. I could use it’s native PowerDNS API support, but that would mean setting up and exposing that API on my primary DNS server, and handling all the authentication for that. Instead I use RFC2136 dynamic DNS updates. These are handled over DNS, and just require a shared secret between the DNS server and the ACME client.
(DNS is core to a lot of security, so if someone were to get hold of that shared secret it would be bad if they could use it to modify the DNS for my zones. So I create a subdomain _acme-challenge.mail.turscar.ie and allow only that subdomain to be updated via dnsupdate, and only from the IP address of the new server. The shared secret can’t be used to modify anything other than ACME challenge records, and it can’t be accessed from anywhere else. That’s an annoying amount of work to do by hand, but I wrote the script to do it once and now it’s a single command.)
There are a lot of certificate authorities out there, and not all of them are as careful about issuing certificates as they should be. We can mitigate that by publishing CAA records in DNS.
A CAA record contains the name of a certificate authority. Any certificate authority that’s about to issue a certificate is required to look up the CAA records for the domain. If it finds any CAA records, and it doesn’t find one with the CA’s name in it, it mustn’t issue the certificate.
We’ve already published CAA records for turscar.ie, so we’re good to go.
LEGO lets you run a script once it’s acquired a certificate and a private key. I mostly use that to write the certificate and private key to the normal places they live (/etc/ssl/certs and /etc/ssl/private) and reload any services that use them. Since I last set up postfix they’ve moved to recommending that the private key and certificate are stored in a single file, rather than separate ones, so I modify the hook script to concatenate the key and certificate and put that chain file where postfix (and only postfix) can read it.
I run LEGO once by hand, to get my certificate, then add it to root’s crontab to run nightly. When it runs it’ll check the expiry date of all the certificates it manages, and renew any that are close to expiring. That’ll keep running forever, and I’ll never need to look at it again.
A TLS certificate is part of a chain of certificates. The one you’re issued is authenticated by a certificate owned by the certificate authority. That one may be authenticated by an intermediate certificate and then that one is authenticated by a trusted root certificate. The list of trusted root certificates is compiled manually, and distributed with your operating system or embedded into your web browser. It’s good to know what certificates are in that chain, and make them all available to your server to offer to clients. If, say, you don’t offer an intermediate certificate then your certificate might not be accepted by a client. Or it might, depending on what certificates the client trusts. And if I wanted to publish TLSA records in DNS to strengthen trust in the certificate I’d need to serve the whole chain back to the root certificate
Running openssl storeutl -noout -text -certs mail.turscar.ie.crt prints out the chain of certificates in that file:
0: Certificate
Certificate:
Data:
Signature Algorithm: ecdsa-with-SHA384
Issuer: C=US, O=Let's Encrypt, CN=E5
Subject: CN=mail.turscar.ie
1: Certificate
Certificate:
Data:
Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Subject: C=US, O=Let's Encrypt, CN=E5Code language: PHP (php)I’ve trimmed out a whole lot of information, leaving just the bits I want to talk about. You can see there are two certificates in this chain. The first one is the one I was just issued. The Subject: field is who it belongs to, mail.turscar.ie1. The Issuer is the entity that vouched for this certificate, Let’s Encrypt’s E5 certificate.
The second certificate belongs to Let’s Encrypt, it’s their intermediate E5 certificate that they use for issuing many of their ECDSA certificates. The issuer is the ISRG Root X1 certificate.
And the ISRG X1 Root is widely recognised, so this certificate chain is good enough, at least until I want to publish TLSA records.
The Signature Algorithm: field shows that this is an ECDSA (elliptic curve) certificate rather than the older RSA type. Client support for ECDSA certificates isn’t universal, so the postfix documentation suggests offering RSA alongside ECDSA. If this were a public smarthost I’d do that, but we’ve been using ECDSA only on our existing smarthost and haven’t seen any problems, so I’m not going to bother.
Postfix
I’ve been using postfix for decades. It’s stable, fairly feature-rich, well supported and well suited for a small business mailserver, inbound or outbound. If I were building something bigger, or more suited to bulk mail usage I’d probably consider KumoMTA instead, but for this setup Postfix is fine. And I’m familiar with how it’s configured and how all the pieces go together.
The most recent release of Postfix is 3.10.0. The Debian repo gives me Postfix 3.7.11. I’d like some of the new features and security mitigations that have been added since then, but I’d like a clean, easily upgradeable install on Debian stable more.
apt install postfix asks me what sort of installation I want. I tell it I want a normal mailserver and it installs and configures postfix, and sets it running via systemd.
Time to configure things.
In /etc/postfix/main.cf I setup TLS
# TLS parameters
smtpd_tls_chain_files = /etc/postfix/ecdsa.pem
smtpd_tls_security_level=may
smtpd_tls_auth_only = yesCode language: PHP (php)The default setup is for a mailserver that listens on port 25. We don’t want that, we want just a submission server on port 587, so off to /etc/postfix/master.cf we go. The default settings are fine, so I just comment out the smtp service and uncomment submission. Restart the service and we’re listening on port 587.
Client Authentication – SASL
We need to be able to authenticate clients, but we don’t need anything sophisticated. Simple username and password will be fine.
So we’ll use Cyrus SASL (simple authentication and security layer) with a PAM provider. Postfix will talk SASL to Cyrus, then Cyrus will use PAM to validate usernames and passwords against the operating systems usernames and passwords. This wouldn’t scale well, but for a few users it’s fine (particularly if those users are also going to want to log in to check the mail delivery logs occasionally).
sudo apt install libsasl2-modules sasl2-binThis installs saslauthd and friends. The default setup is almost what we need, but not quite. Postfix communicates with saslauthd via a unix socket, but postfix is configured to run in a chroot, so can’t access the default socket location. The needed change is well documented in comments in the /etc/default/saslauthd configuration file.
Skipping over a lot of debugging here. Some days I hate being a sysadmin.
Unfortunately, the Debian 12 packaging of saslauthd just doesn’t work. There’s a bad interaction with SysV startup scripts and systemd meaning that it just silently fails, whatever you do. There are a smattering of open bug reports about it, and it’s supposedly fixed in what will become Debian 13. So I create a proper systemd service file for “saslauthd2” and that runs fine.
/usr/sbin/testsaslauthd -f /var/spool/postfix/var/run/saslauthd/mux \
-u steve -p <password>Code language: JavaScript (javascript)This gives me a nice OK "Success." response for a valid password and NO "authentication failed" for an invalid one.
Client Authentication – postfix
Now we tell postfix to use it by creating /etc/postfix/sasl/smtpd.conf
pwcheck_method: saslauthd
mech_list: PLAIN LOGINCode language: HTTP (http)PLAIN and LOGIN are the two plain text authentication styles, which we need to be able to support them with this setup. They’re tunneled over TLS, so that’s fine.
The submission service in master.cf already has smtpd_sasl_auth_enable=yes and main.cf has smtpd_recipient_restrictions=permit_sasl_authenticated,reject so after reloading the postfix service we should be good to go.
Testing with SWAKS, finally
Time for a smoke test, using swaks.
swaks --to steve@blighty.com --server mail.turscar.ie \
--protocol ESMTPS -p 587 --auth-user steve --auth-password mypwdCode language: CSS (css)The SMTP transaction scrolls by, swaks uses STARTTLS to switch to using TLS, then sends AUTH LOGIN to authenticate. The mail is accepted, then sent on to the final destination.
Where it ends up in my spam folder. Nothing’s authenticated yet (and we have no Message-ID, because SWAKS, and it’s 5321.From and 5322.From domains don’t have an MX, because SWAKS), so no great surprise. We’ll do a more realistic test once authentication is set up.
SMTP Authentication – DKIM
A common way – for postfix and sendmail, at least – to plug most sorts of processing into a mailserver is to use a milter. That’s a portmanteau for mail filter, and it’s an API that allows postfix to pass an email to an external process. It’s been around for a couple of decades or more, so it’s pretty stable.
That external process, the milter, can make decisions about how the mail should be routed, and it can modify the header or content.
The commonly used DKIM milter is OpenDKIM. It can both verify inbound mail and sign outbound mail, and can be convinced to sign mail from different senders with different domains and keys, making it possible to have dkim authentication align with the 822.From header.
But OpenDKIM has some open bugs – nothing serious, but some unexpected behaviour in grubby corners. Its last production release dates from 2015. Debian seem to be distributing the most recent beta release from 2018. Don’t get me wrong, it’s good enough to use in production, but it’s not a codebase I’m particularly comfortable diagnosing or modifying. So what else is available?
The usual code I use for authentication is go-msgauth. It’s solid, understandable code and it does include a milter implementation. But it’s very basic, and can only sign with a single domain, configured as a commandline flag.
Rspamd’s dkim_signing module might be another option, but that’s a huge “intended for inbound mail filtering” dependency to pull in for a smarthost. And there’s a few pieces of very nice rust code for authentication, and a milter based on one, but I’d need to build and package those myself.
I guess I’m using OpenDKIM for now.
OpenDKIM
sudo apt install opendkim opendkim-toolsThis installs a basic configuration file in /etc/opendkim.conf. Just like saslauthd it communicates with postfix via a unix socket, so again I need to configure it to create that socket inside the directory postfix is chrooted to.
I’m using opendkim because I want to have aligned dkim signatures, and that means I need to sign with different keys2 depending on the domain in the From: address. That’s configured using a SigningTable and a KeyTable, so I add a couple of lines to /etc/opendkim.conf:
KeyTable /etc/opendkim/KeyTable
SigningTable refile:/etc/opendkim/SigningTableCode language: JavaScript (javascript)KeyTable is just a text file with one record per line – starting with an arbitrary name (I’m using the full hostname where the TXT record will be published, because it’s easy to remember) followed by the DKIM selector, the DKIM SDID (“d=”) and the private key to sign with, all separated by colons:
blueberry._domainkey.wordtothewise.com wordtothewise.com:blueberry:/etc/opendkim/keys/wordtothewise.com/blueberry.privateCode language: JavaScript (javascript)SigningTable is an “refile”, which is used to map the value in the From: header to one of the entries in the KeyTable. “refile” stands for “regular expression file”. Despite that, it does not contain regular expressions, just simple wildcards using an asterisk. Again, it has one record per line, starting with a wildcard match for the From: header, followed by the arbitrary name for the key data from the KeyTable.
*@wordtothewise.com blueberry._domainkey.wordtothewise.comCode language: CSS (css)Next up, signing keys.
First we create somewhere secure to store them, as root:
# mkdir -p /etc/opendkim/keys/wordtothewise.com
# chown -R opendkim:opendkim /etc/opendkim/keys
# chmod go-rwx /etc/opendkim/keysCode language: PHP (php)Then we create a key pair, a private key for signing outbound mail and a public key to publish in DNS. We have a few choices to make – what selector to use, how strong a key we want, whether the key should be explicitly only used for email:
# opendkim-genkey --append-domain \
--bits=2048 \
--directory=/etc/opendkim/keys/wordtothewise.com \
--domain=wordtothewise.com \
--restrict \
--selector=blueberry
# chown -R opendkim:opendkim /etc/opendkim/keysCode language: PHP (php)That creates two files for us. One is blueberry.private – the signing key, generated in the place we’ve already told opendkim to find it. The other is blueberry.txt – the public key, in bind format ready to add to our nameserver:
blueberry._domainkey.wordtothewise.com. IN TXT ( "v=DKIM1; h=sha256; k=rsa; s=email; "
"p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0FOof0FrJBTMxNm/3KbLSnUgBwX5jVkRILFEJJltBbaH0ZqduoUau/NcjioYgDSO8ktF6f4YnUem7VjARTzkl7mnQA9qlhF0Ix0W72oL5cDd6EptuoNn88ws9nBRvDBkeSjFNo/ftvrr6wEMet93EC0mxXKZXT9jgPTAii+cXl1Jg7QkO64DySFUDAodmaBMN9mVtr8P6drO0P"
"sG8RxH9KfvEtLS4L1a42TB7CtydMeIGQJKW51C55cIRhLVXzZ8emdTpZ067tdYNdeHFX7WsSEa5XBIJoDE8LI8RHrYSIdUbtGqkWncy9U0yYjVPJj369Q7yBgWZiwGrjbUQBr3XQIDAQAB" ) ; ----- DKIM key blueberry for wordtothewise.comCode language: JavaScript (javascript)I add that to our main nameservers, and we’ve published our DKIM public key.
There are tools for testing a milter directly, but let’s hope we can skip that and just send some email.
Plugging opendkim-milter in to postfix is just a few lines in /etc/postfix/main.cf:
smtpd_milters = unix:opendkim/opendkim.sock
non_smtpd_milters = $smtpd_milters
milter_default_action = acceptCode language: PHP (php)I add postfix to the opendkim group, so they can both access the socket inside postfix’ chroot, make sure the directory /var/spool/postfix/opendkim is owned by opendkim and has reasonable permissions, then restart both postfix and opendkim.
Testing with SWAKS, again
This time we have to provide an email address for the From: header, to trigger the DKIM signing.
swaks --to steve@blighty.com --from steve@wordtothewise.com \
--server mail.turscar.ie --protocol ESMTPS -p 587 \
--auth-user steve --auth-password mypwdCode language: JavaScript (javascript)The first attempt is deferred by the smarthost. Checking the opendkim logs the ownership of the private key file is wrong, so it can’t sign the mail. Fixing that, the second attempt delivers successfully, but DKIM fails because it can’t find the public key in DNS…
Authentication-Results: mx.turscar.ie;
dkim=fail reason="key not found in DNS" header.d=blueberry header.i=@blueberry header.a=rsa-sha256 header.s=wordtothewise.com header.b=umSfgt6s;
dkim-atps=neutral
Code language: JavaScript (javascript)header.s = wordtothewise.com, header.d=blueberry? I got the two columns in KeyTable the wrong way around. Easy fix. (I’ve fixed it in this post too, as I really don’t want to leave bad configuration where someone might copy and paste it).
Now it works.
Authentication-Results: mx.turscar.ie;
dkim=pass (2048-bit key; secure) header.d=wordtothewise.com header.i=@wordtothewise.com header.a=rsa-sha256 header.s=blueberry header.b=AJgmgJCN;
dkim-atps=neutral
Final test, send an email to aboutmy.email – the results look good.
SPF and DKIM are aligned, DMARC passes. Mail is being sent via TLS 1.3, over an IPv6 connection. Round trip reverse DNS looks good.
What’s next?
I need to add the other domains we use for email to the signing tables, and update their SPF records to acknowledge the new server.
I need to do something about key rotation. There’s not really a great way to automate that around opendkim. It would be possible, but would require generating new keys, pushing them in to the DNS, updating the opendkim signing tables (either by editing the files or, more likely, by moving to use database backed tables). I’m going to think about that, and see if anyone has written anything to help with it.
Conclusions
You can configure a perfectly solid, best practices compliant smarthost using off-the-shelf open source components.
But documentation is sparse, contradictory or wrong. Packaging, at least in the Debian world, is a wreck. The default configuration isn’t going to provide best practices, and the bits of configuration you need to do so aren’t on the simple path.
You should strongly consider having someone else run it for you, or using a commercial (supported, good practices by default) smarthost.
And next time I’m going to try KumoMTA instead, even for a small business smarthost.
- This is a huge oversimplification of what’s in a TLS certificate – you should really be looking at the X509v3 Subject Alternative Name – but it’s good enough for this. ↩︎
- Technically I could sign with the same private key, publish the single public key under multiple domains and still use a d= that matches the domain in the From: header. ↩︎
RSS - Posts