Authentication

Some notes on some of the different protocols used for authentication and authentication-adjacent things in email. Some of this is oral history, and some of it may be contradicted by later or more public historical revision.

SPF

Associates an email with a domain that takes responsibility for it.

Originally Sender Permitted From, now Sender Policy Framework. It allows a domain owner to announce which IP addresses mail that uses a particular return path to be sent from, and whether a recipient should accept mail sent from other IP addresses as legitimate.

It authenticates the domain in the return path, not any hostname that’s visible to the recipient.

Original intent

By allowing recipients to detect “probably forged” return paths, SPF allows recipient ISPs to avoid backscatter, by not sending asynchronous bounces in response to spam with forged return paths.

Secondarily, it provided a way to tie a message to domain name – the one hidden in the return path. Simple blocking of mail that violated SPF stopped a lot of spam, though not in a particularly sustainable way.

It allowed senders to take responsibility for mail they sent (authentication) and deny responsibility for mail they didn’t send (repudiation).

There’s a lot of history documented here.

Current usage

SPF on it’s own is primarily seen as a way for a domain to advertise which IP addresses it expects to send email from, and so to associate an email with a domain that takes responsibility for it. This allows the reputation of a mail stream to be monitored keyed on the associated domain.

It’s widely used for authentication, but use of SPF for repudiation (blocking mail that fails SPF with a -all result) has pretty much disappeared.

It’s also a major building block for DMARC. Having the return path and the domain in the visible From: header be “in the same domain” has additional benefits on it’s own as well as being part of the DMARC process.

Failure modes

SPF authentication is tied to the peer IP address of the SMTP transaction. That means that most sorts of forwarding – vanity domains, mailing lists etc. – will cause SPF to fail. There were attempts to mitigate that, by having the forwarder rewrite the return path (sender rewriting scheme) but as recipient ISPs moved to ignoring SPF failures as far as delivery decisions were concerned they didn’t make that much impact (but see DMARC and ARC).

Benefits and risks

An SPF pass is considered a positive sign, at least if the associated domain has some history. A missing or failed SPF check isn’t widely considered a negative (unless the sender has opted-in to it being used that way, by publishing DMARC records).

Most of the benefits of SPF require that you use a domain you control in your return path rather than one you don’t own, such as your ESP’s bounce domain. Most ESPs should be able to support that, but it will require delegating control to the ESP via adding DNS records to your domain.

It’s generally cheap to deploy for most senders and deploying it has no real risks, but management of it can get more complex when multiple mail streams are in use.

SenderID

Original intent

To be a “better” SPF, and one that authenticated based on the hostname visible to the recipient in the From: header. It had a new style policy record that started with “spf2.0/pra” rather than “v=spf1” but would fall back to the old-style v=spf1 records.

Current usage

Not much. The working group that was working on it gave up due to a range of disagreements, many around intellectual property claims from Microsoft.

Microsoft still appear to check for traditional SPF records based on the domain in From: header, which is technically a SenderID check, but they say they don’t look for spf2.0/ style SenderID records just v=spf1 style SPF records.

Benefits and risks

Based on anecdotal evidence publishing an SPF record for the domain in the visible From: can improve deliverability at Microsoft properties.

There’s not really any risk to publishing that SPF record, whether accurate or not, and the only costs are the maintenance overhead and the (valuable) space it’s DNS record takes up in the root of a zone.

Publishing a spf2.0/ style SenderID record is probably pointless.

DKIM

DKIM allows the sender of an email to attach a hostname to a message in a way that can be cryptographically validated by a recipient.

It authenticates the domain (often called the “d=”) in the DomainKey-Signature header, not any hostname that’s visible to the recipient.

Original intent

DKIM was intended to allow a sender to take responsibility for an email via an attached domain name, allowing recipients to track reputation via that domain name rather than via, e.g., sending IP addresses.

Current usage

DKIM is used much as it was intended, to associate a domain with an email.

It’s also a major building block for DMARC. Having the d= domain and the domain in the visible From: header be “in the same domain” has additional benefits on it’s own as well as being part of the DMARC process.

Failure modes

DKIM relies on a cryptographic signature of the body of the email and a – sender-chosen – subset of the email headers. If any of those are changed at all then the DKIM signature will be broken.

There are obvious ways that the content can be modified – mailing lists adding annotations to the subject line or footers to the body, for instance. But there are also a lot of subtle ways it can be modified. If mail is sent using an “unusual” structure – overly long lines, unwrapped headers, unusual content transfer encoding – then an intermediate mail system may “fix up” the mail in a way that doesn’t change the semantics of the message and which wouldn’t be visible changes to the recipient, but which will break the DKIM signature.

Benefits and risks

DKIM is much more robust against being broken in transit than SPF, and gives similar reputation advantages. It also allows participation in some feedback loops.

It is more complex to deploy, requiring emails to be cryptographically signed as they’re sent, but that’s well supported on current MTAs.

To get most of the advantages of DKIM you need to sign it with a domain you control rather than with a domain you don’t own, such as that of your ESP. Most ESPs should be able to support that, but it will require adding DNS records to your domain to delegate control to your ESP.

DMARC

DMARC allows a domain owner to state that all mail they send with their domain in the visible From: header will by authenticated by them (via SPF or DKIM). It’s for repudiation, not authentication. It effectively changes the semantics of SPF and DKIM from having them being a positive signal to not having them being a (very) negative one.

It also allows a domain owner to request notifications about mail that appears to be sent from their domain but which isn’t correctly authenticated. This is critical to being able to check that you really are authenticating (most of) your email, and to do so before you ask recipients to discard potentially legitimate mail that’s not authenticated. This is the critical feature that distinguishes DMARC from previous attempts at email repudiation such as SSP, ADSP and SPF.

Original intent

Brand protection and anti-phishing.

Current usage

Mostly brand protection. Anti-phishing is still given lip service, and DMARC does mitigate phishing from the most naive phishers, but it’s not particularly effective against an adversary who’s learned to adapt in the years since DMARC began to be widely deployed.

It has also changed some of the semantics of SPF and DKIM authentication. DMARC introduced the idea of “DMARC-aligned” authentication, meaning that the return path used by SPF or the d= used by DKIM is “in the same domain as” the email address in the From: field. Many ISPs will provide preferential treatment to email that is authenticated in a DMARC-aligned way, even if the domain is not publishing DMARC records.

Benefits and risks

Using DMARC in “reporting only” mode, either “p=none” or “p=quarantine pct=0” is an extremely useful tool for mapping out your mail flows and finding sources of legitimate email that aren’t correctly authenticated. This is useful in itself, as well as being an essential step towards DMARC enforcement. It will potentially generate a lot of data, though, and you’ll need to budget for infrastructure and personnel time to handle and analyze those reports. There are very few deliverability or usability risks in this mode, though there are a few mailing lists which will modify their behaviour in a potentially user-surprising way for users of your domain.

To deploy DMARC in enforcing mode effectively requires deploying SPF and DMARC everywhere as a prerequisite (while DMARC only requires that either SPF or DKIM passes the two authentication approaches are fragile and will occasionally break, so you want to have them both in place to minimize the risk of DMARC failing). It also requires ongoing management and monitoring to ensure that authentication hasn’t stopped working or a new mailstream has been deployed without it.

DMARC in enforcing mode is likely to reduce your deliverability rather than increase it, as anything unexpected in mail flow – whether it be a mistake on the senders part, a mistake by the receiver ISP or some sort of forwarding – will cause mail to be lost. In theory an ISP may be using the existence of enforcing DMARC as a positive signal but that’s probably far outweighed by the positives of the DKIM and SPF you had to put in place to get there. (There’re a lot of “soft” effects that DMARC might conceivable have indirectly on recipient behaviour that could have an impact, but until someone independent does real research on that speculation about it is a bit ineffable.)

For domains that are entirely dedicated to sending bulk mail, and which have no humans sending mail with normal mail clients the overhead of deploying DMARC can be tiny, especially if use of DMARC is baked in from the beginning. For more complex domains, or for retrofitting DMARC to a domain that is already in use the effort required to deploy DMARC while being sure that ongoing operations aren’t going to be impacted can be significant, potentially requiring months and person-years of work.

Enforcing DMARC is a prerequisite for BIMI.

ARC

Naive or uncaring use of DMARC breaks mailing lists and forwarding. ARC fixes some cases of that.

Original intent

ARC is Authenticated Received Chain. It allows mail forwarders to communicate whether the mail they’re forwarding was authenticated before they forwarded it.

Current usage

It’s standardized, it’s being deployed by receiver ISPs and forwarders. It seems to do what it says on the box.

Benefits and risks

It’s something that most people don’t have to care about, other than knowing that by making forwarding of authenticated mail more robust it mitigates some of the risks of DMARC.

BIMI

BIMI lets a sender have their logo displayed next to their email in the inbox at some email providers.

Original intent

To allow email senders to loot their companies marketing budget to pay for the costs of deploying DMARC.

No, really. But it’s probably a good thing.

BIMI allows whitelisted senders to display an image next to their email if their mail passes DMARC and they’re vetted in some manner. There’s not really any technical reason for BIMI to require DMARC (as opposed to DKIM and SPF), but it’s a good carrot that mailbox providers can use to encourage senders to deploy DMARC.

Current usage

Gradually being rolled out at some providers.

Benefits and risks

Data on whether having the brands logo displayed next to an email helps with recipient trust or avoiding phishing has been mixed. There’s no real downside, though, beyond the risks of deploying DMARC and the costs of being vetted.

If you already have DMARC in place you should consider BIMI. If you’re considering deploying DMARC you should probably include BIMI as part of that proposal.

STARTTLS

STARTTLS is an extension to SMTP that allows mailservers to exchange mail over an encrypted channel rather than as plain text.

Current usage

Opportunistic STARTTLS is widely supported by receiver ISPs. It’s used for transport security, protecting traffic from passive interception during delivery.

Benefits and risks

Much as most of the web is quietly moving from unencrypted http to TLS protected https the same is true of email. Supporting it when sending email is typically just a configuration setting, and there’s no real drawback from turning it on beyond a slight increase in CPU usage (which is unlikely to be a limiting factor for delivery rates anyway).

Google have stated that they believe in TLS everywhere, to the extent that they provide search benefits to sites that offer it. It’s safe to assume that they’d like to see TLS on their inbound email too.

Null MX

Not exactly authentication, rather a way to say “this domain doesn’t accept email”.

Original intent

Ideally when you’re sending email you’d look up the MX records of the recipient in DNS, then send to one of those. If there were no MX records you wouldn’t send the email. But for backwards compatibility with the internet of the 1980s you also have to check for an A record if there are no MX records.

If a domain doesn’t want email, but does want a webserver, they’ll publish an A records for the webserver and so senders will try and deliver mail there.

Null MX is a formal way to publish MX records saying “don’t even bother trying to send mail”.

Current usage

Universally supported, and many smarthosts will special-case it so as to not even try to send the mail and immediately suppress recipients at those domains without any retry attempts.

Prehistory

Repudiated Mail From, Mail Transmitter, RMX, DMP

These were all early attempts, starting in the late 80s but crystalizing in the early 90s, to identify legitimate return paths in email. They evolved into SPF.

DomainKeys, Identified Internet Mail

DomainKeys was developed by Yahoo, and Identified Internet Mail by Cisco to verify the source of email. Rather than having two competing standards the authors merged them to give DKIM.

DomainKeys had an outbound signing policy that added repudiation of the From: field, much as DMARC does to DKIM/SPF.

SSP, ASP and ADSP

DKIM removed DomainKeys outbound signing policy, avoiding tying all the operational problems with it from the DKIM standardization process by deferring it to a separate standard.

That standard was called variously SSP (Sender Signing Practices), ASP (Author Signing Practices) and ADSP (Author Domain Signing Practices). They were standardized in 2009, but abandoned due to being unused. The experience from that experiment informed DMARC’s development.