ARC: Authenticated Received Chain

On Friday I talked a little about DMARC being a negative assertion rather than an authentication method, and also about how and when it could be deployed without causing problems. Today, how DMARC went wrong and a partial fix for it that is coming down the standards pipeline.
What breaks?

DMARC (with p=reject) risks causing problems any time mail with the protected domain in the From: field is either sent from a mailserver that is not under the control of the protected domain, or forwarded by a mailserver not under the control of the protected domain (and modified, however trivially, as it’s forwarded). “Problems” meaning the email is silently discarded.
This table summarizes some of the mail forwarding situations and what they break – but only from the original sender’s perspective. (If forwarding mail from a users mailbox on provider A to their mailbox on provider-Y breaks because of a DMARC policy on provider-A that’s the user’s problem, or maybe provider-A or provider-Y, but not the original sender’s.)

Use caseSPFDKIMDMARC
alias forwarding
procmail forwarding?*?
discussion mailing list
recipient vanity domain forwarding
recipient forwarding service (e.g. alumni domains)
recipient forwarding e.g. yahoo to gmail✔*
recipient mailbox forwarding via POP
consumer email addresses using other mailbox providers
consumer email addresses sending from ESPs
consumer email addresses using, e.g. zendesk

(* some situations may modify the body or headers of the message, breaking the DKIM signature and causing DMARC failures)
(If you’re diagnosing – or trying to avoid – DMARC issues it’s worth remembering that DKIM signatures can break, apparently randomly, due to issues with how the original email was constructed. If the mail wasn’t forwarded the SPF will be valid and you might not notice the broader DKIM issue.)
Where does it break?
The issues that break DMARC tend only to apply to “loosely controlled” domains – domains that don’t have full control over their mail streams, or which have humans who feel they own addresses in those domains and so want to use them outside the limits DMARC places on them. AOL, Yahoo and LinkedIn are some domains who’ve done that.
There’ve been some rather hacky workarounds for some specific situations. Some discussion mailing lists put a fake email address in the From header, rather than the original author (which prevents DMARC policy triggered bounces, but causes problems for others on the mailing list). AOL added C0nstant Contact, Sailthru and Zendesk to their SPF record.
DMARC doesn’t force receiving ISPs to take any particular action – it’s just advice. Some ISPs treat it naively and discard or reject any mail with a DMARC p=reject domain in the From: header that fails authentication.
Others take a more realistic approach and acknowledge that, for example, a mailing list server with a good reputation that’s emitting unauthenticated email from users at AOL or Yahoo is far more likely to be breaking authentication for legitimate email it’s forwarding than to be an evil phisher.
What can we mitigate?
It would be really nice to be able to say “I trust the forwarder that sent this to me, and I trust them when they tell me that the mail authenticated correctly when they received it” in a more mechanical manner than “I trust this long list of mailing list servers”.
That’s the plan behind Authenticated Received Chain (ARC).
Authenticated Received Chain
In much the same way that regular Received: headers record the series of mailservers that an email passes through as it’s delivered, ARC’s ARC-Seal: headers record the series of administrative domains a company passes through.
An administrative domain is, pretty much, a single company or single email system. While an email going through Microsoft might go through give different mailservers with each mailserver adding a new Received: header the same email would have just on ARC-Seal: header recording it entering the microsoft.com email system.
Other than that, they record where an email has gone, just like a Received: header. Unlike a Received: header, though, they’re cryptographically authenticated so that the organization adding the header can take responsibility for that email at that point in the delivery. If you think that sounds similar to the DKIM concept of taking responsibility for an email as it’s sent, you’re right. An ARC-Seal: header is like a simplified, stripped-down DKIM signature that signs just the ARC-related headers of the message.
Each ARC-Seal header can also have a couple of other ARC headers associated with it.
One is ARC-Message-Signature:. It’s pretty much the same as a DKIM-Signature: header, letting each hop of the delivery add a new DKIM-esque signature of the body of the message and the headers of the message as they send it on.
The other is ARC-Authentication-Results:. It contains the same information as an Authentication-Results: header added by that hop would – did SPF validation pass? was the DKIM signature valid? (and potentially all sorts of other authentication results, from authentication methods that are obsolete or are yet to be invented).
A recipient can step through the series of ARC-Seal: headers and as long as they trust each participant they have authenticated information about the content of the message and the authentication status of that message as it was received by that participant.
This lets an ISP – if they invest in the reputation tracking needed to make the “trust” decision about each participant – identify email that would have been validly authenticated if it hadn’t been through a forwarder or a mailing list and treat that mail as though it were validly authenticated, for making decisions about rejecting it due to DMARC policies.
Who does this affect?
If you’re a user at a domain that publishes DMARC p=reject then ARC has the potential for making your use of forwarders and mailing lists much, much more reliable.
If you’re a participant in discussion mailing lists then as long as the mailing list manager deploys ARC and your ISP is ARC aware you’re less likely to see messages vanish due to DMARC issues, and your mailing list manager will (hopefully) remove some of the gross workarounds they’ve previously put into place to mitigate those problems.
If you develop mailing list management software, or anything that forwards email, you should be reading the ARC specs and following the ARC mailing list and be thinking about how you’re going to implement this.
If you use mailing list software, you should check up on whether it’s implementing ARC. If they are, deploy it when you can. If they’re not, ask them to.
If you’re a mailbox provider you should probably find out more. ARC is something you’re going to want to plug in to your existing reputation based filters.
If you’re a typical ESP and you send email on behalf of your customers who provide the content to you via a web interface or an API and who want to send “From” their consumer email address then ARC is not going to help you. This is not the fix for “individuals and small businesses with consumer email addresses want to run mailing lists” you were looking for. Sorry.
Senders of all flavours. There’s nothing you need to change. As ARC is rolled out it will make some forms of authentication breakage less common, and that may affect your decisions as to whether or when to deploy DMARC reject or quarantine policies. If you’re using DMARC p=none with reporting today you might see those changes happening over then next year or so.
Status
AOL and Gmail are showing ARC results today.
Gmail is adding ARC headers to mail they send or forward.
There are interoperable implementations of the core algorithms in python, C, milter and perl. And a test suite.
Mailman and Sympa are actively adding support.
Want to know more?
ARC home page: arc-spec.org
ARC specification: draft-ietf-dmarc-arc-protocol-03
Recommended usage: draft-ietf-dmarc-arc-usage-01
Mailing list: subscribe or read the archives
 
 

Related Posts

More on ARC

ARC – Authenticated Received Chain – is a way for email forwarders to mitigate the problems caused by users sending mail from domains with DMARC p=reject.
It allows a forwarder to record the DKIM authentication as they receive a mail, then “tunnel” that authentication on to the final recipient. If the final recipient trusts the forwarder, then they can also trust the tunneled DKIM authentication, and allow the mail to be delivered despite the DMARC p=reject published by the sending domain.
The specification and interoperability testing are progressing nicely and it’s definitely going to be useful for discussion list operators and vanity forwarders soon. It’s not something that’s as likely to help ESPs targeting small organizations and individuals, so all y’all shouldn’t be holding your breath for that.
There’s a more information about it at arc-spec.org and they’ve just published a great presentation with a technical overview of how it works:

Read More

The philosophy of DMARC

We know that legitimate email sent with valid SPF and a DKIM signature often breaks in transit.
SPF will fail any time mail is forwarded – via a mailing list, a forwarding service used by the recipient, or just ad-hoc forwarding.
DKIM will fail any time the message is modified in transit. That can be obviously visible changes, such as a mailing list tagging a subject header or adding a footer to the body. It can also be less obvious changes, such as intermediate MTAs wrapping lines that are too long, reencoding content or repackaging the message altogether – perhaps when delivering from a mailserver that is 8BITMIME compliant to one that isn’t.

(This image has absolutely nothing to do with email authentication, but searching for stock photography about email or authentication or chains or, well, pretty much anything like that leads to horribly depressing corporate imagery. So, no. Have something colourful and optimistic instead.)
As SPF and DKIM are typically used, none of this is much of a problem. A message being authenticated provides a little extra information to the receiving mailserver, and the domain attached to the authentication can be used to look up a senders reputation, giving a potential boost to the chances of the mail being sent to the inbox. If the authentication is broken, though, the mail will still be judged on it’s merits – is it coming from an IP address that’s a source of good mail, does the content look legitimate, and all the other things a spam filter looks at.
That authentication is a (potentially big) positive signal, but lack of authentication isn’t really any signal at all is why SPF and DKIM being fragile wasn’t an issue. SPF and DKIM are positive assertions – “IF this mail IS authenticated THEN IT IS from me”.
That changed when DMARC became popular, though.
DMARC allows the owner of a domain to say “We send no mail that is not authenticated, and we promise that none of that authentication will be broken in transit”. DMARC is a negative assertion – “IF this mail IS NOT authenticated THEN it IS NOT from me”. It converts the absence of a positive assertion into a negative assertion.
This isn’t the first attempt to layer a “we authenticate everything” negative assertion on top of fragile email authentication. SPF did it, with the -all flag (which is universally ignored, leaving SPF purely as a positive assertion). DomainKeys did it, with DomainKeys policy records (which you occasionally still see published, but were never really used to reject mail). DKIM did it with ADSP – which didn’t see much use either.
The reason none of them were used much is because even when senders were telling the truth about “we send no email that is not authenticated” they were always lying, to varying degrees, about “none of the authentication will be broken in transit”.
If your domain that is solely used for bulk email. If it’s never for used mail sent by human beings, not even customer support employees. If it’s a newly created domain with no legacy usage that only sends email from a very tightly controlled infrastructure. If you only send email that’s been created via a well implemented message composition pipeline that ensures the content of the is not just RFC compliant but also “well formed”, with short lines, simple widely implemented encoding, vanilla mime structure and so on. And it’s sent out via conservatively configured smarthosts that deliver directly to the end recipients MX. And if you know that the demographics of your recipients are such that the minority that are forwarding that mail elsewhere (e.g. from their Yahoo account to their Google account or via an alumni mail alias) is a small enough group that you don’t care about them…
If all of those things are true, then your domain is going to be able to deploy DMARC pretty easily and safely. If not, though, how can you tell?
That’s the place where DMARC improves over it’s predecessors. It allows you not only to publish a DMARC policy record in test mode, so it’s not actually used to filter your mail (well, mostly, but that’s a longer story) but also to ask recipients to notify you of mail that seems to be from you but which isn’t authenticated.
You can publish a “p=none” DMARC record with notification addresses in it and wait and see what happens. You’ll get notification of mail that has your domain in the From: field but which isn’t authenticated.
As a first round of action that lets you see where you’re sending email from that you didn’t know about. Sysadmin notification email. That marketing splinter group in Sasketchwan. The outsourced survey company.
Once you’ve cleaned all that up, and made sure everyone is authenticating their mail then you can look at what’s left. The next step is likely to be mistakes you’re making in authentication or message composition that’s causing some of your mail – typically depending on content, and source and recipient domain – to become unauthenticated. Clean that up, make sure all your message composition is squeaky clean, make sure employees aren’t sending mail using that domain in ways you don’t authorize (interacting with mailing list, for example).
By that point you’ll have reduced the torrent of reports you’re getting to two types. One is mail that you send that has it’s authentication broken in transit through some process you have no control over. The other type is mail that has your domain in the “From” field but which you didn’t send. Some of that may be legitimate use of your domain by your employees, such as forward-to-a-friend services, signing up for document delivery via email, third-party notification services. By deploying DMARC you are declaring all that sort of usage to be illegitimate, and you’ll need to get all your employees to stop doing it (or, at least, know that it’s going to stop working). The rest of it is likely a mix of spam and phishing mail. The spam, that’s just using your domain in random from addresses, you probably don’t care about. The phishing you do.
You’ve finally cleaned up your mail infrastructure and policies enough to gather the data you need. How much of my legitimate email will have it’s authentication broken (and hence be silently thrown away by DMARC)? And how much hostile phishing mail is targeting my users (and using the exact domain you are)?
Then you have the information you need to make an informed decision as to how badly deploying DMARC will break your legitimate use of email (after you’ve done everything you can to minimize that) and some idea of whether it will provide you any benefit, at least in the shorter term.
That testing phase, where senders can use other peoples mail infrastructure to investigate their sending practices, gradually fix any problems and finally gather some metrics is what made gave the developers of the DMARC spec confidence that it wouldn’t break things, and made it much more deployable than previous approaches to negative assertion.
On Monday, how all that optimistic reasoning went to hell, what it broke and how we’re trying to fix it.

Read More