Compared with some of the more complex pieces of the email protocol the From: header seems deceptively simple. But I’ve heard several people be confused about what it’s made up of over the past couple of months, so I thought I’d dig a bit deeper into how it’s defined and how it’s used in practice.
Here’s a simple example:
There are two interesting parts.
The first is what’s technically called the display-name, but more commonly known as the “friendly from” in the bulk email industry. It has no meaning within the email protocol, it’s just text that’s displayed to the recipient to describe who an email was sent by. Because it’s just text, you can put anything you like in there, but it’s usually either the name of the person who wrote the mail or the name of the company or brand that sent it.
The second is the actual email address, the thing with an at-sign in it. Surprisingly, this isn’t used at all during the actual delivery of the email; there’s a hidden field (called the return path or the 5321.MailFrom or the envelope sender or the bounce address) that’s used instead. For person-to-person email it’s usually the same address, but for bulk mail it’s often different.
So what does the actual email address, the 5322.From, mean? For that we go to the document that specifies what email headers mean – RFC 5322, “Internet Message Format”. (RFC 5322 is the updated replacement of the older RFC 822 – and that’s why the actual email address is often called the 822.From or 5322.From when people are being precise about exactly which email address they’re talking about).
RFC 5322 says “The From: field specifies the author of the message, that is, the mailbox of the person or system responsible for the writing of the message.” and “In all cases, the From: field SHOULD NOT contain any mailbox that does not belong to the author of the message”. It’s the email address of the author of the message.
(In some cases the email may have been written by the author, but then sent on their behalf by someone else. RFC 5322 says that in that situation the email address in the From field is still the author of the message. The person who sent the message gets their own field, “Sender:”).
What is the 5322.From used for? During the delivery process it’s used for some sorts of filtering and authentication. In particular, if you’re reading about DMARC you’ll see “identifier alignment” mentioned a lot – which basically means “the only domain we care about authenticating is the one in the 5322.From”. It’s also the usual field that’s used in user-visible mail filtering such as whitelisting email addresses that are in the users address book.
In the mail client itself the most obvious use of the 5322.From is that when you hit reply, that’s the email address your reply will go to by default. The author of the mail can override that by adding a Reply-To field, containing one or more email addresses if they want different behaviour. It’s also commonly used to filter email and to group mails by author.
What’s displayed to the end user? Originally the entire content of the From: header was shown in the recipients mailbox but it’s now fairly common to display just the friendly from, with no mention of the email address at all. That started in mobile clients, where space is at a premium and the friendly from is just, well, friendlier – but it’s spread to desktop and webmail clients too. In Yahoo webmail the 5322.From isn’t displayed anywhere at all unless you find the View Full Header menu option and dig through the raw headers, and my phone doesn’t display it anywhere obvious and only recently made it possible to see it at all.
The anatomy of From:
T
RSS - Posts
So, from the way I read RFC5322, it seems to me that mailing lists should preserve the original From header, as it ‘should not’ display a mailbox not owned by the author of the message, and should add a ‘Sender’ header with the mailing list address. This is what they do; and yet, the DKIM check causing so much trouble with Yahoo.com is forcing us to change the From header to be the address of the list, in apparent violation of RFC5322.
This would seem to me to indicate that in fact the DKIM check is poorly defined; it should not include such headers as Received and Sender, and probably not Subject either (since lists could add tags to a subject). The SPF check works on envelope sender (no problem in mailing lists as this is replaced) and the SIDF check works on PRA (which is Sender if present else From) which is again fine for mailing lists.
The Yahoo.com DKIM signature unusually includes Received headers, so even if your list does not add a footer or a Subject tag, it will still fail, unless you modify the From header and remove the DKIM signature.
Yes, that’s what mailing lists have should do with the From: field (and have done since they were invented). They don’t tend to set the Sender: field (because it doesn’t add any value to recipients – they have the List-Id: field and it’s friends instead).
There’s absolutely no problem with DKIM signing here – DKIM doesn’t care about the domain in the From: address. For mailing lists it’s probably good practice to leave any existing DKIM signature from the author in place (it might be broken by the list, it might not) and to add it’s own DKIM signature.
Problems appear if you try and use strict DMARC on domains that are used by real users – mailing lists are the most obvious point that DMARC stumbles over there, but there are others.
But this isn’t a flaw with DMARC, or with mailing list handling, even though the combination of the two can cause operational problems (including causing unrelated recipients to be removed from the mailing list due to DMARC-induced bounces).
An ISP that publishes a strict DMARC record is saying “we do not allow our users to post to mailing lists”. The only response from mailing list operators that respects those wishes without causing problems for other mailing list subscribers is to block any subscriber with a domain with strict DMARC from posting to the mailing list.
If respecting the wishes of the ISP/domain owner is less important than the recipients there are other options (but all of those will violate RFC 5322).
The DKIM signature does care about the From domain; depending on settings, and if there are multiple signatures, the domain of the DKIM signature is matched against the domain of the From.
Also, the Sender header is set by mailing list software (eg, Sympa), and is recognised by many mail clients (eg, Outlook), and is also part of the SIDF PRA checks.
No. DKIM absolutely does not care about the From header. You can sign with any d= you like, and there is absolutely no requirement within DKIM for that domain to be the same as the one in the From: header, or anywhere else in the message.
Other tools layered on top of DKIM may care about that, but that’s quite a different thing.
(Yeah, I’m quite forceful in saying that – because it’s a fairly widespread misconception about DKIM that I’ve been trying to correct for *years* 🙂 ).
Actually, at OnlineGroups.net we do set the Sender header but no-one seems to pay any attention to that. In response to new strict DMARC settings, we have chosen to set the From header to an address at our domain (explained at http://onlinegroups.net/blog/2014/04/10/yahoo-dmarc-better-mailing-list-manager/). Yes, that also shows some disregard for RFC5322 but what choice is there? We are taking responsibility for the sending, and delivery of the message on behalf of the author. That isn’t a particularly new thing for Mailing List Managers to do.
Although a DKIM signature can be for any domain independent of the From, and there can be multiple of them even, the problem is that the DMARC rules state that the DKIM signature we want is by the domain of the From address. All others are ignored. Since the DMARC test doesn’t take account of Sender (unlike SIDF which does), and treats a missing DKIM signature as a fail, we have to break RFC5322 and change the From instead of adding a Sender.
To my mind this is a failing in the DMARC specification.
In addition, adding too many headers to the DKIM signature (eg, Received) makes the signature invalid after relay even if you don’t modify Subject or add a footer. This just makes things harder.