I don’t always do predictions posts, even though they’re popular. Most years I skip them because I don’t see major changes in the email space. And, I’m not the type to just write a prediction post just to post a prediction.
This year, though, I do see changes for everyone in the email space. Most of them center on finally having to deal with the technical debt that’s been accumulating over the past few years. I see ISPs and ESPs spending a lot of development effort to cope with the ongoing evolution authentication requirements.
When people started seriously looking at how to authenticate email, the first goal was getting organizations to implement the protocols. This was a practical concession; in order for a new protocol to be used it needs to be widely implemented. Phase one of authenticating email was simply about publishing protocols and getting organizations to use them.
During phase one, the organization that authenticated a mail hasn’t been important. In fact, the SPF spec almost guarantees that the ESP domain is the authenticated domain. In DKIM, the spec says any domain could sign as long as they could publish a public key in that domain’s domainkeys record.
ESPs took full advantage of this and lowered their own development overhead by taking most of the authentication responsibility on themselves. Their domains were in the 5321.from and they published the SPF records. Domains they control were in the d= and they generated and published the DKIM keys. Mail was authenticated without ESP customers having to do much.
We’ve hit the end of phase one. Most of the major players in the email space are authenticating outbound email. Many of the major players are checking authentication on the inbound. Phase one was a success.
We’re now entering phase two, and that changes thing. In phase two, SPF and DKIM are used as the foundation for user visible authentication. Neither SPF nor DKIM were designed to be user visible protocols. To understand what they’re authenticating you have to understand SMTP and email. Even now there are days when I begin talking about one of them and have to take a step back and think hard about what is being authenticated. And I use these things every day!
DMARC is the first of these end user visible protocols built on SPF and DKIM. It uses the established and widespread authentication to validate the user visible from address. This authentication requires that the d= value or the 5321.from address belong belong to the same domain in the visible from address. While you can pick whether the alignment between the visible from and the authentication is “strict” or “relaxed” you have no choice about the alignment.
Prior to DMARC no one really paid much attention to the domain doing the authentication. Authentication was a yes or a no question. If the answer was yes, then receivers could use the authenticated domain to build a reputation. But they weren’t really checking much in the way of who was doing the authentication.
In the push to deploy authentication, ESPs assumed the responsibility for authentication deployed ESPs took the responsibility and did most of the work. For many or most customers, authentication was as simple as clicking a checkbox during deployment. Some ESPs do currently let customers authenticate the mail themselves, but there’s enough overhead in getting that deployed that they often charged extra to cover the costs.
DMARC is rapidly becoming an expectation or even a full on requirement for inbox delivery. In order to authenticate with DMARC, the authenticating domain must be in the same domain space as the visible from. If senders want to use their own domain in the visible from, DNS records have to be present in that domain space. Whether it’s a SPF TXT record or a domainkeys record the email sender customer needs to publish the correct information in DNS. Even now, if you try to authenticate with DKIM through google apps, they require you to publish DNS records.
ESPs aren’t in a situation where they can effectively manage authentication alignment for all their customers. Hosting companies are in even worse shape when it comes to letting customers authenticate email. Developers are facing the fact they need to go back and rework their authentication code. Businesses are facing the fact they need to change their processes so customers can authenticate with DMARC.
It’s not just the infrastructure providers that are facing challenges with authentication. Senders are going to discover they can no longer hand authentication off to their ESPs and not worry about it. They’re going to have to get DNS records published by their own staff.
Getting DNS updates through some big companies is sometimes more difficult than it should be. I had one client a few years ago where getting rDNS changed to something non-generic took over a month. From an IT standpoint, changing DNS should require approvals and proper channels. Marketers may find this new process challenging.
And, if organizations want to publish reject policies for their domains, then they will have to publish records for every outside provider they use. Some of those providers can’t support DMARC alignment right now.
In 2016 a lot of companies will discover their current infrastructure can’t cope with modern authentication requirements. A lot of effort, both in terms of product development and software development, will need to be spent to meet current needs. This means a lot of user visible features will be displaced while the technical debt is paid.
These changes will improve the security and safety of email for everyone. It won’t be very user visible, which will give the impression this was a slow year for email development. Don’t let that fool you, this will be a pivotal year in email.
What to expect in 2016
W
RSS - Posts
Interesting.
Laura, this domain matching, can the envelope sender domain be a subdomain of the From: domain? That seems to be how ESPs currently handle custom sending domains in order to still able to process bounces. But you wouldn’t want your From: or Reply-To: pointed to a bounce handler.
And if so, would you want your DKIM signatures to reference the envelope sending domain or the parent From: domain?
@Alan The envelope sender domain can be a subdomain of the From: domain, in DMARC terms that would be relaxed alignment.
@Alan Some receivers are not handling sub-domains the way in which you might expect. Therefore setting separate records for sending subdomains is my current recommendation. If it is possible in your configuration to add dkim records for your sub-domain, I would certainly be doing that to try and minimise on possible issues.
….
One small thing to note however Laura, we are not quite there with Apps yet. You are not required to make DNS changes to authenticate your new sending domain. It is just Googles recommended method. You are still able to authenticate by uploading an html file, adding a meta-tag or via webmaster tools and thereby avoid DNS changes. I Just double confirmed to be sure it was still currently possible to do it that way.
It sounds like the email environment will continue to become less and less friendly to discussion-type mailing lists.
The authentication you’re talking about, Andrew, is authenticating that you own the domain.
The authentication I’m talking about is signing DKIM with your own DKIM key. That does require publishing the public key in your domainkeys DNS record. I checked before I posted the blog post.
If there is alignment between the envelope sender and the visible from, and there is a valid SPF record, then the mail is considered aligned no matter what the d= is. DMARC just requires alignment and passing with either SPF or DKIM.
It’s not very hard to use custom domains in the envelope from. bounce@esp.customer.example that points to bouncehandler.esp.example. It does require that the customer publish a DNS record for esp.customer.example, though. And that is going to slow down deployment for a lot of senders and ESPs.
Actually, Jacob, I think it’s going to start being better than the last year or two, at least in respect to well-run discussion lists.
The ARC proposal (which we mentioned and provided some links to at https://wordtothewise.com/2015/10/dmarc-news-gmail-preject-and-arc/ ) is going to undo some of the damage that has been done to discussion lists, and let them play nice with authentication changes.
Thanks Laura, I see you edited your post to clarify. I must be being a little slow today as I am more lost as to what you mean now than on my first reading. Really not sure how else you would be signing DKIM with your own DKIM key and not publish a public key in your DNS.
I thought for a moment you may have meant Google would not DKIM sign my messages if I did not DKIM sign. But that is not the case, just tested and I see they sign with *.*gappssmtp.com even when no DKIM or SPF exists on my side.
dkim=pass header.i=@cpo-news.20150623.gappssmtp.com
Thanks, Steve — I missed that the first time around.
Andrew, google signing with *gappssmtp.com is google taking responsibility for the message. That was what a lot of companies did in the push to get authentication adopted.
As we move forward, that type of signature isn’t going to be enough for a message to count as authenticated. The authentication is going to have to come from the organization sending the message. Yes, google will sign the message. But it’s not going to matter as the signature domain doesn’t align with the visible from domain.
It’s a big change and I can understand why some people find it confusing. We’ve spent years now just making sure mail is authenticated without caring who is doing the authentication. Now, we’re transitioning to a place where who is doing the authentication matters. It’s fine, for now and for the short term, to continue to let google sign your mail with their domain. The whole point of the blog post is that kind of authentication isn’t going to be sufficient in the future.
Thanks for responding Laura, but I think I should clarify. I do not have Google sign my real mail, I have specific domains and Google Apps accounts setup with a variety of different configurations for the purposes of testing. The same at Outlook and at other vendors as well bare-metal, various instances in the cloud and on a variety of different VPS’ and dozens of accounts deployed at various providers of webmail solutions. It is part of my deliverability arsenal.
My primary accounts have been signed. Although some configurations are more relaxed than others, for example mail you would see from me on lists. My ‘real email address” for p2p comms is SPF, DKIM and DMARC authenticated and set to p=reject.
https://dmarcian.com/dmarc-inspector/ab.email
Nor am I confused by the principles involved in SPF, DKIM nor DMARC, or implementation of those technologies. I have got a fair grasp of the fundamentals surrounding DMARC, I have even released a tool for generating DMARC records at http://dmarc.in (in Beta use at your own risk).
I understood the general point you attempted to make in your post, I passed comment about a statement I considered to be inaccurate and in my understanding without basis, ” Even now, if you try to authenticate through google apps, they require you to publish DNS records”. I was therefore genuinely keen to learn what the basis was for that statement, as I would rather be corrected than hold to an incorrect assumption and misadvise my clients.
Your response to my query appeared to be presented as a rebuttal. I was asking if I had missed something. You intimated I misunderstood your point and amended your post, now stating “to authenticate with DKIM through google apps, they require you to publish DNS records” leaving the impression that was the original verbage. You then elaborated in your comments to me >>”The authentication I’m talking about is signing DKIM with your own DKIM key. That does require publishing the public key in your domainkeys DNS record.”<Apps->Gmail->Authenticate) and as discussed on the Mailops list and confirmed by Brandon Long from Google in December. That change is not to require customers to make DNS changes at all.
The change is to force DKIM authentication on to all users, if a user does not enable authentication and make the required TXT entries into their DNS then Google will simply DKIM sign with its own domain (GAPPSMTP.com) and key.
Apologies if you feel this is pedantic on my part but I think its fundamentally important such changes are reported accurately, especially on a blog as influential as yours.
Google signing with *gappssmtp.com is google signing the messages with DKIM. The messages fail DMARC Authentication, as they did last month and will continue to fail next month. As I have not published an SPF record that too fails and will continue to fail. The only way to fix that would be to add the relevant DNS TXT Records to my domain.
<>
I disagree, it counts as being authenticated by DKIM today and I am willing to bet it will still count as being authenticated by DKIM in 2017.
It is not particularly effective as it is an unverifiable signature against my DNS records, but a verifiable DKIM signature will still be considered DKIM authenticated, even when it is non-aligned, and it will have value.
There may well be additional value at many mailboxes by having additional authentication configured. I do not doubt that. Bu that message I used as an example fails DMARC authentication today, as it would have in October 2015 and dare say will continue to fail DMARC throughout 2016 and beyond unless there is a substantive change in what I do, or Google does in handling email from that test account. Either I publish DNS records or Google starts to amend the email address in the header and sends mail “On Behalf of” or it will continue to fail DMARC. It will not suddenly be non compliant in respect of DKIM .
>> “…for a message to count as authenticated. The authentication is going to have to come from the organization sending the message.”
This is not true, even if we assume you mean DMARC Authenticated.
For that to be close to accurate you would need to qualify your statement significantly:
. Omit either qualifier and it will be inaccurate. This is important because many senders only care about what is displayed to the human eye, and that is not alway what you are referring to as the visible from address.
DMARC cares only that 5321.MailFrom and 5322.From email address are aligned and that they align with SPF or DKIM. To refer to a ‘visible from’ is a little misleading in the current day and age. Often neither of those address are necessarily particularly visible to average Joe.
The average person see’s the display-name (aka Friendly From), this often contains no email address at all and displays simply as a name, and they also see the reply-to address which does not need to be the 5322.From or the 5321.MailFrom
Neither the display-name nor an additional forced reply-to address does DMARC care an iota about. That alone will be enough for some senders to be happy in respect of branding concerns and leaves all the authentication worries back with the ESP.
I am not saying that there will not be time invested this year in coming up with workarounds, just that ESPs have faced similar issues before and are well versed in dealing with such things. There will be vendors like WttW and my own Deliverability Ltd, and others that will undoubtedly pick up additional work in helping ESPs resolve issues pertaining to DMARC. I simply contend that your blowing the significance and scale out of proportion and occassionally straying from absolute fact. There is more than one low overhead method for ESPs to send in a manner that is compliant and minimises overhead on the customer.
I don’t think it is such a big change in respect of the ESP world. In my experience where you refer to some ESPs allowing something, the majority have allowed it. The workarounds that ESPs found for handling DKIM will serve most very well in respect of DMARC with a little adapting.
One thing, I am sure with sub $1 domains now available we will see a much wider use of cousin domain registrations for email sending, and that will mean authentication using a domain owned by the client but under complete management of the ESP.
That aside, in respect of senders having to make DNS changes themselves, they have had to before DMARC if they wanted to sign their own email and often, as in the case of DKIM even when delegating responsibility to others. As you said “In DKIM, the spec says any domain could sign as long as they could publish a public key in that domain’s domainkeys record.” Publishing that key required a TXT change to the DNS.
From a sender perspective not so dramatic a change either.
Wow, that is indeed an overly pedantic response. I can’t even read all of it.
All I can tell you, Laura’s right, in that if you want to sign with d=your custom domain, you’re setting up something in DNS. Even if you use Google Apps. I’m a Google Apps user myself, and I see both the Google DKIM signature and my custom DKIM signature.
The point being, if you want to pass DMARC, you really have to be/should be/are going to be implementing DKIM with d= for your own domain. Somewhat of a separate issue than “will the mail still be signed with DKIM in 2017.” Nobody’s saying Google’s signatures are going to magically stop working. But they will not magically start working for purposes of DMARC.
If you decide to publish an SPF record (only) instead, great! You (still) just set up a custom DNS record.
“DMARC is rapidly becoming an expectation or even a full on requirement for inbox delivery. “… I don’t believed this to be true. If so, for which service provider(s)?
Thanks Al and indeed if that was what had been said there would be no discussion.
Separate point, the blog post has had clarification added as a result of my comments.
Setting something up in DNS is a given, that was one of my points, the post intimates there was some halcyon era when you could authenticate your domains mail and not touch DNS.
In respect of it magically not working down the road, actually Laura stated it does not matter “it isn’t going to be enough for a message to count as authenticated”.
I had a genuine question as I believed I had missed something, I had not. As opposed to acknowledging a clarification would benefit the reader the average reader was left with the impression I was confused by DKIM & DMARC.
Finally the whole d= question does depend on how RFC 6541 pans out. I am confident at some point we will see Authorized Third-Party Signatures one way or another.
Al got the point correctly. He, and a lot of other people, seem to have understood the point I was making just fine. If you read the post, my predictions are clearly about DMARC authentication becoming the standard.
Since you’re still confused about what I’m saying, let me see if I can make it simpler.
I believe in the future, DMARC is going to become the primary standard for authentication. We already have ISPs stating “we check for DMARC authentication first, and then fall back to DKIM and SPF.” Receivers are going to start checking for alignment and treating aligned mail better. Just having an unaligned DKIM signature or SPF record isn’t going to be enough.
The problem is, a lot of systems can’t currently support DMARC alignment. The technology isn’t there at the moment. I used Google Apps as an example because it’s something many people have experience with. The reality it, there are hundreds and thousands of SaaS systems that send mail that can’t support DMARC alignment. That’s going to need to change. And the development effort to make that change on the backend is going to limit the other kinds of development that can happen in the email space.
Remember, too, this is all my opinion of what’s going to happen. You seem to have a different opinion and think that DMARC isn’t going to be important moving forward. I used to think that, too, until I started paying attention to what the receivers were saying. Listening to what they thought was important helped me develop a vision for the future. And DMARC authentication is coming and it’s going to change how we do things.
There is no serious consideration being given to any third party signature mechanism at the moment. The scaling problems far outweigh the benefits.
To cover the example given, it’s been four years since RFC6541 (ATPS) was published as an RFC. At that time, we added ATPS to OpenDKIM so that it was available for free to anyone that wants to play with it and see if it was a useful thing to adopt. There has been no uptake at all since we did so (our collected usage data found exactly one site that tried it, one of our beta testers), and to my knowledge, no other implementations.
In the DMARC Working Group, there was proposed a mechanism for augmenting DKIM to incorporate the idea of “conditional signatures”. Support for that idea is also lukewarm at best. The next OpenDKIM will include an implementation for people to try, but I have my doubts it will see much adoption.
There is a new proposal being developed that provides end-to-end domain-level authentication of sorts, but it is nascent.
I will preface by saying most of the earlier comments I have made are no longer relevant or appear not to make sense due to the edits you have subsequently made to the blog post.
I have clarified several times, so once more as it seems you do not understand Laura. When you made the comment about Gmail (since edited, then deleted completely), I thought you were aware of something I was not, that you had additional information, that was not the case. No explanation needed on your part.
Had my earlier comments not been relevant, and not made sense to you, I have to wonder, why the repeated edits. I doubt very much I will make the effort to provide any feedback directly in the near future however.
Please do not assume to know what I am thinking ref: “I used to think like that too” – I am not sure how anyone would come to the assumption that I believed DMARC was not going to be important. If that were the case I would not have invested limited time and resources available to me in projects such as dmarc.in
As an aside I have found significant issues with Gmails handing of DMARC within GfW, hence subsequently dropping p=reject personally.
As you seem determined to make this adversarial and desire to focus on my disagreeing with your opinion/conjecture/predictions Laura, I will oblige you.
I have disagreed with your summation of the issue specifically:
Stating hundreds of thousands of SaaS systems simply cannot support DMARC alignment. Untrue, likely you mean they cannot align with the senders primary sending domain.
Stating that in 2016 development of email innovation in the email SaaS space will be held back because platforms need to fundamentally change to enable support for DMARC alignment.
As I said before, some may make those significant changes and invest considerable development resource in doing so, but not all. Many will find workarounds , as has been the case with many ESPs who have had issues with clients implementation of DKIM before. There are workarounds that do not involve aligning with a brands primary sending domain.
The simplest workaround for many has been to register cousin domains and I rather suspect with the number of sub $1 domain registrations that are now available, this will only become an increasingly viable option.
I am unable to edit my comments, so do not have that luxury, I will stand by what I said. Unless there are major changes to DMARC the majority of SaaS systems unable to support it will simply find a workaround.
I am confident you are incorrect on that issue, so I will make a public pledge. If the majority of ESPs are sending their clients mail from their clients primary domains signed with a p=reject I will donate $1000 to a charity of your choice.
Lets see in Jan 2017 how 2016 panned out.
Thanks Murray, where ATOS might fail (have failed already?), you indicate there is the possibility of “conditional signatures’, and with that not being received as well as it might, new proposals are surfacing.
My prediction is that before p=reject becomes the norm or to use another phrase, the standard then a viable solution for the legitimate usage of email that doesn’t conform to current dmarc specifications will need to be addressed. Otherwise lots of mail signing p=none would not seem to be of all that much benefit to receivers, albeit the senders will still have the benefit of reports, therefore not making it redundant
Andrew: There was one edit made to the blog post (I added “with DKIM” into the sentence “Even now, if you try to authenticate with DKIM through google apps, they require you to publish DNS records.”)
Any other edits are a figment of your imagination.
Correct, you added with DKIM, I thought the whole sentence was deleted thereafter, evidently not. That said if you want to authenticate with DKIM you always needed to publish DNS records, Google or otherwise, always have, always will. The “Even now” was what caused me to wonder if you had fresh information, which you did not. I will leave you to have the last word, it is after all your blog. My point is the sentence was pointless and totally superfluous.