DKIM

Google, Alignment and DMARC

Google has been making a number of changes to their systems over the last few weeks. Folks are seeing a lot of changes in Google postmaster tools and they’re seeing changes in how Google is displaying headers in the “show original” tab.

Setting up a smarthost

We run most of our own network services – inbound and outbound email, DNS and web presence. We run separate services for inbound and outbound email to give us more flexibility in how we set things up.

Do we care about SPF alignment?

SPF and DKIM are the two main ways we associate a domain name with a stream of email in an authenticated way. We can choose the DKIM signing domain fairly freely – we can choose any domain or subdomain we control and put it in the d= field of the DKIM signature. But our choice for the SPF domain is more constrained.

Comparing DKIM keys

Sometimes we have a client who has done something wrong when setting up authentication. Their DKIM signing fails due to something being wrong with the public key they’ve published.

Sending domains and hostnames

Lots of times I see someone asking a question and they talk about their sending domain. And it’s sometimes not 100% clear which domain they mean by that – and when we’re talking about alignment and reputation it can make a difference. So here’s a list of (some of?) the different places a mailserver uses a domain.

A series of red arrows pointed left and one green arrow pointing right.

DMARC is the newest of the authentication protocols. It compares the domain in the From: address to the domains authenticated by SPF and DKIM. If either SPF or DKIM pass and they are in the same organizational domain as the domain in the From: address then the email is authenticated with DMARC.

DNS for white label authentication with SproutDNS

I wrote last year about using “stunt” nameservers for customer subdomain authentication – i.e. dynamically generating all the authentication records needed in DNS for each customer as needed.

TXT Records

DKIM public keys live in DNS TXT records. A DNS TXT record contains strings of text, and each string is limited to be no more than 255 characters long.

Customer subdomain authentication

EDIT: Now with a production-ready implementation I talk about more here.

Wildcards and DKIM and DMARC, oh my!

If you’re an ESP with small customers you may have looked at the recent Google / Yahoo requirements around DMARC-style alignment for authentication and panicked a bit.

Don’t break the (RFC) rules

It looks like Microsoft are getting pickier about email address syntax, rejecting mail that uses illegal address formats. That might be what’s causing that “550 5.6.0 CAT.InvalidContent.Exception: DataSourceOperationException, proxyAddress: prefix not supported – ; cannot handle content of message” rejection.

Some Microsoft thoughts

Right at the end of January, Microsoft appears to have made couple of changes to how they’re handling authentication. The interesting piece of this is that, in both cases, Microsoft is taking authentication protocols and using them in ways that are slightly outside the spec, but are logical extensions of the spec.

Cost of authentication

At the end of last year, Steve wrote a post about the different types of authentication. I thought I’d build on that and write about the costs associated with each type. While I know a lot of my readers are actually on the sending side, I’m also going to talk about the costs associated with the receiving side and a little bit about the costs for intermediaries such as CRM systems or ESPs.

Why is DMARC failing?

Multiple times over the last few weeks folks have posted a screenshot of Google Postmaster tools showing some percentage of mail failing DMARC. They then ask why DMARC is failing. Thanks to how DMARC was designed, they don’t need to ask anyone this, they have all the data they need to work this out themselves.

d= for data

A few ISPs use the d= value in the DKIM signature as a way to provide FBL and reputation data to senders. This has some good bits, in that senders can get FBLs and other information regardless of the IP address they’re using and whether or not they have sole access to it.

Authentication is about Identity, not Virtue

I just got some mail claiming to be from “Bank of America <secure@bofasecure.com>”.
It passes SPF:

About that DMARC "exploit"

A security researcher has identified a rendering flaw that allows for “perfect” phishing emails. From his website:

The feds are deploying DMARC

The US National Cybersecurity Assessments & Technical Services Team have issued a mandate on web and email security, including TLS+HSTS for web servers, and STARTTLS+SPF+DKIM+DMARC for email.
It’s … pretty decent for a brief, public requirements doc. It’s compatible with a prudent rollout of email authentication.

Are they using DKIM?

It’s easy to tell if a domain is using SPF – look up the TXT record for the domain and see if any of them begin with “v=spf1”. If one does, they’re using SPF. If none do, they’re not. (If more than one does? They’re publishing invalid SPF.)
AOL are publishing SPF. Geocities aren’t.
For DKIM it’s harder, as a DKIM key isn’t published at a well-known place in DNS. Instead, each signed email includes a “selector” and you look up a record by combining that selector with the fixed string “._domainkey.” and the domain.
If you have DKIM-signed mail from them then you can find the selector (s=) in the DKIM-Signature header and look up the key. For example, Amazon are using a selector of “taugkdi5ljtmsua4uibbmo5mda3r2q3v”, so I can look up TXT records for “taugkdi5ljtmsua4uibbmo5mda3r2q3v._domainkey.amazon.com“, see that there’s a TXT record returned and know there’s a DKIM key.
That’s a particularly obscure selector, probably one they’re using to track DKIM lookups to the user the mail was sent to, but even if a company is using a selector like “jun2016” you’re unlikely to be able to guess it.
But there’s a detail in the DNS spec that says that if a hostname exists, meaning it’s in DNS, then all the hostnames “above” it in the DNS tree also exist (even if there are no DNS records for them). So if anything,_domainkey.example.com exists in DNS, so does _domainkey.example.com. And, conversely, if _domainkey.example.com doesn’t exist, no subdomain of it exists either.
What does it mean for a hostname to exist in DNS? That’s defined by the two most common responses you get to a DNS query.
One is “NOERROR” – it means that the hostname you asked about exists, even if there are no resource records returned for the particular record type you asked about.
The other is “NXDOMAIN” – it means that the hostname you asked about doesn’t exist, for any record type.
So if you look up _domainkey.aol.com you’ll see a “NOERROR” response, and know that AOL have published DKIM public keys and so are probably using DKIM.
(This is where Steve tries to find a domain that isn’t publishing DKIM keys … Ah! Al’s blog!)
If you look up _domainkey.spamresource.com you’ll see an “NXDOMAIN” response, so you know Al isn’t publishing any DKIM public keys, so isn’t sending any DKIM signed mail using that domain.
This isn’t 100% reliable, unfortunately. Some nameservers will (wrongly) return an NXDOMAIN even if there are subdomains, so you might sometimes get an NXDOMAIN even for a domain that is publishing DKIM. shrug
Sometimes you’ll see an actual TXT record in response – e.g. Yahoo or EBay – that’s detritus left over from the days of DomainKeys, a DomainKeys policy record, and it means nothing today.

The philosophy of DMARC

We know that legitimate email sent with valid SPF and a DKIM signature often breaks in transit.
SPF will fail any time mail is forwarded – via a mailing list, a forwarding service used by the recipient, or just ad-hoc forwarding.
DKIM will fail any time the message is modified in transit. That can be obviously visible changes, such as a mailing list tagging a subject header or adding a footer to the body. It can also be less obvious changes, such as intermediate MTAs wrapping lines that are too long, reencoding content or repackaging the message altogether – perhaps when delivering from a mailserver that is 8BITMIME compliant to one that isn’t.

(This image has absolutely nothing to do with email authentication, but searching for stock photography about email or authentication or chains or, well, pretty much anything like that leads to horribly depressing corporate imagery. So, no. Have something colourful and optimistic instead.)
As SPF and DKIM are typically used, none of this is much of a problem. A message being authenticated provides a little extra information to the receiving mailserver, and the domain attached to the authentication can be used to look up a senders reputation, giving a potential boost to the chances of the mail being sent to the inbox. If the authentication is broken, though, the mail will still be judged on it’s merits – is it coming from an IP address that’s a source of good mail, does the content look legitimate, and all the other things a spam filter looks at.
That authentication is a (potentially big) positive signal, but lack of authentication isn’t really any signal at all is why SPF and DKIM being fragile wasn’t an issue. SPF and DKIM are positive assertions – “IF this mail IS authenticated THEN IT IS from me”.
That changed when DMARC became popular, though.
DMARC allows the owner of a domain to say “We send no mail that is not authenticated, and we promise that none of that authentication will be broken in transit”. DMARC is a negative assertion – “IF this mail IS NOT authenticated THEN it IS NOT from me”. It converts the absence of a positive assertion into a negative assertion.
This isn’t the first attempt to layer a “we authenticate everything” negative assertion on top of fragile email authentication. SPF did it, with the -all flag (which is universally ignored, leaving SPF purely as a positive assertion). DomainKeys did it, with DomainKeys policy records (which you occasionally still see published, but were never really used to reject mail). DKIM did it with ADSP – which didn’t see much use either.
The reason none of them were used much is because even when senders were telling the truth about “we send no email that is not authenticated” they were always lying, to varying degrees, about “none of the authentication will be broken in transit”.
If your domain that is solely used for bulk email. If it’s never for used mail sent by human beings, not even customer support employees. If it’s a newly created domain with no legacy usage that only sends email from a very tightly controlled infrastructure. If you only send email that’s been created via a well implemented message composition pipeline that ensures the content of the is not just RFC compliant but also “well formed”, with short lines, simple widely implemented encoding, vanilla mime structure and so on. And it’s sent out via conservatively configured smarthosts that deliver directly to the end recipients MX. And if you know that the demographics of your recipients are such that the minority that are forwarding that mail elsewhere (e.g. from their Yahoo account to their Google account or via an alumni mail alias) is a small enough group that you don’t care about them…
If all of those things are true, then your domain is going to be able to deploy DMARC pretty easily and safely. If not, though, how can you tell?
That’s the place where DMARC improves over it’s predecessors. It allows you not only to publish a DMARC policy record in test mode, so it’s not actually used to filter your mail (well, mostly, but that’s a longer story) but also to ask recipients to notify you of mail that seems to be from you but which isn’t authenticated.
You can publish a “p=none” DMARC record with notification addresses in it and wait and see what happens. You’ll get notification of mail that has your domain in the From: field but which isn’t authenticated.
As a first round of action that lets you see where you’re sending email from that you didn’t know about. Sysadmin notification email. That marketing splinter group in Sasketchwan. The outsourced survey company.
Once you’ve cleaned all that up, and made sure everyone is authenticating their mail then you can look at what’s left. The next step is likely to be mistakes you’re making in authentication or message composition that’s causing some of your mail – typically depending on content, and source and recipient domain – to become unauthenticated. Clean that up, make sure all your message composition is squeaky clean, make sure employees aren’t sending mail using that domain in ways you don’t authorize (interacting with mailing list, for example).
By that point you’ll have reduced the torrent of reports you’re getting to two types. One is mail that you send that has it’s authentication broken in transit through some process you have no control over. The other type is mail that has your domain in the “From” field but which you didn’t send. Some of that may be legitimate use of your domain by your employees, such as forward-to-a-friend services, signing up for document delivery via email, third-party notification services. By deploying DMARC you are declaring all that sort of usage to be illegitimate, and you’ll need to get all your employees to stop doing it (or, at least, know that it’s going to stop working). The rest of it is likely a mix of spam and phishing mail. The spam, that’s just using your domain in random from addresses, you probably don’t care about. The phishing you do.
You’ve finally cleaned up your mail infrastructure and policies enough to gather the data you need. How much of my legitimate email will have it’s authentication broken (and hence be silently thrown away by DMARC)? And how much hostile phishing mail is targeting my users (and using the exact domain you are)?
Then you have the information you need to make an informed decision as to how badly deploying DMARC will break your legitimate use of email (after you’ve done everything you can to minimize that) and some idea of whether it will provide you any benefit, at least in the shorter term.
That testing phase, where senders can use other peoples mail infrastructure to investigate their sending practices, gradually fix any problems and finally gather some metrics is what made gave the developers of the DMARC spec confidence that it wouldn’t break things, and made it much more deployable than previous approaches to negative assertion.
On Monday, how all that optimistic reasoning went to hell, what it broke and how we’re trying to fix it.

Tools!

I just added a DMARC validation tool over on tools.wordtothewise.com.

You can give it a domain – such as ebay.com – and it will fetch the DMARC record, then explain and validate it. Or you can paste the DMARC record you’re planning to publish into it, to validate it before you go live.
If you’ve not seen our tools page before, take a look. As well as DMARC we have a DKIM validator, SPF expander and optimizer, general DNS lookup tools, a bunch of RFCs covering all sorts of protocols, and base64 and quoted-printable decoders.
There’s also a widget that lets you add those little unicode pictures to your subject lines, whether you need a snowman ⛄, a forest ????, or a pig getting closer ???.
The results pages all have easily copyable URLs so they’re pretty good for sharing with co-workers or customers if you need that sort of thing.
(And if you need a cidr calculator, whois, or easy access to abuse.net & Microsoft SNDS check out Al’s xnnd.com.)

December 2016: The Month in Email

Happy New Year! We’re looking forward to some interesting new projects this year, both for our clients and for Word to the Wise. Stay tuned!
December was a slow month for blogging, with everything going on. But we’re back on the horse now and ready to blog for 2017.

List and subscription management continue to be hot topics, especially in the wake of the listbombing attacks earlier this year. Earlier this month, I presented a webinar on listbombing for the EEC and DMA to review the attacks and discuss best practices for companies to manage subscriptions. For Ask Laura, I wrote about the unsubscribe process and how senders can best manage those requests to keep their lists current and compliant.
With all the holiday mail flying around, Steve wrote up a good post about the challenges of DNS hosting and issues customers may have reaching your site. He also wrote about canonicalization, a process for comparing things to see if they are the same, which is useful for understanding how messages change during the delivery process. It’s important to understand how this works with DKIM, as that process specifically looks at changes to messages in delivery to validate them.
I wrote a post about how delivery at Gmail is a bit different from other mail providers, which can lead to intermittent delivery problems, and got some useful information in the comments about some upcoming process changes. And as always, unwanted email is SPAM. It doesn’t matter if you call it outreach or prospecting, or “here’s something you might find interesting!” Still SPAM.

One of these things is just like the other

Canonicalization is about comparing things to see if they’re the same. Sometimes you want to do a “fuzzy” comparison, to see if two things are interchangeable for your purposes, even if they’re not exactly identical.
As a concrete example, these two email addresses:

Gmail showing authentication results to endusers

A bit of older news, but worth a blog post. Early in August, Gmail announced changes to the inbox on both the web interface and the android client. They will be pushing authentication results into the interface, so end users can see which emails are authenticated.

These are not deliverability changes, the presence or absence of authentication will not affect inbox delivery. And the gmail Gmail support pages clarify that lack of authentication is not a sign that mail is spam.
This isn’t a huge change for most ESPs and most senders. In fact, Gmail has reported more than 95% of their mail is authenticated with either SPF or DKIM. Now, Gmail does a “best guess” SPF – if it looks like an IP should be authorized to send mail for a domain (like the sending IP is the same as the MX) then it’s considered authenticated.
It’s good to see authentication information being passed to the end user.

Don't just follow the HOWTO

There are so many moving parts to ensure good email deliverability. Email marketers need to know marketing, they need to know email and they need to know design. The technical bits of email can be a challenge to learn, and many folks who write tutorials and How-Tos write them for a different audience than marketers.
One of the things I’m trying to do is demystify the technical end of email for marketers. Today I talked about authentication in the Only Influencers newsletter. Check it out!
Understanding the technical: Authentication
Authentication in general

A DKIM primer resurrected

I was looking for some references today back in old blog posts. This means I discover some old links are dead, blog posts are gone or moved, and information is lost.
In this case it’s a post by J.D. Falk on deliverability.com. The link is dead (it looks like the whole website is dead), but I found a copy of his post and am reproducing it here. I don’t have permission, because I can’t get permission from him, but the content is extremely useful and I don’t want it lost.

Ask Laura: Can you help me understand no auth / no entry?

Dear Laura,
I’m a little confused by the term “no auth / no entry”. Gmail and other major receivers seem to be moving towards requiring authentication before they’ll even consider delivery.
Does this just mean SPF and DKIM, or does this mean the much more stringent DMARC, as well?
Thanks,
No Shirt, No Shoes, No What Now?

Ask Laura: Do I have to publish DMARC?

Should you publish DMARC?

I’ve been hearing a lot lately about DMARC. Being at M³AAWG has increased that. Last night we were at dinner and heard from the next table “And they’re not even publishing DMARC!!!!”
I know DMARC is the future. I know folks are going to have to start publishing DMARC records. I also know that the protocol is the future. I am also not sure that most companies are ready for DMARC.
So lets take a step back and talk about DMARC, what it is and why I’m still a little hesitant to jump on the PUBLISH DMARC NOW!! bandwagon.

Ask Laura: Confused about Authentication

Things you need to read: 2/5/16

Ask the Expert: How Can Email Marketers Stay Out of Gmail Jail and in the Inbox? The expert in question is an old friend of mine, Andrew Barrett. I met Andrew online in the late 90s, and we worked together (briefly) at MAPS. He was out of email for a while, but I’m pleased he came back to share his talents with us. The information in the article is valuable for anyone who struggles with getting to the Gmail inbox.
Unclutter Your Inbox, Archive & Keep Your Messages. Shiv Shankar talks about some new features at Yahoo Mail. With a simple click, you can archive email so it’s available to search, but not cluttering up your inbox. One of the things that jumped out at me from that article is that Yahoo is providing 1 TB of storage. That’s more than Google!
The EEC is doing a survey on the impact of CASL and want to hear from marketers. Go check out their blog post and take their survey.
Sparkpost has a guest blog from Alex Garcia-Tobar, co-founder of Valimail about common DKIM failures. I’ve met Alex a few times and I’ve always found him a pleasure to talk to. Alex is somewhat new in the email space, but he really gets some of the challenges in the authentication space. A lot of the issues he mentions in that blog post like lack of key rotation and shared keys are some of the technical debt I was talking about in my predictions for 2016 post.
What links have you read this week that are worth sharing?

Things you need to read

The email solicitation that made me vow to never work with this company again. When sending unsolicited email, you never know how the recipient is going to respond. Writing a public blog post calling you out can happen.
The 2016 Sparkies. Sparkpost is looking for nominations for their email marketing awards. Win a trip to Insight 2016!
5 CAN SPAM myths. Send Grid’s General Counsel speaks about CAN SPAM myths. Personally, asking for an email to unsubscribe is annoying. I never know if the unsubscribe request worked or not. Give me a link any day.
The most misunderstood statistic in email marketing. A good discussion of why raw complaint rates isn’t the metric the ISPs use, and how it can mislead folks about their email program.
Office 365 is expanding it’s DKIM signing. Terry Zink discusses the upcoming changes to how Office365 handles DKIM signatures. This is exactly the kind of changes I was talking about in my 2016 predictions post – background changes that are going to affect how we authenticate email. He even specifically calls out whether or not a particular signature is DMARC aligned or not.

What to expect in 2016

I don’t always do predictions posts, even though they’re popular. Most years I skip them because I don’t see major changes in the email space. And, I’m not the type to just write a prediction post just to post a prediction.
This year, though, I do see changes for everyone in the email space. Most of them center on finally having to deal with the technical debt that’s been accumulating over the past few years. I see ISPs and ESPs spending a lot of development effort to cope with the ongoing evolution authentication requirements.
When people started seriously looking at how to authenticate email, the first goal was getting organizations to implement the protocols. This was a practical concession; in order for a new protocol to be used it needs to be widely implemented. Phase one of authenticating email was simply about publishing protocols and getting organizations to use them.
During phase one, the organization that authenticated a mail hasn’t been important. In fact, the SPF spec almost guarantees that the ESP domain is the authenticated domain. In DKIM, the spec says any domain could sign as long as they could publish a public key in that domain’s domainkeys record.
ESPs took full advantage of this and lowered their own development overhead by taking most of the authentication responsibility on themselves. Their domains were in the 5321.from and they published the SPF records. Domains they control were in the d= and they generated and published the DKIM keys. Mail was authenticated without ESP customers having to do much.
We’ve hit the end of phase one. Most of the major players in the email space are authenticating outbound email. Many of the major players are checking authentication on the inbound. Phase one was a success.
We’re now entering phase two, and that changes thing. In phase two, SPF and DKIM are used as the foundation for user visible authentication. Neither SPF nor DKIM were designed to be user visible protocols. To understand what they’re authenticating you have to understand SMTP and email. Even now there are days when I begin talking about one of them and have to take a step back and think hard about what is being authenticated. And I use these things every day!
DMARC is the first of these end user visible protocols built on SPF and DKIM. It uses the established and widespread authentication to validate the user visible from address. This authentication requires that the d= value or the 5321.from address belong belong to the same domain in the visible from address. While you can pick whether the alignment between the visible from and the authentication is “strict” or “relaxed” you have no choice about the alignment.
Prior to DMARC no one really paid much attention to the domain doing the authentication. Authentication was a yes or a no question. If the answer was yes, then receivers could use the authenticated domain to build a reputation. But they weren’t really checking much in the way of who was doing the authentication.
In the push to deploy authentication, ESPs assumed the responsibility for authentication deployed ESPs took the responsibility and did most of the work. For many or most customers, authentication was as simple as clicking a checkbox during deployment. Some ESPs do currently let customers authenticate the mail themselves, but there’s enough overhead in getting that deployed that they often charged extra to cover the costs.
DMARC is rapidly becoming an expectation or even a full on requirement for inbox delivery. In order to authenticate with DMARC, the authenticating domain must be in the same domain space as the visible from. If senders want to use their own domain in the visible from, DNS records have to be present in that domain space. Whether it’s a SPF TXT record or a domainkeys record the email sender customer needs to publish the correct information in DNS. Even now, if you try to authenticate with DKIM through google apps, they require you to publish DNS records.
ESPs aren’t in a situation where they can effectively manage authentication alignment for all their customers. Hosting companies are in even worse shape when it comes to letting customers authenticate email. Developers are facing the fact they need to go back and rework their authentication code. Businesses are facing the fact they need to change their processes so customers can authenticate with DMARC.
It’s not just the infrastructure providers that are facing challenges with authentication. Senders are going to discover they can no longer hand authentication off to their ESPs and not worry about it. They’re going to have to get DNS records published by their own staff.
Getting DNS updates through some big companies is sometimes more difficult than it should be. I had one client a few years ago where getting rDNS changed to something non-generic took over a month. From an IT standpoint, changing DNS should require approvals and proper channels. Marketers may find this new process challenging.
And, if organizations want to publish reject policies for their domains, then they will have to publish records for every outside provider they use. Some of those providers can’t support DMARC alignment right now.
In 2016 a lot of companies will discover their current infrastructure can’t cope with modern authentication requirements. A lot of effort, both in terms of product development and software development, will need to be spent to meet current needs. This means a lot of user visible features will be displaced while the technical debt is paid.
These changes will improve the security and safety of email for everyone. It won’t be very user visible, which will give the impression this was a slow year for email development. Don’t let that fool you, this will be a pivotal year in email.

A brief history of TXT Records

txt
When the Domain Name System was designed thirty years ago the concept behind it was pretty simple. It’s mostly just a distributed database that lets you map hostname / query-type pairs to values.
If you want to know the IP address of cnn.com, you look up {cnn.com, A} and get back a couple of IP addresses. If you want to know where to send mail for aol.com users, you look up {aol.com, MX} and you get a set of four hostname / preference pairs back. If you want to know the hostname for the IP address 206.190.36.45 you look up {45.36.190.206.in-addr.arpa, PTR} and get a hostname back.
There’s a well-defined meaning to each of those query types – A is for IP addresses, MX is for mailservers, PTR is for hostnames – and that was always the intent for how DNS should work.
When DNS was first standardized, though, there was one query type that didn’t really have any semantic meaning:

Four things to check before your next mailing

Like many bits of technology, email is often set-and-forget. Everything is checked and rechecked during setup, and then no one goes back and looks at it again. But mail programs are not static, and people make changes. These changes don’t really break things, but over time they can create their own set of problems.
Setting aside some time every quarter or even every year to check and make sure all the bits of mail are configured correctly is a good idea.

Office365/EOP IPv6 changes starting today

Terry Zink at Microsoft posted earlier this week that Office365/Exchange Online Protection will have a significant change this week. Office365 uses Exchange Online Protection (EOP) for spam filtering and email protection. One of the requirements to send to EOP over IPv6 is to have the email authenticated with either SPF or DKIM. If the mail sent to Office365/EOP over IPv6 is not authenticated with SPF or DKIM, EOP would reject the message with a 554 hard bounce message. Most mail servers accept the 554 status code and would not retry the message. After multiple 5xx hard bounces to an email address, many mail servers would unsubscribe the user from future email campaigns. The update starting today April 24, will change the error status code for unauthenticated mail to EOP from a 554 hard bounce to a 450 soft bounce and a RFC-compliant and properly configured mail server would then retry the message.
Prior to April 24, 2015, EOP responds to unauthenticated mail with a status code of: “554 5.7.26 Service Unavailable, message sent over IPv6 must pass either SPF or DKIM validation”.

Authentication and Repudiation

Email Authentication lets you demonstrate that you sent a particular email.
Email Repudiation is a claim that you didn’t send a particular email.

SPF is only for email authentication¹
DKIM is only for email authentication
DMARC is only for email repudiation

¹ SPF was originally intended to provide repudiation, but it didn’t work reliably enough to be useful. Nobody uses it for that now.

Salesforce and DKIM

Last month I wrote about how Salesforce was implementing the ability to sign emails sent from Salesforce CRM with DKIM. The Spring 15 update is now live as is the ability to use an existing DKIM key or allow Salesforce to create a new one for you.
Setting up DKIM within Salesforce is straightforward. A Salesforce Administrator would go to Setup->Email Administration->DKIM Keys.
sf-dkim-step0
You can either allow Salesforce to create you a new DKIM key or you can import an existing key. For this example, I am going to create a new DKIM key for the domain wttwexample.com with a DKIM selector of 2015Q1.
Step 1 – Creating a new key within Salesforce, you enter the Selector for the key (2015Q1), the domain for the key (wttwexample.com), and the strictness of the key allowing either the exact domain only, subdomains of the domain only, or Exact domain and subdomains.
sf-dkim-step1
Step 2 – The next screen will display both the Public Key and the Private Key.
sf-dkim-step2
Step 3 – With the key being created, we need to store the Public Key within our DNS for the domain by created a TXT record with a hostname of 2015Q1._domainkey.
sf-dkim-step3
Using a DKIM check tool like ours http://tools.wordtothewise.com/authentication, we can see if the DKIM key is in the DNS and if the key is valid.
Step 4 – Once we have confirmed the key is valid and in DNS, we can go back to Salesforce and activate the key.
Step 5 – Emails sent from the Salesforce CRM Sales Cloud will now be signed with the new DKIM key and the emails will have a new header added called DKIM-Signature.
Signing with DKIM allows us to tell the recipient ISP that “yes, I sent this email” and this allows the ISP to track our reputation by the domain instead of just by the IP address. This means that some fraction of our good reputation will be associated with these emails that are sent from Salesforce CRM. If we have not established any reputation yet, signing with DKIM is a good key to enable services like feedback loops as it includes the proof that you’re sending the FBL reports to someone responsible, not a random third party.
If you have plans to consider utilizing DMARC, you need to have ALL of your sources of mail authenticated. DMARC looks for a passing SPF or DKIM validation during its evaluation of the message. Utilizing both SPF and DKIM for DMARC validation is recommended.
Having emails signed with DKIM, having a valid SPF, setting up sensible reverse DNS, having good hostnames all show that you are doing your part to send legitimate and valid mail. Signing with DKIM does not give you a free pass to send spammy emails, it just tells the receiving party who is taking responsibility for sending the message.

February 2015 – The month in email

This was a short and busy month at WttW!

We attended another great M3AAWG conference, and had our usual share of interesting discussions, networking, and cocktails. I recapped our adventures here, and shared a photo of the people who keep your email safe while wearing kilts as well. We also commended Jayne Hitchcock on winning the Mary Litynski award for her work fighting abuse and cyberstalking.

Salesforce SPF and now DKIM support

Salesforce has published a SPF record for sending emails from Salesforce for years and with the Spring ’15 release, they will provide the option to sign with DKIM.
The SPF record is straight forward, include:_spf.salesforce.com which includes _spf.google.com, _spfblock.salesforce.com, several IP address blocks, mx, and ends with a SoftFail ~all.
Salesforce Knowledge Article Number: 000006347 goes in-depth with information regarding their SPF Record.

Email Authentication in a nutshell

There are 3 types of authentication currently in use for email.

Office365 checking DMARC on the inbound

According to a recent blog post, Office365 is starting to evaluate incoming messages for DMARC. I talked a little bit about DMARC in April when Yahoo started publishing a p=reject message.

Superstition, correlation and reality

I’m not a huge baseball fan, probably a side effect of growing up in a city with no MLB team. But I do enjoy the social aspects of rooting for local teams when they’re winning big games. Last night I was following the World Series score online and switched over to watch the last inning. I posted something about the game on FB just about 30 seconds before the Giant’s outfield bobbled what should have been a single (at best). I immediately posted an apology, “Sorry about that, shouldn’t have said anything!”
Do I really think that my post somehow cursed two outfielders and caused them to bobble a simple play? No, of course not. But it is a very human response. In fact, there’s an entire advertising campaign centered around the the weird things people do while watching sports.
There is a lot of superstition in email delivery, too. I think that’s a combination of filtering necessarily being a black box, human’s built in tendency to see patterns in random data, and a need to be able to control and affect outcomes.
Figuring out cause and effect in the real world is not trivial. In my research days we set out to control as many confounding factors as possible so we could demonstrate the cause and the effect. That’s really hard to do when you’re not at a lab bench. In the real world, we can’t always control things directly. Instead, we have to rely on statistics and representative (or non-representative) samples.
Delivery isn’t even close to a science and one of the major issues is that filters are always changing. I’ve certainly seen occasions where multiple clients, or colleagues, were having problems delivering to one ISP or another. One of my clients made a change and saw their delivery improve. They patted themselves on the back for figuring out the problem. At the same time, though, other folks saw their delivery improve without making any changes. I can’t always convince people that whatever they did had nothing to do with their delivery improving.
The flip side is I can’t always convince people to stop doing somethings that they don’t need to do. I see a lot of mail with both DomainKeys and DKIM signatures. In most cases both signatures have the same selectors. DomainKeys is deprecated. No one, and I mean no one with a modern email system, is checking DomainKeys without checking DKIM. Senders can safely stop signing with DomainKeys and have nothing happen. It doesn’t matter, lots of ESPs and sender sign with both. They’re not going to change it. I’ve had multiple groups tell me they’re afraid to stop signing because it might hurt their delivery.
The reality is I didn’t make the Giant’s outfield bobble the ball because I posted to FB that I was watching the bottom of the 9th inning. The reality is that DomainKeys is deprecated and there’s no benefit to signing with both DomainKeys and DKIM. The reality is we are humans and we are inherently superstitious. Most of the times our superstitions are harmless. But sometimes they cause us more work than we need to do and provide no tangible benefits.

Spam, Phish or Malware?

Some mornings I check mail from my phone. This showed up this morning.

My first thought was “oh, no, Pizza Hut is spamming, wonder who sold them my address.”
Then I remembered that iOS is horrible and won’t show you anything other than the Friendly From and maybe it was some weird phishing scheme.
When I got to my real mail client I checked headers, and sure enough, it wasn’t really from Pizza Hut. I’m guessing actually malware, but I don’t have a forensics machine to click the link and I’m not doing it on anything I can’t wipe (and have isolated from the rest of my network).
The frustrating thing for me is that this is an authenticated email. It not from Pizza Hut, the address belongs to some company in France. Apparently, that company has had their systems cracked and malware sent through them. Fully authenticated malware, pretending to be Pizza Hut, and passing authentication on various devices.
Pizza Hut isn’t currently publishing a DMARC record, but in this case, a DMARC record for Pizza Hut wouldn’t matter. None of the email addresses in the headers point to Pizza Hut.
I spent last week listening to a lot of people discussing DMARC and authentication and protecting people from scams and headers. But those all the protocols in the world won’t protect against this kind of thing. Phishing and malware can’t be fixed by technology alone. Even if every domain on the planet published a p=reject policy, mail like this would still get through.

Alice and Bob Sign Messages

Alice and Bob can send messages privately via a nosy postman, but how does Bob know that a message he receives is really from Alice, rather than from the postman pretending to be Alice?
If they’re using symmetric-key encryption, and Bob is sure that he was talking to Alice when they exchanged keys, then he already knows that the mail is from Alice – as only he and Alice have the keys that are used to encrypt and decrypt messages, so if Bob can decrypt the message, he knows that either he or Alice encrypted it. But that’s not always possible, especially if Alice and Bob haven’t met.
Alice’s shopping list is longer for signing messages than for encrypting them (and the cryptography to real world metaphors more strained). She buys some identical keys, and matching padlocks, some glue and a camera. The camera isn’t a great camera – funhouse mirror lens, bad instagram filters, 1970s era polaroid film – so if you take a photo of a message you can’t read the message from the photo. Bob also buys an identical camera.
sign1
Alice takes a photo of the message.
sign2
Then Alice glues the photo to one of her padlocks.
sign3
Alice sends the message, and the padlock-glued-to-photo to Bob.
sign4
Bob sees that the message claims to come from Alice, so he asks Alice for her key.
sign5
(If you’re paying attention, you’ll see a problem with this step…)
Bob uses Alice’s key to open the padlock. It opens (and, to keep things simple, breaks).
Bob then takes a photo of the message with his camera, and compares it with the one glued to the padlock. It’s identical.
sign6
Because Alice’s key opens the padlock, Bob knows that the padlock came from Alice. Because the photo is attached to the padlock, he knows that the attached photo came from Alice. And because the photo Bob took of the message is identical to the attached photo, Bob knows that the message came from Alice.
This is how real world public-key authentication is often done.

May 2014: The month in email

It’s been a busy and exciting month for us here.
Laura finished a multi-year project with M3AAWG, the Messaging, Malware and Mobile Anti-Abuse Working Group (look for the results to be published later this year) and continued working with clients on interesting delivery challenges and program opportunities. Steve focused on development on the next version release of Abacus, our flagship abuse desk tool, which will also be available later this year.
And as always, we had things to say about email.
The World of Spam and Email Best Practices
We started the month with a bit of a meta-discussion on senders’ fears of being labeled spammers, and reiterated what we always say: sending mail that some people don’t want doesn’t make you evil, but it is an opportunity to revisit your email programs and see if there are opportunities to better align your goals with the needs of people on your email lists. We outlined how we’ve seen people come around to this position after hitting spamtraps. That said, sometimes it is just evil. And it’s still much the same evil it’s been for over a decade.
We also wrote a post about reputation, which is something we get asked about quite frequently. We have more resources on the topic over at the WiseWords section of our site.
Gmail, Gmail, Gmail
Our friends over at Litmus estimate Gmail market share at 12%, which seems pretty consistent with the percentage of blog posts we devote to the topic, yes? We had a discussion of Campaign Monitor’s great Gmail interview, and offered some thoughts on why we continue to encourage clients to focus on engagement and relevance in developing their email programs. We also wrote a post about how Gmail uses filters, which is important for senders to understand as they create campaigns.
SMTP and TLS
Steve wrote extensively this month about the technical aspects of delivery and message security. This “cheat sheet” on SMTP rejections is extremely useful for troubleshooting – bookmark it for the next time you’re scratching your head trying to figure out what went wrong.
He also wrote a detailed explanation of how TLS encryption works with SMTP to protect email in transit, and followed that with additional information on message security throughout the life of the message. This is a great set of posts to explore if you’re thinking about security and want to understand potential vulnerabilities.
DKIM
Steve also wrote a series of posts about working with DKIM (DomainKeys Identified Mail), the specification for signing messages to identify and claim responsibility for messages. He started with a detailed explanation of DKIM Replay Attacks, which happens when valid email is forwarded or otherwise compromised by spammers, phishers or attackers. Though the DKIM signature persists (by design) through a forward, the DKIM specification restricts an attacker’s ability to modify the message itself. Steve’s post describes how senders can optimize their systems to further restrict these attacks. Another way that attackers attempt to get around DKIM restrictions is by injecting additional headers into the message, which can hijack a legitimately signed message. If you’re concerned about these sort of attacks (and we believe you should be), it’s worth learning more about DKIM Key Rotation to help manage this. (Also of note: we have some free DKIM management tools available in the WiseTools section of our site.)
As always, we’re eager to hear from you if there are topics you’d like us to cover in June.

DKIM Key Rotation

Several people have asked me about how to rotate DKIM keys in the past few days (as if you’re modifying anything to mitigate replay attacks, you need to invalidate the signatures of all the mail you sent before you made those changes).

DKIM and injected headers

If you look at the DKIM-Signature header in any piece of email signed with DKIM you’ll see that one of the fields it contains, the h= field, lists some email header names, for example:

DKIM replay attacks

Replay attacks on DKIM signed messages
When you receive an email validly signed with DKIM by example.com that might not mean that example.com sent the email to you, or that they even sent this email at all.
What it does tell you is that at some point in the past, example.com signed an email with exactly the same headers and body and sent it to someone. That’s often close enough to the same thing. But if that original recipient were to resend the email to you completely unchanged then the DKIM signature would still validate when you received it. That’s not a bug; it’s one of the design features of DKIM that it typically survives mail forwarding.
That original recipient could also forward the exact same email to a million of their closest friends, and the DKIM signature would validate at each of those million recipients ISPs. This is one form of a replay attack, and it isn’t something DKIM prevents.
DKIM doesn’t prevent replay, but does mitigate it
Completely eliminating replay attacks over SMTP is difficult – it’s inherently a store-and-forward protocol, so there’s no way to have the sender and recipient do any sort of handshake to ensure that a particular signature is only used once. It’s not unheard of for email to be delayed for days, and delays of hours aren’t unusual, so allowing a signature to be valid for only a few seconds after it’s sent won’t work. And the design requirement that DKIM signature survive forwarding means that it has to survive the final recipient’s email address not being the same as the email address the mail was originally sent to so you can’t include the envelope recipient in the signature.
So what does DKIM do to mitigate replay attacks? The answer to that is surprising – almost everything DKIM does is there to mitigate them. The DKIM signature depends on the body of the message, the subject line and the content of any other headers the sender chooses to include; changing any of that will invalidate the signature. That means that while anyone can grab a copy of an email sent by, for example, paypal and forward it on to someone else, if they modify the content at all it will no longer have paypal’s signature. So an attacker can’t just grab someone else’s signed email and replace it with modified content – and if they can’t do that, where’s the benefit to a spammer or phisher to replay a message?
But all that work is for naught if you allow the attacker to choose the content before you sign the message. There are several ways an attacker can do that, but one example that’s particularly relevant today is ESP trial accounts.
I’m stealing your reputation
If you allow anonymous signups for trial accounts that let a potential customer try out your system you’ll want to put very tight limits on how it can be used, so as to avoid spammers signing up and spamming through your servers. Maybe you’ll limit the number of email addresses the trial user can upload, or the number of emails they can send. At the most extreme you might even limit the trial account to sending mail solely to the trial users own (confirmed) email address.
But if an attacker can send even one piece of email they create through your trial account to themselves, and you sign that email, they can take it and send it to a million recipients – and it’ll still have your DKIM signature on it so it’ll use your reputation to avoid filters and end up in the recipients inbox. And then the recipients will report it as spam, and all that spam will be counted against the reputation associated with your DKIM identifier. If you share a DKIM identifier (“d=”) across all your customers that could cause all your customer mail to start being rejected or sent to the spam folder. (Even if you don’t it could still affect your delivery negatively, as spam filtering systems – both automated and human – sometimes aren’t entirely rational or predictable).
Spam that’s sent like this will be a little “off”, compared to legitimate email – the To: field won’t have the email address of the recipient, for instance, and there’ll be no personalization in the Subject or body of the message. It’s no worse than most spam, and it’s more than balanced out by being able to hijack someone else’s reputation.
So if you provide any way for unvetted non-customers to send email through your systems you should consider adding some DKIM limitations to the constraints you already have on that mail path. Not signing with DKIM at all avoids the problem altogether, but also means you can’t demonstrate your DKIM prowess to legitimate potential customers. You might want to sign with a DKIM d= domain that’s different to your production signatures, perhaps even a completely different top level domain to avoid any risk of confusion (but don’t try and hide that it’s your domain – that’s what spammers do).
Other operational mistakes
There are some grubby corners of the email and DKIM specs that sometimes interact to cause other holes that this sort of reputation hijacker can exploit. I’ll talk about header duplication tomorrow.

A brief DMARC primer

DMARC stands for Domain-based Message Authentication, Reporting and Conformance. What DMARC does is allow domain owners to publish policy statements in DNS telling receiver domains what to do with messages that do not authenticate. In addition, DMARC introduces the concept of “domain alignment.” What this means is that the authentication has to be from the same domain (or a sub-domain) as the address in the header-from: line. The idea behind DMARC is that organizational owners can use SPF and DKIM authentication to authenticate their actual domain in the header-from line. This moves authentication from a important but behind the scenes technology out to an end user visible technology.

Gmail FBL update

Last week Gmail started contacting ESPs that signed up for their new FBL with more information on how to set up mailings to receive FBL emails.
One of the struggles some ESPs are having is the requirement for DKIM signing. Many of the bigger ESPs have clients that sign with their own domains. Gmail is telling these ESPs to insert a second DKIM signature to join the FBL.
There are a couple reasons this is not as simple or as doable as Gmail seems to think, and the challenges are technical as well as organizational.
The technical challenges are pretty simple. As of now, not all the bulk MTAs support multiple signatures. I’ve heard that multiple signatures are being tested by these MTA vendors, but they’re not in wide use. This makes it challenging for these ESPs to just turn on multiple signatures. For ESPs that are using open source software, there’s often a lot of customization in their signing infrastructure. Even if they have the capability to dual sign, if they’re not currently using that there is testing needed before turning it on.
None of the technical challenges are show stoppers, but they are certainly show delayers.
The organizational challenges are much more difficult to deal with. These are cases where the ESP customer doesn’t want the ESP to sign. The obvious situation is with large banks. They want everything in their infrastructure and headers pointing at the bank, not at their ESP. They don’t want to have that second signature in their email for multiple reasons. I can’t actually see an ESP effectively convincing the various stakeholders, including the marketing, security and legal staff, that allowing the ESP to inset a second signature is good practice. I’m not even sure it is good practice in those cases, except to get stats from Gmail.
Hopefully, Gmail will take feedback from the ESPs and change their FBL parameters to allow ESPs to get information about their customers who sign with their own domain.

DMARC: Please Be Careful!

(Cross posted from Spam Resource.)
Every couple of days, somebody new pops up on the DMARC-Discuss mailing list to ask some question or share an observation. It’s great to see people interested and joining the conversation. Clearly, DMARC interest and adoption are growing. What’s really frustrating, though, is that for about a quarter of the new subscribers, their first mailing list message goes to the spam folder in my Gmail account. It has become sort of an intelligence test I apply to new subscribers — I’ve stopped digging those messages out of the spam folder. I’m figuring that if they can’t figure out how to implement a DMARC record, or they don’t understand that it’s not really compatible with mailing lists nor is it meant for hobbyist domains, then I think perhaps they’ve got some things they’ve got to figure out before they’re ready to join the discussion.
To that end, let me take a moment to jot down some recommendations for folks who are considering implementing DMARC.

DKIM and DomainKeys, Spam and Ham

I’ve been preaching “DKIM is great! DomainKeys is obsolete, get rid of it!” for several years now. I thought I’d take a look at my mailbox and see who was using authentication.
I’ve divided this into “Ham” and “Spam”. Spam is, well, all the spam I’ve received over the past couple of years. Ham is the non-spam mail in my inbox, whether personal, business, bulk or transactional. I’ve excluded most of the discussion mailing lists I’m on (not least because many of them consist of people in the email industry or are email standards development mailing lists, so have email authentication levels that are way outside the norm).

Troubleshooting tools

There have been a number of comments on my post about Hotmail moving to SPF authentication having to do with troubleshooting authentication failures. I have been helping clients troubleshoot these issues, and am able to take on new clients to solve authentication problems. Contact me for more information.
Of course, many of these issues can be solved with access to the right tools. Steve’s been working on a number of tools that may help the troubleshooting process and we’ve recently launched them on Emailstuff.org. The website itself contains a number of DNS and data related tools we use for investigations and thought we’d share with the public at large.
One of the really useful tools is the SPF record expander. Plug in any domain, like google.com, and see what IP addresses they authorize to send mail.

Hotmail moves to SPF authentication

Hotmail has recently stopped using Sender ID for email authentication and switched to authenticating with SPF. The protocol differences between SenderID and SPF were subtle and most senders who were getting a pass at Hotmail were already publishing SPF records.
From an email in my inbox from September:

DKIM and Gmail

After they were a a little embarrassed by their own DKIM keys being poorly managed a few months ago, Google seem to have been going through their inbound DKIM handling and tightening up on their validation so that badly signed mail that really shouldn’t be treated as DKIM signed, won’t be treated as signed by Gmail.
This is a good thing, especially as things like DMARC start to be layered on top of DKIM, but it does mean that you really need to check your signing configuration and make sure you’re not doing anything silly.

Gmail sending out warnings for 512 bit DKIM keys

As an update to yesterday’s post, Gmail is contacting postmasters at domains signing with 512 bit keys to warn them of the upcoming changes. This message also clarifies “DKIM keys failing.” Messages signed with 512 bit keys or less will be treated as unsigned by Gmail in the next week or so.

Is Google failing DKIM keys shorter than 512 bits?

Today’s Wednesday question comes from Andrew B. and got pushed to Thursday so I could check a few more facts.

How long is your DKIM key?

While we were at M³AAWG, Wired published an article talking about how simple it was to crack DKIM keys. I didn’t post about it at the time because it didn’t really seem like news. DKIM keys smaller than 1024 are vulnerable and not secure and the DKIM spec does not recommend using keys smaller than 1024. When I asked the DKIM-people-who-would-know they did tell me that the news was that the keys had been cracked and used in the wild to spoof email.
Fair enough.
If you are signing with DKIM, use a key 1024 or longer. Anything shorter and your risk having the key cracked and your mail fraudulently signed.
This morning M³AAWG published recommendations on keeping DKIM keys secure.

Setting up DNS for sending email

Email – and email filtering – makes a lot of use of DNS, and it’s fairly easy to miss something. Here are a few checklists to help:

DMARC Interoperability

Facebook hosted a DMARC interoperability event earlier this week. In terms of protocol development, interoperability events are a sign that the protocol is ready for more widespread use.

Return Path partners with Symantec

Today Return Path announced a partnership with Symantec to improve their anti-phishing product. Return Path is incorporating the Symantec Trusted Domain List into their authentication and filtering product to help customers protect their brands. Press Release
Phishing scams affect everyone, and having a brand that is used in phishing can reduce consumer trust in that brand. Protecting brands in email has been one of the more difficult challenges facing the email community. With the adoption of DKIM and DMARC by major brands and ISPs it has become easier to track and address phishing.

DKIM deployment challenges

Cloudmark has an interesting blog post pointing out some of the challenges of signing mail with DKIM in a large company with a diverse mail system.

DMARC: an authentication framework

A new email industry group was announced this morning. DMARC is a group of industry participants, including large senders, large receivers and relevant intermediaries working on a framework to reduce the harm from phishing.
DMARC is working on a standard to allow senders to publish sending policies and receivers to act on those policies. Currently, senders who want receivers to not deliver unauthenticated email have to negotiate private agreements with the ISPs to make that happen. This is a way to expand the existing programs. Without a published standard, the overhead in managing individual agreements would quickly become prohibitive.
It is an anti-phishing technique built on top of current authentication processes. This is the “next step” in the process and one that most people involved in the authentication process were anticipating and planning for. I’m glad to see so many big players participating.

Links Sept 29, 2011

Al Iverson has a post up about his experiences with customers who try to acquire email addresses through appending.
J.D. Falk has a post up about the history of DKIM.

DKIM is Done

This was posted to the IETF DKIM Working Group mailing list this morning:

Gmail and the via

I was hoping to have a detailed post up today about the conditions where gmail presents the user with a “via” but time seems to have gotten away from me. But I can give you the conclusions.

Gmail shows authentication data to the recipient

Yesterday Gmail rolled out some changes to their interface. One of the changes is that they are now showing end users authentication results in the user screen.
It’s really the next step in email authentication, showing the results to the end user.
So how does Google do this? Google is checking both SPF and DKIM. If mail is authenticated and the authentication matches the from address then they display the email as:

If we click on “details” for that message, we find more specific information.
In this case the mail went through our outgoing mailserver to gmail.
Mailed-by indicates that the message passed SPF and that the IP address is a valid source of mail from wordtothewise.com.
Signed-by shows the domain in the DKIM d=. In this case, we signed with the subdomain dt.wordtothewise.com. That’s what happens when you sign using the domain in the From address (or a subdomain of it).
For a lot of bulk senders, though, their mail is signed using their ESP’s domain instead. In that case Gmail shows who signed the mail as well as the from address.

And when we click on “details” for that message we see:
This is an email from a sender using Madmimi as an ESP. Madmimi is handling both the SPF authentication and the DKIM authentication.
As an aside, this particular sender has a high enough reputation that Gmail is offering me an unsubscribe option in their interface.
Gmail is distinguishing between first party and third party signatures in authentication. If the mail is authenticated, but the authentication appears to be handled by a separate entity, then Gmail is alerting recipients to that fact.
What does this mean for bulk senders?
For senders that are signing with a domain that matches their From: domain, there is no change. Recipients will not see any mention of your ESP in the headers.
However, if you are using an ESP that is signing your mail with a domain they own, then your recipients will see that information displayed in the email interface. If you don’t want this to be displayed by Gmail, then you will need to move to first party signing. Talk to your ESP about this. If they’re unsure of how to manage it, you can point them to DKIM Core for an Email Service Provider.
Gmail blogpost about the changes
Gmail help page about authentication results

ESPs, Non-portable Reputation and Vendor Lock-in

I’ve seen some mentions recently of ESPs suggesting that if you use your own domain in the From: of mail you send through an ESP then that ESP can’t “do email authentication” properly unless they require you to edit your domains DNS settings. That’s not really so, but there is a kernel of truth in there.
The real situation is, unsurprisingly, a bit more complicated.
What authentication features should you look for in an ESP?

Link roundup June 18, 2010

Hotmail has released a new version of their software with some changes. Return Path discusses the changes in depth, but there are a couple that senders may find helpful.

Who can you trust?

I’ve been recently dealing with a client who is looking at implementing authentication on their domains. He’s done a lot of background research into the schemes and has a relatively firm grasp on the issue. At this point we’re working out what policies he wants to set and how to correctly implement those policies.
His questions were well informed for the most part. A few of them were completely out of left field, so I asked him for some of his references. One of those references was the EEC Email Authentication Whitepaper.
My client was doing the best he could to inform himself and relies on industry groups like the EEC to provide him with accurate information. In this case, their information was incomplete and incorrect.
We all have our perspectives and biases (yes, even me!) but there are objective facts that can be independently verified. For instance, the EEC Authentication whitepaper claimed that Yahoo requires DKIM signing for access to their whitelist program. This is incorrect, a sender does not have to sign with DKIM in order to apply for the Yahoo whitelist program. A bulk sender does have to sign with DKIM for a Y! FBL, but ISPs are given access to an IP based FBL by Yahoo. I am shocked that none of the experts that contributed to the document caught that error.
Independent verification is one reason I publish the Delivery Wiki. It’s a resource for everyone and a way to share my knowledge and thought processes. But other experts can “check my work” as it were and provide corrections if my information is outdated or faulty. All too often, senders end up blaming delivery problems on evil spirits, or using “dear” in the subject line or using too much pink in the design.
Delivery isn’t that esoteric or difficult if you have a clear understanding of the policy and technical decisions at a range of ESPs and ISPs, the history and reasoning behind those decisions, and enough experience to predict the implications when they collide.
Many senders do face delivery challenges and there is considerable demand for delivery experts to provide delivery facts. That niche has been filled by a mix of people, of all levels of experience, expertise and technical knowledge, leading to the difficult task of working out which of those “experts” are experts, and which of those “facts” are facts.

A quick marketers guide to DKIM

J.D. Falk posted a brief but comprehensive guide to the different DKIM flags: what they mean and how they may affect delivery. (The original link seems to be dead so I reproduced the blog post for reference It’s just that good. A DKIM Primer Resurrected

DKIM: what it's not

An ESP twittered this past week about their new DKIM implementation going live. They were quite happy with themselves. Unfortunately, in their blog post, they mentioned 3 things that DKIM would provide for their customers, and got it wrong on all 3 points. Their confusion is something that a lot of people seem to get wrong about DKIM so I thought I would explain what was wrong.

DKIM implementation survey: prelim results

First off, I want to thank everyone who participated in the DKIM implementation survey. This week has been pretty hectic so far, so I haven’t had a chance to actually dig down into the data from the survey, but I thought I’d post some preliminary results.
The ESP survey had 45 respondents. 30% of those sent more than 15 million emails a month.
Of all the respondents: 40% are signing with Domain Keys, 51.1% are signing with DKIM.
Of all respondents: 79.5% are signing with Domain Keys and 78.8% are signing with DKIM to access services (whitelists or FBLs) provided by the ISPs.
50% of those not signing with Domain Keys are not doing so because customers have not requested it. 61% of those not signing with DKIM are not doing it because of technical difficulties with deployment.
The ISP survey had 16 respondents, with 37.5% handling less than 500,000 mailboxes and 18.8% handling more than 15 million mailboxes. 75% of respondents said they are not checking Domain Keys on inbound mail. 56% said they are not currently checking DKIM on inbound mail.
Only 10 ISPs answered the question if they plan to check either Domain Keys or DKIM.

DKIM implementation survey

DKIM has been a hot topic of discussion on some of my mailing lists today. One of the open questions is what is holding up adoption of DKIM. I have my own theories, but thought I’d throw out some questions to see how ESPs and ISPs are currently using domain based reputation.
I have set up two surveys one for ESPs and one for ISPs. Responses are anonymous.
I’ll collect responses for a week and share the results.

New Delivery tools

A couple nifty new delivery tools were published over the weekend.
Mickey published Bounce P.I. where senders can paste in an error message or bounce and it will tell you what filter generated it. If the rejection is unrecognized, it will flag the message internally and it will be researched to see if the filter can be identified.
Steve has a new tool at the DKIMCore site. The key generating tool and the record checking tool have been up for a while. This weekend, though, he published a tool to check the validity the DKIM record published in DNS. Tool output shows if the record is valid, the version and the public key.

Links for 7/8/9

With all the traveling I did last month, I’m still not back to full blogging speed. I have been slowly reading through the backlog of unread posts from my RSS feeds and there was lots of good stuff published.
Three myths about DKIM by John Levine. A very good explanation taking down some of the myths of DKIM. Also on the DKIM front, RFC 5585 DKIM Service Overview was published last month. According to Cisco, DKIM adoption is climbing. More information about DKIM is available at dkim.org and our own dkimcore.org.
The always awesome guys at Mailchimp have embraced twitter as part of their platform. Not only have they set up their own service for link shortening so that links can be tweeted, but have also incorporated twitter stats into their mail dashboard.
Al has an insightful post on delivery, spam filtering vendors and the differences (or lack thereof) between B2C and B2B marketing. As I tell my customers, there is no switch inside the filtering scheme for “I know this person, they’re OK, let the mail in.”
Terry Zink has started a series about blacklists triggered by the recent SORBS announcement. His first post, My take on blacklists, part 2, discusses how some people go about building a blocklist from scratch.
Happy 7-8-9 everyone.

AOL and DKIM

Yesterday, on an ESPC call, Mike Adkins of AOL announced upcoming changes to the AOL reputation system. As part of these changes, AOL will be checking DKIM on the inbound. Best estimates are that this will be deployed in the first half of 2009, possibly in Q1. This is something AOL has been hinting at for most of 2008.
As part of this, AOL has deployed an address where any sender can check the validity of a DKIM signature against the AOL DKIM implementation. To check a signature, send an email to any address at dkimtest.aol.com.
I have done a couple of tests, from a domain not signing with either DK or DKIM, from a domain signing with DK and from a domain signing with both DK and DKIM. In all cases, the mail is rejected by AOL. The specific rejection messages are different, however.
Unsighng domain: host dkimtest-d01.mx.aol.com[205.188.103.106] said: 554-ERROR: No DKIM header found 554 TRANSACTION FAILED (in reply to
end of DATA command)
DK signing domain: “205.188.103.106 failed after I sent the message.
Remote host said: 554-ERROR: No DKIM header found
554 TRANSACTION FAILED”
DK/DKIM signing domain: “We tried to delivery your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 554 554-PASS: DKIM authentication verified
554 TRANSACTION FAILED (state 18).”
As you can see, in all cases mail is rejected from that address. However, when there is a valid DKIM signature, the failure message is “554-PASS.”
As I have been recommending for months now, all senders should be planning to sign with DKIM early in 2009. AOL’s announcement that they will be using DKIM signatures as part of their reputation scoring system is just one more reason to do so.

How not to handle unsubscribes

On the heels of my unsubscribe experience last week where an ESP overreacted and unsubscribed addresses that did not belong to me, I encountered another deeply broken unsubscribe process. This one is the opposite, there is no way to unsubscribe from marketing mail at all. Representatives of PayPal have only been able to suggest that if I do not want their mail, that I block PayPal in my email client.
I had a PayPal account years and years ago. They made some extensive privacy policy changes back in 2003 and when I did not actively agree to the new policies, they closed the account. That account closure seemed to take, I heard nothing from PayPal. In early 2008, I made a purchase at a vendor that only accepted credit cards through PayPal. Normally, I do not do business with vendors who only accept payment through PayPal, but there appeared to be a way to make the payment without establishing a PayPal account, so I went ahead and made the purchase.
The receipt from that purchase came from PayPal, and mentioned that I had an existing PayPal account. I figured that because the address was the same as the 2003 account that the boilerplate did not understand ‘closed accounts’. I brushed off the notice and did not worry about it.
On June 23, I received marketing email from PayPal. The mail offered 10% off my first eBay purchase, if I set up an eBay account using the same address on my PayPal account. Yay. Spam. Oh, well, no big deal, there was an unsub link at the bottom of the email. It is PayPal, they are a legitimate company, they will honor an unsubscribe. It will all be fine.
Or. Not.
Clicking on the unsubscribe link in the email takes me to a webpage that tells me I had to login to my account to unsubscribe. But I do not have an account!
They clearly think I have an account linked to the email address they mailed. I decide to see if I can recover the account and then unsubscribe. I put in the email address they sent the marketing email to, the password I probably would have used had I actually set up this account and hit “submit.” PayPal now asks me to set up 3 questions to use to recover my account in case I forget the login in the future. Uh. What? No. I do not want to set up an account, I want them to stop sending me email. I abandon that webpage.
I then attempt to recover the password to the account. Put in the email address that PayPal is sending email to and hit “forgot password”. PayPal, as expected, sends me an email. Click this magic link to recover your account. PayPal then asks me to input the full number of the credit card associated with the account – the credit card number I do not have. What account? What credit card number? Is this from my 2003 subscription that was closed? Is this from the purchase I made in February? I abandon that webpage.
The recover password email helpfully lists a phone number I can call for assistance so I call. In order to be able to talk to someone I have to enter my phone number. And the credit card number associated with my account. I resorted to randomly pounding on “0” and telling the voice recognition software I wanted help. Eventually, it got so confused it transfered me to a real human.
Tragically, the voicemail system was actually more helpful than the real human on the other end. Distilling down hours of sitting on the phone with them, I am told the following:

Predictions for 2008

I did not have a lot of predictions for what will happen with email at the beginning of the year so I did not do a traditional beginning of the year post. Over the last 3 – 4 weeks, though, I have noticed some things that I think show where the industry is going.
Authentication. In January two announcements happened that lead me to believe most legitimate mail will be DK/DKIM signed by the end of the year. AOTA announced that approximately 50% of all email was currently authenticated. They did not separate out SPF/SenderID authentication from DK/DKIM authentication, but this still suggests email authentication is being widely adopted. AOL announced they will be checking DKIM on their inbound mail. I expect more and more email will be DKIM signed in response to this announcement.
Filtering. The end of 2007 marked a steady uptick in mail being filtered or blocked by recipient domains. I expect this trend to continue throughout 2008. Recipient domains are rolling out new technology to measure complaints, evaluate reputation and monitor unwanted email in ways that tease out the bad actors from the good. This means more bad and borderline email will be blocked. Over the short term, I expect to see more good email blocked, too, but expect this will resolve itself by Q2/Q3.
Sender Improvements. As the ISPs get better at filtering, I expect that many borderline senders will discover they cannot continue to have sloppy subscription practices and still get their mail delivered. Improved authentication and better filtering let ISPs pin-point blocks. Instead of having to block by IP or by domain, they can block only some mail from a domain, or only some mail from an IP. There are a number of senders who are sending mail that users do not want mixed with mail that recipients do want. Right now, if there is more mail that recipients want in that mix, then ISPs let the mail through. This will not continue to happen through 2008. Senders will need to send mail users actively want in order to see good delivery.
Less is more. A lot of other email bloggers have talked about this, and I will echo their predictions. Less email is more. Send relevant mail that your customers want. Target, target, target. Good mailers will not send offers to their entire database, instead they will send mail to a select portion of their database.
Feedback loops. Use of feedback loops by recipient domains will continue to grow.
Mobile email. More recipients will be receiving email on mobile devices.
Suggestions for 2008

DKIM "i=" vs "d=" and Reputation

This really should be part seven of a twelve part series or some such as it deals with an aspect of DKIM that’s really important, but is way down in the details of implementation. (dkim.org is a reasonable place to start for a general overview of DKIM).
There’s an apparently endless thread on the DKIM-SSP spec development mailing list at the moment about the differences between two fields in a DKIM signature that could be used to tie a senders reputation to. Several ESP delivery folks asked me to explain what everyone was talking about, and this post is a first cut at that.
“i=” vs “d=”
There are two possible fields in a DKIM signature that could be used to identify the sender of a message, and so to tie a sender history and reputation record to. They are the so-called “i=” and “d=” field, from the syntax used to include them in the signature.