A little over a year ago, Kristin Bond posted an article (reprinted here) looking at the diversity of speakers at marketing conferences. As with many articles pointing out gender issues in technology there was quite a bit of discussion about it on a related mailing list. Some of the comments were supportive and open to the idea that gender diversity is an overall good. Some of the comments, while well meaning, indicated the commenters didn’t understand some of the more systemic issues that result in conferences with speaker lists that consist primarily of white men.
Kristin, I, Jen Capstraw and April Mullen started talking privately about the issue. What I discovered during those conversations is that I wasn’t alone in how I felt about some spaces. Being a woman in tech I expect to feel left out in many places. When I go to a conference, or I participate in an online space or I meet up with colleagues in social situations, I expect that someone will say something sexist. As a woman I regularly feel like an outsider. What I didn’t realize is other women in those same spaces felt the same way. By not saying something I was missing an opportunity to find a supportive atmosphere with other women who also thought spaces were unfriendly or toxic to women.
But we didn’t just complain; we decided to take action. What would happen if we created a space to help conferences find women speakers? What would happen if we set up a framework for women to find mentors? What did we have to lose by trying? Thus, Women of Email™ was formed.
A year later:
Women of Email has had an incredibly successful first year! We’ve filled 17 speaker slots at various conferences. We’ve had our first round of mentorships. We host an active Facebook community where women working in the email space talk about all the things that affect our careers. We have had dozens of meetups around the world.
Last month Jen, April, Kristin and I all flew to Vegas to have an in-person board meeting and look to the future. It was great to finally see each other in person and talk about our goals for the organization. We’re working to get our corporate status solidified and getting IRS approval as a 501(c)3 non-profit. That’s a work in progress. We were hoping to be able to announce it on the anniversary of our founding, but bureaucracy is always slower than one hopes.
Women of Email isn’t simply for marketers, we welcome all women working in email to join Women of Email, including folks working for agencies, postmaster teams, abuse desks, and compliance teams. For those of you on Facebook, we have a closed group for members to discuss email, delivery, work and life.
In the coming months, we’ll be continuing to build out our speaker’s bureau (so add your name!). If you’re organizing a conference we’re happy to recommend names to you or share your conference info with our speakers. We’re looking forward to having another flash-mentoring event in the fall. We’re also looking for volunteers who would like to help us expand our programs.
On a more introspective note, we are aware that the current board is very middle-class and white and heterosexual. This is not very diverse and we know it. As we expand the leadership of the organization, we are actively looking for women who aren’t like us. We want this organization to address the needs of women in email, not just white heterosexual women. I can’t promise we’ll always do things right, but we’re going to do our best and accept when we’ve failed.
I’m incredibly lucky in my career. I run my own business, and have rarely had direct experience with sexism that affected my job or career. In the early days the most common problem was that some men would insist on having Steve on calls. It was annoying and frustrating, but it wasn’t career damaging. Plus, we’d charge them for Steve’s time as well as mine. He would also spend most of the calls telling them he wasn’t the right person to answer the question, they should really ask me about it. His actions helped deal with those sexist clients directly, but they also did more. They significantly increased my confidence and left me knowing I had the authority to speak on my own. Even now, clients will sometimes ask for him on a call. I don’t acquiesce any longer; they hired me, they get me.
Every woman deserves acknowledgement of their knowledge and expertise. They deserve the opportunity to learn and grow in their career. As an industry, we are better when we have more and more diverse voices. Women of Email can’t fix everything. We can’t meet everyone’s needs. What we promise is doing our best in the space we’ve claimed.
On May 31, British broadband provider EE discontinued service for a number of email domains: Orange.net, Orangehome.co.uk, Wanadoo.co.uk, Freeserve.co.uk, Fsbusiness.co.uk, Fslife.co.uk, Fsmail.net, Fsworld.co.uk, and Fsnet.co.uk.
These domains were acquired by EE as part of multiple mergers and acquisitions. On their help page, EE explains that the proliferation of free email services with advanced functionality has led to a decrease in email usage at these domains.
Yesterday, Terra.co.br announced they were discontinuing email to a number of their free domains as of June 30, 2017: terra.com, terra.com.ar, mi.terra.cl, terra.com.co, terra.com.mx, terra.com.pe, terra.com.ve, and terra.com.ec.
I’m not surprised to see these domains going away and I think we’ll see more of it going forward. The reasons are pretty simple. Mail is not an easy service to run. Mail doesn’t bring in a lot of money. Dedicated mailbox providers do a great job and the addresses from them are portable.
Mail is not an easy service
Managing a mail server is not an easy task. There’s so much to pay attention to and monitor to keep the network and users safe. Spammers are always changing tactics and modifying their methods. They work tirelessly to find ways to get their mail in front of people. Filters cannot be set and forgotten. Someone must manage and tweak them constantly. Sure, you can outsource it to commercial filters, but that’s still a cost.
It’s not just spam filtering that requires expertise, it’s also virus and malware filtering. Think about the botnets and worms affecting users recently. They’re often infecting machines by way of email. But they use broadband networks to spread. Broadband providers, at least the responsible ones, have dedicated security teams to monitor infections, cut off infected users, and assist them in cleaning up and getting back online.
All of these functions take money, which leads me to the second point.
No one wants to pay for mail
OK, maybe not NO one. But, in general, consumers won’t pay extra for email service. It’s a core feature, not an add-on. This means that broadband providers have to pay for spam and virus filtering out of general revenues. They can’t add features and then bump rates. Consumers expect all the bells and whistles with their email accounts, and if it’s not there, well, they’ll go to Gmail.
Which leads me to my third point.
Free mail providers are driving innovation
Mailbox providers, like Gmail and Microsoft are driving innovation in the inbox. Both companies have announced new products over the last few years like Sweep, Tabs, and Focused inbox. They’re also driving standards and innovation in the backend email space. Gmail has already started using ARC, they support TLS, and they have one of the most advance spam filtering systems in the world.
All of these factors are contributing to the decrease in mail usage at broadband providers. Even better, a free mail address isn’t tied to your location. If you move out of your broadband provider’s area, you can lose your email address. Freemail addresses are portable and stick with you forever. I’ve had one Hotmail address for over 20 years now, and the same username at Gmail since someone sent me one of the coveted invites to the Gmail beta test.
Ironically, over the years there’s been a push by marketers to find a users real email address. The theory was that the free mail addresses weren’t the addresses recipients really used, and so weren’t as valuable as the real address. But that’s not what happens. Many people use freemail addresses as their primary addresses.
Advice for marketers
As domains continue to disappear, marketers are going to have to up their game when it comes to bounce handling and data hygiene. Unless marketers allow users to update their email addresses, they risk forever losing contact with those customers. That’s a loss. But there’s a bigger loss hiding in these domains. Filtering companies and public blocklists use abandoned domains as a data source. Sure, they’ll bounce mail for 12 – 24 months, but down the road these addresses could drive spam blocking.
Data hygiene is a fact of life. Domains, and email addresses, going away are a fact of life. Planning ahead and incorporating ongoing maintenance into processes will lessen the events of domains going away. Companies that have preference centers or the ability to change addresses can react swiftly to events like this. A domain is going away? All they have to do is grab subscribers at those domains and send a few emails asking for the new address. Companies that don’t have processes in place to handle these events, are going to lose subscribers. They risk blocking in the long term.
Failing to implement data hygiene processes will lead to poor delivery. Don’t let it happen to you.
A few months ago a colleague sent me, and every other person on his overly large LinkedIn list, an email looking for some help hiring. It starts off with “Greetings LinkedI Connections” and ends with… an unsubscribe link.
P.S. If you don’t want to hear from me, here’s an unsubscribe link – that’s the easiest way. My LinkedIn network has gotten so ridiculously large that [unfortunately] I have to use an ESP to send messages like this out – which I do very infrequently – maybe a handful of times a year – I promise. 🙂
The part that really annoyed me was this person didn’t use my LinkedIn address. No, instead they found a different address for me and used that for their blast1 email. I pointed out to them that the address they mailed wasn’t my LinkedIn address. They replied they knew, that they’d mapped all their LinkedIn connections to corporate email addresses.
I asked them not to do that with my email address in the future, at which point they unfriended me on LinkedIn because it was easier.
This is the major problem with appending. I have an address associated with my LinkedIn account. It’s the address I have dedicated to handling requests from my LinkedIn network. But this person decided that it was better to use a different address to send me this email. Why? Dunno, you have to ask them. Probably because they thought, somehow, they’d get a better response if they ended up in my “primary” mailbox.
I suspect this person doesn’t like Gmail tabs either.
There are two major points I want to make with this story.
The first is that it’s a really bad idea to make assumptions about which email address to use for people, especially when they have given you an address to use. I do check my LinkedIn folder regularly but I do it on my time. My corporate address is for business. It’s for my clients, employees, and customers. Random requests for networking? I want those emails to go into my networking folder (and, yes, it really is named “networking”).
The second is, when someone says “please don’t do that again” don’t get all huffy about it. That only makes you look petulant and thin-skinned. In this case, the sender is an executive for a large player in the email space. Do you really think I’m going to recommend them to my clients? Fair or unfair, the interaction was unpleasant.
This is a very personal example of appending. A single person decided to find an address for me and specifically use that instead of the one I gave them. This was appending, but not in bulk. Nevertheless, it was someone else deciding they could override my decisions because they wanted to get in my inbox. It’s rude.
It’s no less rude when thousands of addresses are automatically appended to a list. In fact, that’s even more rude. Instead of just one annoyed person, there are thousands. Appending is a bad practice, whether it’s one or one million.
1: Yes, I called it a blast. It was a blast. Even the sender admits it’s a blast. They’re not doing anything other than importing hundreds of (collected, not opted-in) email addresses into an ESP and sending the same email to everyone of them, whether or not it’s relevant.
Spamhaus has listed a number of domains belonging to French politicians recently. In their blog post about it, they mention that the listings are directly related to address lists provided to candidates by the French government.
We learned of this issue recently when two different French candidates became entangled in two of our automated spam detection systems, the DBL and the CSS. The candidates whose IPs and domains were listed by us, when contacting us to resolve the listings, independently told us that the lists they were sending to were provided by the French government:
“I am a candidate for the French election next week and need to send as soon as possible an email to the 100000 people eligible to vote to explain my program and motivations. Emails has been given by french authorities.”
“Our lists are opt-in and have been provided by the French Government.
I’ve talked about purchased lists in the past (But my purchased list is TARGETED!!!, Where can I mail a purchased list?, Trust the list broker, Your purchased list … is spam, Purchased Lists and ESPs). But there’s always more to talk about.
No money needs to change hands for a “purchased” list.
This is a huge issue. I, and other deliverability experts, have repeatedly heard some variation on “it’s not purchased, it was given to me.” “It’s not purchased, it was part of my conference registration.” “It’s not purchased, it’s a professional organization I belong to and they give us the addresses.” “It’s not purchased, it’s rented.” “It’s not purchased, it’s co-reg.” The reality, though, is all those list types are what deliverability people mean when they say”purchased.”
Specifically, in the French election case, the candidates may or may not have paid for the lists, but they were still purchased.
What is a purchased list?
It’s pretty simple: if you didn’t compile the list yourself, if some third party did it, then the list is purchased. Those appended addresses? Purchased. That trade show list? Purchased. That highly targeted list of executives? Purchased. That list of French voters? Purchased.
Are purchased lists always bad?
Yes. There is zero incentive for third parties to make sure their address lists are clean and deliverable. Most email marketers know this. What’s the first thing most companies do when they purchase a list? Run it through a hygiene provider. Even the people buying lists don’t expect them to be clean. People and companies compiling lists make their money by collecting as many emails addresses as possible. They prioritize quantity over quality. Getting permission, checking accuracy, and bounce handling lower the quantity of email addresses available for sale. Their bottom line suffers when they do any sort of data hygiene.
I have no idea how the French government collected the email addresses. If it was part of voter registration they’re unlikely to do any verification. Here in California, voters have a choice about providing an email address. But the state does nothing to verify the address belongs to the voter. They also do sell that information for certain non-commercial purposes.
This list is transparent and has real permission.
Everybody mails purchased addresses.
Not everyone does, I know that for sure. I also know it’s fairly common. And I know some purchased lists are low complaint and low bounces so they don’t look bad to ESPs. The term for those kinds of lists is waterfalling. It’s simple. List compiler runs their list though enough different ESPs and removes addresses that complain, bounce or unsubscribe. Eventually, there’s a list where anyone who might unsubscribe has and anyone who might complain has. As ESPs are mostly reactive, the lists slide under the radar and never get flagged as purchased. The challenge for these lists is Gmail, and a lot of senders who have great delivery most places but fail at Gmail. Some of this is related to Gmail’s refusal to send complainers addresses back to ESPs.
Is there any best practice advice for mailing purchased lists?
Not really, no. If you ask most of the deliverability folks, ISP representatives or filter companies they’ll tell you don’t mail a purchased list. Still, there are a number of senders who try and come up with scenarios where a purchased list is fine and they can make money doing it. I’m not going to argue that some companies will see a short boost in revenue when they mail a purchased list. It can work over the short term. Over the long term, however, it drives down deliverability and makes it harder to reach the inbox.
It’s easy to tell if a domain is using SPF – look up the TXT record for the domain and see if any of them begin with “v=spf1”. If one does, they’re using SPF. If none do, they’re not. (If more than one does? They’re publishing invalid SPF.)
AOL are publishing SPF. Geocities aren’t.
For DKIM it’s harder, as a DKIM key isn’t published at a well-known place in DNS. Instead, each signed email includes a “selector” and you look up a record by combining that selector with the fixed string “._domainkey.” and the domain.
If you have DKIM-signed mail from them then you can find the selector (s=) in the DKIM-Signature header and look up the key. For example, Amazon are using a selector of “taugkdi5ljtmsua4uibbmo5mda3r2q3v”, so I can look up TXT records for “taugkdi5ljtmsua4uibbmo5mda3r2q3v._domainkey.amazon.com“, see that there’s a TXT record returned and know there’s a DKIM key.
That’s a particularly obscure selector, probably one they’re using to track DKIM lookups to the user the mail was sent to, but even if a company is using a selector like “jun2016” you’re unlikely to be able to guess it.
But there’s a detail in the DNS spec that says that if a hostname exists, meaning it’s in DNS, then all the hostnames “above” it in the DNS tree also exist (even if there are no DNS records for them). So if anything,_domainkey.example.com exists in DNS, so does _domainkey.example.com. And, conversely, if _domainkey.example.com doesn’t exist, no subdomain of it exists either.
What does it mean for a hostname to exist in DNS? That’s defined by the two most common responses you get to a DNS query.
One is “NOERROR” – it means that the hostname you asked about exists, even if there are no resource records returned for the particular record type you asked about.
The other is “NXDOMAIN” – it means that the hostname you asked about doesn’t exist, for any record type.
So if you look up _domainkey.aol.com you’ll see a “NOERROR” response, and know that AOL have published DKIM public keys and so are probably using DKIM.
(This is where Steve tries to find a domain that isn’t publishing DKIM keys … Ah! Al’s blog!)
If you look up _domainkey.spamresource.com you’ll see an “NXDOMAIN” response, so you know Al isn’t publishing any DKIM public keys, so isn’t sending any DKIM signed mail using that domain.
This isn’t 100% reliable, unfortunately. Some nameservers will (wrongly) return an NXDOMAIN even if there are subdomains, so you might sometimes get an NXDOMAIN even for a domain that is publishing DKIM. shrug
Sometimes you’ll see an actual TXT record in response – e.g. Yahoo or EBay – that’s detritus left over from the days of DomainKeys, a DomainKeys policy record, and it means nothing today.
I recently received a 419 spam that had a message at the top of the email.
Yup, a 419 spammer is trying to convince me there are millions of dollars waiting for me, but he won’t pay his software vendor 29.99 to comply with a license.
This is only the most recent in a long line of examples of spammers being cheap and attempting to steal services.
Back when I was working abuse almost every ISP had a story about a spammer who refused to pay their bill. Or spammers who were so high maintenance they cost the company money.
The company I worked for had a spammer that was on our system for far too long. Eventually they were cut off for non-payment and their hardware was confiscated. Still, the spammer came in and managed to remove the hardware before the building guards were alerted. It was disappointing, but at least they weren’t spamming off our network any longer.
Even now, ESPs share stories of customers who come in, spam and never pay their bill. Works for the spammer, they can get a few weeks of spamming in without having to pay for the service. They spew their stuff and leave a giant mess for the ESP to clean up. Next week, they’re on to the next ESP.
The real problem with this is that with enough ESPs and enough sends you can clean a list. This list can then be sold, or moved to a credible ESP without any of the tell tale signs of a purchased list. It’s so common it even has a name: waterfalling. It’s profitable, though, and there are enough small ESPs out there with little compliance experience that it can work.
I regularly get questions from folks who’ve worked themselves into a hole about swapping IPs or domains in order to get out of the hole. My answer is always the same. Changing identity might work in the short term, but it won’t work longer term. I also tell them that spammers have been trying to avoid filters for a lot longer than they have. Spammers are good at it, and still get caught in filters. Better to spend time trying to fix the underlying problem – typically users aren’t engaged with your mail – then trying to obfuscate who is sending the message to avoid filters.
Focus on sending good email that users want, rather than trying to avoid filters. That’s the key to getting into the inbox.
The world is beginning to go all https, all the time, but until recently good practice was to make a web page available via both http and https.
The problem is that if you try and load a resource from an http URL from a page that was loaded via https it’ll complain about it, and not load the resource.
And if your users web browser is loading the http version of your page because it’s Internet Explorer 6 and doesn’t speak modern SSL then it’ll be unable to load anything over https, including any of your resources.
So whether you choose https or http protocol for loading your page resources it’s going to break for someone.
One common trick to avoid the problem is to use a protocol-relative URL. That looks like “//example.com/blahblah.css”, and it’ll load the resource over the same protocol that the page was loaded over.
While we can safely use “https://…” everywhere now, “//…” URLs are still a common idiom for things like loading things like CSS libraries from public content-distribution networks as well as your own resources.
I was reading long-but-excellent writeup about Stack Overflow’s migration to TLS (hey, I read this sort of stuff for fun) and they point out something I hadn’t considered – mail clients don’t really have any sensible way to use protocol-relative URLs. The mail client loaded the “page” from a mailbox and so has no base document protocol to work from (even if there’s a <base> element in the content it’s not likely to affect a mail client). If the user is reading it via webmail or a mail client that’s using an embedded web browser to render HTML it might work, sometimes, but it’s not going to reliably load that resource in general.
So, if you’re copy-pasting content from your web collateral to reuse in an email, make sure you’re not loading anything external – including images – via a “//…” style URL. Rewrite them to use “https://…”.
On Friday I talked a little about DMARC being a negative assertion rather than an authentication method, and also about how and when it could be deployed without causing problems. Today, how DMARC went wrong and a partial fix for it that is coming down the standards pipeline.
DMARC (with p=reject) risks causing problems any time mail with the protected domain in the From: field is either sent from a mailserver that is not under the control of the protected domain, or forwarded by a mailserver not under the control of the protected domain (and modified, however trivially, as it’s forwarded). “Problems” meaning the email is silently discarded.
This table summarizes some of the mail forwarding situations and what they break – but only from the original sender’s perspective. (If forwarding mail from a users mailbox on provider A to their mailbox on provider-Y breaks because of a DMARC policy on provider-A that’s the user’s problem, or maybe provider-A or provider-Y, but not the original sender’s.)
|discussion mailing list
|recipient vanity domain forwarding
|recipient forwarding service (e.g. alumni domains)
|recipient forwarding e.g. yahoo to gmail
|recipient mailbox forwarding via POP
|consumer email addresses using other mailbox providers
|consumer email addresses sending from ESPs
|consumer email addresses using, e.g. zendesk
(* some situations may modify the body or headers of the message, breaking the DKIM signature and causing DMARC failures)
(If you’re diagnosing – or trying to avoid – DMARC issues it’s worth remembering that DKIM signatures can break, apparently randomly, due to issues with how the original email was constructed. If the mail wasn’t forwarded the SPF will be valid and you might not notice the broader DKIM issue.)
Where does it break?
The issues that break DMARC tend only to apply to “loosely controlled” domains – domains that don’t have full control over their mail streams, or which have humans who feel they own addresses in those domains and so want to use them outside the limits DMARC places on them. AOL, Yahoo and LinkedIn are some domains who’ve done that.
There’ve been some rather hacky workarounds for some specific situations. Some discussion mailing lists put a fake email address in the From header, rather than the original author (which prevents DMARC policy triggered bounces, but causes problems for others on the mailing list). AOL added C0nstant Contact, Sailthru and Zendesk to their SPF record.
DMARC doesn’t force receiving ISPs to take any particular action – it’s just advice. Some ISPs treat it naively and discard or reject any mail with a DMARC p=reject domain in the From: header that fails authentication.
Others take a more realistic approach and acknowledge that, for example, a mailing list server with a good reputation that’s emitting unauthenticated email from users at AOL or Yahoo is far more likely to be breaking authentication for legitimate email it’s forwarding than to be an evil phisher.
What can we mitigate?
It would be really nice to be able to say “I trust the forwarder that sent this to me, and I trust them when they tell me that the mail authenticated correctly when they received it” in a more mechanical manner than “I trust this long list of mailing list servers”.
That’s the plan behind Authenticated Received Chain (ARC).
Authenticated Received Chain
In much the same way that regular Received: headers record the series of mailservers that an email passes through as it’s delivered, ARC’s ARC-Seal: headers record the series of administrative domains a company passes through.
An administrative domain is, pretty much, a single company or single email system. While an email going through Microsoft might go through give different mailservers with each mailserver adding a new Received: header the same email would have just on ARC-Seal: header recording it entering the microsoft.com email system.
Other than that, they record where an email has gone, just like a Received: header. Unlike a Received: header, though, they’re cryptographically authenticated so that the organization adding the header can take responsibility for that email at that point in the delivery. If you think that sounds similar to the DKIM concept of taking responsibility for an email as it’s sent, you’re right. An ARC-Seal: header is like a simplified, stripped-down DKIM signature that signs just the ARC-related headers of the message.
Each ARC-Seal header can also have a couple of other ARC headers associated with it.
One is ARC-Message-Signature:. It’s pretty much the same as a DKIM-Signature: header, letting each hop of the delivery add a new DKIM-esque signature of the body of the message and the headers of the message as they send it on.
The other is ARC-Authentication-Results:. It contains the same information as an Authentication-Results: header added by that hop would – did SPF validation pass? was the DKIM signature valid? (and potentially all sorts of other authentication results, from authentication methods that are obsolete or are yet to be invented).
A recipient can step through the series of ARC-Seal: headers and as long as they trust each participant they have authenticated information about the content of the message and the authentication status of that message as it was received by that participant.
This lets an ISP – if they invest in the reputation tracking needed to make the “trust” decision about each participant – identify email that would have been validly authenticated if it hadn’t been through a forwarder or a mailing list and treat that mail as though it were validly authenticated, for making decisions about rejecting it due to DMARC policies.
Who does this affect?
If you’re a user at a domain that publishes DMARC p=reject then ARC has the potential for making your use of forwarders and mailing lists much, much more reliable.
If you’re a participant in discussion mailing lists then as long as the mailing list manager deploys ARC and your ISP is ARC aware you’re less likely to see messages vanish due to DMARC issues, and your mailing list manager will (hopefully) remove some of the gross workarounds they’ve previously put into place to mitigate those problems.
If you develop mailing list management software, or anything that forwards email, you should be reading the ARC specs and following the ARC mailing list and be thinking about how you’re going to implement this.
If you use mailing list software, you should check up on whether it’s implementing ARC. If they are, deploy it when you can. If they’re not, ask them to.
If you’re a mailbox provider you should probably find out more. ARC is something you’re going to want to plug in to your existing reputation based filters.
If you’re a typical ESP and you send email on behalf of your customers who provide the content to you via a web interface or an API and who want to send “From” their consumer email address then ARC is not going to help you. This is not the fix for “individuals and small businesses with consumer email addresses want to run mailing lists” you were looking for. Sorry.
Senders of all flavours. There’s nothing you need to change. As ARC is rolled out it will make some forms of authentication breakage less common, and that may affect your decisions as to whether or when to deploy DMARC reject or quarantine policies. If you’re using DMARC p=none with reporting today you might see those changes happening over then next year or so.
AOL and Gmail are showing ARC results today.
Gmail is adding ARC headers to mail they send or forward.
There are interoperable implementations of the core algorithms in python, C, milter and perl. And a test suite.
Mailman and Sympa are actively adding support.
Want to know more?
ARC home page: arc-spec.org
ARC specification: draft-ietf-dmarc-arc-protocol-03
Recommended usage: draft-ietf-dmarc-arc-usage-01
Mailing list: subscribe or read the archives
We know that legitimate email sent with valid SPF and a DKIM signature often breaks in transit.
SPF will fail any time mail is forwarded – via a mailing list, a forwarding service used by the recipient, or just ad-hoc forwarding.
DKIM will fail any time the message is modified in transit. That can be obviously visible changes, such as a mailing list tagging a subject header or adding a footer to the body. It can also be less obvious changes, such as intermediate MTAs wrapping lines that are too long, reencoding content or repackaging the message altogether – perhaps when delivering from a mailserver that is 8BITMIME compliant to one that isn’t.
(This image has absolutely nothing to do with email authentication, but searching for stock photography about email or authentication or chains or, well, pretty much anything like that leads to horribly depressing corporate imagery. So, no. Have something colourful and optimistic instead.)
As SPF and DKIM are typically used, none of this is much of a problem. A message being authenticated provides a little extra information to the receiving mailserver, and the domain attached to the authentication can be used to look up a senders reputation, giving a potential boost to the chances of the mail being sent to the inbox. If the authentication is broken, though, the mail will still be judged on it’s merits – is it coming from an IP address that’s a source of good mail, does the content look legitimate, and all the other things a spam filter looks at.
That authentication is a (potentially big) positive signal, but lack of authentication isn’t really any signal at all is why SPF and DKIM being fragile wasn’t an issue. SPF and DKIM are positive assertions – “IF this mail IS authenticated THEN IT IS from me”.
That changed when DMARC became popular, though.
DMARC allows the owner of a domain to say “We send no mail that is not authenticated, and we promise that none of that authentication will be broken in transit”. DMARC is a negative assertion – “IF this mail IS NOT authenticated THEN it IS NOT from me”. It converts the absence of a positive assertion into a negative assertion.
This isn’t the first attempt to layer a “we authenticate everything” negative assertion on top of fragile email authentication. SPF did it, with the -all flag (which is universally ignored, leaving SPF purely as a positive assertion). DomainKeys did it, with DomainKeys policy records (which you occasionally still see published, but were never really used to reject mail). DKIM did it with ADSP – which didn’t see much use either.
The reason none of them were used much is because even when senders were telling the truth about “we send no email that is not authenticated” they were always lying, to varying degrees, about “none of the authentication will be broken in transit”.
If your domain that is solely used for bulk email. If it’s never for used mail sent by human beings, not even customer support employees. If it’s a newly created domain with no legacy usage that only sends email from a very tightly controlled infrastructure. If you only send email that’s been created via a well implemented message composition pipeline that ensures the content of the is not just RFC compliant but also “well formed”, with short lines, simple widely implemented encoding, vanilla mime structure and so on. And it’s sent out via conservatively configured smarthosts that deliver directly to the end recipients MX. And if you know that the demographics of your recipients are such that the minority that are forwarding that mail elsewhere (e.g. from their Yahoo account to their Google account or via an alumni mail alias) is a small enough group that you don’t care about them…
If all of those things are true, then your domain is going to be able to deploy DMARC pretty easily and safely. If not, though, how can you tell?
That’s the place where DMARC improves over it’s predecessors. It allows you not only to publish a DMARC policy record in test mode, so it’s not actually used to filter your mail (well, mostly, but that’s a longer story) but also to ask recipients to notify you of mail that seems to be from you but which isn’t authenticated.
You can publish a “p=none” DMARC record with notification addresses in it and wait and see what happens. You’ll get notification of mail that has your domain in the From: field but which isn’t authenticated.
As a first round of action that lets you see where you’re sending email from that you didn’t know about. Sysadmin notification email. That marketing splinter group in Sasketchwan. The outsourced survey company.
Once you’ve cleaned all that up, and made sure everyone is authenticating their mail then you can look at what’s left. The next step is likely to be mistakes you’re making in authentication or message composition that’s causing some of your mail – typically depending on content, and source and recipient domain – to become unauthenticated. Clean that up, make sure all your message composition is squeaky clean, make sure employees aren’t sending mail using that domain in ways you don’t authorize (interacting with mailing list, for example).
By that point you’ll have reduced the torrent of reports you’re getting to two types. One is mail that you send that has it’s authentication broken in transit through some process you have no control over. The other type is mail that has your domain in the “From” field but which you didn’t send. Some of that may be legitimate use of your domain by your employees, such as forward-to-a-friend services, signing up for document delivery via email, third-party notification services. By deploying DMARC you are declaring all that sort of usage to be illegitimate, and you’ll need to get all your employees to stop doing it (or, at least, know that it’s going to stop working). The rest of it is likely a mix of spam and phishing mail. The spam, that’s just using your domain in random from addresses, you probably don’t care about. The phishing you do.
You’ve finally cleaned up your mail infrastructure and policies enough to gather the data you need. How much of my legitimate email will have it’s authentication broken (and hence be silently thrown away by DMARC)? And how much hostile phishing mail is targeting my users (and using the exact domain you are)?
Then you have the information you need to make an informed decision as to how badly deploying DMARC will break your legitimate use of email (after you’ve done everything you can to minimize that) and some idea of whether it will provide you any benefit, at least in the shorter term.
That testing phase, where senders can use other peoples mail infrastructure to investigate their sending practices, gradually fix any problems and finally gather some metrics is what made gave the developers of the DMARC spec confidence that it wouldn’t break things, and made it much more deployable than previous approaches to negative assertion.
On Monday, how all that optimistic reasoning went to hell, what it broke and how we’re trying to fix it.
All the authentication and DMARC in the world can’t save you from stupid.
I just got a survey request from my bank. Or, at least, it claimed to be from my bank.
From: Barclays International Banking Survey <email@example.com>
The mail passed SPF (though the SPF record suggests this is being mailed from all over the place) and was validly DKIM signed for barclayssurveys.com. And that domain has a DMARC policy
But there’s nothing in any of that that tells me – or mail filters – that this has anything to do with Barclays Bank.
“barclayssurveys.com” is what’s know as a cousin domain in the phishing world. It’s a domain that has absolutely nothing to connect it to the legitimate domain of the phishing target, but which looks plausible to a recipient.
This one didn’t actually look that plausible, though. The website is hosted on a RackSpace VPS with no reverse DNS configured. The domain is registered by “chime.plc.uk” – whose website is just an Outlook Web Access instance:
All of which would be suspicious enough if it came from my local dive bar, but this is coming from an international bank that’s big enough, rich enough and technically savvy enough that they own their own top level domain.
No institution can claim to care about phishing or account takeover as an issue when the legitimate email they send is less plausible than a typical phishing mail. This is just setting up their customers to fall for phishing mail.
And, yes, it’s from a legitimate survey firm. One that’s quite widely used in the United Kingdom and Éire. How do I know it’s widely used? Because the mail they send out leaks information about their customers:
X-Confirmit-FixedSenderDomain: factssurvey.co.uk, feedback-waveutilities.co.uk, feedback-anglianwaterbusiness.co.uk, npowersurveys.com, o2surveys.co.uk, gustosurveys.co.uk, customersatisfaction.rbs.co.uk, customersatisfaction.natwest.com, mail.customersatisfaction.rbs.co.uk, mail.customersatisfaction.natwest.com, panel.uk.com, virgintrainseastcoastsurveys.com, barclayssurveys.com, sunnyloanssurveys.com, sagafeedback.co.uk, boxcleversurveys.co.uk, surveys.ulsterbank.ie, sagafeedback.co.uk, barclays.com, titanfeedback.co.uk, barclaycardsurveys.com, aegonfeedback.co.uk, directionsurveys.co.uk
Just from the names I recognize that’s five major high street banks, a payday loan outfit, several utility companies, travel companies and a major cellphone company that are sending survey email that’s this badly done. And that’s probably just the ones that are being sent from this particular mailserver.
I went back and checked where my bank usually sent email from, and how their authentication was normally set up. The previous mail I got from them was a timely warning about “Phishing” and “Smishing” and “Vishing” warning me to be very careful about clicking on links in mail claiming to be from my bank, for fear of being phished.
It was addressed to “%first name%”.