Mailbox providers want happy recipients
Mailbox providers want their users to be happy with the mail they receive and the service they get. That’s driven by stark business reasons: acquiring new users is costly, happy users bring in revenue – whether directly, or indirectly via advertising – and their word of mouth helps bring in more users, and hence more revenue. That’s still true when the email service is bundled as part of a larger package, such as broadband service or domain registration.
For consumer mailboxes user happiness is affected by a lot of things, but a major requirement is accurate email filtering. An inbox full of spam makes the whole experience unpleasant, and drives users to try competing services. One important email missed because it was rejected or dropped or sent to the spam folder can make them really quite angry.
Some email it’s easy to classify correctly. Mail from the user’s daughter, who they’ve corresponded with for years and whose email address is in their address book? Pretty easy to decide that it should go in the inbox.
Mail that consists solely of this …
Subject: X5O!P%A@P[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
… that goes to the spam folder, at best.
But most mail it’s harder to make a decision about. On the spectrum of obviously unwanted to obviously wanted there’s a grey area in the middle where it’s not immediately clear. That grey area is where even the best spam filters may make mistakes.
Some of those mistakes don’t matter – there’s a lot of mail that recipients are … meh … about, and are fine with it if it’s in their inbox, but won’t miss it if it ends up in their spam folder. (We call it “greymail”).
But it’s mail that falls in that grey area in the middle that’s likely to lead to filtering mistakes that will make users unhappy, so mailbox providers want to make that grey are as small as possible.
Legitimate senders want happy recipients
Legitimate bulk senders mostly want to send email that people want to people who want it. They want recipients to be happy to receive their mail.
Maybe they’re not overjoyed to receive a bank statement, or a receipt or a fathers day shapewear special offers email, but it’s all email they’re generally content to have in their inbox.
They actively want, or are at least content to receive, email that they’ve signed up to receive. If they weren’t, they wouldn’t have signed up for it.
And legitimate senders want the inboxes of their recipients to be as spam free as possible. If it’s a user’s happy place they’ll spend more time looking at mail there, rather than just skimming through it to find the stuff they have to read and marking all the spam as spam. The good, wanted mail they’re sending won’t be lost in the junk. A healthy email ecosystem directly benefits those senders.
Why is this so hard?
So … legitimate senders are sending mail recipients want to receive, mailbox providers want to deliver that mail to the recipients who want to receive it.
Why is this so hard to do?
The problem is that there are a lot of email senders – a lot – who aren’t legitimate. There’s the malware, the phishing, the weird spam from power supply manufacturers in Shenzhen, the never ending stream of B2B “can we schedule a call” spam, the … just check your spam folder, you know what I mean.
There are also a lot of otherwise legitimate companies who buy lists of email addresses and send the same advertising to them as they send to their customers. The exact same email may be unwanted spam to one recipient, while being a newsletter another recipient signed up for and wants to retrieve.
There’s also a lot of email that if you look at the content it’s obviously spam that nobody would ever sign up for – but when you investigate you discover that a lot of your users did sign up for it, and do want it, and will get mad when it’s in their spam folder.
It’s very hard for anyone other than the recipient to reliably recognize whether a piece of email is wanted or unwanted – but that’s what mailbox providers need their spam filtering technology to do.
So … Deliverability
It’s all about mail in the grey area, where it’s hard to tell what’s wanted and what isn’t, and where filters tend to make mistakes. The less mail in the grey area, the fewer filtering mistakes, happier users. That’s what we want.
A large chunk of deliverability is collaboration – both explicit and tacit – between legitimate senders and mailbox providers to make all this happen, trying to ensure that recipients get more of the email they want and less of the email they don’t want, by helping mailbox providers make the grey area in the middle smaller.
Senders authenticate email to make it easier for mailbox providers to reliably recognize their streams of mail, so their filters can use the history of how previous mail was engaged with by recipients to decide that this mail should go in the green “wanted mail” box.
We publish DMARC records so that mailbox providers can say “this mail fails DMARC, we’ll put it in the red unwanted mail box”.
We send wanted mail, so that mailbox providers can see that the mail we send is wanted by their recipients, and so future mail from us can go in the wanted mail box.
We differentiate different sorts of email – by sending IP, or by authentication – to make it easier to recognise those mail streams.
Mailbox providers provide feedback loops and other sorts of feedback (such as email deferrals) to tell us which of our mail streams are problematic, and allow us to fix them.
Mailbox providers pay attention to the different mail streams we send, for example understanding that transactional flows will have different recipient behaviour to marketing flows, and holding them to different standards.
Mailbox providers dig through the vast amount of data they gather about the email they receive, wanted, unwanted and “who knows?” to see what data points are useful to distinguish between the three types of email. And then they tell us. Do this and you’ll differentiate yourself from a lot of spammers.
Even simple advice like “make sure your mailservers have sensible reverse DNS” lets senders distinguish themselves from malware-infested windows machines. That might sound obvious to you, because of course you do that, it’s just normal practice. But it wasn’t normal practice until ISPs and spam filter developers said “Uh, you should do this, so we know you’re intentionally a mailserver.”
There’s a lot of baseline behaviour that it was reasonable to expect a legitimate sender to support in 2023 – using email authentication, say, or offering unsubscription. Large mailbox providers looked at their data and saw that a lot of unwanted email didn’t meet this baseline set of requirements, and a lot of wanted email did. It would be nice to use that behaviour to move some of the mail that’s in the grey “unknown” box into the red or green – but there were too many legitimate senders who weren’t meeting that baseline.
If those legitimate senders in the grey box were all to meet that baseline behaviour suddenly the spam filters would be able to move all bulk mail streams that didn’t meet that behaviour into the red “unwanted” box, significantly shrinking the grey box and making it possible for the spam filters to be more accurate.
More accurate spam filters makes for happier users – and should make for happier senders, as their wanted mail will mostly be in the inbox and won’t be battling so much unwanted email there for positive attention.
Yahoo and Google announced last year that you had to be this tall to ride the ride. If you were sending bulk email and you didn’t meet that baseline behaviour then you were going to end up in the red box sooner or later. That baseline behaviour is just some of the basics that pretty much every deliverability group would agree were best practices that everyone should do – but now there’s incentive to find resources and make decisions to make meeting those requirements something every legitimate bulk sender can do.
Yahoo and Google could have just started filtering mail based on those requirements (and, in many cases, already were doing so). Announcing the requirements publicly was a gift to delivery folk, giving them a tool they could use to deal with their organizations tech debt and update their processes. And they worked with delivery folks to adjust the requirements and timelines and documentation to make the changes more effective and more palatable.
And mailbox provider and filter vendor staff are talking to deliverability folks all the time to spot where something has gone wrong, mitigate a filtering bug, suggest better practices on both sides of the mail transaction.
Mailbox provider staff are often limited in what they can say publicly, or how specific they can be about an issue, as they’re defending against malicious bad actors. It’s very, very common for spammers to pretend to be representatives of an ESP, for example, while trying to extract information about how an ISP filters spam, so they can craft a way to get their unwanted spam delivered. But mailbox provider staff still collaborate with trusted folks on the deliverability side, to understand what concerns and issues there are, and to quietly get the word out about changes or problems or other things it’ll benefit legitimate senders to know.
While it’s easy to forget sometimes when dealing with “why is my mail being deferred?” or “did this ISP filter do something weird again?” or “why is someone clicking on all my links?” deliverability folks at legitimate email senders and staff at mailbox providers or filter vendors are generally on the same side. We all want a healthy email ecosystem, we want happy recipients, we want wanted email from legitimate senders to slip into the inbox without hitting any speedbumps.
If it doesn’t feel like that in the moment it’s probably not intentional. We’re all – filter maintainers and deliverability folks both – working with complex, slightly unreliable technology at a ridiculously large scale, with a team of overworked, underfunded staff who mostly believe in what they’re doing. Cut each other some slack and assume good intent.
RSS - Posts