Despite what a lot of SEO slop will try and tell you there’s no way to measure deliverability performance across multiple ESPs in any way that’s meaningful.
(The SEO slop tends to do things like sign up for free accounts at a bunch of ESPs, send three emails to probe accounts and see how many reach the inbox at mailbox providers served by the probe account service. If you’re a reader of this blog you can probably already see why that’s meaningless, but I could write several thousand words just on Why These People Are Wrong and Why Their SEO Slop Is Bad.)
Could we do the same sort of testing but … better? Not really. You’d need to be sending real content to a real audience – not probe accounts – consistently from each ESP long enough for metrics to stabilise to get a fair rough measure. And moving that audience to another ESP would change things enough that you couldn’t really compare with the first one, let alone by the tenth ESP. And even if you did all that it would, at best, give a vague comparison that’d be valid for folks sending the same sort of content to the same sort of audience.
So, what can you do? And how much does your ESP affect your deliverability?
If you have dedicated everything
First, is your reputation going to be isolated from their other customers?
If you’re going to be sending from a dedicated IP address, and all your reverse DNS, SPF and DKIM is using your domain rather than one owned by the ESP then your sending infrastructure is pretty well isolated. If you’re using your own domain in link tracking and image loading URLs, and your own domain in any CDNs used for images, rather than one owned by the ESP then your content is pretty well isolated too.
If everything you’re doing is white-labeled to your domain then you are going to get the deliverability your content and practices deserve. The direct impact of your ESP choice is going to be minimal.
If you have shared anything
If any of the identifiers mail filters use to detect bad actors – spam, malware, phishing – are shared between your mail and mail from other customers of the ESP then things get a little trickier.
At it’s simplest, if one customer is sending bad mail from an IP address, all mail from that IP address will be treated as suspicious (and, fairly commonly, all blocked at some mailbox providers). If there’s a lot of unwanted mail being sent with using a particular hostname in links in the message then any other mail using those same links will be viewed with suspicion.
Sending from a shared IP pool is the most obvious shared identifier, but sharing SPF and DKIM identifiers are also important. And an often overlooked aspect of sharing is the URLs in the content of the message – the ESP will wrap links for click tracking, add hosted images for “open” tracking and host images on a CDN.
In this setup you are sharing to some extent the reputation of all the other ESP customers using the same identifiers.
Sometimes the result of that can be catastrophic – if another customer on the same shared pool you’re using is sending bad content, or if other customers consistently send bad content, then a mailbox provider may outright block the IP address of that shared pool. Suddenly none of the mail from that shared pool is delivered to that mailbox provider, including yours.
Sometimes it can be less obvious. If a significant fraction of mail being sent using the ESPs shared click tracking domain is unwanted, then use of that click tracking domain will be recognised as a sign of unwanted email. Real spam filtering is usually more complex than a SpamAssassin score, but if you think of it as “sends from this ESP” adding three to your emails SpamAssassin score that leaves you a lot less slack to avoid hitting a score of five and ending up in the spam folder.
Most of this filtering tracks behaviour over time. A single bad send, or even a single bad customer is unlikely to be enough to impact delivery for other ESP customers (though it could cause IP blocking of the shared pool in some cases, at some providers). But an ongoing pattern of low quality mail from an ESP will affect delivery of all their shared customers.
Most large consumer mailbox providers are pretty good at separating mail streams from shared IP pools, so that even if you’re using shared IPs and shared domains they can mostly distinguish your email stream from those of your ESPs other customers. If you’re a B2C sender this isn’t really something you need to care about at the Gmails, the Yahoos and the Microsofts. It’s something you’re more likely to see issues with at smaller providers, or with enterprise filters.
So what should I be looking for?
Some folks may, at this point, say “Well, obviously I should be demanding a dedicated IP!”.
Dedicated IPs are expensive resources. If you’re sending huge volumes of mail then you need one, for network engineering reasons. But if you’re sending less…
There’s probably no real threshold of volume below which you absolutely shouldn’t be using a dedicated IP, but if you’re sending small amounts of mail – less than a million a week, perhaps? – building initial good deliverability on a dedicated IP may be harder.
Also, shared reputation across customers isn’t always a bad thing. If you’re one of a hundred customers in a pool and you do a bad thing, like sending to your suppression list, then the impact of that will be mitigated by the other customers using the shared identifiers. As long as they’re all good customers, mostly sending good email they’ll be supporting each other.
If you’re a smaller sender, you’re probably happy on your ESPs shared pools and – as long as most of your ESPs customers are good senders – will be able to build and maintain excellent deliverability.
You might still want to use your own domains for authentication and click tracking. That’ll isolate your reputation somewhat from other customers using the pool, and it’ll make it easier to move the good reputation you’ve built to another ESP. So if the ESP offers white label DKIM and SPF authentication that’s good. White label click tracking is nice to have too.
Are my ESPs other customers good?
This is an important thing to find out, but it’s really hard to measure directly. Spam filter operators will have a fairly shrewd idea, but that’s not data they’ll share (and it’ll change over time). You could check some of their IP addresses or domains on one of the multi-blacklist lookup sites – but most of the lists they check aren’t at all useful, and even for the few that are they only provide a snapshot of whether the ESP is listed now – and even some of the best ESPs may occasionally get listed.
But we can get some hints as to whether they allow customers to send spam from their network. Check their webpage. Is there a link to their acceptable use policy or terms of service? Does it forbid users from spamming? That might be phrased as requiring evidence of consent, requiring permission, an explicit ban on purchased lists, banning sending unsolicited messages, or banning use of third-party lists.
If you can easily find their AUP, and it forbids their customers from sending spam that’s a good sign that they don’t encourage spammers on their network. We maintain a list of ESPs that require good practices if you’d like somewhere to start. If you can’t find it, or it allows spam then that’s a bad sign.
Clearly stated policies are a good start, but whether they’re enforced matters too. That’s hard to measure, but a critical part of policy enforcement is having staff to do the work, and the infrastructure needed to do it.
If you’re a potential customer you might send an email to abuse@your-new-esp.com explaining that you’re evaluating them, and can they send you a link to their acceptable use policy.
If the email bounces, the ESP doesn’t have a functional abuse desk. If it returns an autoresponse saying the mail has not been read, they don’t have a functional abuse desk. If it requires you to fill out a web form, they don’t have a functional abuse desk.
550-5.7.1 Please do not reply to this email. If you wish to reply, please click
This is really not a good sign
550-5.7.1 on this <link> to update your
550-5.7.1 ticket, track the status, or close your ticket directly via the
550 5.7.1 console. – gcdp 5614622812f47-3f3af4a1d55si1924262b6e.281
If it just vanishes and you never hear back, or you get an automatic response and nothing more that’s not a great sign, but it might just mean their abuse desk is overwhelmed and doesn’t have the capacity to reply to email, or is forbidden by policy from doing so. And sometimes it means that they accept and discard all mail sent to their abuse alias (yes, I’ve been told by abuse staff at very large providers whose name you’d recognise that they do this). It’s hard to distinguish between those cases, unfortunately.
If you get a response with useful information, whether it be boilerplate or from a human being that suggests there’s a functional abuse desk there, with the resources and policies to do something. And that’s a really, really good sign that their customers are probably, on average, over time sending mostly wanted email. And that’s what you want your co-customers on an ESP to be doing.
Indirect ESP effects on deliverability
If we’re concerned about deliverabilty there are some things we’d really like an ESP to do for us that aren’t directly connected to their ability to deliver our email to the inbox.
Most of the time the reason we’re having problems delivering email is because of something we’re doing. Identifying why, and fixing it, requires some resources from the ESP.
Do they have in-house delivery expertise to help? This is expensive to offer, and likely not available to small customers but can be extremely valuable.
Do they provide access to any third party delivery monitoring services, such as probe accounts? The value of these is arguable, but if there’s easy access to that additional data that’s good to know.
Does their reporting of delivery data provide what you need to diagnose a problem? Does it let you track metrics over time? Does it let you break down metrics by recipient mailbox provider (even at the level of gmail / google apps vs office 365 vs yahoo vs everyone else)? Does it let you get the actual rejection messages for bounced email?
It’s not all about deliverability
While deliverability is a critical aspect for an email campaign – if the mail doesn’t make it to the recipients inbox then every other aspect you worked on is wasted – it’s definitely not the most important measure for an ESP. Not least because deliverability is mostly driven by your behaviour, not the ESPs.
Ease of use, decent automation, segmentation, responsive customer support, great templating, A/B testing, reporting, integration with other tools you use and just “are they going to be a pleasant company to work with” are all at least as important.
RSS - Posts