Bounces, complaints and metrics

In the email delivery space there are a lot of numbers we talk about including bounce rates, complaint rates, acceptance rates and inbox delivery rates. These are all good numbers to tell us about a particular campaign or mailing list. Usually these metrics all track together. Low bounce rates and low complaint rates correlate with high delivery rates and high inbox placement.

A totally fake graph I just made up
An example of bounce and complaint metrics correlating with inbox delivery
But sometimes the numbers lie and there isn’t a clear correlation between the metrics and the inbox delivery. In fact, these bounce rates and complaint rates are exactly the same as above, but the complaint and bounce rates don't correlate with delivery.
Another graph I just totally made up
Inbox delivery is decreasing even as complaints and bounces stay the same.

Why does this happen?

There are a number of different reasons that mail with low complaint and bounce rates will have low inbox delivery rates. Some of them are signs of improper behaviour on the part of the sender, some of them are simply the consequence of how mail is currently filtered. Whatever the reason it can cause confusion on the part of a lot of senders. To many people having low complaint rates and low bounce rates means good inbox delivery.
It’s probably partially the fault of delivery experts that so many people have such tight mental connections between low complaints and bounces and inbox delivery. When approached with delivery problems by customers and clients, many of us will look at complaint and bounce rates and advise that both complaint rates and bounce rates should be lower. As we work with clients to lower rates, then their inbox delivery often improves. So, clearly, low bounce and complaint rates mean higher inbox rates.
The problem is, though, that complaint rates and bounce rates are proxy measurements. They’re used to measure how much a mail is wanted by recipients and how clean the mailing list is. A list with high complaints and bounces is usually a list that doesn’t have much permission associated with it. Recipients generally don’t want mail that mail from that sender.
It’s important to remember, though, that complaints and bounce rates don’t specifically measure how wanted a particular mail is. The reason we focus on them is that they are easy to measure and they are correlated with how wanted an email is. We can use them to give us information. Delivery experts use that information to craft solutions to delivery problems. But we’re not actually fixing complaint rates and we’re not actually fixing bounce rates. Often the things I tell clients don’t directly lower complaint rates and they don’t directly lower bounce rates. Instead, I focus on fixing the policies and processes that are causing poor delivery. As those things get cleaned up, the complaint rates and bounce rates decrease and inbox rates increase.

Correlation is not causation
The correlation is that good lists and good senders have low bounce rates and low complaint rates. It’s not that they do anything special to address complaints and bounces, but rather that all the things they do focus on keep their lists healthy. Healthy lists have low bounce and complaint rates.
It’s very possible to have low complaint rates and low bounce rates without having a healthy list. That’s often what’s going on when senders have “great stats” and “zero complaints” but are still seeing poor inbox rates. These senders will focus on getting the great stats, because they think that it’s the great stats that lead to the good inbox rate. But they have it backwards. It’s that good list management, hygiene and engagement lead to good inbox rates. One way to measure list management and hygiene and engagement is to measure complaint and bounce rates.
Not every list with good stats has those goods stats because of good list management, hygiene and engagement. These rates are fairly easy to manipulate and some senders spend a lot of money and time manipulating their stats. Manipulating delivery stats did result in better inbox delivery, a bit, which is why so many companies spent so much time doing it. ISPs are adapting, though, and this is why we’re seeing senders with “great stats” have poor inbox delivery.
 
 

Related Posts

Before you build a list

I can’t add much more to Steve Denner’s article about building list size.

Read More

Don't take my subscribers away!

Tom Sather has a good summary of the problems with inactive email addresses and why data hygiene is critical to maintain high deliverability. These recommendations are some of the most difficult to convince people to implement.
Some of my clients even show me numbers that show that a recipient that hadn’t opened or read and email in 18 months, suddenly made a multi-hundred dollar purchase. Another client had clear numbers that showed even recipients that didn’t open for an entire year were responsible for 10% of revenue.
They tell me I can’t expect them to let their customers go. These are significant amounts of money and they won’t let any potential revenue go without a fight.
I understand this, I really do. The bottom line numbers do make it tough to argue that inactive subscribers should be removed. Particularly when the best we can offer is vague statements about how delivery may be affected by sending mail to unengaged users.
I don’t think many senders realize that when they talk about unengaged users they are actually talking about two distinct groups of recipients.
The first group is that group of users that actively receive email, but who aren’t opening or reading emails from particular senders. This could be because of their personal filters, or because the mail is going to the bulk folder or even simply because they don’t load images by default. This is the pool that most senders think of when they’re arguing against removing unengaged users.
The second group is that group of users that never logs in ever. They have abandoned the email address and never check it. I wrote a series of posts on Zombie Emails (Part 1, 2, 3) last September, finishing with suggestions on how to fight zombie email addresses.
Unlike senders ISPs can trivially separate the abandoned accounts from the recipients who just don’t load images. Sending to a significant percentage of zombie accounts makes you look like a spammer. Not just because spammers send mail to really old address lists, but a number of spammers pad their lists with zombie accounts in order to hide their complaint rates. The ISPs caught onto this trick pretty quickly and also discovered this was a good metric to use as part of their filtering.
I know it’s difficult to face the end of any relationship. But an email subscription isn’t forever and if you try to make it forever then you may face delivery problems with your new subscribers.

Read More

Data hygiene and bouncing zombies

There are a number of folks who tell me there can be no zombie addresses on their lists, they aggressively remove any address that bounces. The problem is that zombie addresses don’t bounce, at least not always. And even when ISPs say they have a policy to bounce email after a certain period of time with no access, that’s not always put into practice.
How do I know that ISPs don’t always deactivate addresses on the schedules they publish? Because I have seen addresses not be deactivated.
I have addresses in a lot of places that I go for long periods of time not checking. It’s rare that they’re taken from me or reject mail – most of the time they’re special test addresses I use when diagnosing issues. This post is based on my experiences with those addresses and how abandoned addresses are treated at some ISPs.
For Gmail I have two examples of addresses not being deactivated.
In July 2011, we set up a test address to look at how Gmail was handling authentication. We sent a matrix of different test emails to it, with valid and invalid SPF and DKIM signatures. We pulled the data from the account. I don’t know for certain when the last time I logged in, but it was August or September of last year. So we have an address that has been dormant since September 2011.
I just sent mail to the account and google happily accepted it.
Mar  2 07:03:22 misc postfix/smtp[11770]: 11CA12DED3: to=<wttwtestacct@gmail.com>, relay=gmail-smtp-in.l.google.com[74.125.127.27]:25, delay=1.8, delays=0.25/0.02/0.56/0.93, dsn=2.0.0, status=sent (250 2.0.0 OK 1330700602 x8si8608852pbi.66)
I have another google account (apparently) that my records show I set up sometime in 2010. The login info was saved October 2010. I don’t know when the last time I logged in was, but given I’d forgotten the existence of the account it’s a good bet that it has been more than a year. That account is also accepting mail as of today.
Mar  2 07:06:25 misc postfix/smtp[11836]: 8D90C2DED3: to=<phphendrie@gmail.com>, relay=gmail-smtp-in.l.google.com[74.125.127.27]:25, delay=1.6, delays=0.26/0.02/0.68/0.66, dsn=2.0.0, status=sent (250 2.0.0 OK 1330700785 a8si4075740icw.96)
For Hotmail I also have quite a bit of history and information. I signed up for my first Hotmail account in 1997. That was an account I used the address to post to usenet, but I didn’t actually use it for mail. I’d check it occasionally (usually when someone said in the newsgroup that they were going to email me) but it wasn’t an address I used regularly. As I moved from posting regularly in usenet, I started checking that account even less.
For a while, if I went more than 6 months checking my Hotmail account they would make me “re-claim” it. What would happen when I’d log in is I’d get a message along the lines of “well, we disabled this account due to inactivity, do you want it back?” I’d say yes, have to go through the setup process again and it would be my account. Mail was deleted during the disabling, and I am guessing they rejected anything new going to that account. I went through this dance for 4 or 5 years. I even had my calendar set to remind me to login every 6 months or so. There was some sentimental value to the address that kept me logging in. I have that same username at every major free ISP: Gmail, Hotmail, Yahoo and AOL, so it’s “my” address.
About 6 or 7 years ago, that behavior changed. I stopped getting the request to reclaim my account. Instead I could just log in. I’d still have mail (mostly spam as the address is on *lots* of lists and millions CDs). I still check it irregularly. I don’t have any idea when the last time I checked it was, but I think it’s been since at least November and probably longer back than that. Hotmail is still accepting mail for that address as well.
It’s anecdotal evidence, at best, but it ‘s the type of evidence that is acceptable even when it’s anecdotal. There are some addresses that are abandoned for long periods of time at the free mailbox providers and they’re are not all automatically pulled from the ranks of active addresses.
What does this mean for senders? It means that data hygiene has to go beyond just removing addresses that bounce. ISPs are not disabling addresses consistently enough for marketers to be able to trust that all addresses on their list are active just because they are accepting email.
This is the root of the recommendation to put in a hygiene program, this is why senders need to look at who is actually engaged with their brand and make some hard decisions about shooting zombies in the head.

Read More