The importance of data hygiene

Over the weekend, one of the major ISPs purged a lot of abandoned accounts from their system. This has resulted in a massive increase in 550 user unknown bounces at that ISP. This ISP is one of those that uses bounces to feed into their reputation system and the purge may cause otherwise good senders to be blocked temporarily.
Talking to clients and other industry folks, it looks like the addresses that have newly bounced off had zero activity for at least 6 months. Nothing. Nada. No clicks. No opens. No interaction.
This is why data hygiene is so critical. Just because the emails are being accepted at the ISP, and even showing inbox placement at the mailbox monitoring companies does not mean that there is actually someone reading your email. Failure to look at overall data means that when an ISP bulk deletes abandoned accounts then bounces will increase. While I don’t expect this to have any real, long term effect on sender reputation I do expect that some senders with a lot of cruft on their list will see some short term delivery problems.
Companies that run re-engagement campaigns saw a whole lot less bouncing and even less blocking as a result of the purge. They were removing addresses that were non-responsive all along and thus didn’t have major deadwood on their list.
Ongoing data hygiene shows you what your list really is, not your list plus abandoned accounts. The addresses that the ISP purged? They were not valuable anyway. No one was reading that mail for at least 6 months.
If you did see a spike in bounces this weekend at a major ISP, you should really look at engagement. If some percentage of recipients at one ISP are actually non-existent, then it’s likely that about that same number are non-existent at other major ISPs as well. What are you going to do to identify and remove those dead addresses from your lists?

Related Posts

20% of email doesn't make it to the inbox

Return Path released their global delivery report for the second half of 2009. To put together the report, they look at mail delivery to the Mailbox Monitor accounts at 131 different ISPs for 600,000+ sends. In the US, 20% of the email sent by Mailbox Monitor customers to Return Path seed accounts doesn’t make it to the inbox. In fact, 16% of the email just disappears.
I’ve blogged in the past about previous Return Path deliverability studies. The recommendations and comments in those previous posts still apply. Senders must pay attention to engagement, permission, complaints and other policy issues. But none of those things really explain why email is missing.
Why is so much mail disappearing? It doesn’t match with the philosophy of the ISPs. Most ISPs do their best to deliver email that they accept and I don’t really expect that ISPs are starting to hard block so many Return Path customers in the middle of a send. The real clue came looking at the Yahoo numbers. Yahoo is one of those ISPs that does not delete mail they have accepted, but does slow down senders. Other ISPs are following Yahoo’s lead and using temporary failures as a way to regulate and limit email sent by senders with poor to inadequate reputations. They aren’t blocking the senders outright, but they are issuing lots of 4xx “come back later” messages.
What is supposed to happen when an ISP issues a 4xx message during the SMTP transaction is that email should be queued and retried. Modern bulk MTAs (MessageSystems, Port25, Strongmail) allow senders to fine tune bounce handling, and designate how many times an email is retried, even allowing no retries on a temporary failure.
What if the missing mail is a result of senders aggressively handling 4xx messages? Some of the companies I’ve consulted for delete email addresses from mailing lists after 2 or 3 4xx responses. Other companies only retry for 12 – 24 hours and then the email is treated as hard bounced.
Return Path is reporting this as a delivery failure, and the tone of discussion I’m seeing seems to be blaming ISPs for overly aggressive spamfiltering. I don’t really think it’s entirely an ISP problem, though. I think it is indicative of poor practices on the part of senders. Not just the obvious permission and engagement issues that many senders deal with, but also poor policy on handling bounces. Perhaps the policy is fine, but the implementation doesn’t reflect the stated policy. Maybe they’re relying on defaults from their MTA vendor.
In any case, this is yet another example of how senders are in control of their delivery problems. Better bounce handling for temporary failures would lower the amount of email that never makes it to the ISP. This isn’t sufficient for 100% inbox placement, but if the email is never handed off to the ISP it is impossible for that email to make it to the inbox.

Read More

Delivery problems are not all spam related

Not every delivery failure is due to poor reputation or spam. Sometimes ISPs just have problems on their mailservers and so mail doesn’t get through. It’s often hard for delivery experts (and their bosses and their customers and their clients) to watch email delays or rejections without being able to do anything about it.
Sometimes, though, there is nothing to do. The rejections are because something broke at the ISP and they have to sort through it. Just this week there’s been a lot of twitter traffic about problems at a major cable company. They are rate limiting senders with very good reputations. They have admitted there is a problem, but they don’t have a fix or an ETA. From what I’ve heard it they’re working with their hardware vendor to fix the problem.
Hardware breaks and backhoes eat fiber. Yes, ISPs should (and all of the large ones do) have backups and redundancies. But those backups and redundancies can’t always handle the firehose worth of mail coming to the ISPs. As a result, the ISPs start rejecting some percentage of mail from everyone. Yahoo even has a specific error message to distinguish between “we’re blocking just you” from “we’re shedding load and temp failing everyone.”

Read More

This is why the ISPs throw up their hands at senders

I recently saw a question from an ESP rep asking if anyone had a personal contact at a particular ISP. The problem was that they had a rejection from the ISP saying: 571 5.7.1 too many recipients this session. The ESP was looking for someone at the ISP in order to ask what the problem was.
This is exactly the kind of behaviour that drives ISPs bonkers about senders. The ISP has sent a perfectly understandable rejection: “5.7.1: too many recipients this session.” And instead of spending some time and energy on the sender side troubleshooting, instead of spending some of their own money to work out what’s going on, they fall back on asking the ISPs to explain what they should do differently.
What, exactly, should you do differently? Stop sending so many recipients in a single session. This is not rocket science. The ISP tells you exactly what you need to do differently, and your first reaction is to attempt to mail postmaster@ the ISP and then, when that bounces, your next step is to look for a personal contact?
No. No. No.
Look, connections and addresses per connections is one of the absolute easiest things to troubleshoot. Fire up a shell, telnet to port 25 on the recipient server, and do a hand SMTP session, count the number of receipts. Sure, in some corporate situations it can be a PITA to do, sometimes you’re going to need to get it done from a particular IP which may be an interface on an appliance and doesn’t have telnet or whatever. But, y’know what? That Is Your Job.  If your company isn’t able to do it, well, please tell me so I can stop recommending that as an ESP. Companies have to be able to test and troubleshoot their own networks.
Senders have been begging ISPs for years “just tell us what you want and we’ll bother you less.” In this case the ISP was extremely clear about what they want: they want fewer recipients per connection. But the ESP delivery person is still looking for a contact so they can talk to the ISP to understand it better.
This is why the ISPs get so annoyed with senders. They’re tired of having to do the sender’s job.

Read More