Troubleshooting

Troubleshooting: part 3

As I continue to think about how people troubleshoot email delivery I keep finding other things to talk about. Today we’re going to talk about the question most folks start with when troubleshooting delivery. “Did ISP change something?”

The variables are not independent

In my previous career I was a molecular biologist. Much of my work was done on bacteria but after I left grad school, I ended up working in a developmental biology lab. Bacteria were (mostly) simple: just about every trait was controlled by a single gene. We could study what that gene did by removing it from the bacteria or adding it to a well characterised bacteria.

Troubleshooting delivery problems

Everyone has their own way of troubleshooting problems. I thought I would list out the steps I take when I’m trying to troubleshoot them.

When you can’t get a response

I’ve seen a bunch of folks in different places looking for advice on what to do when they can’t get a response from a postmaster team, or a filtering company. I was all set to write yet another post about how silence is an answer. Digging through the archives, though, I see I’ve written about this twice already in the last 18 months.

Email filters and small sends

Have you heard about the Baader-Meinhoff effect?

The Baader-Meinhof effect, also known as frequency illusion, is the illusion in which a word, a name, or other thing that has recently come to one’s attention suddenly seems to appear with improbable frequency shortly afterwards (not to be confused with the recency illusion or selection bias). Baader–Meinhof effect at Wikipedia

There has to be an corollary for email. For instance, over the last week or so I’ve gotten an influx of questions about how to fix delivery for one to one email. Some have been from clients “Oh, while we’re at it… this happened.” Others have been from groups I’m associated with “I sent this message and it ended up in spam.”

Anyone know why…

Countless questions about email troubleshooting start with “does anyone know why.” Unfortunately, most of these questions don’t contain enough detail to get a useful answer.

In the case of email, even the smallest redactions, like the IP address and the domain in question, can make it difficult for anyone to provide help. Details matter.
Every detail matters, sending IP and domain are just the beginning. Who’s doing the sending? What is their authentication setup? What IP are they using? How were the addresses collected? What is their frequency? What MTA is used? Are they linking to outside sites? Are they linking to outside services? Where are images hosted? Is the mail going to the bulk folder or being rejected? What ISPs or filters are involved?
The relevant questions go on and on and on.
We send fairly detailed question lists to clients. I regularly look at them to try and make them shorter. But the reality is these are questions that are relevant. Without enough information we simply cannot troubleshoot delivery problems.

Troubleshooting and codes

Microsoft is still in the process of rolling out new mail servers. One thing that is new about these is some new codes on their error messages. This has led to questions and speculations as to what is going on.

Troubleshooting delivery is hard, but doable

Even for those of us who’ve been around for a while, and who have a lot of experience troubleshooting delivery problems things are getting harder. It used to be we could identify some thing about an email and if that thing was removed then the email would get to the inbox. Often this was a domain or a URL in the message that was triggering bulk foldering.
Filters aren’t so simple now. And we can’t just randomly send a list of URLs to a test account and discover which URL is causing the problem. Sure, one of the URLs could be the issue, but that’s typically in context with other things. It’s rare that I can identify the bad URLs sending mail through my own server these days.
There are also a lot more “hey, help” questions on some of the deliverability mailing lists. Most of these questions are sticky problems that don’t map well onto IP or domain reputation.
One of my long term clients recently had a bad mail that caused some warnings at Gmail.
We tried a couple of different things to try and isolate the problem, but never could discover what was triggering the warnings. Even more importantly, we weren’t getting the same results for identical tests done hours apart. After about 3 days, all the warnings went away and all their mail was back in the inbox.
It seemed that one mailing was really bad and resulted in a bad reputation, temporarily. But as the client fixed the problem and kept mailing their reputation recovered.
Deliverability troubleshooting is complicated and this flowchart sums up what it’s like.

Here at Word to the Wise, we get a lot of clients who have gone through the troubleshooting available through their ESPs and sometimes even other deliverability consultants. We get the tough cases that aren’t easy to figure out.
What we do is start from the beginning. First thing is to confirm that there aren’t technical problems, and generally we’ll find some minor problems that should be fixed, but aren’t enough to cause delivery problems. Then we look at the client’s data. How do they collect it? How do they maintain it? What are they doing that allows false addresses on their list?
Once we have a feel for their data processes, we move on to how do we fix those processes. What can we do to collect better, cleaner data in the future? How can we improve their processes so all their recipients tell the ISP that this is wanted mail?
The challenging part is what to do with existing data, but we work with clients individually to make sure that bad addresses are expunged and good addresses are kept.
Our solutions aren’t simple. They’re not easy. But for clients who listen to us and implement our recommendations it’s worth it. Their mail gets into the inbox and deliverability becomes a solved problem.

Thoughts on "ISP relations"

I’ve been thinking a lot about the field of ISP relations and what it means and what it actually is. A few years ago the answer was pretty simple. ISP relations is about knowing the right people at ISPs in order to get blocks lifted.
The fact that ISPs had staff just to deal with senders was actually a side effect of their anti-spam efforts. In many places blocking was at least partially manual, so there had to be smart, technical, talented people to handle both the blocking and unblocking. That meant there were people to handle sender requests for unblocking.
Spam filters have gotten better and more sophisticated. Thus, the ISPs don’t need smart, technical, talented and expensive people in the loop. Most ISPs have greatly scaled back their postmaster desks and rely on software to handle much of the blocking.
Another issue is that some people on the sender side rely too heavily on the ISPs for their data. This makes the ISP reps, and even some spam filtering company reps, reluctant to provide to much help to senders. I’ve had at least 3 cases in the last 6 months where a sender contacted me to tell me they had spoken with someone at an ISP or filtering company and were told they would get no more help on a particular issue. In talking with those reps it was usually because they were drowning under sender requests and had to put some limits on senders.
All of this means ISP Relations is totally different today than it was 5 years ago. It’s no longer about knowing the exact right person to contact. Rather it’s about being able to identify problems without ISP help. Instead of being able to ask someone for information, ISP Relations specialists need to know how to find data from different sources and use that data to identify blocking problems. Sure, knowing the right person does help in some cases when there’s an obscure and unusual issue. But mostly it’s about putting together any available evidence and then creating a solution.
We still call it “ISP Relations” but at a lot of ISPs there is no one to contact these days. I think the term is a little misleading, but it seems to be what we’re stuck with.

Where do you accept reports?

One of the things that is most frustrating to me about sending in spam reports is that many ESPs and senders don’t actively monitor their abuse address. A few months ago I talked about getting spam from Dell to multiple email addresses of mine.
What I didn’t talk about was how badly broken the ESP was in handling my complaint. The ESP was, like many ESPs, an organization that grew organically and also purchased several smaller ESPs over the course of a few years. This means they have at least 5 or 6 different domains.
The problem is, they don’t effectively monitor abuse@ for those different domains. In fact, it took me blogging about it to get any response from the ESP. Unfortunately, that initial response was “why didn’t you tell us about it?”
I pointed out I’d tried abuse@domain1, abuse@domain2, abuse@domain3, and abuse@domain4. Some of the addresses were in the mail headers, others were in the ESP record at abuse.net. Three of those addresses bounced with “no such user.” In other words, I’d tried to tell them, but they weren’t accepting reports in a way I could access.
Every ESP should have active abuse addresses at domains that show up in their mail. This means the bounce address domain should have an abuse address. The reverse DNS domain should have an abuse address. The d= domain should have an abuse address.
And those addresses should be monitored. In the Dell case, the ESP did have an active abuse@ address but it was handled by corporate. Corporate dropped the ball and never forwarded the complaint to the ESP reps who could act on the spam issue.
ESPs and all senders should have abuse@ addresses that are monitored. They should also be tested on a regular basis. In the above case, addresses that used to work were disabled during some upgrade or another. No one thought to test to see if they were working after the change.
You should also test your process. If you send in a complaint, how does it get handled? What happens? Do you even have a complaint handling process outside of “count and forward”?
All large scale senders should have appropriate abuse@ addresses that are monitored. If you don’t, well, you look like a spammer.

Guide to resolving ISP issues

I often get a chuckle out of watching some people, who are normally on the blocking end of the delivery equation, struggle through their own blocking issues. A recent situation came up on a mailing list where someone who has very vehement opinions about how to approach her particular blocklist for delisting and that the lists policies are immutable. The company she works for is having some delivery issues and she’s looking for a contact to resolve the issues.
While digging through my blog posts to see if there was any help I could provide, I realized I don’t have a guide to resolving blocking issues at ISPs. Much of the troubleshooting can be done without ever contacting the ISPs or the blocklists.
Identify the issue.
There are a number of techniques that ISPs use to protect their users from malicious or problematic mail, from rate-liming incoming mail, putting mail in the bulk folder, or blocking specific IP addresses. Step one to resolving any delivery problem is to identify what is happening to the mail. In order to resolve the issue, you have to know what the issue is.
All too often, the description of a delivery problem is: My mail isn’t getting delivered. But that isn’t very clear as to what the actual problem is. Are you being temp failed? Is mail being blocked? Is mail going to the bulk folder? Is this something affecting just you or is it a widespread problem?
Troubleshoot your side.
Collect as much data about the problem as you can. Dig through logs and get copies of any rejection messages. Follow any URLs that are present in the bounce messages. Try sending a bare bones email to yourself at that ISP with just URLs, is it still blocked? What if you send from a different IP, does the same thing happen?
There is a lot of troubleshooting a sender can do without having to contact an ISP, and the information can lead to resolution that doesn’t involve having to contact the ISP. Also, many current ISP blocks are dynamic, they come up and go down without any human intervention. Those blocks that require contact to get them resolved have clear instructions in the bounce message.
Fix your stuff.
Whether it’s a reputation issue or a minor technical issue, fix the problem on your end. Just moving IP addresses or changing a URL isn’t a sustainable fix. There is a reason mail is being blocked or filtered and if you don’t fix that issue, the blocks are just going to come back. After you do fix your stuff, expect to see changes in a few days or a week. The ISP filters are generally quite responsive to sender improvements so if you’ve fixed the stuff you should see changes pretty quickly. Expect unblocking or filtering to take a little longer than the block was in place.
If you can’t figure out what the problem is, hire a consultant. Here at Word to the Wise we can often quickly identify a problem and provide a path to resolution. Sometimes the problem isn’t even the ISPs, we’ve had multiple cases where our clients were using custom software and their software wasn’t SMTP compliant and we were able to identify the problem and get their mail working again. There are a host of other independent consultants out there that can also help you identify and resolve blocking problems.
Contact the ISPs.
If there is a hard block or after fixing what you think the underlying problem is, you’ll have to contact the ISP. Many ISPs provide self service websites and contact forms to facilitate this process. Generally, though, most issues aren’t going to require contact.

I'm on a blocklist! HELP!

Recently, an abuse desk rep asked what to do when customers were complaining about being assigned an IP address located on a blocklist. Because not every blocklist actually affects mail delivery it’s helpful to identify if the listing is causing a problem before diving in and trying to resolve the issue.

Troubleshooting email delivery

Mark Brownlow has a post up explaining how he discovered some problems with delivery at Gmail by digging deeper into his statistics. Mark goes through his thought process including his initial conjecture on what might be causing the problems and then how he looked at the data to see if his supposition fit the data.
I love this post. It is so refreshing to watch someone document how they asked questions, then looked at data to find out the answers. Too many people treat best practices in email delivery as a set of rules that are meant to be broken. Instead of actually asking questions and determining what is best for their market and their recipients they implement best practices.
Following best practices isn’t exactly a bad thing, the reason they’re best is because they’re easy to communicate practices that will not result in bad outcomes. But, they’re not always the ideal practices for a specific situation. Best practices are ones that work across a wide range of senders and situations. Blindly implementing best practices will not always result in the best outcome for each situation.
Mark’s post is a tutorial in the art of looking at email delivery. I think there is a need for more of those kinds of posts, explaining the process from identifying an email problem through to confirming that is actually the problem and then testing potential fixes. I’ll be posting troubleshooting guides here over the next few weeks and months. If you have an issue you think would be an interesting case study drop me an email and we’ll go through it.

Troubleshooting the simple stuff

I was talking with one of my Barry pals recently and was treated to a rant regarding deliverability experts that can’t manage simple things. We’ve been having an ongoing conversation recently about the utterly stupid and annoying questions some senders ask. Last week, I was ranting about a delivery person asking what “5.7.1. Too many receipts this session” meant. This morning I got an IM.