As I continue to think about how people troubleshoot email delivery I keep finding other things to talk about. Today we’re going to talk about the question most folks start with when troubleshooting delivery. “Did ISP change something?”
At least once a week I check some delivery or email fora and some form of the question is sitting there.
“Did X change something? We haven’t done anything different and our delivery went way down overnight.”
Did Y change their filters? Our delivery is tanking and all our authentication is fine.”Anyone hear of a change at Z? We have been having increasing difficulty reaching the inbox and we don’t understand why. Looking for suggestions.
In reality, the answer to this question Does Not Matter and asking it is only going to delay actually resolving your delivery issue.
When filters change
The reality is, filters are continually changing. ISPs and filtering companies are always tuning filters. These changes are roughly in 3 categories.
- Ongoing tweaking and improvement to provide a better experience for their users
- Changes done to address a emergent threat (Yahoo deploying p=reject is one example of this)
- Specific changes to catch a type of spam they had previously been unable to effectively identify and filter.
Filters are not static. They are continually adjusting based on a number of things. We can always assume the answer to the question is yes. Something changed. Now what?
There are basically 3 situations here.
- The filters did something unexpected and caught mail it wasn’t intended to catch, causing recipients to complain to the ISP.
- The filter change was intentional but caught more mail than was intended, causing recipients to complain to the ISP.
- The filter change was intentional and caught exactly the mail that was intended and the recipients didn’t care enough to notice that mail was missing.
In the first two cases, the ISP is going to fix things. They’re going to listen to their users and adjust the filters. In the first case, I expect to see changes and rollback within 24 – 48 hours. In the second, I expect to see changes in 24 – 96 hours.
The third case is the interesting one. Does anyone care about mail they don’t care about going to the bulk folder? Folks sending mail, even opt-in mail, that the users don’t complain about when it’s missing is the definition of grey mail. Filter maintainers listen to their users. If users complain they’ll change things, if users don’t complain they’ll assume the filters are working as intended.
The answer to the question did the filters changed tells you nothing. Of course the filters changed. Either they’re doing something that the maintainers don’t intend, which means they’ll be fixed or they’re catching mail they’re intended to catch.
Instead of asking if the filters changed, flip the question. Why are my users not interested enough in my mail to notice it when it’s gone? Start your troubleshooting from that perspective.
RSS - Posts
This is, at best, very tangential to the topic at hand, but it’s something I found myself thinking about regarding anti-spam measures.
Let’s say I come out with a new anti-spam measure: Reject all mail from domains that contain the letter ‘e’. Initially, this measure catches a large amount of spam, and unavoidably a certain amount of legitimate mail as well.
I continue to advocate for this filter, and point out just how much spam the rule catches. Other sites start to adopt it.
As more and more people adopt the rule, senders begrudgingly start to change their setup so that they no longer send mail from domains containing the letter ‘e’. There is a period of time when spam is actually more likely to pass this test than non-spam, since spammers tend to rotate through domains pretty quickly anyway and aren’t particularly attached to any one of them. But, eventually, the overall mail ecosystem adjusts, and legitimate mail mostly passes the test, with most of the mail caught by the filter being spam. Of course, inexperienced or new administrators, mostly of small sites, may still accidentally set things up so that their mail comes from a domain containing the letter ‘e’, but they eventually mostly figure it out, and, anyway, you can’t make an omelette without breaking some eggs.
Anyway that’s all very dumb, but I do wonder if there’s a good way to determine whether a particular anti-spam measure works better than that.
There are, in fact, a significant number of blocklists out there that work on how much spam they block regardless of how much real mail they block. These blocklists are not really used by receivers of any size. The situation is just as you describe, the blocklist creators come up with a set of rules to block mail, either by domain or IP, and then sell it based on the amount of spam it catches. This type of filters also catches a lot of real mail, and 20+ years of experience says that type of filter won’t be widely deployed. We had the situation you described (with IPs rather than domains) and widespread blocking didn’t happen.
To answer your other question: There are lots of ways to determine how effective filters are. Some coming out of academic research and a lot done by the large filter maintainers themselves. I found a trove of research papers on arXiv a few weeks ago. Both Google and Microsoft have some developer blogs that discuss things, too. Most filters aren’t just about all the spam that is blocked. They’re about how they meet the needs of the users. There is feedback from the users about how accurate the filters are and every filter maintainer I’ve talked to pays attention to that.