Email filters and small sends

Have you heard about the Baader-Meinhoff effect?

The Baader-Meinhof effect, also known as frequency illusion, is the illusion in which a word, a name, or other thing that has recently come to one’s attention suddenly seems to appear with improbable frequency shortly afterwards (not to be confused with the recency illusion or selection bias). Baader–Meinhof effect at Wikipedia

There has to be an corollary for email. For instance, over the last week or so I’ve gotten an influx of questions about how to fix delivery for one to one email. Some have been from clients “Oh, while we’re at it… this happened.” Others have been from groups I’m associated with “I sent this message and it ended up in spam.”

The challenge is, what we do to fix delivery of bulk mail doesn’t really apply to one to one mail. The underlying theory is the same: Send mail people expect to receive and if it gets delivered to the bulk folder have them go fish it out. But when we are sending bulk mail we have a whole population of recipients to work with. When we’re sending one to one mail we only have one person to work with.

Most people don’t know what their filters do under the covers.I know we have a fairly stock install of SpamAssassin and there are some bayesian filters built into mail.app. It’s pretty easy to ID why something was filtered by SA, it tells you. But the built in filters are a black box. All I know is that they learn from what mail I mark as spam.

I can see the results. For instance, almost every time I do a password reset the “here’s your temporary password” message ends up in my spam folder. Doesn’t really matter what provider it is or how regularly I get mail from the vendor or anything like that. If I’m doing the password reset dance then 90% of the time I have to go dig the message out of my spam folder.

It’s possible that I could reset the filters built into mail.app and have this mail come into the inbox. But that will also mean I have to go retrain my filters by manually sorting through the 40 – 50 spams that get through spam assassin every day. That’s tedious and not a lot of fun and there’s no guarantee that the filters won’t re-learn that password reset style messages are spam more often than not.

“Why did this actual, real, one-to-one message go to spam?” is a question we can almost never answer. Sure, sometimes a domain reputation is bad enough or the message is in the wrong language or there’s something blindingly obvious with the content that makes it clear why the message went to spam. But those cases are not as common as we may like. Sometimes the filters just decide this mail should be delivered there.

The point here is that a lot of what we do for deliverability works for bulk, because we’re managing populations and statistics and are sending enough mail we can move the needle on machine learning. When we’re sending very small quantities of mail, then we’re relying on individual users knowing why their mail is going to bulk. Almost no one does, it’s just gotten way too complex.

Which leaves us in a position where email is unreliable for some forms of communication. I don’t think this is a permanent status. I think we’re in a period of filtering changes where folks are trying lots of things to see what works. A decade ago it was whitelists and blacklists and FBLs and paid certification. Now, it’s machine learning and recipient behaviour and individualised inbox experiences.

Filters are continually adapting to spammers. Spammers are continually adapting to filters. This competition is driving rapid evolution on both sides. It’s like punctuated equilibrium for email.

Related Posts

Gmail, machine learning, filters

I’m sure by now readers have seen the article from Gmail “Spam does not bring us joy — ridding Gmail of 100 million more spam messages with TensorFlow.” If you haven’t seen it, go read it. It’s not often companies write about their filtering philosophy and what tools they’re using to manage incoming bad mail.

Read More

Ask Laura: Should we allow tagged addresses?

AmsCanal1

Read More

AHBL Wildcards the Internet

AHBL (Abusive Host Blocking List) is a DNSBL (Domain Name Service Blacklist) that has been available since 2003 and is used by administrators to crowd-source spam sources, open proxies, and open relays.  By collecting the data into a single list, an email system can check this blacklist to determine if a message should be accepted or rejected. AHBL is managed by The Summit Open Source Development Group and they have decided after 11 years they no longer wish to maintain the blacklist.
A DNSBL works like this, a mail server checks the sender’s IP address of every inbound email against a blacklist and the blacklist responses with either, yes that IP address is on the blacklist or no I did not find that IP address on the list.  If an IP address is found on the list, the email administrator, based on the policies setup on their server, can take a number of actions such as rejecting the message, quarantining the message, or increasing the spam score of the email.
The administrators of AHBL have chosen to list the world as their shutdown strategy. The DNSBL now answers ‘yes’ to every query. The theory behind this strategy is that users of the list will discover that their mail is all being blocked and stop querying the list causing this. In principle, this should work. But in practice it really does not because many people querying lists are not doing it as part of a pass/fail delivery system. Many lists are queried as part of a scoring system.
Maintaining a DNSBL is a lot of work and after years of providing a valuable service, you are thanked with the difficulties with decommissioning the list.  Popular DNSBLs like the AHBL list are used by thousands of administrators and it is a tough task to get them to all stop using the list.  RFC6471 has a number of recommendations such as increasing the delay in how long it takes to respond to a query but this does not stop people from using the list.  You could change the page responding to the site to advise people the list is no longer valid, but unlike when you surf the web and come across a 404 page, a computer does not mind checking the same 404 page over and over.
Many mailservers, particularly those only serving a small number of users, are running spam filters in fire-and-forget mode, unmaintained, unmonitored, and seldom upgraded until the hardware they are running on dies and is replaced. Unless they do proper liveness detection on the blacklists they are using (and they basically never do) they will keep querying a list forever, unless it breaks something so spectacularly that the admin notices it.
So spread the word,

Read More