Return Path have an interesting post up about content filtering. I like the model of 3 different kinds of filters, in fact it’s one I’ve been using with clients for over 18 months. Spamfiltering isn’t really about one number or one filter result, it’s a complex interaction of lots of different heuristics designed to answer the question: do recipients want this kind of mail?
What makes the best email filter? There isn’t really a single answer to that question. Different people and different organizations have different tolerances for how false positives versus false negatives. For instance, we’re quite sensitive to false positives here, so we run extremely conservative filtering and don’t block very much at the MTA level. Other people I know are very sensitive to false negatives and run more aggressive filtering and block quite a bit of mail at the MTA level.
For the major ISPs, the people who plan, approve, design and monitor the filters usually want to maximize customer happiness. They want to deliver as much real mail as possible while blocking as much bad mail. Blocking real mail and letting through bad mail both result in unhappy customers and increase the ISP’s costs, either through customer churn or through support calls. And this is a process, filters are not static. ISPs roll out new filters all the time, sometimes they are an improvement and sometimes they’re not. When they’re not, they’re pulled out of production. This works both for positive filters like Return Path and negative filters like blocklists.
Then there is mail filtering that doesn’t have to do with spam. Business filters, for instance, often block non-business mail. Permission of the recipient often isn’t even a factor. Companies don’t often go out of their way to block personal mail, but if personal mail gets blocked (say the vacation plane ticket or the amazon receipt) they don’t often unblock it. But when you think about why a business provides email, it makes perfect sense. The business provides email to further its own business goals. Some personal usage is usually OK, but if someone notices and blocks personal email then it’s unlikely the business will unblock it, even if the employee opted in.
In the case of email filters, the free market does work. Different ISPs filter mail differently. Some people love Gmail’s filters. Other people think Hotmail has the best filtering. There are different standards for filtering, and that makes email stronger and more robust. Consumers have choices in their mail provider and spamfiltering.
Return Path published their most recent Global Deliverability report this morning. (Get the Report) This shows that inbox placement of mail has decreased 6% in the second half of 2011. This decrease is the largest decrease Return Path has seen in their years of doing this report.
To be honest, I’m not surprised at the decrease. Filters are getting more sophisticated. This means they’re not relying on simply IP reputation for inbox delivery any longer. IP reputation gets mail through the SMTP transaction, but after that mail is subject to content filters. Those content filters are getting a lot better at sorting out “wanted” from “unwanted” mail.
I’m also hearing a lot of anecdotal reports that bulk folder placements at a couple large ISPs increased in the first quarter of 2012. This is after the RP study was finished, and tells me increased bulk folder placement is more likely to be a trend and not a blip.
One of the other interesting things from the RP study is that the differences are not across all mail streams, but are concentrated in certain streams and they vary across different regions.
I made that joke the other night and most of the folks who heard it didn’t get the reference. It made me feel just a little bit old.
Anyhow, Mickey beat me to it and posted much of what I was going to say about Ken Magill’s response to a very small quote from Neil’s guest post on expiring email headers last week.
I, too, was at that meeting, and at many other meetings where marketers and the folks that run the ISP spam filters end up in the same room. I don’t think the marketers always understand what is happening inside the postmaster and filtering desks on a day to day basis at the ISPs. Legitimate marketing? It’s a small fraction of the mail they deal with. Ken claims that marketing pays the salaries of these employees and they’d be out of a job if marketing didn’t exist. Possibly, but only in the context that they are paid to keep their employers servers up and running so that the giant promises made by the marketing team of faster downloads and better online experiences actually happen.
If there wasn’t an internet and there weren’t servers to maintain, they’d have good jobs elsewhere. They’d be building trains or designing buildings or any of the thousands of other jobs that require smart technical people.
Ken has no idea what these folks running the filters and keeping your email alive deal with on a regular basis. They deal with the utter dregs and horrors of society. They are the people dealing with unrelenting spam and virus and phishing attacks bad enough to threaten to take down their networks and the networks of everyone else. They also end up dealing with law enforcement to deal with criminals. Some of what they do is deal with is unspeakable, abuse and mistreatment of children and animals. These are the folks who stand in front of the rest of us, and make the world better for all of us.
They should be thanked for doing their job, not chastised because they’re doing what the people who pay them expect them to be doing.
Yes, recipients want the mail they want. But, y’know, I bet they really don’t want all the bad stuff that the ISPs protect against. Ken took offense at a statement that he really shouldn’t have. ISPs do check their false positive rates on filtering, and those rates are generally less than 1% of all the email that they filter. Marketers should be glad they’re such a small part of the problem. They really don’t want to be a bigger part.