Pattern matching primates

Why do we see faces where there are none? Paradolia
Why do we look at random noise and see patterns? Patternicity
Why do we think we have discovered what’s causing filtering if we change one thing and email gets through?
It’s all because we’re pattern matching primates, or as Michael Shermer puts it “people believe weird things because of our evolved need to believe nonweird things.”
Our brains are amazing and complex and filter a lot of information so we don’t have to think of it. Our brains also fill in a lot of holes. We’re primed at seeing patterns, even when there’s no real pattern. Our brains can, and do, lie to us all the time. For me, some of the important part of my Ph.D. work was learning to NOT trust what I thought I saw, and rather to effectively observe and test. Testing means setting up experiments in different ways to make it easier to not draw false conclusions.
Humans are also prone to confirmation bias: where we assign more weight to things that agree with our preconceived notions.
Take the email marketer who makes a number of changes to a campaign. They change some of the recipient targeting, they add in a couple URLs, they restructure the mail to change the text to image ratio and they add the word free to the subject line. The mail gets filtered to the bulk folder and they immediately jump to the word free as the proximate cause of the filtering. They changed a lot of things but they focus on the word free. 
Then they remove the word free from the subject line and all of a sudden the emails are delivering. Clearly the filter in question is blocking mail with free in the subject line.
Well, no. Not really. Filters are bigger and more complex than any of us can really understand. I remember a couple years ago, when a few of my close friends were working at AOL on their filter team. A couple times they related stories where the filters were doing things that not even the developers really understood.
That was a good 5 or 6 years ago, and filters have only gotten more complex and more autonomous. Google uses an artificial neural network as their spam filter.  I don’t really believe that anything this complex just looks at free in the subject line and filters based on that.
It may be that one thing used to be responsible for filtering, but those days are long gone. Modern email filters evaluate dozens or hundreds of factors. There’s rarely one thing that causes mail to go to the bulk folder. So many variables are evaluated by filters that there’s really no way to pinpoint the EXACT thing that caused a filter to trigger. In fact, it’s usually not one thing. It could be any number of things all adding up to mean this may not be mail that should go to the inbox.
There are, of course, some filters that are one factor. Filters that listen to p=reject requests can and do discard mail that fails authentication. Virus filters will often discard mail if they detect a virus in the mail. Filters that use blocklists will discard mail simply due to a listing on the blocklist.
Those filters address the easy mail. They leave the hard decisions to the more complex filters. Most of those filters are a lot more accurate than we are at matching patterns. Us pattern matching primates want to see patterns and so we find them.
 

Related Posts

Language as filtering criteria

A few months ago I was working on a delivery audit for a client who sends mail in multiple languages. We discovered that the language of an email has a significant delivery impact. The same email in different languages was delivered differently, particularly at Gmail. Emails in a language I don’t normally receive email in were delivered to my bulk folder.
Other folks have commented on similar things. Some filters really do look at preferred language of the recipient and treat mail in other languages as problematic. I don’t think that’s unreasonable. I do get a lot of foreign language spam and there’s no real way to stop it. Many countries don’t require opt-out links, and so there isn’t a clear way to even unsubscribe.
Writing in the recipient’s local language is one way to minimize inappropriate blocking, even when you have permission to send mail.
 
 

Read More

Dodging filters makes for effective spamming

Spam is still 80 – 90% of global email volume, depending on which study look at. Most of that spam doesn’t make it to the inbox; ISPs reject a lot of it during the SMTP transaction and put much of rest of it in the bulk folder. But as the volumes of spam have grown, ISPs and filters are relying more and more on automation. Gone are the days when a team of people could manually review spam and tune filters. There’s just too much of it out there for it to be cost effective to manually review filters.
In some ways, though, automatic filters are easier to avoid than manual filters. Take a spam that I received at multiple addresses today. It’s an advertisement for lists to “meet my marketing needs.” I started out looking at this mail to walk readers through all of the reasons I distrusted this mail. But some testing, the same sorts of testing I do for client mails, told me that this mail was making it to the inbox at major ISPs.
What told me this mail was spam? Let’s look at the evidence.
listsellingspam_thumb

Read More

Email problems are costly

Last week Zulily released their quarterly earnings. Their earnings’ report was disappointing, resulting in a drop in their stock prices. The chairman of the company told reporters on a conference call that part of the reason for the drop in earnings were due to deliverability problems “at a large ISP.”

Read More