Pattern matching primates

Why do we see faces where there are none? Paradolia
Why do we look at random noise and see patterns? Patternicity
Why do we think we have discovered what’s causing filtering if we change one thing and email gets through?
It’s all because we’re pattern matching primates, or as Michael Shermer puts it “people believe weird things because of our evolved need to believe nonweird things.”
Our brains are amazing and complex and filter a lot of information so we don’t have to think of it. Our brains also fill in a lot of holes. We’re primed at seeing patterns, even when there’s no real pattern. Our brains can, and do, lie to us all the time. For me, some of the important part of my Ph.D. work was learning to NOT trust what I thought I saw, and rather to effectively observe and test. Testing means setting up experiments in different ways to make it easier to not draw false conclusions.
Humans are also prone to confirmation bias: where we assign more weight to things that agree with our preconceived notions.
Take the email marketer who makes a number of changes to a campaign. They change some of the recipient targeting, they add in a couple URLs, they restructure the mail to change the text to image ratio and they add the word free to the subject line. The mail gets filtered to the bulk folder and they immediately jump to the word free as the proximate cause of the filtering. They changed a lot of things but they focus on the word free. 
Then they remove the word free from the subject line and all of a sudden the emails are delivering. Clearly the filter in question is blocking mail with free in the subject line.
Well, no. Not really. Filters are bigger and more complex than any of us can really understand. I remember a couple years ago, when a few of my close friends were working at AOL on their filter team. A couple times they related stories where the filters were doing things that not even the developers really understood.
That was a good 5 or 6 years ago, and filters have only gotten more complex and more autonomous. Google uses an artificial neural network as their spam filter.  I don’t really believe that anything this complex just looks at free in the subject line and filters based on that.
It may be that one thing used to be responsible for filtering, but those days are long gone. Modern email filters evaluate dozens or hundreds of factors. There’s rarely one thing that causes mail to go to the bulk folder. So many variables are evaluated by filters that there’s really no way to pinpoint the EXACT thing that caused a filter to trigger. In fact, it’s usually not one thing. It could be any number of things all adding up to mean this may not be mail that should go to the inbox.
There are, of course, some filters that are one factor. Filters that listen to p=reject requests can and do discard mail that fails authentication. Virus filters will often discard mail if they detect a virus in the mail. Filters that use blocklists will discard mail simply due to a listing on the blocklist.
Those filters address the easy mail. They leave the hard decisions to the more complex filters. Most of those filters are a lot more accurate than we are at matching patterns. Us pattern matching primates want to see patterns and so we find them.
 

Phishing costs company $46 million
A brief history of TXT Records

Related Posts

Thoughts on Gmail filtering

Gmail has some extremely complex filters. They’re machine learning based and measure hundreds of things about incoming mail. The filters are continually adjusting to changes and updating how they treat specific mail.
One consequence of continually adjusting machine learning filters is that filtering is not static. What passes to the inbox now, may not pass in a couple hours.
One of the other challenges with Gmail filters is that they look at all the mail mentioning a particular domain and so affiliate mail and 3rd party mail can affect delivery of corporate mail.
The good news is that continually adjusting filters adapt to positive changes as well as negative ones. In fact, I recently made a segmentation suggestion to a client and they saw a significant increase in inbox delivery at Gmail the next day.
Gmail can be a challenge for delivery, but send mail users want and mail does go to the inbox.

Read More

When spam filters fail

Spam filters aren’t perfect. They sometimes catch mail they shouldn’t, although it happens less than some people think. They sometimes fail to catch mail they should.
One of the reason filters fail to catch mail they should is because some spammers invest a lot of time and energy in figuring out how to get past the filters. This is nothing new, 8 or 9 years ago I was in negotiations with a potential client. They told me they had people who started working at 5pm eastern. Their entire job was to craft mail that would get through Hotmail’s filters that day. As soon as they found a particular message that made it to the inbox, they’d blast to their list until the filters caught up. When the filters caught up, they’d start testing again. This went on all night or until the full list was sent.
Since then I’ve heard of a lot of other filter bypass techniques. Some spammers set up thousands of probe accounts at ISPs and would go through and “not spam” their mail to fool the filters (ISPs adapted). Some spammers set up thousands of IPs and rotate through them (ISPs adapted). Some spammers register new domains for every send (ISPs adapted). Some spammers used botnets (ISPs adapted)
I’m sure, even now, there are spammers who are creating new techniques to get through filters. And the ISPs will adapt.

Read More

What to do when an important email bounces

Some emails are more important than others. I know, I know, all emails are important, but really, some are more important than others.
I’ve recently been decluttering by the simple expedient of enrolling in paperless statements for some of our accounts. We have a 1TB NAS, I’m not going to run out of storage space and I will have so much less paper to deal with. Plus, electronic searches are easier than digging through a file I’ve just shoved statements in for the whole year.
Some companies just let you sign up for statements online and don’t take any extra steps to verify your email address or tell you what happens if your email breaks. But at least one company has gone the extra mile to establish how they handle email bounces.
consentforpaperlessdocs
First, to sign up for paperless notifications I have to give my consent to receive docs. Even better, when I look at the important information it expressly details what happens if my email address bounces.

Read More