Updating the filtering model

One thing I really like about going to conferences is they’re often one of the few times I get to sit and think about the bigger email picture. Hearing other people talk about their marketing experiences, their email experiences, and their blocking experiences usually triggers big picture style thoughts.
Earlier this week I was at Activate18, hosted by Iterable. The sessions I attended were interesting and insightful. Of course, I went to the deliverability session. While listening to the presentation, I realized my previous model of email filtering needed to be updated.

The old model

The old model was pretty simple. It was based on the idea that ISPs ask two fundamental questions about email when it comes in.

  1. Is it safe?
  2. Is it wanted?

If the answer to both those questions is yes, the mail is delivered to the inbox.
Safe is something we don’t talk much about in the marketing space, because generally our mail is safe. But our mail has to go through the exact same filters that are set out to catch the bad guys. And sometimes we do things that trigger that set of filters unintentionally.
The wanted is where marketers can really shine. ISPs look at their user behavior to determine if mail is wanted or not. While the measurements are slightly different than what marketers use, Marketers must also look at how wanted mail is. Engagement with the email is part of it. But, you can also use other metrics you have. Do they visit your website? Are they active on your FB page? What other data do we have that says this person is engaged with your brand and won’t object to increased volume.
Spammers send unwanted mail. Spammers send mail to a lot of addresses where the address owner doesn’t log in. Spammers send mail to people who don’t want it, so they delete it immediately. If you are sending lots of mail and your recipient demographic looks like the typical spammer recipient demographic, then your mail will be treated like spam. It’s ALL about the recipient. That’s what the ISP uses to measure your mail. And if your recipients are reacting to your mail in the same way they react to spammer mail, then you’re going to face deliverability challenges.
Increasing volume is expected, but you need to be strategic with how you increase it. I’m working with one of my clients right now to help them with the final touches on the holiday marketing program. They are increasing mail, but we made some changes to their proposed program to protect them against deliverability challenges. They’re in the early stages of warmup (yes, you need to warmup for significant volume changes!) and it’s looking good.
(That’s from an email I wrote to answer a question on the Only Influencers mailing list back in 2015)

The newer model

This week, though, I realized that another question snuck into the equation. The ISPs are still asking if mail is safe. Does it have harmful content? Phishing? Viruses? Is it coming from a botnet? ISPs use IP reputation and domain/URL reputation to answer the bulk of these questions during the SMTP transaction. If an ISP determines the mail isn’t safe, they’ll reject it out of hand.
But with technology improvements and machine learning, ISPs are able to split the second question into two sub questions.

  1. Is it unsolicited?
  2. Is it wanted by the recipient?

The unsolicited piece was always part of the equation. Many of the metrics used to answer the “is it wanted” question were actually trying to determine if the mail was unsolicited. These measurements include the things we talk about: bounce rate, complaint rate, unknown user rate.  There are two reasons I think this is worth pointing out. The first is that I have often glossed over how much unsolicited mail “legitimate” senders actually send. Every company who purchases a list, who uses lead gen, who collects addresses at point of sale send unsolicited email. They may not mean to, but they do. The second reason is this is the piece of spam filtering that data hygiene companies are addressing. Everyone who cleans your list, identifies bounces, finds bad users, they’re specifically addressing this filtering. And many senders don’t understand why their mail is still going to bulk even after they’ve purchased very expensive hygiene services.
The reality is, that hygiene services make mail look less unsolicited, by removing many of the markers that tell ISPs the mail is unsolicited. But that doesn’t make the mail any more wanted by the recipient. Hence the current focus on engagement and individual delivery metrics. These are metrics that can’t be faked by the sender. Third parties can’t identify those recipients that want a particular piece of mail. And, with privacy laws like GDPR, it’s unlikely those business models will be cost effective.

Related Posts

Do system administrators have too much power?

Yesterday, Laura brought a thread from last week to my attention, and the old-school ISP admin and mail geek in me felt the need to jump up and say something in response to Paul’s comment. My text here is all my own, and is based upon personal experience as well as those of my friends. That said, I’m not speaking on their behalf, either. 🙂
I found Paul’s use of the word ‘SysAdmin’ to be a mighty wide (and — in my experience — probably incorrect) brush to be painting with, particularly when referring to operations at ISPs with any significant number of mailboxes. My fundamental opposition to use of the term comes down to this: It’s no longer 1998.
The sort of rogue (or perhaps ‘maverick’) behavior to which you refer absolutely used to be a thing, back when a clean 56k dial-up connection was the stuff of dreams and any ISP that had gone through the trouble to figure out how to get past the 64k user limit in the UNIX password file was considered both large and technically competent. Outside of a few edge cases, I don’t know many system administrators these days who are able to (whether by policy or by access controls) — much less want to — make such unilateral deliverability decisions.
While specialization may be for insects, it’s also inevitable whenever a system grows past a certain point. When I started in the field, there were entire ISPs that were one-man shows (at least on the technical side). This simply doesn’t scale. Eventually, you start breaking things up into departments, then into services, then teams assigned to services, then parts of services assigned to teams, and back up the other side of the mountain, until you end up with a whole department whose job it is to run one component of one service.
For instance, let’s take inbound (just inbound) email. It’s not uncommon for a large ISP to have several technical teams responsible for the processing of mail being sent to their users:

Read More

AHBL Wildcards the Internet

AHBL (Abusive Host Blocking List) is a DNSBL (Domain Name Service Blacklist) that has been available since 2003 and is used by administrators to crowd-source spam sources, open proxies, and open relays.  By collecting the data into a single list, an email system can check this blacklist to determine if a message should be accepted or rejected. AHBL is managed by The Summit Open Source Development Group and they have decided after 11 years they no longer wish to maintain the blacklist.
A DNSBL works like this, a mail server checks the sender’s IP address of every inbound email against a blacklist and the blacklist responses with either, yes that IP address is on the blacklist or no I did not find that IP address on the list.  If an IP address is found on the list, the email administrator, based on the policies setup on their server, can take a number of actions such as rejecting the message, quarantining the message, or increasing the spam score of the email.
The administrators of AHBL have chosen to list the world as their shutdown strategy. The DNSBL now answers ‘yes’ to every query. The theory behind this strategy is that users of the list will discover that their mail is all being blocked and stop querying the list causing this. In principle, this should work. But in practice it really does not because many people querying lists are not doing it as part of a pass/fail delivery system. Many lists are queried as part of a scoring system.
Maintaining a DNSBL is a lot of work and after years of providing a valuable service, you are thanked with the difficulties with decommissioning the list.  Popular DNSBLs like the AHBL list are used by thousands of administrators and it is a tough task to get them to all stop using the list.  RFC6471 has a number of recommendations such as increasing the delay in how long it takes to respond to a query but this does not stop people from using the list.  You could change the page responding to the site to advise people the list is no longer valid, but unlike when you surf the web and come across a 404 page, a computer does not mind checking the same 404 page over and over.
Many mailservers, particularly those only serving a small number of users, are running spam filters in fire-and-forget mode, unmaintained, unmonitored, and seldom upgraded until the hardware they are running on dies and is replaced. Unless they do proper liveness detection on the blacklists they are using (and they basically never do) they will keep querying a list forever, unless it breaks something so spectacularly that the admin notices it.
So spread the word,

Read More

It's not fair

In the delivery space, stuff comes in cycles. We’re currently in a cycle where people are unhappy with spam filters. There are two reasons they’re unhappy: false positives and false negatives.
False positives are emails that the user doesn’t think is spam but goes into the bulk folder anyway.
Fales negatives are emails that the user does thing is spam but is delivered to the inbox.
I’ve sat on multiple calls over the course of my career, with clients and potential clients, where the question I cannot answer comes up. “Why do I still get spam?”
I have a lot of thoughts about this question and what it means for a discussion, how it should be answered and what the next steps are. But it’s important to understand that I, and most of my deliverability colleagues, hate this question. Yet we get it all the time. ISPs get it, too.
A big part of the answer is because spammers spend inordinate amounts of time and money trying to figure out how to break filters. In fact, back in 2006 the FTC fined a company almost a million dollars for using deceptive techniques to try and get into filters. One of the things this company did would be to have folks manually create emails to test filters. Once they found a piece of text that would get into the inbox, they’d spam until the filters caught up. Then, they’d start testing content again to see what would get past the filters. Repeat.
This wasn’t some fly by night company. They had beautiful offices in San Francisco with conference rooms overlooking Treasure Island. They were profitable. They were spammers. Of course, not long after the FTC fined them, they filed bankruptcy and disappeared.
Other spammers create and cultivate vast networks of IP addresses and domains to be used in snowshoeing operations. Still other spammers create criminal acts to hijack reputation of legitimate senders to make it to the inbox.
Why do you still get spam? That’s a bit like asking why people speed or run red lights. You still get spam because spammers invest a lot of money and time into sending you spam. They’re OK with only a small percentage of emails getting through filters, they’ll just make it up in volume.
Spam still exists because spammers still exist.
 

Read More