Gmail, machine learning, filters

laura
Industry
February 18, 2019

I’m sure by now readers have seen the article from Gmail “Spam does not bring us joy — ridding Gmail of 100 million more spam messages with TensorFlow.” If you haven’t seen it, go read it. It’s not often companies write about their filtering philosophy and what tools they’re using to manage incoming bad mail.

There were a few parts of the article that confirmed some of my theories about Gmail and a few things that were unexpected.

Open source tools

It’s no surprise that Google uses a machine learning engine built in house. What I didn’t know was it was called TensorFlow and was open sourced by Google. Many companies in the email space open source some of their tools. Exacttarget open sourced FuleUX long before they were SFMC and maintain a GitHub account with a number of tools. Mailchimp also maintains an account with their open source code. Steve releases a bunch of tools and code he writes both for work and for fun.

Open source software runs a whole lot more of the internet than many people know. Some of the primary contributors do the work on their own time. But many companies, large and small, understand how vital open source tools are to their business. They hire and support open source developers to maintain and extend the software.

Catching the hard spam

Google catches a lot of spam, and they’re always trying to catch the stuff that falls through the cracks. My recent call volume about going to spam at Gmail told me that Gmail had implemented some new filters. Many people were telling me that things were fine and then, with no change in what they were doing, mail started going to bulk. Other delivery folks were also talking about their customers getting caught up in filters.

We’ve gotten to the point, particularly with Google but also with the other webmail providers, where the bulk of egregious spam is blocked. What’s left is not some spammer sending 10MM messages, but a much more difficult problem. Spam that reaches the inbox is sent in much smaller quantities. It’s also heavily targeted. Spammers are trying to look like legitimate marketers but still sending mail without permission.

This targeted spam is something I’ve been thinking about a lot lately. Mostly because anti-spammers did a pretty good job making not-spamming look like it was beneficial to senders. Many deliverability recommendations boil down to stop spamming but phrased in a way that makes the advice more palatable. Much of the type of spam that’s getting caught in the new filters follows deliverability recommendations. The piece it misses is that it’s not being sent with the permission of the recipient.

Believe it or not, spam filters started out as protecting users from mail they didn’t ask for. As the internet as grown and email has become a channel for crime the focus of filters have changed. But, fundamentally, deep down, the original purpose of keeping mail boxes useful by stopping unsolicited mail is still there. The ML filters are giving Google, and others, tools to actually address that mail better.

The trend is clear. Filters are getting more an more able to address unsolicited email in a complex sender and user environment. Machine learning is driving a lot of that, and Google is at the front of the pack. They’re doing their best to stop the small scale spammers that have avoided a lot of the last generation of filters.

Filtering is not just about spam

A lot of filters started out just as filters against spam. But over the years they’ve morphed into more general blocks against dangerous or problematic email. There’s a lot of crime and bad behavior on the internet, much of it using email as a conduit or vector. Filtering is so much more than stopping spam now. It’s as much, or more, about stopping crime.
Email filters are essential to protect us from scammers. Sometimes I forget this, and then I read about a grandmother getting swindled by a Nigerian scammer and ending up dead.
There are real consequences to poor filtering and there is real crime facilitated by email. It’s easy to forget this as we deal with the email that gets caught in filters when they shouldn’t.
Filters are one of the first lines of defense against online crime.
Not only does filtering stop crime, but they also keep email working. An unfiltered mail stream is an ugly, unreadable, unworkable mess.

Permission and B2B spam

Two of the very first posts I wrote on the blog were about permission (part 1, part 2). Re-reading those posts is interesting. Experience has taught me that recipients are much more forgiving of implicit opt-in than that post implies.
The chance in recipient expectations doesn’t mean, however, that permission isn’t important or required. In fact, The Verge reported on a chatbot that will waste the time of spammers. Users who are fed up with spam can forward their message to Re:Scam and bots will answer the mail.
I cannot tell you how tempted I am to forward all those “Hey, just give me 10 minutes of your time…” emails I get from B2B spammers. I know, those are actually bots, but there is lovely symmetry in bots bothering one another and leaving us humans out of it.

Speaking of those annoying emails, I tweeted about one (with horrible English…) last week. I tagged the company in question and they asked for an example. After I sent it, they did nothing, and I continued to get mail. Because of course I did.
These types of messages are exactly why permission is so critical for controlling spam. Way more companies can buy my email address and add me to their spam automation software than I can opt-out of in any reasonable time frame. My inbox, particularly my business inbox, is where I do business. It’s where I talk with clients, potential clients, customers and, yes, even vendors. But every unsolicited email wastes my time.
It’s not even that the mail is simply unwanted. I get mail I don’t want regularly. Collecting white papers for my library, RSVPing to events, joining webinars all result in me getting added to companies’ mailing lists. That’s fair, I gave them an email address I’ll unsubscribe.
The B2B companies who buy my address are different. They’re spamming and they understand that. The vendors who sell the automation filters tell their customers how to avoid spam filters. Spammers are told to use different domains for the unsolicited mail and their opt-in mail to avoid blocking. The software plugs into Google and G Suite account because very few companies will block Google IPs.
I’ve had many of these companies attempt to pay me to fix their delivery problems. But, in this case there’s nothing to fix. Yes, your mail is being blocked. No, I can’t help. There is nothing I can say to a filtering company or ISP or company to make them list that block. The mail is unwanted and it’s unsolicited.
The way to get mail unblocked is to demonstrate the mail is wanted. If you can’t do that, well, the filters are working as intended.

Filters working as intended

One of the toughest deliverability problems to deal with is when mail is blocked or going to spam because the filters are working as intended. Often the underlying issue is a lack of permission.