Gmail suddenly puts mail in the bulk folder

One of the delivery challenges that regularly comes up in various delivery discussion spaces is the “Gmail suddenly put my mail in spam.” From my perspective, there is rarely a “suddenly” about Gmail’s decision making process.

As I was answering one of these questions I had a number of thoughts. I’ll share them here on the blog so I can find them in the future.

The first thing that occurred was that I’d shifted my thinking to considering Gmail filtering, in particular, as a filter based on a mailstream, not on a message. Thus my tweet:

The other was a realization that most people don’t consider how what they’re doing might be not OK, but might not be bad enough to be filtered this time. That not OK behaviour builds up over time and eventually tips into mail being filtered. Not because of one message, but because of the dozen or 40 or 75 or hundred messages behind it.

Gmail is really good about just watching and monitoring and watching and scoring. They measure mainstreams over time, and reputation is the sum of all your sends. There may be nothing different or new about a particular send, but Gmail’s been seeing overall reputation decrease just a little bit every time you send.

To put it more succinctly: Senders see themselves “doing the same things” and not realising that every time they do this, they’re slowly eroding away at their reputation with Gmail. Once the erosion hits the tipping point

As I explained it on Facebook.

The effect (moving mail to spam) happens suddenly. The cause builds up over time. Gmail is really good about just watching and monitoring and watching and scoring. They measure mainstreams over time, they don’t really measure individual sends.

What happens is that Gmail starts moving mail sent to people who don’t engage with it to the bulk folder. Your open rates don’t change because these people aren’t opening the mail anyway. But every message delivered to the bulk folder is a ding on your reputation. Those dings build up, until your reputation hits a tipping point and all mail to some of your engaged users sometimes goes to bulk. You might see a small decrease in open rates, but nothing major. You continue mailing as you are and then “all of a sudden” your mail is going to bulk. But it’s just mail to the people who were receiving it in the inbox before, the majority of your mail was always going to bulk. 

This is why any changes you might make to mail, changing an IP or a domain name or using a slightly different format, can sometimes work for a little while. It’s basically decoupling that individual message from the broader history of sending. So you can keep doing the same things over and over again and not hit the inbox.

If, however, you don’t fix the underlying problem, one of two things will happen. Google will connect the new mailstream with the old mailstream and your reputation will fall to the old reputation in a few days. Alternatively, Google won’t connect the new mailstream with the old mailstream and you’ll have many months to slowly erode the reset reputation. In either case it’s just a matter of time before you end up back in the bulk folder again.

Gmail looks at the whole message stream when making decisions. In order to improve delivery at gmail we must also look at the whole message stream.

Related Posts

Why do ISPs do that?

One of the most common things I hear is “but why does the ISP do it that way?” The generic answer for that question is: because it works for them and meets their needs. Anyone designing a mail system has to implement some sort of spam filtering and will have to accept the potential for lost mail. Even the those recipients who runs no software filtering may lose mail. Their spamfilter is the delete key and sometimes they’ll delete a real mail.
Every mailserver admin, whether managing a MTA for a corporation, an ISP or themselves inevitably looks at the question of false positives and false negatives. Some are more sensitive to false negatives and would rather block real mail than have to wade through a mailbox full of spam. Others are more sensitive to false positives and would rather deal with unfiltered spam than risk losing mail.
At the ISPs, many of these decisions aren’t made by one person, but the decisions are driven by the business philosophy, requirements and technology. The different consumer ISPs have different philosophies and these show in their spamfiltering.
Gmail, for instance, has a lot of faith in their ability to sort, classify and rank text. This is, after all, what Google does. Therefore, they accept most of the email delivered to Gmail users and then sort after the fact. This fits their technology, their available resources and their business philosophy. They leave as much filtering at the enduser level as they can.
Yahoo, on the other hand, chooses to filter mail at the MTA. While their spamfoldering algorithms are good, they don’t want to waste CPU and filtering effort on mail that they think may be spam. So, they choose to block heavily at the edge, going so far as to rate limit senders that they don’t know about the mail. Endusers are protected from malicious mail and senders have the ability to retry mail until it is accepted.
The same types of entries could be written about Hotmail or AOL. They could even be written about the various spam filter vendors and blocklists. Every company has their own way of doing things and their way reflects their underlying business philosophy.

Read More

Thoughts on Gmail and the inbox

Over the last few months more and more marketers are finding their primary delivery challenge is the Gmail inbox. I’ve been thinking about why Gmail might be such a challenge for marketers. Certainly I have gotten a lot of calls from people struggling to figure out how to get into the Gmail inbox. I’ve also seen aggressive domain based filtering from Gmail, where any mention of a particular domain results in mail going to the bulk folder.
It’s one of those things that’s a challenge, because in most of these cases there isn’t one cause for bulk foldering. Instead there’s a whole host of things that are individually very small but taken together convince Gmail that the mail doesn’t need to be in the inbox.
A pattern that I’m starting to see is that Gmail is taking a more holistic look at all the mail from a sender. If the mail is connected to an organization, all that mail is measured as part of their delivery decision making. This is hurting some ESPs and bulk senders. I’ve had multiple ESPs contact me in the last 6 months looking for help because all their customer emails are going to bulk folder.
Gmail’s filtering is extremely aggressive. From my perspective it always has been. I did get an invite for a Gmail account way back in the day. I moved a couple mailing lists over to that account to test it with some volume and discussion lists. I gave up not long after because no matter what I did I couldn’t get gmail to put all the mail from that list into the tag I had set up for it. Inevitably some mail from some certain people would end up in my spam folder.
Gmail has gotten better, now they will let you override their filters but give you a big warning that the message would have been delivered to spam otherwise.
Gmail_NotSpam
What are mailers to do? Right now I don’t have a good answer. Sending mail people want is still good advice for individual senders. But I am not sure what can be done about this ESP wide filtering that I’m starting to see. It’s possible Gmail is monitoring all the mail from a particular sender or ESP and applying a “source network” score. Networks letting customers send mail Gmail doesn’t like (such as affiliate mail or payday mail, things they mentioned specifically at M3AAWG) are having all their customers affected.
I suspect this means that ESPs seeing problems across their customer base are going to have to work harder to police their customers and remove problematic mail streams completely. Hopefully, ESPs that can get on the Gmail FBL can identify the problem customers faster before those customers tank mail for all their senders.

Read More

Filtering by gestalt

One of those $5.00 words I learned in the lab was gestalt. We were studying fetal alcohol syndrome (FAS) and, at the time, there were no consistent measurements or numbers that would drive a diagnosis of FAS. Diagnosis was by gestalt – that is by the patient looking like someone who had FAS.
It’s a funny word to say, it’s a funny word to hear. But it’s a useful term to describe the future of spam filtering. And I think we need to get used to thinking about filtering acting on more than just the individual parts of an email.

Filtering is not just IP reputation or domain reputation. It’s about the whole message. It’s mail from this IP with this authentication containing these URLs.  Earlier this year, I wrote an article about Gmail filtering. The quote demonstrates the sum of the parts, but I didn’t really call it out at the time.

Read More