Feedback loops

There are a lot of different perspectives on Feedback Loops (FBLs) and “this is spam” buttons across the email industry.
Some people think FBLs are the best thing since sliced bread and can’t figure out why more ISPs don’t offer them. These people use use the data to clean addresses off their lists, lower complaints and send better mail. They use the complaints as a data source to help them send mail their recipients want. Too many recipients opted out on a particular offer? Clearly there is a problem with the offer or the segmentation or something.
Other people, though, think the existence of “this is spam” buttons and FBLs is horrible.  They call people who click “this is spam” terrorists or anti-commerce-net-nazis. They want to be able to dispute every click of the button. They think that too many ISPs offer this is spam buttons and too many ESPs and network providers pay way to much attention to complaints. The argue ISPs should remove these buttons and stop paying attention to what recipients think.
Sadly, I’m not actually making up the terminology in the last paragraph. There really are who think that the problem isn’t with the mail that they’re sending but that the recipients can actually express an opinion about it and the ISPs listen to those opinions. “Terrorists” and “Nazis” are the least of the things they have called people who complain about their mail.
One of the senior engineers at Cloudmark recently posted an article talking about FBLs and “this is spam” buttons. I think it’s a useful article to read as it explains what value FBLs play in helping spam filters become more accurate.

We have found over time that the most effective systems are those that learn to classify undesirable content based on feedback from users. The user is truly the best judge of what is and isn’t spam. The faster that consistent feedback becomes available, the sooner a filter can be re-trained to detect and respond to new attacks.

Spam filters are not there to stymie senders, their real job is to protect recipients. Faster adapting filters may seem like a bad thing, but they adapt in both directions. Send good mail that people want and it gets through. Send bad mail that people complain about and filters quickly adapt to block it.

Related Posts

Why do ISPs do that?

One of the most common things I hear is “but why does the ISP do it that way?” The generic answer for that question is: because it works for them and meets their needs. Anyone designing a mail system has to implement some sort of spam filtering and will have to accept the potential for lost mail. Even the those recipients who runs no software filtering may lose mail. Their spamfilter is the delete key and sometimes they’ll delete a real mail.
Every mailserver admin, whether managing a MTA for a corporation, an ISP or themselves inevitably looks at the question of false positives and false negatives. Some are more sensitive to false negatives and would rather block real mail than have to wade through a mailbox full of spam. Others are more sensitive to false positives and would rather deal with unfiltered spam than risk losing mail.
At the ISPs, many of these decisions aren’t made by one person, but the decisions are driven by the business philosophy, requirements and technology. The different consumer ISPs have different philosophies and these show in their spamfiltering.
Gmail, for instance, has a lot of faith in their ability to sort, classify and rank text. This is, after all, what Google does. Therefore, they accept most of the email delivered to Gmail users and then sort after the fact. This fits their technology, their available resources and their business philosophy. They leave as much filtering at the enduser level as they can.
Yahoo, on the other hand, chooses to filter mail at the MTA. While their spamfoldering algorithms are good, they don’t want to waste CPU and filtering effort on mail that they think may be spam. So, they choose to block heavily at the edge, going so far as to rate limit senders that they don’t know about the mail. Endusers are protected from malicious mail and senders have the ability to retry mail until it is accepted.
The same types of entries could be written about Hotmail or AOL. They could even be written about the various spam filter vendors and blocklists. Every company has their own way of doing things and their way reflects their underlying business philosophy.

Read More

Light blogging for a while

Sorry for the lack of substantive posts, things seem to have gone completely out of control and I’m not finding a lot of extra cycles to sit down and blog. I’ll try and get some stuff up this week, but I’m also getting ready for MAAWG and the sessions I’m a part of there.
There was an interesting post by Romer over on his personal blog. If you don’t know, Romer helps maintain one of the commercial mail filters. He recently got spammed by one of his vendors and talked about how this is probably not the best idea. Al adds his own take on companies assuming permission. I’ve talked about taking permission in the past but haven’t touched on things like “spamming the guy who runs the filter.”
You’d be surprised, or maybe you wouldn’t, about how many people who run filters for large organizations get spammed regularly. You wouldn’t be surprised to find out that those people do factor in their own personal spam load when adjusting their organizational filters.

Read More

Content based filters

Content based filters are incredibly complex and entire books could be written about how they work and what they look at. Of course, by the time the book was written it would be entirely obsolete. Because of their complexity, though, I am always looking for new ways to explain them to folks.
Content based filters look at a whole range of things, from the actual text in the message, to the domains, to the IP addresses those domains and URLs point to. They look at the hidden structure of an email. They look at what’s in the body of the message and what’s in the headers. There isn’t a single bit of a message that content filters ignore.
Clients usually ask me what words they should change to avoid the filters. But this isn’t the right question to ask. Usually it’s not a word that causes the problem. Let me give you a few examples of what I mean.
James H. has an example over on the Cloudmark blog of how a single missing space in an email caused delivery problems for a large company. That missing space changed a domain name in the message sufficiently to be caught by a number of filters. This is one type of content filter, that focuses on what the message is advertising or who the beneficiary of the message is. Some of my better clients get caught by these types of filters occasionally. A website they’re linking to or a domain name they’re using in the text of the message has a bad reputation. The mail gets bulked or blocked because of that domain in the message.
One of my clients went from 100% inbox every day to random failures at different domains. Their overall inbox was still in the 96 – 98% range, but there was a definite change. The actual content of their mail hadn’t changed, but we kept looking for underlying causes. At one point we were on the phone and they mentioned their new content management system. Sure enough, the content management company had a poor reputation and the delivery problems started exactly when they started using the content management. The tricky part of this was that the actual domains and URLs in the messages never changed, they were still clickthrough.clientdomain.example.com. But those URLs now pointed to an IP address that a lot of spammers were abusing. So there were delivery problems. We made some changes to their setup and the delivery problems went away.
The third example is one from quite a long time ago, but illustrates a key point. A client was testing email sends through a new ESP. They were sending one-line mail through the ESPs platform to their own email account. Their corporate spamfilter was blocking the mail. After much investigation and a bit of string pulling, I finally got to talk to an engineer at the spamfiltering company. He told me that they were blocking the mail because it “looked like spam.” When pressed, he told me they blocked anything that had a single line of text and an unsubscribe link. Once the client added a second line of text, the filtering issue went away.
These are just some of the examples of how complex content based filters are. Content is almost a misnomer for them, as they look at so many other things including layout, URLs, domains and links.

Read More