Blocklists

What Spamtraps Tell Us

Many blocklists use spamtraps to detect poor sending practices and will cite spamtrap hits as the reason for the blocks. Senders legitimately fear spamtraps showing up on their lists because of this. If spamtraps weren’t used by blocklists no one would really care about them. They’re just another kind of bad address.

Rethinking public blocklists

Recently, a significant majority of discussions of email delivery problems mention that neither the IPs or domains in use are on any of the public blocklists. I was thinking about this recently and realised that, sometime in the past, I stopped using blocklists as a source of useful information about reputation.

Another day another dead blacklist

FADE IN
EMAILGEEKS.SLACK.COM #email-deliverability
It is morning in the channel. The regular crowd is around discussing the usual.
JK, smart, competent head of deliverability at an ESP asks: Anyone familiar with SECTOOR EXITNODES listings and have insight into what’s going on if listed?
ME: Uh, that’s the Tor Exit Nodes list. They think your IP is used by Tor. That’s all sorts of weird. Let me do some digging.
5 minutes of google searches, various dig commands and a visit to the now non-existent sectoor.de website show that the sectoor.de domain expired and is now parked.
ME (back in channel): It looks like the blacklist domain expired and is now parked. So they’re listing the world and nothing to worry about. Not your problem, and not anything you can fix.
JK: Like a UCEProtect fiasco – not just us but everyone?
ME: No, more like the spamcannibal fiasco. The domain expired and so it’s listing the world.
ME: The world would be a better place without MXToolbox worrying about every stupid blocklist. Or even if they would follow the blocklist RFC check for expired domains before panicking the world.
SCENE

Reading between the lines

Reading between the lines an important skill in deliverability.
Why? Over the last few years there’s been an increasing amount of collaboration between deliverability folks at ESPs and ISPs. This is great. It’s a vast improvement on how things were 10 years ago. However, there are still ongoing complaints from both sides. There probably always will be. And it’s not like a blog post from me is going to fix anything. But I see value in talking a bit about how we can improve our ability to collaborate with one another.

Asking for help with a blocklist

There are often questions arising about how to go about getting off a particular blocklist. A few years ago I led the MAAWG effort to document what to if if you were On a Blocklist (pdf link). That document was aimed primarily at MAAWG members and deliverability experts with working knowledge of blocklists. I think, even now, it’s a good background on how to deal with a listing and mail being blocked.

There have been discussions on multiple mailing lists over the last week or so about how to deal with listings at different blocklists. Many folks on these lists have extensive experience, so these are good places to ask. With that being said, a lot of the requests lack sufficient details to help.
So, if you’re ever on a blocklist and want some help from a mailing list about the problem, here’s a short guide for how to ask for help.

Thoughts on filters

One of the questions we received during the EEC16 closing keynote panel was why isn’t there a single blocklist that everyone uses and why don’t ISPs share data more. It would be so much easier for senders if every ISP handled mail the same as every other. But the world isn’t that simple, and it’s not always clear which mail stream is spam and which is good mail.

Just Block It

I tend to go back and forth about reporting spam these days. On one level I know that it’s all a numbers game, and policy enforcement is more about the quantity of complaints than the quality. Knowing this I don’t often send in complaints. I do make a few exceptions: when I know the policy enforcement team or when it’s a current or former client.

Listen to me talk about filtering, blocklists and delivery

I did an interview with Practical eCommerce a few weeks ago. The podcast and transcript are now available.
I want to thank Kerry and the rest of the staff there for the opportunity to talk email and filtering with their readers.
Happy Thanksgiving everyone in the US.

Just go read here…

I wrote earlier this week about bad ways to evaluate and choose an ESP. It was all going to end today in an insightful and profound post telling all of you exactly how to find the best ESP.
Then Smartinsights published an insightful and useful article on choosing an ESP yesterday.
So, yeah, just go read what Jordie has to say. I have a couple other things to add, but I’ll drop those in another post.

I do not think that means what you think it means

Yesterday, I looked at the analysis of ESP delivery done by Mr. Geake. Today we’ll look at some of his conclusions.
“Being blacklisted most likely suggests that sender IP either sends out to a great deal of unknown or angry recipients.” That’s not how most blocklists work. Most blocklists are driven by spam traps or by the personal mailboxes of the list maintainers. The only blocklist that took requests from the public was the old MAPS RBL, and I don’t believe that is the case any longer.
Blocking at ISPs is often a sign of sending out a lot of mail to unknown or angry / unengaged recipients. But most ISPs don’t make their lists public. Some allow anyone to look up IP addresses, and if we had the IPs we could check. But we don’t, so we can’t.
“[…] if you share this IP with Phones4U then only 62% of your emails will be accepted by a recipient’s email server. That’s before they hit the junk filter. I wouldn’t want to pay for that.” This conclusion relies on the Sender Score “accepted rate” number. Accepted Rate is a figure I don’t rely on for much. I’ve never been able to reconcile this number with what client logs tell me about accepted rate. For instance, I have one IP address that has a 4.4% acceptance rate. But I know that 19 out of 20 emails from this IP do not bounce. In fact, it’s rare to see any mail from this IP bounce.
The one thing that Mr. Geake gets right, in all of this, is that if you’re on a shared IP address with a poor sender, then you share that sender’s reputation. Their reputation can hurt your delivery.
But a dedicated IP isn’t always your best bet, either. Smaller senders may not have the volume or frequency required to develop and keep a good reputation on an static IP. In these cases, sharing an IP address with similar senders may actually increase delivery.
For some senders outsourcing the email expertise is a better use of resources than dedicating a person to managing email delivery. For other senders, bringing mail in house and investing in staff to manage email marketing is better.
Tomorrow: how do you really evaluate an ESP?

Twisting information around

One of my mailing lists was asking questions today about an increase in invitation mailings from Spotify. I’d heard about them recently, so I started digging through my mailbox to see if I’d received one of these invites. I hadn’t, but it clued me into a blog post from early this year that I hadn’t seen before.
Research: ESPs might get you blacklisted.
That article is full of FUD, and the author quite clearly doesn’t understand what the data he is relying on means. He also doesn’t provide us with enough information that we can repeat what he did.
But I think his take on the publicly available data is common. There are a lot of people who don’t quite understand what the public data means or how it is collected. We can use his post as a starting off point for understanding what publicly available data tells us.
The author chooses 7 different commercial mailers as his examples. He claims the data on these senders will let us evaluate ESPs, but these aren’t ESPs. At best they’re ESP customers, but we don’t know that for sure. He claims that shared IPs means shared reputation, which is true. But he doesn’t claim that these are shared IPs. In fact, I would bet my own reputation on Pizza Hut having dedicated IP addresses.
The author chooses 4 different publicly available reputation services to check the “marketing emails” against. I am assuming he means he checked the sending IP addresses because none of these services let you check emails.
He then claims these 4 measures

Blocklist changes

Late last year we wrote about the many problems with SORBS. One of the results of that series of posts was a discussion between a lot of industry professionals and GFI executives. A number of problems were identified with SORBS, some that we didn’t mention on the blog. There was an open and free discussion about solutions.
A few months ago, there were a bunch of rumors that GFI had divested themselves from SORBS. There were also rumors that SORBS was purchased by Proofpoint. Based on publicly available information many of us suspected that GFI was no longer involved in SORBS. Yet other information suggested that Proofpoint may truly have been the purchaser.
This week those rumors were confirmed.

Are blocklists always a good decision?

One of the common statements about blocklists is that if they have bad data then no one will use them. This type of optimism is admirable. But sadly, there are folks who make some rather questionable decisions about blocking mail.
We publish a list called nofalsenegatives. This list has no website, no description of what it does, nothing. But the list does what it says it does: if you use nofalsenegatives against your incoming mailstream then you will never have to deal with a false negative.
Yes. It lists every IP on the internet.
The list was set up to illustrate a point during some discussion many years ago. Some of the people who were part of that discussion liked the point so much that they continued to mention the list. Usually it happens when someone on a mailing list complained about how their current spamfiltering wasn’t working.
Some of the folks who were complaining about poor filtering, including ones who should know better, did actually install nofalsenegatives in front of their mailserver. And, thus, they blocked every piece of mail sent to them.
To be fair, usually they noticed a problem within a couple hours and stopped using the list.
This has happened often enough that it convinced me that not everyone makes informed decisions about blocking. Sure, these were usually small mailservers, with maybe a double handful of users. But these sysadmins just installed a blocklist, with no online presence except a DNS entry, without asking questions about what it does, how it works or what it lists.
Not everyone makes sensible decisions about blocking mail. Our experience with people using nofalsenegatives is just one, very obvious, data point.

You can't always get what you want

It’s a problem anyone who has done any delivery work has faced. There’s a client who is having blocklist problems or ISP delivery problems and they won’t pay any attention to what you say. They insist that you talk to the blocklist or the ISP or hand over contacts directly so they can “dialog with” someone internally. They don’t like what they’re hearing, and they hope that the answer will be different if they find a new person to talk to.
The reality is many of the people at ISPs and blocklists don’t want to talk to these types of senders. They may answer a friendly question from someone they know and trust, but sometimes not even then.
Some very large ISPs and major blocklists don’t even take sender questions. They won’t communicate with anyone about any delivery issues.
I’ve had to tell more than a few clients recently that various ISPs and blocklists weren’t interested in helping those clients with their delivery problems. There are two classes of reactions I get from clients. Some clients focus on moving forward. “OK, now what? How can we identify the issue, what data do we have and how can we figure out what the problem is?”
Other clients continue to look for ways to talk to whomever is blocking their mail. They’re convinced if they can just “explain their business model” or be told what they’re doing wrong, that all their delivery problems will magically disappear.
Needless to say those clients who focus on moving forward and looking at the information they do have have much better success resolving their delivery problems. What many senders don’t understand is the wealth of data they have that will help them resolve the issue. And even if they know it’s buried in their files, they don’t always know where to start looking or even what they’re looking for.
But that is, of course, why you hire someone like me who understands spamfiltering and email. I help senders understand how email filters work and identify what parts of their programs are likely to be responsible for delivery issues. I often find the most valuable service I provide to clients is a fresh set of eyes that can see the forest. With my help, they manage to stop obsessing unproductively about one particular symptom and focus on the underlying problems.
Senders who think the holy grail of problem resolution is speaking to the right person at an ISP or blocklist generally are disappointed, even when they hire someone who knows all the right people at the ISPs. They can’t always get what they want. But I can often help them get what they need.

Blocklist BCP

As many of you may be aware there is a draft document working its way through the Internet Research Task Force (IRTF) discussing best common practices for blocklists. The IRTF is a parallel organization to the IETF and is charged with long term research related to the Internet. The Anti-Spam Working Group was chartered to investigate tools and techniques for dealing with spam.
Recently the ASRG posted a draft of a best practices document aimed at those running blocklists (draft-irtf-asrg-bcp-blacklists-07). This document has been under development for many years. The authors have used this document to share their experiences with running blocklists and their knowledge of what works and what doesn’t.
Best practices documents are never easy to write and consensus can be difficult. But I think that the authors did a good job capturing what the best practices are for blocklists. I do support the document in principle and, in fact, support many of the specific statements and practices outlined there. As with any best practices documents it’s not perfect but overall it reflects the current best practices for blocklists.
Ken Magill’s article about the BCP
Anti-Abuse buzz article about the BCP

Blocklists, delisting and extortion

As I’m sure many of you have heard by now there is a new blocklist called ‘nszones.’ This blocklist is apparently stealing data from a number of other publicly accessible blocklists, combining the data and then charging folks for delisting.
This is a scam attempting to extort money from people. The blocklist has no way to actually remove IPs from the parent zones and I’m pretty sure they won’t even remove IPs from their own zones. In this case, the blocklist is clearly a scam, but there are other lists that are actually used by some mailservers that do charge for removal.
No legitimate blocklist will ever expect a listee to pay for delisting. Ever.
I feel very strongly about this. In fact, one of the major blocklists is run off a domain owned by Word to the Wise. Occasionally, I get contacted by folks looking for help with a listing on that list and I will not take them on as a client. I will provide general advice and make sure that they are correctly contacting the blocklist but nothing more.
This is, to my mind, the only ethical thing to do. I don’t even want a hint of impropriety surrounding either myself or the blocklist. Charging money for delisting only feeds the conspiracy theories.
Charging listees for removal (or listing listees so those charges can be a revenue source) is likely to lead to poor quality data and a blocklist that’s not terribly accurate nor effective. Furthermore, if a list operator is unethical or confrontational in their interactions with listees, they’re probably equally unprofessional in their interactions with potential list users. This results in few recipient domains actually using the list to block mail. Lists that charge are not widely used and being listed on them often does not affect email delivery in any appreciable manner.