Blocklist

Terminology

There is a lot more to say here, and I’m working on a longer post to really talk about the underlying racism in tech and how we as an industry have failed.

Recently, a significant majority of discussions of email delivery problems mention that neither the IPs or domains in use are on any of the public blocklists. I was thinking about this recently and realised that, sometime in the past, I stopped using blocklists as a source of useful information about reputation.

It’s a new year, do you know what your filters are doing?

Yesterday the NJABL domain expired. The list was disabled back in 2013 but the domain continued to be maintained as a live domain. With the expiration, it was picked up by domain squatters and is now listing everything. Steve wrote about how and why expired blocklist domains list the world last year.

Spamhaus DBL

Over the last few months I’ve gotten an increasing number of questions about the Spamhaus DBL. So it’s probably time to do a blog post about it.

Thoughts on policy

A particular blocklist, once again, listed a major ESP this week. Their justification is “this is our policy.” Which is true, it is their policy to list under these circumstances. That doesn’t make it a good policy, or even an effective policy. It’s simply a policy.

Another day another dead blacklist

FADE IN
EMAILGEEKS.SLACK.COM #email-deliverability
It is morning in the channel. The regular crowd is around discussing the usual.
JK, smart, competent head of deliverability at an ESP asks: Anyone familiar with SECTOOR EXITNODES listings and have insight into what’s going on if listed?
ME: Uh, that’s the Tor Exit Nodes list. They think your IP is used by Tor. That’s all sorts of weird. Let me do some digging.
5 minutes of google searches, various dig commands and a visit to the now non-existent sectoor.de website show that the sectoor.de domain expired and is now parked.
ME (back in channel): It looks like the blacklist domain expired and is now parked. So they’re listing the world and nothing to worry about. Not your problem, and not anything you can fix.
JK: Like a UCEProtect fiasco – not just us but everyone?
ME: No, more like the spamcannibal fiasco. The domain expired and so it’s listing the world.
ME: The world would be a better place without MXToolbox worrying about every stupid blocklist. Or even if they would follow the blocklist RFC check for expired domains before panicking the world.
SCENE

SpamCannibal is dead

The SpamCannibal blacklist – one that didn’t affect your email too much but which would panic users who found it on one of the “check all the blacklists!” websites – has gone away.
It was silently abandoned by the operator at some point in the past year and the domain registration has finally expired. It’s been picked up by domain squatters who, as usual, put a wildcard DNS record in for the domain causing it to list the entire internet.
Al has more details over at dnsbl.com.
If you run a blacklist, please don’t shut it down this way. Read up on the suggested practice in RFC 6471. If you just can’t cope with that consider asking people you know in the industry for help gracefully shutting it down.
Blacklist health checks
If you develop software that uses blacklists, include “health check” functionality. All relevant blacklists publish records that show they’re operating correctly. For IP based blacklists that means that they will always publish “127.0.0.2” as listed and “127.0.0.1” as not listed. You should regularly check those two IP addresses for each blacklist and if 127.0.0.1 is listed or 127.0.0.2 isn’t listed immediately disable use of that list (and notify whoever should know about it).
For IPv6 blacklists the always listed address is “::FFFF:7F00:2” and the never listed address is “::FFFF:7F00:1”. For domain-based blacklists the always listed hostname is “TEST” and the never listed hostname is “INVALID”. See RFC 5782 for more details. (And, obviously, check that the blacklists your software supports out of the box actually do implement this before turning it on).
If you use someone else’s blacklist code, ask them about their support for health checks. If your mail filter doesn’t use them you risk either suddenly having all your mail go missing (for naive blacklist based blocking) or having some fraction of wanted mail being delivered to your spam folder (for scoring based filters).

UCEProtect and GDPR fallout

First thing this morning I got an email from a client that they were listed on the UCEProtect Level 3 blacklist. Mid-morning I got a message from a different client telling me the same thing. Both clients shared their bounce messages with me:

What kind of mail do filters target?

All to often we think of filters as a linear scale. There’s blocking on one end, and there’s an inbox on the other. Every email falls somewhere on that line.
Makes sense, right? Bad mail is blocked, good mail goes to the inbox. The bulk folder exists for mail that’s not bad enough to block, but isn’t good enough to go to the inbox.
Once we get to that model, we can think of filters as just different tolerances for what is bad and good. Using the same model, we can see aggressive filters block more mail and send more mail to bulk, while letting less into the inbox. There are also permissive filters that block very little mail and send most mail to the inbox.
That’s a somewhat useful model, but it doesn’t really capture the full complexity of filters. There isn’t just good mail and bad mail. Mail isn’t simply solicited or unsolicited. Filters take into account any number of factors before deciding what to do with mail.

Reading between the lines

Reading between the lines an important skill in deliverability.
Why? Over the last few years there’s been an increasing amount of collaboration between deliverability folks at ESPs and ISPs. This is great. It’s a vast improvement on how things were 10 years ago. However, there are still ongoing complaints from both sides. There probably always will be. And it’s not like a blog post from me is going to fix anything. But I see value in talking a bit about how we can improve our ability to collaborate with one another.

January 2017: The Month in Email

Between client work and our national political climate, it’s been a very busy month around here and blogging has been light. Things show no sign of slowing down in February, so we’d love to hear from you with questions and suggestions of what you’d most like to see us focus on in our limited blogging time this month. We got a great question about how senders can access their Google Postmaster tools, and I wrote up a guide that you might find useful.

We’re also revisiting some older posts on often-requested topics, such as spamtraps, so feel free to comment below if there are topics you’d like us to address or update. One topic that comes up frequently, both on the blog and in our consulting practice, is about what to do when you’re on a blocklist. I revisited an old-but-still-relevant post on that topic as well.
On the Best Practices front, I wrote about how brands can use multiple channels to connect with customers and prospective customers to promote and enhance email delivery. I also took a moment to look back over 2016 and forward to 2017 in the realm of email security.
I continue to be annoyed by B2B spam, and have started responding to those “requests” for my time directly. Steve also wrote a long post about B2B spam, focusing on how these spammers are using Google and Amazon to try to work around reputation issues.
In case you missed it, I contributed some thoughts to a discussion on 2017 email trends over at Freshmail with my exhortation to “Make 2017 the year you turn deliverability into a KPI.”
I’m also still in the process of completing my 2017 speaking schedule, so I’m looking for any can’t-miss conferences and events you’d recommend. Thanks for keeping in touch!

Asking for help with a blocklist

There are often questions arising about how to go about getting off a particular blocklist. A few years ago I led the MAAWG effort to document what to if if you were On a Blocklist (pdf link). That document was aimed primarily at MAAWG members and deliverability experts with working knowledge of blocklists. I think, even now, it’s a good background on how to deal with a listing and mail being blocked.

There have been discussions on multiple mailing lists over the last week or so about how to deal with listings at different blocklists. Many folks on these lists have extensive experience, so these are good places to ask. With that being said, a lot of the requests lack sufficient details to help.
So, if you’re ever on a blocklist and want some help from a mailing list about the problem, here’s a short guide for how to ask for help.

How many blocklists do we need?

There’s been a discussion on the mailop list about the number of different blocklists out there. There are discussions about whether we need so many lists, and how difficult the different lists make it to run a small mail system (80K or so users). This discussion wandered around a little bit, but started me thinking about how we got to a place where there are hundreds of different blocklists, and why we need them.

There is a lot of history of blocklists, and it’s long, complicated and involves many strong and passionate personalities. Some of that history is quite personal to me. Not only do I remember email before spam, I was one of MAPS’ first few employees, albeit not handling listings. I’ve talked with folks creating lists, I’ve argued with folks running lists. For a while I was the voice behind a blocklist’s phone number.
The need, desire and demand for different lists has come up over the years. The answer is pretty simple: there are many different types of abuse. One list cannot effectively address all abusive traffic nor have policies that minimize false positives.
Lists need different policies and different delisting criteria. The SBL lists based on volume of email to addresses that are known to have not opted in to receive mail. The PBL lists IPs where the IP owner (usually an ISP) says that the IPs are not supposed to be sending mail by their policy. URIBL and SURBL list domains, not IPs. Some lists have delisting requirements, some let listees remove themselves.
The policies of listing and delisting are not one size fits all, nor should they be.
There are two widely used lists that have significantly different delisting policies: the SBL and the CBL.
The SBL focuses on IP addresses they believe are under the control of or supporting the services of spammers. They measure this by primarily relying on spamtraps, but they also accept forwarded mail from some trusted individuals. Getting delisted from the SBL means explaining to Spamhaus what steps were taken to stop the spam from coming. It’s a manual process with humans in the loop and can require significant business process changes for listees. (We’ve helped dozens of companies resolve SBL listings over the years, contact us if you need help.)
On the other hand, the CBL is a mostly automated list. It lists ources of mail that aren’t real mail servers sending real mail, but are sending a lot of stuff. As they describe it:

Ongoing subscription attack

Brian Krebs posted a couple days ago about his experience with the subscription bomb over the weekend. He talks about just how bad it was over the weekend.

Spamhaus comments on subscription attack

Steve Linford, CEO of Spamhaus commented on my blog post about the current listings. I’m promoting it here as there is valuable information in it.

Subscription bombing, ESPs and Spamhaus

A number of ESPs woke up to a more-than-usually-bad Monday morning. Last night Spamhaus listed 10s of networks, including ESPs, on the SBL. The listings all contained the following note:

December 2014: The month in email

2014 has been a busy and exciting year at Word to the Wise (look for more on that in a year-end wrap-up post next week!) and this month was particularly thrilling for us as we officially doubled our size with the addition of Josh and Meri on our client services team.
If you’re a regular reader of our blog, you’ve probably spotted Josh’s byline on a few posts: Google’s Inbox Team answers questions on Reddit, which looks at what this new email client portends for both consumers and email marketers, and M3AAWG Recommends TLS, which reviews M3AAWG’s recommendation that mailbox providers phase out SSL encryption in favor of TLS. Look for more smart insights from Josh in 2015.
Steve contributed a post on the proper syntax for displaying a friendly email address, and a very helpful guide for generating useful test data that doesn’t compromise personally identifiable information from your actual customer data. He also detailed the brief DBL false positive from Spamhaus’ new “Abused-Legit” sub-zone and best practices for handling unrecognized responses.
I wrote about some of the subtleties inherent in how brands decide to “converse” with customers in email and other channels. We’ll just keep saying it: companies need to respect the inbox as personal space. I want to thank both Steve and Josh for picking up my slack on blogging. 7+ years is a long time to try and say new things on the blog and I needed a bit of a break.

Some email related news

A couple links to relevant things that are happening in email.
M³AAWG released the Help! I’m on a Blocklist! (PDF link) doc this week. This is the result of 4 years worth of work by a whole lot of people at M³AAWG. I was a part of the working group (“doc champion” in M³AAWG parlance) and want to thank everyone who was involved and contributed to the process. I am very excited this was approved and published so people can take advantage of the collective wisdom of M³AAWG participants.
In other announcements, Gmail announced today on their Google+ page that that they were putting a new “unsubscribe” link next to the sender name when mail is delivered to the Promotions, Social or Forums tab. This appears to be the official announcement of the functionality they announced at the SF M³AAWG last February. It likely means that all users are currently getting the “unsubscribe” link. What Gmail doesn’t mention in that blog post is that this functionality uses the “List-Unsubscribe” header, not the link in the email, but I don’t think anyone except bulk mailers really care about how it’s being done, just that it is.
Also today Gmail announced they were going to recognize usernames with non-Latin or accented characters in the name. Eventually, they claim, they’ll also allow people to get Gmail addresses with accented characters.

Fake DNSBLs

Spamhaus ~~recently~~ announced a few years ago that they have discovered a company that is pirating various blocklists, relabeling them and selling access to them. Not only is the company distributing the zones, they’re also running a “pay to delist” scheme whereby senders are told if they pay money, they’ll be removed from the lists.
The fake company does remove the listing from the fake zones, but does nothing to remove the IP from the original sender. This company has been caught in the past and was blocked from downloading Spamhaus hosted zones in the past, but have apparently worked around the blocks and are continuing to pirate the zone data.
It’s not clear how many customers the blocklist has, although one ESP rep told me they were seeing bounces referencing nszones.com at some typo domains.
No legitimate DNSBL charges for delisting. While I, and other people, do consult for senders listed on the major blocklists, this is not a pay for removal. What I do is act as a mediator and translator, helping senders understand what they need to do to get delisted and communicating that back to the blocklist. I work with senders to identify good, clean addresses, bad address segments and then suggest appropriate ways to comply with the blocklist requirements.

Questions about Spamhaus

I have gotten a lot of questions about Spamhaus since I’ve been talking about them on the blog and on various mailing lists. Those questions can be condensed and summed up into a single thought.

NJABL blocklist closed for good

The NJABL blocklist has shut down for good. The zone files are empty and are expected to stay that way for a while.
Anyone using NJABL, either for yes/no delivery decisions or as part of a scoring system, should remove that blocklist from their setup.

RFC-i blocklist shutting down

The RFC-ignorant blocklist announced on 9/15 that it will be shutting down service.
I commend Derek and his team for how they’re handling the shut down. All too many blocklists have been shut down due to owner burnout with disastrous consequences. Most of these lists did things like listing the whole world or just pulling domains out of the roots. Both types of shutdown methods cause problems for unrelated entities. In one case, a major DNS provider was dealing with what looked like a DOS attack after one blocklist shut down.
I’ve certainly had my differences of opinion with the folks behind RFC-i over the years, but I commend them for announcing the shut down ahead of time and gracefully shutting down RFC-i.

Five-Ten blacklist retired

The Five-Ten website has a notice that they have retired the blacklist. Five-Ten wasn’t the greatest list for blocking mail, they aggressively listed senders and there were a number of false positives against a standard mail stream. But it was useful as a touchpoint. If I had a client that wasn’t listed on Five-Ten that told me something about their normal practices.

Comcast changes

I updated the Wiki a few weeks ago when I heard, but don’t think I posted anything here. Comcast has changed their delisting form page to http://postmaster.comcast.net/block-removal-request.html. The old form is currently non-functional. You can fill it in, but it’s unconnected to anything on the back end and it won’t result in an IP being delisted from the various Comcast blocklists.
My understanding is that the old form may come back to life at some point, but it’s much safer to use the new form and the new Comcast Postmaster Site.

New blocklisting process

There is a new type of blocking designed to interrupt the ability of users to click and visit phishing sites.
DNS Response Policy Zones allows companies running recursive resolvers to create a zone that will not resolve specific domains. This is a second layer of filtering, if a spammer manages to get an email with a malicious link into the inbox then the ISP can still protect the user from becoming a victim from the scam. For more detailed information about RPZ, check out the helpful slides published by ISC.
Two blocklists announced this morning that they were publishing lists in RPZ format so ISPs can import the data into their DNS recursive resolver. SURBL is currently offering their list as RPZ. Spamhaus is currently running a beta for the DBL in a RPZ format. If you’re a current DBL user, talk to Spamhaus about checking out their new format.

I'm on a blocklist! HELP!

Recently, an abuse desk rep asked what to do when customers were complaining about being assigned an IP address located on a blocklist. Because not every blocklist actually affects mail delivery it’s helpful to identify if the listing is causing a problem before diving in and trying to resolve the issue.

Changes at Comcast Postmaster

Two changes at the Comcast Postmaster page that I think are worthy of mentioning.