Dnsbl

List the world!

We often say that a blacklist has “listed the world” when it shuts down ungracefully. What exactly does that mean, and why does it happen?
Blacklists are queried by sending a DNS lookup for an A record, just the same as you’d find the address of a domain for opening a webpage there. The IP address or domain name that’s being queried is encoded in the hostname that’s looked up.
For example, if you wanted to see whether the IP address 82.165.36.226 was listed on the SpamHaus SBL you’d ask DNS for an A record for the hostname 226.36.165.82.sbl.spamhaus.org. If that returns an answer, the IP address is listed. If it doesn’t, it isn’t.
If a blacklist returns an answer for any IP address (or domain) you ask it about it’s “listing the world” or “listing the internet”, saying that everyone you ask about is listed.
Sometimes this is done intentionally as an attempt to get people to stop using a blacklist. If it blocks all your mail, you’ll stop using it. Unfortunately, that never works. Most blacklists aren’t used to block mail, they’re used as part of a scoring based spam filter. And a blacklist that’s poorly run or unmaintained enough that it shuts down ungracefully probably wasn’t trusted much, so added a very small spamminess value to a spam filters score … so nobody notices when they start listing every address.
More often it’s done when a blacklist is abandoned, leaving it’s base domain name to expire.
When a domain expires it reverts to the control of the registrar and eventually is resold, typically to a domain squatter. (A domain squatter is someone who buys up domains when they become available and hopes to sell them on at vastly inflated prices).
Both the registrar and the squatter really want to resell the domain, for a lot of money. But while they control the domain they might as well make tiny amounts of money from it. The way they do that is to run advertising on the site, typically with low end banner or text ads (cheap to serve, low standards as to where they can be run) along with a link to “Buy This Domain For A Lot Of Money!”.
Every bit of traffic that went to websites in the expired domain is valuable to them – every misdirected open from someone looking for the expired content is now an advertising view. They don’t know what hostnames in the domain were actually in use. www.example.com and example.com are a safe bet, but there may also have been forums.example.com, webmail.example.com, chat.example.com and so on …
They don’t know, or care, what hostnames were in use. They just want as many page views as possible to inflate the tiny amount of money they’re getting from their text ads.
So they set up wildcard DNS for the domain, pointing it at a webserver that’s configured to show a domain-specific advertising page for any hostname pointed at it.
*.example.com -> 192.0.2.25
That means that forums.example.com will resolve to 192.0.2.25, as will www.example.com.
And so will 226.36.165.82.nfn.example.com – so anyone using nfn.example.com as a blacklist will get a valid A record response for any IP address the look up. It “listed the world”.

SpamCannibal is dead

The SpamCannibal blacklist – one that didn’t affect your email too much but which would panic users who found it on one of the “check all the blacklists!” websites – has gone away.
It was silently abandoned by the operator at some point in the past year and the domain registration has finally expired. It’s been picked up by domain squatters who, as usual, put a wildcard DNS record in for the domain causing it to list the entire internet.
Al has more details over at dnsbl.com.
If you run a blacklist, please don’t shut it down this way. Read up on the suggested practice in RFC 6471. If you just can’t cope with that consider asking people you know in the industry for help gracefully shutting it down.
Blacklist health checks
If you develop software that uses blacklists, include “health check” functionality. All relevant blacklists publish records that show they’re operating correctly. For IP based blacklists that means that they will always publish “127.0.0.2” as listed and “127.0.0.1” as not listed. You should regularly check those two IP addresses for each blacklist and if 127.0.0.1 is listed or 127.0.0.2 isn’t listed immediately disable use of that list (and notify whoever should know about it).
For IPv6 blacklists the always listed address is “::FFFF:7F00:2” and the never listed address is “::FFFF:7F00:1”. For domain-based blacklists the always listed hostname is “TEST” and the never listed hostname is “INVALID”. See RFC 5782 for more details. (And, obviously, check that the blacklists your software supports out of the box actually do implement this before turning it on).
If you use someone else’s blacklist code, ask them about their support for health checks. If your mail filter doesn’t use them you risk either suddenly having all your mail go missing (for naive blacklist based blocking) or having some fraction of wanted mail being delivered to your spam folder (for scoring based filters).

Help! We're on Spamhaus' list

While trying to figure out what to write today, I checked Facebook. Where I saw a post on the Women of Email group asking for help with a Spamhaus listing. I answered the question. Then realized that was probably useable on the blog. So it’s an impromptu Ask Laura question.

We’re listed on Spamhaus’ list, any advice on how to get off? Our email provider has a plan, just looking for more input.
If you’re on the SBL, there’s a problem (somewhere) with your data collection process. You’re getting addresses that don’t actually belong to your customers / subscribers / whatever.
The fastest way off it to cut WAY back on who you are mailing to. Mail only to addresses you know, for sure, based on activity in the email, want your mail. Then you can start to go through the other addresses and make decisions about how to verify that those addresses belong to the people you think they do.
If you’re at an ESP, do what they tell you to do. Most ESPs have dealt with this before.
One thing to think about, once you get past the crisis stage, is that if you’re on the SBL, it’s likely your delivery is overall pretty bad. These aren’t folks that dramatically list for a single mistake, there’s a pattern. ISPs look at different patterns, but will often find the same answers and delivery will be bad.
It’s important to realize that Spamhaus has 4 or 5 different lists that have different listing criteria. This is for the SBL, there’s also CSS, CBL, PBL, DBL and XBL. They address different problems and have different listing and delisting criteria.

News in the email space

Various things happening in the email space recently that are worth mentioning but don’t have enough to justify a whole blog post.
Verizon announced a new umbrella company for the AOL and Yahoo media properties, including things like Engadget, Huffington Post. Based on the various press articles I’ve seen this doesn’t appear to affect the email handling for either set of domains.

Asking for help with a blocklist

There are often questions arising about how to go about getting off a particular blocklist. A few years ago I led the MAAWG effort to document what to if if you were On a Blocklist (pdf link). That document was aimed primarily at MAAWG members and deliverability experts with working knowledge of blocklists. I think, even now, it’s a good background on how to deal with a listing and mail being blocked.

There have been discussions on multiple mailing lists over the last week or so about how to deal with listings at different blocklists. Many folks on these lists have extensive experience, so these are good places to ask. With that being said, a lot of the requests lack sufficient details to help.
So, if you’re ever on a blocklist and want some help from a mailing list about the problem, here’s a short guide for how to ask for help.

DNSBLs, wildcards and domain expiration

Last week the megarbl.net domain name expired. Normally this would have no affect on anyone, but their domain registrar put in a wildcard DNS entry. Because of how DNSBLs work, this had the effect of causing every IP to be listed on the blocklist. The domain is now active and the listings due to the DNS wildcard are removed.

How many blocklists do we need?

There’s been a discussion on the mailop list about the number of different blocklists out there. There are discussions about whether we need so many lists, and how difficult the different lists make it to run a small mail system (80K or so users). This discussion wandered around a little bit, but started me thinking about how we got to a place where there are hundreds of different blocklists, and why we need them.

There is a lot of history of blocklists, and it’s long, complicated and involves many strong and passionate personalities. Some of that history is quite personal to me. Not only do I remember email before spam, I was one of MAPS’ first few employees, albeit not handling listings. I’ve talked with folks creating lists, I’ve argued with folks running lists. For a while I was the voice behind a blocklist’s phone number.
The need, desire and demand for different lists has come up over the years. The answer is pretty simple: there are many different types of abuse. One list cannot effectively address all abusive traffic nor have policies that minimize false positives.
Lists need different policies and different delisting criteria. The SBL lists based on volume of email to addresses that are known to have not opted in to receive mail. The PBL lists IPs where the IP owner (usually an ISP) says that the IPs are not supposed to be sending mail by their policy. URIBL and SURBL list domains, not IPs. Some lists have delisting requirements, some let listees remove themselves.
The policies of listing and delisting are not one size fits all, nor should they be.
There are two widely used lists that have significantly different delisting policies: the SBL and the CBL.
The SBL focuses on IP addresses they believe are under the control of or supporting the services of spammers. They measure this by primarily relying on spamtraps, but they also accept forwarded mail from some trusted individuals. Getting delisted from the SBL means explaining to Spamhaus what steps were taken to stop the spam from coming. It’s a manual process with humans in the loop and can require significant business process changes for listees. (We’ve helped dozens of companies resolve SBL listings over the years, contact us if you need help.)
On the other hand, the CBL is a mostly automated list. It lists ources of mail that aren’t real mail servers sending real mail, but are sending a lot of stuff. As they describe it:

Fake DNSBLs

Spamhaus ~~recently~~ announced a few years ago that they have discovered a company that is pirating various blocklists, relabeling them and selling access to them. Not only is the company distributing the zones, they’re also running a “pay to delist” scheme whereby senders are told if they pay money, they’ll be removed from the lists.
The fake company does remove the listing from the fake zones, but does nothing to remove the IP from the original sender. This company has been caught in the past and was blocked from downloading Spamhaus hosted zones in the past, but have apparently worked around the blocks and are continuing to pirate the zone data.
It’s not clear how many customers the blocklist has, although one ESP rep told me they were seeing bounces referencing nszones.com at some typo domains.
No legitimate DNSBL charges for delisting. While I, and other people, do consult for senders listed on the major blocklists, this is not a pay for removal. What I do is act as a mediator and translator, helping senders understand what they need to do to get delisted and communicating that back to the blocklist. I work with senders to identify good, clean addresses, bad address segments and then suggest appropriate ways to comply with the blocklist requirements.