The view from a blacklist operator

We run top-level DNS servers for several blacklists including the CBL, the blacklist of infected machines that the SpamHaus XBL is based on. We don’t run the CBL blacklist itself (so we aren’t the right people to contact about a CBL listing) we just run some of the DNS servers – but that means that we do get to see how many different ways people mess up their spam filter configurations.
This is what a valid CBL query looks like:

  • “14.23.177.10.cbl.abuseat.org”

It’s just the IP address being queried (10.177.23.14) with the numbers reversed, with “.cbl.abuseat.org” added on the end. Not rocket science.
Here’s a tiny sample of some of the invalid queries:

  • “70.46.6.10.abuseat.org”
  • “202.204.219.10cbl.abuseat.org”
  • “252.94.193.10.ns1-cbl.abuseat.org”
  • “255.190.244.10 cbl.abuseat.org”
  • “166.193.222.10#cbl.abuseat.org”
  • “214.6.224.10.*@cbl.abuseat.org”
  • “212.9.185.10.http://cbl.abuseat.org”
  • “76.207.80.10.bl.abuseat.org”
  • “185.124.73.10.cbb.abuseat.org”
  • “201.54.179.10.cbl-xbl.abuseat.org”
  • “54.191.254.10.opm.abuseat.org”
  • “181.4.133.10.sbl-xbl.abuseat.org”
  • “176.33.165.10.cbl.abuseat.orgcbl.abuseat.org”
  • “101.126.133.10.cbl.abuseat.org:Mail from %IP% refused by blackhole site cbl.abuseat.org”

Those are just 15 of about 1800 different misconfigurations I have on file, just for queries to the CBL. I’ve seen similar things at other domains I host, and I’ve heard of just the same sort of thing from other people who own domains that are similar in some way to a domain used by a blacklist. It’s not unusual.
What happens when someone misconfigures a blacklist lookup in this way? Because of the way DNS based blacklists work the response to any of these invalid queries will be “no, that IP address isn’t listed”. So all these people are attempting to use the CBL to filter out spam and haven’t noticed that it’s never actually stopped any email. And all the time they’re doing this, they’re hammering my DNS servers (and many other peoples) with millions of pointless queries every day.
What can the DNS server operators do about that? Because of the way DNS works, blocking the broken queries will actually increase the amount of traffic they have to deal with by several times. Contacting all the people making the queries and pointing out the problem would be a huge task, and even when I have tracked down contact information and notified people by email I’ve never had a response and the problem has never been fixed.
So the only remaining option is to make the misconfiguration more obvious to the user – by responding to the invalid queries with “yes, that IP address is listed” and hoping that causing them to reject all the mail sent to their users will encourage them to fix their configuration. I check my nameserver statistics every so often and add “poison” entries for the more obvious misconfigurations I find. I did that for a bunch of misconfigurations manually yesterday, which will probably cause a lot of domains to reject a bunch of email they didn’t want to this morning.
There are fairly simple ways to make sure you’re querying a real blacklist – pretty much all of the legitimate blacklists include the IP address “127.0.0.2” as a test entry. You can use that to check that a blacklist is live manually – if the blacklist domain is sbl.spamhaus.org then a dns lookup for “2.0.0.127.sbl.spamhaus.org” should return an answer (typically 127.0.0.2) while a dns lookup for “1.0.0.127.sbl.spamhaus.org” should return “not found” / “NXDOMAIN”. If either of those tests fails, the blacklist is broken in some way, and you shouldn’t use it.
The choice of 127.0.0.2 for the test entry wasn’t arbitrary: 127.0.0.2 is a “local” address that’s always available on machine, though it’s usually never used for anything. But you can use it – if you open a commandline on your mailserver you can run an SMTP transaction by hand (as I discussed yesterday) from 127.0.0.2 using “telnet -b 127.0.0.2 your.hostname 25” (on Linux-ish systems, anyway – some other telnets use “-s” instead of “-b”). That way you can see whether you’re really rejecting based on a blacklists, and what error you’re giving. (It would be nice if every blacklist also had another test entry in 127.* as well as 127.0.0.2, so you could check them individually, but they don’t. Hint to blacklist operators.).
It’s very easy for spam filter authors to check those test entries once a day for each of the blacklists they were configured to use, and to disable the ones that failed. If you’re a postmaster who uses blacklists as part of your spam filter (and you probably should) you should check with the people who provide the filter whether it makes those checks – and if it doesn’t, ask them to add them. That will protect you from misconfigurations, blacklists being shut down, blacklists being abandoned and bought up by domain squatters and all sorts of other things that can cause you to lose a lot of mail.

Related Posts

Why do ISPs limit emails per connection?

A few years ago it was “common knowledge” that if you were sending large amounts of email to an ISP the most polite way to do that, the way that would put the least load on the receiving mailserver, was to open a single SMTP session to the mailserver and then to send all the mail for that ISP down that single connection.
That’s because the receiving mailserver is concerned about two main resources when handling inbound email – the pool of “slots” assigned one per inbound SMTP session, and the bandwidth (network and disk, and related resouces such as memory and CPU) consumed by the inbound mail – and this approach means the sender only uses one slot, and it allows the receiving mailserver to control the bandwidth used simply by accepting data on that one connection at a given rate. It also amortizes all the connection setup costs over multiple emails. It’s a beautiful thing – it just doesn’t get any more efficient than that.
That seems perfect for the receiving ISP – but ISPs don’t encourage bulk senders to do this. Instead many of them have been moving from “one connection, lots of mail through it” to “multiple connections, a few messages through each”. They’re even limiting the number of deliveries permitted over a single connection. Why would that be?
The reason for this is driven by three things. One is that the number of simultaneous inbound SMTP sessions that a mailserver can handle is quite tightly limited by the architecture of most mailservers. Another is that the amount of mail that’s being sent to large ISP mailservers keeps going up and up – so there are sometimes more inbound SMTP sessions asking for access than the mailserver can handle. The third is that ISPs know that there are different categories of email being sent to their users – 1:1 mail from their friends that they want to see as soon as possible, wanted bulk mail that their users want to see when it arrives and spam; lots and lots of spam.
So ISPs want to be able to do things like accept 1:1 mail all the time, while deferring bulk mail and spam to allow them to shed traffic at times of peak load. But they can only make decisions about whether to accept or defer delivery in an efficient way at SMTP connection time – they pick and choose amongst the horde of inbound connection attempts to prioritize some and defer others, letting them keep within the number of inbound sessions that they can handle simultaneously.
But once the ISP lets a bulk mailer connect to deliver their mail, they lose most of the ability to further control that delivery as the sender might send thousands of emails down that connection. (Even if the ISP has the ability to throttle bandwidth – as some do to control obvious spam – that just means that the sender would tie up an expensive inbound delivery slot for longer).
So, in order to allow them to prioritize inbound connections effectively the ISP needs to terminate the session after a few deliveries, and then make that sender start competing with other senders for a connection again.
So ISPs aren’t limiting the number of deliveries per SMTP connection to make things difficult for senders, or because they don’t understand how mail works. They’re doing it because that lets them prioritize wanted email to their users. The same is true when they defer your mail with a 4xx response.
It might be annoying to have to deal with these limits on delivery, but for legitimate bulk mail senders all this throttling and prioritization is a good thing. Your mail may be given less priority than 1:1 mail – but, if you maintain a good reputation, you’re given higher priority than all the spam, higher priority than all the email borne viruses, higher priority than all the junk email, higher priority than the 419 spams. And higher priority than mail from those of your competitors who have a worse reputation than yours.

Read More

Which is better UTF-8 or ISO-?

Someone asked today on a mailing list whether they should be using UTF-8 or “ISO” encoding for sending email. What’s the best choice depends on some of the details of the situation, but here’s the answer I gave:
UTF-8 will work for pretty much anything, as it’s just an 8 bit encoding scheme for Unicode (which is supposed to be the one character encoding to rule them all). It’s well supported in most languages and development environments – Windows has been native UTF-16 under the covers since the mid 90s, for instance – and typical messages that use mainstream glyphs should render well from utf-8 in most western MUAs and browsers.
There are still a very few old or broken clients out there that will not handle UTF-8 well but (outside the asian language market, where there’s still some non-ASCII, non-Unicode legacy usage) they’re typically ones that don’t really handle any character set encoding well and the only thing safe to send to them is either plain ASCII or whichever ASCII superset their OS happens to support natively (which is probably an argument for sending Windows-1252 codepage, but not a terribly strong one).
The various extended ASCIIs (such as ISO-8859-*) will only work for messages that are written solely using characters from that character set. If you have even one character in a message that cannot be expressed in ISO-8859-1, then you can’t use ISO-8859-1 to send that message.
ISO-8859-1 (aka Latin1) is fairly sloppy in some respects – it has no apostrophe, nor single quotes, for instance – but it can handle an awful lot of languages, from Kurdish to Swahili. It can’t handle Dutch, Estonian, Finnish, Hungarian and Welsh particularly well, nor can it show the Euro symbol (ISO-8859-14 or -15 are needed for some characters there).
A common problem is that many people (and the software they write) think that Windows uses Latin1. It doesn’t, it uses Windows-1252. If you accept messages written on Windows, using the Windows-1252 code page, and throw them out on the wire as ISO-8859-1 what you end up with is not quite right. It mostly works, as the two codepages overlap quite a bit, but they have different glyphs in the 0x80-0x9f range. So if you use single or double quotes (“smart quotes”), or the Euro symbol, or ellipses, or bullet, or the trademark symbol in your message they’ll be garbled. This is so common that some mail clients and web browsers will actually treat a document that claims to be ISO-8859-1 as Windows-1252, but that’s a bug workaround and not something it’s really safe to rely on.
If you’re doing personalized messages, and you’re sending one of them to Győző and one of them to Eiður then you may have to use different character sets for the two messages. If you’re talking about Győző and personalizing it for Eiður then you might find things break horribly.
Someone probably has some concrete data on mail client character set support, broken down by region and language, but my understanding is that this is a reasonable approach:

Read More

How to disable a domain

Sometimes you might want to make it clear that a domain isn’t valid for email.
Perhaps it’s a domain or subdomain that’s just used for infrastructure, perhaps it’s a brand-specific domain you’re only using for a website. Or perhaps you’re a target for phishing and you’ve acquired some lookalike domains, either pre-emptively or after enforcement action against a phisher, and you want to make clear that the domain isn’t legitimate for email.
There are several things to check before disabling email.
1. Are you receiving email at the domain? Is anyone else?
Check the MX records for the domain, using “host -t mx example.com” from a unix commandline, or using an online DNS tool such as xnnd.com.
If they’re pointing at a mailserver you control, check to see where that mail goes. Has anything been sent there recently?
If they’re pointing at a mailserver that isn’t yours, try and find out why.
If there are no MX records, but there is an A record for the domain then mail will be delivered there instead. Check whether that machine receives email for the domain and, if so, what it does with it.
Try sending mail to postmaster@ the domain, for instance postmaster@example.com. If you don’t get a bounce within a few minutes then that mail may be being delivered somewhere.
2. Are you sending email from the domain? Is anyone else?
You’re more likely to know whether you’re sending mail using the domain, but there’s a special case that many people forget. If there’s a server that has as it’s hostname the domain you’re trying to shut down then any system software running no that server – monitoring software, security alerts, output from cron and so on – is probably using that hostname to send mail. If so, fix that before you go any further.
3. Will you need mail sent to that domain for retrieving passwords?
If there are any services that might have been set up using an email address at the domain then you might need a working email address there to retrieve lost passwords. Having to set email back up for the domain in the future to recover a password is time consuming and annoying.
The domain registration for the domain itself is a common case, but if there’s any dns or web hosting being used for the domain, check the contact information being used there.
4. How will people contact you about the domain?
Even if you’re not using the domain for email it’s quite possible that someone may need to contact you about the domain, and odds are good they’ll want to use email. Make sure that the domain registration includes valid contact information that identifies you as the owner and allows people to contact you easily.
If you’re hosting web content using the domain, make sure there’s some way to contact you listed there. If you’re not, consider putting a minimal webpage there explaining the ownership, with a link to your main corporate website.
5. Disabling email
The easiest way to disable email for a domain is to add three DNS records for the domain. In bind format, they look like:

Read More