Act 1 • Act 2 • Intermezzo • Act 3 • Act 4 • Act 5
Management Summary, Redistributable Documents and Links
In the past week we’ve demonstrated that the SORBS reputation data is riddled with mistakes, poor practices, security holes and operational problems, and that the quality of the end result is really too poor to be useful.
Today I’m looking at how this information should affect your choice of spam filtering technology.
Should my spam filters use GFI/SORBS data?
Simply, NO. The quality of reputation data GFI provide is too false-positive prone to rely on in production, even as part of a scoring system.
After all, a false positive is far worse than a false negative, as far as RBL (or general filtering system) usability is concerned.@delivery_kitty
And the problem isn’t just false positives.
Because it takes a long time before a spamming IP address reliably appears on the blacklist, not much spam is stopped. SORBS appears completely unsuitable to the most common way of spamming, via botnets.abuse department at xs4all, a major EU ISP
Your ISP
If you receive mail via your ISP then you’re unlikely to have problems with SORBS blocking your mail, as very few successful ISPs will use it for blocking outright. If you’re at a smaller ISP then they may well be using spam filters such as SpamAssassin, with a dependency on GFI / SORBS data sources, though.
But it’s not worth contacting your ISP unless you find out mail is being bounced or put into a junk folder due to a SORBS listing, or if you can tell by looking at the headers of email you receive that it’s being scored against SORBS.
(If you’re concerned about use of third-party reputation sources by your ISP, you could ask them to provide – or, better, publish – a list of the data sources they use, so their customers can make well-informed decisions about their filtering.)
Your Mailserver
If you run your own inbound mailserver, make sure it is not configured to use any of the SORBS blacklists for blocking email. How to do that varies depending on the server, but for commonly used linux mailservers grepping the configuration files for the string “sorbs” is probably a good place to check.
(There are some great blacklists, with very low false positive rates, to consider using instead – for IP based reputation: spamhaus zen, spamcop, cbl – and for URL reputation: spamhaus dbl, uribl, surbl)
SpamAssassin
SpamAssassin is a widely used server-side score based spam filter. Unfortunately it seems to ship with SORBS blacklists turned on “out of the box”.
I believe that adding the following to /etc/spamassassin/local.cf will disable it – I could be wrong, and would appreciate feedback from any SpamAssassin experts out there.
score RCVD_IN_SORBS_BLOCK 0 score RCVD_IN_SORBS_DUL 0 score RCVD_IN_SORBS_HTTP 0 score RCVD_IN_SORBS_MISC 0 score RCVD_IN_SORBS_SMTP 0 score RCVD_IN_SORBS_SOCKS 0 score RCVD_IN_SORBS_WEB 0 score RCVD_IN_SORBS_ZOMBIE 0
The default SpamAssassin scores are pretty low, so it doesn’t pay that much attention to SORBS – but that a spam filter as influential as SpamAssassin uses such a poor source of data at all is a bit of a problem. Hopefully the SpamAssassin developers will look at the issue for a future release.
Commercial Products
If you’re using a commercial spam filter, check where they’re getting their reputation data from. If you have an existing commercial filter that can use external blacklists, make sure it’s use of SORBS is disabled.
If you’re considering purchasing a new commercial spam filter, there are two things you need to consider. First, if the filter supports using SORBS or other GFI-derived reputation data make sure that can be disabled. Second, if you’re considering a commercial product that uses SORBS or GFI data out of the box, despite the multi-year history of false positives and other problems, think about how solid their other product engineering decisions might be.
I for one will not be considering any products from GFI. I had budgeted and received approval for $20K worth of GFI NSM next year. I will not be making that purchase after this latest episode with SORBS.Skyhawk
Outsourced Services
Outsourced spam filtering services can be very opaque about what approaches they use to decide whether or not email is spam, and will often hide their use of external reputation services.
Some of them are more open than others. Proofpoint posted SORBS DUHL DNS Block List Causing Widespread Email Deliverability Issues Once Again (note that GFI told Proofpoint the problem was fixed on Nov 30th, which we know isn’t true), but it’s rare that a SaaS provider will be that open about how a problem is caused by their reliance on a third-party service. Kudos to Proofpoint for their openness (though they should look elsewhere for reputation data).
Edit: Proofpoint have clarified that they were discussing the problems some of their customers had sending email due to false SORBS listings – not that they were using any data from GFI themselves. Sorry, guys. So if you’re looking for a filtering appliance or outsourced service that’s GFI/SORBS-free (and also quite a nice product), Proofpoint is worth a look.
If your SaaS or outsourced spam filter provider has a clear statement in their product description or contract which third-party data sources they use, then you have the information you need. If not, you should probably contact your support representative and find out whether they use SORBS or not. If they decline to make any statement on it, assume the worst.
In the case of SORBS, this is (at least) the second major misclassification issue we’ve observed in the last 90 days. Email administrators who currently rely on SORBS should be aware of these issues and take action as necessary.Proofpoint
There’s nothing wrong with an outsourced provider using reputable third-party services but if they’re relying on poor quality data sources you may find mail to you being bounced for no good reason, at any time. If that’s the situation you’d be well advised to consider looking at alternative filtering providers.
And Finally
There’s a lot more that could be said, but I’m sure you’re interested in seeing some non-GFI/SORBS content on this blog (and there’s a limit to the amount of technical and business analysis I really want to do for someone other than a paying customer).
Laura will probably revisit the subject soon, going into some more detail about the policy problems that I just touched lightly on and looking more generally about what other companies can learn. And I know several other industry bloggers are planning on discussing GFI and SORBS in the next week or two.
I’ll be gathering links and some other information, including a PDF version of this series of articles suitable for mailing out, at https://wordtothewise.com/sorbs/ over the next day or two.
RSS - Posts
Well. That was indeed a comprehensive overview of all the reasons why SORBS is really Not A Very Good Blocklist Indeed, and what people can do about it. Bravo! The only thing I’m curious about is, that I’m not sure quite what prompted such an in-depth destruction of SORBS – although I certainly don’t disagree with anything you’ve written.
Looking forward to future posts (or series of posts) with the same level of detail and research. Nicely done.
SORBS has been problematic for a long time, both before their acquisition by GFI and after.
Since their acquisition by GFI their operational practices and data quality have, if anything, been getting worse. They’ve screwed up their database spectacularly badly twice in the past couple of months – and each time they’ve made false statements about what’s gone on, blaming it on a DDoS and so on.
But SORBS also have a history of blacklisting and harassing anyone who publicly calls them on their practices – I’ve seen entire companies blacklisted for multiple years with no cause other than calling out SORBS for poor practices on a private mailing list.
If your business is related to sending email, speaking truth about SORBS can get you blacklisted and cause you significant operational costs. If your employers business is related to sending email, speaking truth about SORBS can get you blacklisted and damage your career path.
Because of that, influential people in the email industry have been loath to call it as it is. I know of at least two major businesses in the email industry who were planning at a director level to denounce SORBS, only to have that quashed as a poor business move by C-level executives (rightly, I suspect).
I, and Laura, and our company are highly respected in the email industry. If GFI tries to badmouth us, or blacklist us, they’ll just be laughed at. So we can safely start the discussion.
Thanks for this very detailed article about SORBS, and thanks for including my blog post at http://blog.proofpoint.com as part of your post! Just a point of clarification regarding your section on Proofpoint:
Proofpoint does *not* use SORBS as part of our SaaS email security solution (we have our own reputation technology that we call Proofpoint Dynamic Reputation).
The reason Proofpoint and certain customers of Proofpoint’s SaaS email security solutions became aware of problems with SORBS is that they were experiencing problems with deliverabiliity of outbound email to domains that *do* use SORBS. That is, outbound email from IPs for which we host the gateway was getting blocked because certain Proofpoint IPs were incorrectly on the SORBS DUHL. Our customers were not having *inbound* message receipt problems.
As my post notes, we encourage email administrators not to rely on a single RBL for vetting inbound email. Unfortunately, in many cases admins don’t even know what RBLs email is being checked against or how to turn off such checks should that be needed. For example, there are commercial anti-spam solutions in the market that incorporate checks against SORBS without making it obvious in their admin interfaces. (But again, Proofpoint is not one of these.)
Best Regards,
Keith
Keith Crosley
Director, Market Development, Proofpoint – http://www.proofpoint.com
Editor, Proofpoint Email Security Blog – http://blog.proofpoint.com
I can’t imagine anyone is still using the SORBS database. I used it years ago but I can’t have any filtering that has false positives at all.
So no SORBS, no SPEWS, nothing like that.
Though I realize a lot of people are somehow braindead, and used that guy Joe’s lookup even after he told everyone to please stop, for a long time too. What was that? Osurusoft, or something? They so many people had a fit when he blacklisted the world by just not making the old list available. How the heck was he gonna get all those braindead people to stop querying him?
Everyone who was paying attention had fair warning. I suspect the people who are still using SORBS are the same ones. Not paying attention.
These spam rating methods change. Antispam operations is a very dynamic process. You don’t just type something in a box on your mailserver and forget it for 9 years.
But some apparently do just that. Ridiculous.
Heh.
Funny, I went and reread the post.
spamhaus zen, spamcop, cbl
are exactly what I am using.
Not really any false positives with those.
Does some spam get through? Yes. Could I get rid of it all? Yes, but that would also get rid of 15-30% of the legitimate, personally typed messages coming through.
Too high a price to pay. At least for me.
Not that users don’t whine about getting 5 spam messages a day and how awful that is! Get real. 5 or 10 a day? I can delete those standing on my head. Heh.
@Steevo Someone on a list I’m on suggested having a day where we all turn off our spam filters, just to show users how insignificant the amount of spam they normally get is compared to what’s being blocked. I’d never do it, of course, but it is quite an amusing idea!
Our small mail server blocks over 12,000 very bad spam emails per day, and accepts the same or more marked as spam, etc. This is only for a few hundred users and a handful of domains. No thanks to turning spam filters off! Not all users get bombed, but those that do very much need the filters.
As for SORBS, their website has been down in entirety for most of the day, yet their DUHL continues to block thousands if not millions of legitimate, static IP blocks incorrectly with no way for any victims to correct the problem . . .
[…] And finally, work with the larger anti-spam community, not against it. That’s where you’ll get your best intelligence, and your most effective […]
[…] Path has turned our recent series of blog posts about SORBS into a handy list for what people SHOULD do when they’re intending to run a blocklist. More […]
Steve:
Yep local.cf is where one would want to put those rule adjustments:
/etc/spamassassin is common, /usr/local/etc/spamassassin, as well as /etc/mail/spamassassin are also somewhat common. (sometimes one is a symlink to one of the others though) Surprised that its still in the default config, although to be fair I havent used a default config for SA in a good while so idk what all is in there.
If one is on a system that allows individual spamassassin prefs, but the administration does not want to remove sorbs for whatever reason, similar rules can also be implemented on a per-user basis.
That being said.. if one uses greylisting, adding an extra hour or so to the default is a relatively safe way to use the spam and dul zones. Low risk when (not if) there are issues with the blacklist, decent FN reduction as a result of giving spammers extra time to get listed elsewhere though. I’ve been using it that way for a while, disabling temporarily as needed. no ill effects. (YMMV)
In spamassassin, as long as the points are low enough and some form of overriding or whitelisting (like DNSWL) is used, it *can* be used in combination with other broad-brush style blocklists to reduce FNs without increasing FPs. But better to start with a safer default (ie completely disabling sorbs in the filters), and then very carefully enabling if desired. Enabling it upfront globally *is* a bit nuts though. (IMHO)
Re: Spamassassin
If I am remembering correctly, if you use those settings in local.cf, it will still generate the outbound network queries to the SORBS servers, and simply zeroes out the score for those hits. Could be remembering incorrectly however.
I believe this single rule in local.cf *may* replace all the ones you listed, but I haven’t used SpamAssassin in quite some time:
score RCVD_IN_SORBS 0
or it may have been deprecated for what you have in your article.
Looking at an old, inactive, NOT-current instance, it looks like the way to prevent network queries to SORBS is to comment out EVERY line in the SORBS section in /usr/share/spamassassin/20_dnsbl_tests.cf, where DNSBL tests are defined. Should be about 60-70 lines or so.
This may or may not be 100% complete — to that end:
# for file in
rpm -ql spamassassin; do grep -Hli sorbs $file; done/etc/mail/spamassassin/local.cf
/usr/lib/perl5/vendor_perl/5.8.8/Mail/SpamAssassin/Conf.pm
/usr/share/spamassassin/20_dnsbl_tests.cf
/usr/share/spamassassin/30_text_de.cf
/usr/share/spamassassin/30_text_fr.cf
/usr/share/spamassassin/30_text_nl.cf
/usr/share/spamassassin/30_text_pl.cf
/usr/share/spamassassin/50_scores.cf
Thanks much for posting this series. It seems like it’s been needed for a very long time and I will probably use it as a default go-to link when any SORBS related issue comes up with a mail admin.
I was going to add a link to here from the Wikipedia sorbs page but someone beat me to it :p
Steven King:
AFAIK if one runs sa-update (via cron or manually) then commenting out the references in /usr/share/spamassassin gets undone on each update. local.cf is where you wanna make site-wide changes if you want them to persist across updates.
[…] SORBS Blocks Gandhi.net. The DNSBL that everyone loves to hate blocked a large, well-known registrar. Some changes have happened since then, such as the dropping of the requirement to pay a fee for delisting. But, Steve Atkins at Word to the Wise has put up a six part series wherein he makes the argument that GFI/SORBS is harmful. […]
[…] SORBS Blocks Gandi.net. The DNSBL that everyone loves to hate blocked a large, well-known registrar. Some changes have happened since then, such as the dropping of the requirement to pay a fee for delisting. But, Steve Atkins at Word to the Wise has put up a six part series wherein he makes the argument that GFI/SORBS is harmful. […]
[…] Block Lists and the Death of a Thousand Cuts Posted: July 20, 2010 by hey4ndr3w in Deliverability Tags: block list, spam 0 Author’s Note: Since this writing, block lists operated by SORBS have had a pair of spectacular, catastrophic failures resulting in the inadvertent and wholly spurious characterization of enormous chunks of the Internet as sources of spam, or dynamic IP space, or both. Poor infrastructure planning and operational security precluded a graceful recovery; in fact, it was nearly a week before operations returned to normal. As such, use of GFI/SORBS as a reliable source of data on production mail servers is deprecated. My colleague Steve Atkins at Word to the Wise provides an exhaustive review of the problems leading up to and exacerbating the failures, and summarizes them neatly. […]
SORBS claimed our issue was resolved and issued an apology, but a few weeks later, the problem is back. Even worse, their website says that my IPs are NOT listed, but a direct DNS query shows 127.0.0.10… the silly “dynamic IP” list.
What’s really a problem is that the email admins of apache.org are using SORBS! I am now unable to post questions regarding blocking spam to the spamassassin mailing list because of SORBS. Meanwhile, a pervasive spammer using IPs at rackvibe is spamming away. Those IPs are clear in SORBS, while my small IP block remains listed despite not having ever send out one single spam. What’s wrong with this picture?
Guess what!
In 2012 this problem dynosaur problem is still around. This amoungst claims that google and Microsft would solve all the SPAM problems once and for all.
They use incomplete and outdated IP registrations and make assumptions on ‘the threat level’ (that only they appear to know) and will claim you a (potential – not dissclosed) spammer or not..
A robot (even dumber than a game console AI) will attempt to interact with you via email!?
Quoting the ‘first ammedment and all! = FTF (they have more money than brains) with guaranteed NO SUCCESS = don’t bother.
Amazing this brick-and-mortar run by automation only (using outdated and not updated information sources) is actually still taken seriously in this day and age.
Just amazing how dynosaur seem to survive.
It is a dynosaur, treatit accordingly!