Act 1 • Act 2 • Intermezzo • Act 3 • Act 4 • Act 5
Management Summary, Redistributable Documents and Links
Yesterday I talked about GFI responsiveness to queries and delisting requests about SORBS listings. Today I’m going to look at data accuracy.
The two issues are tightly intertwined – a blacklist that isn’t responsive to reports of false positive listings will end up with a lot of stale or inaccurate data, and a blacklist that has many false positives will likely be overwhelmed with complaints and delisting requests, and won’t be able to respond to them – leading to a spiral of dissatisfaction and inaccurate data feeding off each other.
Because it is so difficult to remove an IP address from the list, SORBS as a blacklist produces many false positives.
XS4ALL also consider SORBS harmful
GFI/SORBS maintains nine different IP address based blacklists, but they’re usually bundled together and treated as a single “Don’t accept email from this address” blacklist. Each of the nine lists have somewhat different listing policies, though there’s been some scope creep and blurring of the lines over the years.
I’m going to focus on just one of them, dul.dnsbl.sorbs.net. This is intended to list “Dynamic IP Address ranges” – consumer internet connections, such as DSL lines, cable modems and dialup modem pools where the IP address assigned to a user will change over time. These systems don’t typically send legitimate email directly to recipients (rather they send mail via their ISPs smarthost) and often contain a lot of consumer windows machines, which tend to get infected and send viruses and spam, so declining to accept mail from this sort of address pool is a fairly sensible decision. (Spamhaus maintain a list with similar goals, the PBL, as do Trend Micro).
GFI/SORBS have had a number of database accidents that have repeatedly caused false listings in a number of their lists, but because the DUL zone tends to list large ranges of IP addresses, data handling mistakes there tend to cause more visible problems.
the SORBS DUHL list has become badly broken, flagging thousands maybe millions of static IP’s as dynamic. This setting will flag as spam email from numerous legitimate sources incorrectly. Numerous attempts by mail admins the world over have failed to get Sorbs to fix the mess yet.SmarterMail Support Forum
ISPs tell me that GFI/SORBS also refuse to accept notifications about false positive listings in their DUL zone. Or they do update their database, but then reload the bad data a few weeks later. And if the ISP asks GFI for a status update about a false listing, their policy is to move that request to “the bottom of the pile”, ensuring that the inaccurate data that’s causing noticeable problems continues to be published.
If an ISP has reported to SORBS that a CIDR is no longer dynamic, and the (repeat) notifications have been ignored for 6-12 months… at what point does it go from lack of responsiveness, to data quality, to negligence, to willful malice?frustrated anonymous system administrator
The dul list was originally seeded based on data acquired from the dynablock list in 2003. I’m told that stale data, possibly dating back to 2003, is repeatedly being loaded into the GFI/SORBS DUL list, leading to a huge number of false positives. I can’t tell whether that is the case, but there’s certainly a lot of bad data leading to false listings.
Your problem in researching bad data in SORBS is not going to be finding examples of false listings, it’s going to be whittling that forest down to a manageable stack of wood.Comment from IRC
Very true. Lets choose a particular example: “n2.bullet.mail.sp2.yahoo.com” aka 67.195.134.51. This is one of the mailservers for Yahoo Groups, and sends a lot of mailing list mail. There’s nothing at all to suggest that it’s an end-user, dynamically assigned address machine. Just the opposite, it’s listed at dnswl.org as a Yahoo server that shouldn’t be blacklisted. It’s listed by ARIN as part of a /16 (65536 addresses) assigned to Yahoo. There’s nothing in the hostname to suggest it’s dynamically assigned, it even has the word “mail” in the hostname, a common sign of a legitimate mailserver. McAfee TrustedSource list it as a clean mailserver with a history of sending significant volumes of email, as do SenderBase.
And yet it’s listed in the dnsbl.sorbs.net zone, with a return value of 127.0.0.10 meaning GFI/SORBS are claiming it’s a dynamic IP address.
Looking up that IP address on the SORBS website was a fairly painful exercise (I’ll go into more detail about that, and other SORBS operational problems on Monday) but this is what I found:
That IP address is categorized, wrongly, by GFI/SORBS as a dynamic address.
Why did I choose this particular server as an example, rather than one of the countless other false positives I could have picked? Well, it’s not just a single IP address that’s listed as a false psitive: GFI/SORBS are listing all of 67.194.0.0/15 – that’s 131,072 Yahoo servers that are categorized wrongly. And i know that GFI staff were explicitly notified about that particular listing early yesterday morning. Yet GFI are still publishing that data (as well as at least dozens of other false positive listings of similar size).
An outage usually means someone works quickly to resolve it. Having it still be an issue after 5 days is gross negligence. #sorbsTwitter
A blacklist should have checks in place that make it unlikely that badly wrong data is published, though even the best blacklists will very occasionally have a problem and publish bad data. How they respond to false positives is really important. If a blacklist is notified of false positive listings of this magnitude the safe thing to do is to pull all dubious listings from the published blacklist data (or if it can’t be narrowed down, pull all listings) until the problem is resolved.
That will eliminate the loss of legitimate email to the blacklists customers (and most of the spam the blacklist might have stopped will likely be blocked or filtered by other parts of the spam filters they use). GFI/SORBS have not done this, rather they’re following the same practice they’ve used during previous database catastrophes – continuing to publish known bad data.
I do not doubt that there are mail admins rationally fearing for their jobs this week. I am lucky enough to no longer be in the sort of pathological enterprise where a burst of excess false positives is a risk to an admin’s employment, but I am sure that not everyone who was using the SORBS DUHL until this week is so fortunate.Senior Security Consultant
More on Monday.
RSS - Posts
[…] They’ve had a year to do that, so how have they done? Yesterday, my good friend Delivery Kitty reminded me to take a look at GFI/SORBS. Today responsiveness. Tomorrow, data quality. […]
If history is any indication, I assume Michelle Sullivan’s response to this will be threats against you, possibly a false blacklisting of your IP addresses, and perhaps even a typo-laden blog post by her in response. I don’t envy you any of that. But thank you for sharing this. It needed to be said.
some people are morons “anyone can blame a mistake on a DDoS” … the problem is the mistake was corrected but the DDoS is preventing the correction from getting to the real world.
I cannot trust anymore Sorbs after this long down time… Some years ago, my registrar had the mail forwarding service down for some weeks and I had to learn quickly how to operate myself a reliable mail server (Postfix) ! My customers are small organizations but cannot suffer wrong blacklisting from ou to their partners because Sorbs has been unresponsive.
The trouble is much bigger than the supposed spam fight benefit.
My opinion is that, due to this big issue, many systems administrators can be forced to leave definitely Sorbs and it will have a positive influence in the future, things working so much better without Sorbs and its 2003 obsolete datas.
Michelle: Stop the lies. You always claim DDoS when SORBS has just another fuckup. Once again SORBS has reactivated old DUL-Listings, e.g. for 85.25.230.x. This happened back in october as well, and that time you also claimed DDoS.
Who shall believe your DDos lies? If you would really be under DDoS attack (for several days now), how comes that your DNS servers are answering in normal (fast) speed? One would expect the answers to be very slow in case of a DDoS.
As I said: Every time when SORBS makes just another mistake, you claim DDoS for either the problem or your inability to fix the mistake. And because you claim this every time, nobody believes you any longer.
If you can’t fix your data, you need to turn off your dnsbl name servers. You are currently harming mail administrators around the world in a big way. I’m still waiting when the 85.25.230.x netblock will finally be removed from DUL again.
Maybe you should just DELETE the old data so that it can’t be reactivated by accident again.
But this would be a sensible thing to do and most people don’t believe that you guys at SORBS (if it isn’t only you) are capable of doing the right thing.
Here is another /24 wrongly listed in DUL for days: 85.25.184.x. FIX IT NOW!
Here is another /21 wrongly listed in DUL for days: 85.25.24.0/21. FIX IT NOW, PLEASE!
One more amazing thing is about the self-signed ssl certificate used by SORBS ! It look’s like a poor homeless website ! A verisign certificate can cost 500 bucks for one year, but I’ve got one inexpensive from Trustico for 16 / year and there is no trouble with all the current browsers and operating systems. It looks like SORBS was running an old 386 server in a garage, not like a worlwide operating service. This is not serious at all !
I have now started to send emails to the various postmasters/hostmasters who block my IP through SORBS. I have explained the error to them, also the bad behaviour of SORBS and a pointer to the XS4ALL page. I have asked them to ditch SORBS in their own interest.
Maybe the best move would be if you would do so as well where you encounter SORBS users. The madness will only stop for good when either SORBS is dismantled (also keep complaining to GFI as the owners!) or nobody uses it anymore.
The “DDOS” is probably just thousands of mail admins trying to get their legitimate servers delisted. A for blocks, as best I can tell with various spot checks it looks like all of 67.x.x.x is blocked — more than 16 million IP’s! The sad thing is, as bad as SORBS DUHL is, the rest of their spam mechanisms do work, and unfortunately catch some spam that no other lists catch. I hope they can get their act together, and maybe just kill the DUHL list as it’s never been one of the better parts of SORBS anyway.
Hans, I would be interested in seeing the wording of the emails you send to postmasters. I’ve found the time or two I’ve tried doing this, it’s fallen on deaf ears, as a lot of amateur sysadmins can not imagine a blacklist that is not responsive and fair.
I’ve been trying to create an account on Sorbs to request a delisting (from DUHL), but I keep getting an error, with perl error code echoed to screen, after waiting for minutes for the register an account to process. If I was waiting for a de-listing and asked for $$$ in an efficient queue, that would be immoral, but not even able to create an account adds incompetent to the adjective list..
How does a DDoS insert bad records into your blacklist? That is the real issue. If that would stop happening, then a DDoS would have no impact on removal of the bad records.
My company is directly affected by this, for the second time in two months. We have a set of address block that falls within a /24 block of 67.x.x.x and the last response I got back from the SORBS automated system (never have I hated the robot R2D2 more than I do today) was that the entire /24 block was ‘inelligible’ for de-listing.
The parent company sites a DDoS attack as well and says their management team is aware of the issue and working to resolve it ASAP. We’ll see what happens…
[…] So that’s what a DoS attack can do – make a service unavailable. What can’t a DoS attack do? It can’t make any changes to the server(s) under attack. It can’t deface a web page. It can’t cause a web service to give wrong answers. It can’t corrupt information stored in a database. It can’t cause a blacklist to add false listings. That last point is fairly important – no DDoS against any blacklist infrastructure can cause it to add false listings. How does a DDoS insert bad records into your blacklist? That is the real issue. If that would stop happening, then a DDoS would have no impact on removal of the bad records.insightful comment on Fridays post […]
I was also hit by this problem for the second time in as many months. I called my ISP, who is AT&T and had them deal with SORBS directly. Our subnet was delisted in less than 24 hours. It helps to have the clout of AT&T behind you when dealing with people like SORBS.
My biggest problem with SORBS is this: They do not communicate. Their home page has not been updated since 2004, despite the massive problems of late. They have an opportunity to communicate with the admin community directly on their home page, but they choose not to.
I could deal with this type of problem much better if I had ANY SMIDGEN OF AN IDEA that SORBS A) knew of a problem going on and B) acknowledged it and gave some brief outline on how it would be handled.
Rather than having Michelle crawl out from under a rock every few weeks to throw up a lame DDOS excuse, WHY DON’T YOU COMMUNICATE DIRECTLY FROM YOUR WEBSITE HOME PAGE?
Instead we get NOTHING from the SORBS website. Even if you register, assuming you can when their site crashes every 15 minutes – you will hear NOTHING from them.
It is the definition of arrogance.
I for one will not be considering any products from GFI. I had budgeted and received approval for $20K worth of GFI NSM next year. I will not be making that purchase after this latest episode with SORBS.
I encourage every mail admin to call GFI directly, and let them know that you will be much less likely to consider their products for purchase or recommend them to another IT professional,beacuse of the staff at SORBS and their lack of professionalism which one can only assume extends to all employees at GFI.
Dear SORBS: I can deal with the occasional glitch. I *cannot* forgive your arrogance or your lack of communication, which does nothing except communicate your lack of care. You have a home page – which is nearly useless. MAKE SOME USE OF IT. USE IT TO COMMUNICATE DURING PROBLEMS.
I’ve got some really nice data showing how SORBS lists you for hitting a seed address of theirs ONCE 24 hours after they purchased the domain which expired. The user who previously owned the address in question COI (confirmed opt-in) into a list 8 months prior and had shown opens and click throughs within a couple months of the domain transfer. This is just an example of how one of their other zones uses terrible data as the basis for a listing.
anyone can block anything for any reason of course but they should be honest about the reason
even if the reason is just: i dont like you
We are also victims – our IP address was listed since 5th june 2006 as dynamic address from range… address is static at least two years, and yet we are consider as a spammers… Cannot login on their servers (Error 500 – internal server errors), no response to us or our ISP from SORBS… Even don’t mention a self signed certificate – a really trustworthy partner… Thanks Michelle
I won’t get pedantic about why you chose that address. 🙂
As you also noted, it is 67.195/16 which is allocated toYahoo, not a /15. SORBS listed at least the /15 and based on what I could see on their website 12/2, they seemed to have listed the enclosing /10 at one point, which is what is behind the “6 DUHL entries” cited in the image you posted. The the first half of the listed range, 67.194/16, is allocated to the University of Michigan. This example illustrates that this was not just a case of SORBS listing more of a network than made sense, they listed unrelated networks together.
[…] Listing policies must be clear and […]
Maybe ms Michelle Sullivan can tell me why my IP addresses are *STILL* showing up in their “service” as Dynamic. I am actually unable to deal with actual spam because I cannot communicate with the SpamAssassin mailing list… why? Apache.org is using SORBS!
Ticket #326879 too. This is not the first time I’ve had to deal with this and frankly, it’s getting old.
I have given them a ticket number, and the IP of my mail server has not changed in years. Yet it’s being blocked as Dynamic.
It’s a couple months after the alleged DDoS and things are STILL broken. I am advising everybody I know that runs a mail server to remove SORBS from their setup, if they haven’t done so already. GFI threw away a ton of money, obviously.
SORBS has decided that my ip address is to be selected for blacklist. They are unable to remove it from their black list for no real reason, their robot says so.
This organisation is corrupt and needs the www community to rebel against it to rid them from the internet. The internet will be a much better place without SORBS,
Don’t get me wrong. Spam is abhorrent, I do not condone it, what I want to see is the removal of the spammers ability to post, but to leave good honest people to do their own business in peace.
GMX blocked from sending mail no reason given automated service does not list blocked from sending
cant paste image of message as requested it just wont work hopeless customer service im not a geek so i dont stand a chance agaist their robot self automated machine !
Hi.
The article was written in 2010. It is now 2106, so I was wondering if anyone is still here, and what the status
of SORBS is? If they have improved?