Ddos
Censorship and free speech online
One of the things I discovered yesterday while looking at Krebs on Security was that Google Alphabet has a program to provide hosting and dDOS protection for journalists. Project Shield, as it’s called, is a free service for approved applicants that keeps up websites that might be taken down otherwise. Eligible organizations include those providing news, information on human rights and monitoring elections.
This is something I hadn’t heard of before and my only reaction is good for Google.
Look, we’ve gotten to the point where attackers have resources beyond the scope that most of us can imagine. It’s expensive even for large organizations to manage and pay for the level of protection they need.
Even more importantly a lot of very important work is done by individuals or small organizations. Brian is a prime example of that. He does an incredible job investigating online crime on his own time. His site and his information is an invaluable resource for many. Losing his site, and losing his information would leave a huge hole in the security community. There are other folks in other spaces who, like Brian, don’t have the resources to protect themselves but do have important things to say and share.
I’m glad to see Google committing their resources and skills to help organizations protect themselves. It’s so important that this work is done and we don’t lose voices just because they can’t afford hundreds of thousands of dollars a year.
There has been abuse and harassment online for as long as I’ve been here. But it seems recently the size and severity of attacks have increased. And a lot of service providers are struggling with how to manage it and what their responsibilities are.
A few weeks ago Facebook deleted an iconic photo from the Vietnam era due to child nudity in the photo. That decision was reversed and discussed in many, many different places. One of the most interesting discussion happened on a friend’s Facebook feed. Many of the participants work at various online providers. They have to make these kinds of decisions and create policy to do the right thing – whatever the right thing is. It was very interesting to be able to follow the discussion and see how many different issues FB and other online providers have to consider when creating these types of policies.
I thing the thing I have to confront the most about the internet is how big it is. And how crucial it’s become to all sorts of issues. Social media can be a cesspool of abuse, there’s no question. But it can also be a force for good. I’m glad companies like Google are stepping up to preserve the good parts of the internet.
The Cyber and The Security
Cybersecurity has been on my mind lately. There is a lot of bad stuff going on, from giant dDOS attacks, to subscription bombing, to the ongoing low level harassment that some people have to deal with on a daily basis. I’ve written a lot about how I think marketers are going to have to step up and stop being a conduit for abuse. I do believe this. There are a lot of different issues to discuss but there are also many, many different stake holders in the issue of cybersecurity.
I’ve been on multiple calls with different groups over the last few weeks discussing the implications of the subscription attack and how it was carried out. The majority of my focus is email and how to protect senders from becoming a conduit for abuse. Other folks participating on the call are looking at what abuse is out there and how to stop it or minimize it.
One thing that came up on a recent call is that the bulk of dDOS traffic that took Brian Krebs’ website down was from various Internet of Things devices. Security cameras, DVD players, televisions, lightbulbs and other connected devices were part of the problem. It’s a huge issue, and one that cannot simply be mitigated by just ISPs and providers. But convincing individuals to secure their lightbulbs can be a challenge, we can’t even protect their computers completely. Convincing companies to stop providing default usernames and passwords or using the same keys for every device is another challenge.
These are big issues that we’re going to have to deal with.
Last night, with 100 million of my virtual friends and a small group of local ones, I watched the first Presidential debate. Part of the debate was about cyber security. To misquote Vice President Biden, “Cybersecurity is a big freaking deal.” We have nation states, and groups with the resources of nation states, conducting covert operations online. We have hacking, compromises, bonnets and other malicious activity occurring every, single day. And, the more complex the site and the more users it has the more likely it is to be compromised. Cybersecurity is a critical part of national security and our own individual security. We must take it seriously and we must address it.
Now, I’ll be honestI don’t think there is a solution to the problem. I think, though, that there are hundreds of things we can do as individuals, as companies, as nations, as volunteer organizations, as NGOs and as coalitions to solve different parts of the problem. We all need to think about what it is and who’s doing the bad stuff.
It’s common to think of hackers as lonely boys in basements who have too much time and too little to do. Back in the ancient days of the spam wars some folks referred to them as “chickenboners“: beer drinking rednecks who ate fried chicken and threw the bones on the floors of their trailers. The reality even then, though, was that many spammers ran businesses and made a lot of money. Admittedly, the descriptions of how the business was run are cringe inducing and full of illegal activity.
Now, much of the hacking is actually organized crime outside the US. This makes it hard to address successfully through legal channels.
It’s all very complicated. But I think we can agree security is a big deal. We are all part of the solution, by securing our sites and our personal devices. We’re also part of the solution by paying attention to the larger issues and events going on around us.
Security issues affect us all
I’ve been talking about security more on the blog. A lot of that is because the security issues are directly affecting many senders. The biggest effect recently has been on companies ending up on the SBL because their signup forms were the target of a subscription attack. But there are other things affecting online spaces that are security related. Right now not much of it is affecting email senders, but it’s good to be aware of.
DDOS attacks
There has been an increase in DDOS attacks against different companies and network. Some of the online game sites have been targeted including EA, Blizzard and others. A group called PoodleCorp is claiming responsibility for those attacks.
Another set of DDOS attacks hit Brian Krebs’ website this week. The site stayed up, but Akamai has told Brian they can no longer host his website. His website is down for now and the foreseeable future.
While this activity doesn’t affect marketers directly, it does tell us that there is active development happening on the less legal side of the internet. The volumes of the recent attacks have sent records. They’re also changing in scope and including new kinds of traffic in an effort to knock sites offline. Even more concerning, they appear to be systematically attempting to discover defenses in order to attack the internet as a whole.
Increase in Spam
Spam has been on the decrease over the last few years. Many of us were treating it as a mostly-solved problem. But a new report from Cisco Talos shows that trend is reversing and spam levels are increasing. Current levels are approaching those last seen more than 5 years ago. Cisco Talos has used a number of different sources of data, all showing an increase in spam directly and indirectly.
This month in email: February 2014
After a few months of hiatus, I’m resurrecting the this month in email feature. So what did we talk about in February?
Industry News
There was quite a bit of industry news. M3AAWG was in mid-February and there were actually a few sessions we were allowed to blog about. Gmail announced their new pilot FBL program. Ladar Levinson gave the keynote talking about the Lavabit shutdown and his new darkmail program. Brian Krebs won the Mary Litynski award for his work in investigating online security issues. The 4 major mailbox providers talked about their spam filters and spam filtering philosophy.
February was also the month where different companies evaluated their success or failure of products. LinkedIn announced the shutdown of their Intro product and Facebook announced the shutdown of their Facebook.com email service.
Security Issues
Cloudmark published their 2013 report on the Global Spam Threat and we discovered that the massive Target breach started through phishing. I also noticed a serious uptick in the amount of phishing mails in my own mailbox. There is new round of denial of service attacks using NTP amplification. We provided information on how to secure your NTP servers.
Address Collection
The Hip Hop group De La Soul released their entire catalog for free, online, using a confirmed opt-in email process. On the flip side, the M3AAWG hotel required anyone logging into the wifi network to give an email address and agree to receive marketing mail. We also discovered that some political mailing lists were being used in ways the politicians and recipients didn’t expect.
Email Practices
I talked about how to go about contacting an ISP that doesn’t have a postmaster page or a published method of contact. Much of that information is actually relevant for contacting ISPs that do have a contact method, too. Finally, I talked about how ISPs measure engagement and how that’s significantly different from how ESPs think it is.
More denial of service attacks
There are quite a lot of NTP-amplified denial of service attacks going around at the moment targeting tech and ecommerce companies, including some in the email space.
What does NTP-amplifed mean? NTP is “Network Time Protocol” – it allows computers to set their clocks based on an accurate source, and keep them accurate. It’s very widely used – OS X and Windows desktops typically use it by default, and most servers should have it running.
NTP is a UDP based service, like DNS, one that works by sending a packet to a server and the server sending a packet back rather than opening a persistent connection to the server as TCP based services (e.g. SMTP, HTTP, …) do. That simpler protocol means that it’s easy for me to send a request to an NTP server with a false source address, claiming I’m someone else – and the NTP server will send it’s reply back to that fake source address rather than to me. So if I want to DoS someone by flooding their network with packets I can send NTP requests to a public NTP server claiming to be the victims server. The NTP server will send the replies back to the victim – and it’ll be almost impossible for the victim, or the NTP server, to trace where I’m sending those request packets from.
As a malicious attacker that already sounds good – but it gets better. The size of a reply is often bigger than the size of a request, sometimes a lot bigger. If I choose the request I make carefully I can easily make sure the reply is at least an order of magnitude bigger than the request. So for every megabit of forged requests I sent to NTP servers, the victim might see at least 10 times that hitting their servers. That’s amplification.
What can you do about it? If you’re running NTP servers that will respond to requests from the general Internet, you ideally need to lock those down so that they’ll only respond to requests from your own clients. You can use the instructions and information at the open NTP project to check to see if you’re running open NTP servers and use the templates provided by Project Cymru as a basis to secure your NTP servers and appliances.
What can you do to prepare for this sort of attack? Have monitoring in place, so that you’re notified if there are large volumes of unexpected traffic. Overprovision your bandwidth, if possible, to give you more time to react. Block “large” (>90 bytes for IPv4, >110 for IPv6) UDP packets with a source or destination port of 123 as far upstream as possible, and all UDP packets that have both a source port of 123 and a destination port of 80 or 25 – this shouldn’t affect legitimate use of NTP by your users. Consider having your production servers use NTP servers operated by you, rather than public NTP servers – that way, if they’re targeted you can block any traffic that looks like NTP to them without affecting their time synchronization. Research DoS mitigation providers – different providers have different strengths and cost structures, and they can be much more reasonably priced if you talk to them before an attack rather than during one.
What if you’re targeted by this sort of attack? If you’re not a sysadmin, stay out of your sysadmins way and make sure there’s coffee, food and a quiet place without interruptions available. If you are a sysadmin, talk to your upstream NOC. They’re in a much better place, in information, resources and knowledge, than you to help mitigate. Reach out to your peers who are also being attacked and offer to share information. Look at Cisco’s mitigation advice. The attack will probably target your publicly visible website. If so, consider moving that to another network (or behind a commercial DoS mitigation provider) so that your production servers and customer portal web presence isn’t impacted.
More information
Arrest made in Spamhaus dDOS
According to a press release by the Openbaar Ministerie (the Public Prosecution Office), a dutch man with the initials SK has been arrested in Spain (English translation) for the dDOS attacks on Spamhaus. Authorities in Spain have searched the house where SK was staying and seized electronic devices including computers and mobile phones.
Brian Krebs has more, including multiple sources that identify SK as Sven Olaf Kamphuis. Sven Olaf Kamphuis was quoted in many articles about the dDOS, including the NY Times and various reports by Ken Magill.
ETA: Spamhaus thanks the LEOs involved in the arrest.
Post-mortem on the Spamhaus DOS
There’s been a ton of press over the last week on the denial of service attack on Spamhaus. A lot of it has been overly excited and exaggerated, probably in an effort to generate clicks and ad revenue at the relevant websites. But we’re starting to see the security and network experts talk about the attack, it’s effects and what it tells us about future attacks.
I posted an analysis from the ISC yesterday. They had some useful information about the attack and about what everyone should be doing to stop from contributing to future attacks (close your open DNS resolver). The nice thing about this article is that it looked at the attack from the point of view of network health and security.
Today another article was published in TechWeekEurope that said many of the same things that the ISC article did about the size and impact of the attacks.
What’s the takeaway from this?
More on the attack against Spamhaus and how you can help
While much of the attack against Spamhaus has been mitigated and their services and websites are currently up, the attack is still ongoing. This is the biggest denial of service attack in history, with as much as 300 gigabits per second hitting Spamhaus servers and their upstream links.
This traffic is so massive, that it’s actually affecting the Internet and web surfers in some parts of the world are seeing network slowdown because of this.
While I know that some of you may be cheering at the idea that Spamhaus is “paying” for their actions, this does not put you on the side of the good. Spamhaus’ actions are legal. The actions of the attackers are clearly illegal. Not only is the attack itself illegal, but many of the sites hosted by the purported source of the attacks provide criminal services.
By cheering for and supporting the attackers, you are supporting criminals.
Anyone who thinks that an appropriate response to a Spamhaus listing is an attack on the very structure of the Internet is one of the bad guys.
You can help, though. This attack is due to open DNS resolvers which are reflecting and amplifying traffic from the attackers. Talk to your IT group. Make sure your resolvers aren’t open and if they are, get them closed. The Open Resolver Project published its list of open resolvers in an effort to shut them down.
Here are some resources for the technical folks.
Open Resolver Project
Closing your resolver by Team Cymru
BCP 38 from the IETF
Ratelimiting DNS
News Articles (some linked above, some coming out after I posted this)
NY Times
BBC News
Cloudflare update
Spamhaus dDOS grows to Internet Threatening Size
Cyber-attack on Spamhaus slows down the internet
Cyberattack on anti-spam group Spamhaus has ripple effects
Biggest DDoS Attack Ever Hits Internet
Spamhaus accuses Cyberbunker of massive cyberattack
CBL website and email back on line
The CBL website is back on line.
It’s possible that your local DNS resolver has old values for it cached. If so, and if you can’t flush your local DNS cache, and you really can’t wait until DNS has been updated then you may be able to put a temporary entry in your hosts file to point to cbl.abuseat.org.
You can get the IP address you need to add by querying the nameserver at ns-2038.awsdns-62.co.uk for cbl.abuseat.org. No, I’m not going to tell you the IP address – if you can’t do a basic DNS query, you shouldn’t be modifying your hosts file and you can just wait a day.
dDOS spreads to the CBL
Spamhaus has mostly mitigated the dDOS against the Spamhaus website and mailserver, but now the CBL is under attack. They have been working to get that under protection as well, but it’s taking some time.
Right now there are no public channels for delisting from the CBL. The Spamhaus Blog will be updated as things change, and I’ll try and keep things updated here as well.
UPDATE: Cloudflare talks about the scope of the attack
Spamhaus under major dDOS
Late last night I, and a number of other folks, received mail from Spamhaus informing us of a major denial of service attack against their servers. The attack is so bad that the website and main mailserver is currently offline.
DNS services, including rsync and the mirrors, are up and running.
Spamhaus is working to bring the mailserver and website back up, and are hoping to have it up later today.
If there are any critical or particularly urgent SBL issues today, contact your ESP delivery team. The folks who were contacted do have an email address for urgent issues. This is not an address for routine queries, however, and most listees are going to have to wait until normal services are restored to have their listing addressed.
If there is something particularly urgent and your ESP or delivery team does not have a contact address, you can contact me an I can see what I can do.
UPDATE: Most of the IPs people have sent me are actually XBL/CBL listings. But right now the CBL webserver is responding slowly due to the DOS.
If you want to look up a listing without using the Spamhaus website you can use the “host” or “dig” command line tools. To do this reverse the digits in the IP address and append zen.spamhaus.org on the end.
So for the IP 10.11.12.13 you would query 13.12.11.10.zen.spamhaus.org
Spamhaus dDOS
I got mail late last night from one of the Spamhaus peeps telling me that they were under a distributed Denial of Service (dDOS) attack. This is affecting email. Incoming email is delayed and they’re having difficulty sending outgoing email. This is affecting their responses to delisting queries.
They are working on mitigation and hopefully will be fully up and running soon.
Updates when I get them.
Update (8/29/2012): mail to Spamhaus should be back.
GFI/SORBS – a DDoS Intermezzo
Act 1 • Act 2 • Intermezzo • Act 3 • Act 4 • Act 5
Management Summary, Redistributable Documents and Links
I’ve been stage-managing for a production of The Nutcracker this week, so musical terminology is on my mind. In opera, the intermezzo is a comedic interlude between acts of an opera series.
This comedic interlude is about the “DDoS” – a distributed denial of service attack. What is a denial of service attack?