Recent Posts

DNS, SERVFAIL, firewalls and Microsoft

When you look up a host name, a mailserver or anything else there are three types of reply you can get. The way they’re described varies from tool to tool, but they’re most commonly referred to using the messages dig returns – NXDOMAIN, NOERROR and SERVFAIL.
NXDOMAIN is the simplest – it means that there’s no DNS record that matches your query (or any other query for the same host name).
NOERROR is usually what you’re hoping for – it means that there is a DNS record with the host name you asked about. There might be an exact match for your query, or there might not, you’ll need to look at the answer section of the response to see. For example, if you do “dig www.google.com MX” you’ll get a NOERROR response – because there is an A record for that hostname, but no answers because there’s no MX record for it.
SERVFAIL is the all purpose “something went wrong” response. By far the most common cause for it is that there’s something broken or misconfigured with the authoritative DNS for the domain you’re querying so that your local DNS server sends out questions and never gets any answers back. After a few seconds of no responses it’ll give up and return this error.
Microsoft
Over the past few weeks we’ve heard from a few people about significant amounts of delivery failures to domains hosted by Microsoft’s live.com / outlook.com, due to SERVFAIL DNS errors. But other people saw no issues – and even the senders whose mail was bouncing could resolve the domains when they queried Microsofts nameservers directly rather than via their local DNS resolvers. What’s going on?
A common cause for DNS failures is inconsistent data in the DNS resolution tree for the target domain. There are tools that can mechanically check for that, though, and they showed no issues with the problematic domains. So it’s not that.
Source ports and destination ports
If you’re even slightly familiar with the Internet you’ve heard of ports – they’re the numbered slots that servers listen on to provide services. Webservers listen on port 80, mailservers on port 25, DNS servers on port 53 and so on. But those are just the destination ports – each connection comes from a source port too (it’s the combination of source port and destination port that lets two communicating computers keep track of what data should go where).
Source ports are usually assigned to each connection pretty much randomly, and you don’t need to worry about them. But DNS has a history of the source port being relevant (it used to always use source port 53, but most servers have switched to using random source ports for security reasons). And there’s been an increasing amount of publicity about using DNS servers as packet amplifiers recently, with people being encouraged to lock them down. Did somebody tweak a firewall and break something?
Both source and destination ports range between 1 and 65535. There’s no technical distinction between them, just a common understanding that certain ports are expected to be used for particular services. Historically they’ve been divided into three ranges – 1 to 1023 are the “low ports” or “well known ports”, 1024-49151 are “registered ports” and 49152 and up are “ephemeral ports”. On some operating systems normal users are prevented from using ports less than 1024, so they’re sometimes treated differently by firewall configurations.
While source ports are usually generated randomly, some tools let you assign them by hand, including dig. Adding the flag -b "0.0.0.0#1337" to dig will make it send queries from  source port 1337. For ports below 1024 you need to run dig as root, but that’s easy enough to do.
A (slightly) broken firewall
“sudo dig -b "0.0.0.0#1024" live.com @ns2.msft.net” queries one of Microsofts nameservers for their live.com domain, and returns a good answer.
“sudo dig -b "0.0.0.0#1023" live.com @ns2.msft.net” times out. Trying other ports above and below 1024 at random gives similar results. So there’s a firewall or other packet filter somewhere that’s discarding either the queries coming from low ports or the replies going back to those low ports.
Older DNS servers always use port 53 as their source port – blocking that would have caused a lot of complaints.
But “sudo dig -b "0.0.0.0#53" live.com @ns2.msft.net” works perfectly. So the firewall, wherever it is, seems to block DNS queries from all low ports, except port 53. It’s definitely a DNS aware configuration.
DNS packets go through a lot of servers and routers and firewalls between me and Microsoft, though, so it’s possible it could be some sort of problem with my packet filters or firewall. Better to check.
“sudo dig -b "0.0.0.0#1000" google.com @ns1.google.com” works perfectly.
So does “sudo dig -b "0.0.0.0#1000" amazon.com @pdns1.ultradns.net“.
And “sudo dig -b "0.0.0.0#1000" yahoo.com @ns1.yahoo.com“.
The problem isn’t at my end of the connection, it’s near Microsoft.
Is this a firewall misconfiguration at Microsoft? Or should DNS queries not be coming from low ports (other than 53)? My take on it is that it’s the former – DNS servers are well within spec to use randomly assigned source ports, including ports below 1024, and discarding those queries is broken behaviour.
But using low source ports (other than 53) isn’t something most DNS servers will tend to do, as they’re hosted on unix and using those low ports on unix requires jumping through many more programming hoops and involves more security concerns than just limiting yourself to ports above 1023. There’s no real standard for DNS source port randomization, which is something that was added to many servers in a bit of a hurry in response to a vulnerability that was heavily publicized in 2008. Bind running on Windows seems to use low ports in some configurations. And even unix hosted nameservers behind a NAT might have their queries rewritten to use low source ports. So discarding DNS queries from low ports is one of the more annoying sorts of network bugs – one that won’t affect most people at all, but those it does affect will see it much of the time.
If you’re seeing DNS issues resolving Microsoft hosted domains, or you’re seeing patterns of unexpected SERVFAILs from other nameservers, check to see if they’re blocking queries from low ports. If they are, take a look and see what ranges of source ports your recursive DNS resolvers are configured to use.
(There’s been some discussion of this recently on the [mailop] mailing list.)

Read More

Increase in bounces at Y!

I’ve been seeing reports over the last few days about an increase in bounces at Yahoo. Reliable people are telling me they’re seeing some increase in “invalid user” bounces.
You may remember Yahoo announced an overhaul of their mail product back in December. Reliable sources tell me that this is more than just interface revamp. In the back end, Yahoo! is removing older products with few users and security problems. This fits in with the changes CEO Mayer has been making with the company: slim down and stop supporting unprofitable products.
It makes sense that while engineers are looking at the guts of the email program and cleaning up the cruft, they will also disable long unused email addresses. This will result in higher unknown users for some senders.
What’s interesting to me is that the reports are somewhat sporadic. Some senders are seeing a huge percentage of bounces, some are seeing the normal percentage. I expect this difference isn’t anything more than how actively a sender purges based on engagement. Senders that purge unengaged addresses are going to have already removed a lot of the addresses Yahoo! is now purging from their database. Senders that keep sending to their whole list, are going to see a lot of unknown user bounces.
I’ve asked a few folks and people who’ve responded told me that spot checks showed all the addresses turning up as invalid had no engagement for long periods of time.
If you are seeing a lot of bounces at Yahoo! over the last few days, you need to remove those addresses from your lists. I also recommend looking at the engagement statistics of these newly purged recipients. This will tell you, approximately, what an abandoned address profile looks like. You can use that information to make good decisions about purging unengaged users at other ISPs as well. Not only does this lower costs, because you’ll be sending to less non-responsive email addresses, it will also improve delivery at many ISPs.

Read More

Goodbye Mr. Ebert

The Chicago Sun Times announced earlier today that Roger Ebert passed away today. Mr. Ebert was a legendary film critic, who hosted multiple shows over the last few decades.
His influence wasn’t just in the film arena, though. Mr. Ebert was an active participant online. In fact it was Roger Ebert, in 1996 at the Conference of World Affairs in Boulder Colorado, that coined “The Boulder Pledge.”

Read More

Marketo files for IPO

Marketo filed documents for a $75M IPO yesterday.

Read More

Maybe the sky is only falling a little bit

There was quite a bit of breathless reporting last week about the DoS against Spamhaus and how it was large enough to break the Internet. As the postmortem has gone on, a few things are becoming clear.

Read More

Post-mortem on the Spamhaus DOS

There’s been a ton of press over the last week on the denial of service attack on Spamhaus. A lot of it has been overly excited and exaggerated, probably in an effort to generate clicks and ad revenue at the relevant websites. But we’re starting to see the security and network experts talk about the attack, it’s effects and what it tells us about future attacks.
I posted an analysis from the ISC yesterday. They had some useful information about the attack and about what everyone should be doing to stop from contributing to future attacks (close your open DNS resolver). The nice thing about this article is that it looked at the attack from the point of view of network health and security.
Today another article was published in TechWeekEurope that said many of the same things that the ISC article did about the size and impact of the attacks.
What’s the takeaway from this?

Read More

Internet Storm Center on the Spamhaus DOS

The Internet Storm Center (ISC) has a blog post up discussing the DOS attack against Spamhaus. They do confirm they saw traffic approaching 300Gbps against Spamhaus. They also point out that most people probably never knew.

Read More

Spamhaus answers about the recent DoS attack

Answers about recent DDoS attack on Spamhaus

Read More

More on the attack against Spamhaus and how you can help

While much of the attack against Spamhaus has been mitigated and their services and websites are currently up, the attack is still ongoing.  This is the biggest denial of service attack in history, with as much as 300 gigabits per second hitting Spamhaus servers and their upstream links.
This traffic is so massive, that it’s actually affecting the Internet and web surfers in some parts of the world are seeing network slowdown because of this.
While I know that some of you may be cheering at the idea that Spamhaus is “paying” for their actions, this does not put you on the side of the good. Spamhaus’ actions are legal. The actions of the attackers are clearly illegal. Not only is the attack itself illegal, but many of the sites hosted by the purported source of the attacks provide criminal services.
By cheering for and supporting the attackers, you are supporting criminals.
Anyone who thinks that an appropriate response to a Spamhaus listing is an attack on the very structure of the Internet is one of the bad guys.
You can help, though. This attack is due to open DNS resolvers which are reflecting and amplifying traffic from the attackers. Talk to your IT group. Make sure your resolvers aren’t open and if they are, get them closed. The Open Resolver Project published its list of open resolvers in an effort to shut them down.
Here are some resources for the technical folks.
Open Resolver Project
Closing your resolver by Team Cymru
BCP 38 from the IETF
Ratelimiting DNS
News Articles (some linked above, some coming out after I posted this)
NY Times
BBC News
Cloudflare update
Spamhaus dDOS grows to Internet Threatening Size
Cyber-attack on Spamhaus slows down the internet
Cyberattack on anti-spam group Spamhaus has ripple effects
Biggest DDoS Attack Ever Hits Internet
Spamhaus accuses Cyberbunker of massive cyberattack

Read More

Some content is just bad; but it doesn't have to be

There are a few segments in the marketing industry that seem to acquire senders with bad mailing practices. Nutraceuticals, male performance enhancing drugs, short term or payday loans and gambling have a lot of senders that treat permission as optional. The content and the industry themselves have garnered a bad reputation.
This makes these industries extremely difficult for mailers who actually have permission to send that content to their recipients. Working with this kind of sender, sometimes it seems impossible to get mail delivered to the inbox, no matter what the level of permission. Even when it’s double confirmed opt-in with a cherry on top, all the care in the world with permission isn’t enough to get inbox delivery.
This doesn’t have to be the case. Look at the porn industry. Early on in the email marketing arena there was a lot of unsolicited image porn. A Lot. So much that complaints by recipients drove many ISPs to disable image loading by default. The legitimate porn companies, though, decided unsolicited image porn was bad for the industry as a whole. Porn marketers and mailers adopted fairly strong permission and email address verification standards.
It was important for the porn marketers that they be able to prove that the person they were mailing actually requested the email. The porn marketers took permission seriously and very few companies actually send photographic porn spam these days. Even the “Russian girls” spam doesn’t have not safe for work images any longer.
Because of their focus on permission, in some cases revolving around age of consent in various jurisdictions, the porn industry as a whole is not looked at as “a bunch of spammers.” Porn content isn’t treated as harshly as “your[sic] pre-approved for a wire transfer” or “best quality drugs shipped overnight.”
Just having offensive content isn’t going to get you blocked. But having content that is shared by many other companies who don’t care about permission, will cause delivery headache after delivery headache. This is true even when you are the One Clean Sender in the bunch.
 

Read More
Categories
Tags