Sometime last night, AOL managed to delete their MX records, causing mail to hard bounce for at least 3 hours, possibly more. Annalivia noticed, contacted the NOC, appropriate people were paged and the records are now functional again.
This morning AOL seems to be having more mail problems, possibly related to everyone retrying mail that was hard bounced last night after the MX record was deleted. Or the company is just finally showing the consequences of laying off so many people last year.
I think the most worrying bit about this is that the AOL NOC didn’t notice there was no mail coming in for 3 hours. I don’t get mail for an hour and I start checking to see if the mailserver has fallen over. I can’t believe no one noticed no incoming mail for 3 hours.
I suggest that anyone who had AOL bounces last night package those up and resend today. But don’t send them all at once, trickle them out over the course of the day. Remember, everyone else is trying to send their mail, too. And AOL is not having a happy day.
UPDATE: The Return Path Received blog points out some of the reasons some of you might still be seeing AOL mail fail. The fix is to flush your DNS cache or reboot your DNS server.
AOL goes kablooey
A
RSS - Posts
I just tried emailing someone to two different aol accounts at 2pm central and they both bounced back:
The reason for the problem:
5.1.2 – Bad destination host ‘DNS Hard Error looking up aol.com (A): domain has no A record’
They better send Anna a gift basket
lmao @ AOL
You may have gotten caught up with your DNS server caching the bad data. You can flush your cache or just wait for it to expire out. Return Path has a good blog post explaining the DNS issue http://www.returnpath.net/blog/received/2010/12/mx-record/
Thanks for the linkage, Laura. It worries me too, the lack of noticing. “My” folks were either asleep or didnt have access to see the issue, but the NOC should have seen it. It makes me very sad.
Thanks for the info – given the time zone difference our caches are probably ok but this is on the way to 2nd line support now
AOL is a company that went from having a very alert and active network management function to it seems, well, none anymore. That is so sad.
But I doubt they are making money anymore so this is a result of that. Without revenues they really cannot employ qualified people.
Since AOL is essentially lacking any paying customers anymore they also have few customers who can complain.
I guess everything eventually seeks it’s level.