AOL: Still broken
I’m still hearing reports that AOL is still having problems accepting mail. I’ve also heard they’re still working on it. There is no information on when a fix may be finished.
I’m still hearing reports that AOL is still having problems accepting mail. I’ve also heard they’re still working on it. There is no information on when a fix may be finished.
Sometime last night, AOL managed to delete their MX records, causing mail to hard bounce for at least 3 hours, possibly more. Annalivia noticed, contacted the NOC, appropriate people were paged and the records are now functional again.
This morning AOL seems to be having more mail problems, possibly related to everyone retrying mail that was hard bounced last night after the MX record was deleted. Or the company is just finally showing the consequences of laying off so many people last year.
I think the most worrying bit about this is that the AOL NOC didn’t notice there was no mail coming in for 3 hours. I don’t get mail for an hour and I start checking to see if the mailserver has fallen over. I can’t believe no one noticed no incoming mail for 3 hours.
I suggest that anyone who had AOL bounces last night package those up and resend today. But don’t send them all at once, trickle them out over the course of the day. Remember, everyone else is trying to send their mail, too. And AOL is not having a happy day.
UPDATE: The Return Path Received blog points out some of the reasons some of you might still be seeing AOL mail fail. The fix is to flush your DNS cache or reboot your DNS server.
As most of you probably know, AOL went through a serious round of layoffs yesterday. Unlike previous layoffs this one did hit the postmaster team pretty hard. Anna posted this morning that she was the only non-programming member of the postmaster team left in the US. This means there are a number of experienced folks looking for work with experience managing delivery for a large outfit. More info is on her blog.
While I don’t have any firm data, I expect that this is going to significantly affect the support that senders see from AOL. I know many of us have held up AOL as the poster child for how ISPs should interact with senders. That era is drawing to a close.
These layoffs come as AOL has migrated to a new mail system and a lot of senders are seeing new and different error messages. I do believe the folks handling the mail system and the migration are still there and are feverishly working to resolve problems caused by the migration. Right now things are in flux and senders should probably expect delays in getting support from AOL for delivery problems.
UPDATE: Matt Vernhout has a list of suggestions for how to deal with AOL delivery issues.
AOL is currently returning “451 4.3.0 <invaliduser@aol.com>: Temporary lookup failure” in some cases when they really mean “550 user unknown.” This message from AOL should be treated as 5xx failure and the message should not be retried (if at all possible) and the failure should be counted as a hard bounce for list management purposes.
This is something broken at AOL’s end, and the guys with the magic fingers that keep the system running are working to fix it. Right now there doesn’t seem to be an ETA on a fix, though.
Even if you are a sender who is able to stop the retries, you may see some congestion and delays when sending to AOL for the time being. Senders who don’t get the message, or who are unable to stop their MTAs from retrying 4xx mail will continue to attempt delivery of these messages until their servers time out. This may cause congestion for everyone and a noticeable slowdown on the AOL MTAs.
AOL blog post on the issue
HT: Annalivia