Microsoft changes

There’s been quite a bit of breakage and delivery failure to various Microsoft domains this month. It started with them changing the MX for hotmail.co.uk, then the MX for hotmail.fr… and both these things seem to have broken mail. I also saw a report this morning that some of the new MXs have TLS certificates that don’t match the hostnames.

What’s going on?

Historically, Microsoft had two completely separate commercial mail systems, their free webmail and a hosted enterprise solution. The free mail systems started when MS purchased Hotmail, evolved into Live and then eventually Outlook. The enterprise solution is currently Office365. It, too, started off when MS purchased a filtering company and then evolved it into Office365.
A few years ago Microsoft decided they no longer wanted two email systems. So they were going to migrate both to the same back end.

That was 2 years ago…

Yes. Much of the backend stuff has been moving around behind the scenes over the last 2 years. Recently, though, they’ve been deploying new MX records for some of the regional Hotmail domains.

Why should we care?

Well, all things being equal we shouldn’t. But, y’know, nothing about email can ever be easy. On Sept. 1, 2017 a number of people started reporting that all their mail to hotmail.co.uk was bouncing with user unknown messages. Even addresses that were known live. That’s when some folks noticed that the MX records had updated from mx[1-4].hotmail.com to *.olc.protection.outlook.com.
Based on some of the discussions surrounding the change, Hotmail intended to make this change quietly and let the old and new MXs run in parallel. But, spammers had to ruin it. There were so many senders that didn’t update their MX records MS decided to set the old mail servers to reject all mail to hotmail.co.uk. This broke a bunch of stuff with senders, particularly for those who cache MX records for a very long time.

They fixed it.

It appears most senders / ESPs got the message and updated their MX records. MS, to their credit, realized this wasn’t an ideal situation and adapted their transition process. It appears hotmail.fr is the next regional domain to move to the new MX. Instead of swapping out all of the records, MS has added *.olc.protection.outlook.com to the MX records for hotmail.fr. Mail to hotmail.fr is being handled by both the old mx[1-4].hotmail.com and the new *.olc.protection.outlook.com servers. I expect that they’ll phase out the old ones at some point.

Mostly.

Of course, the only reason I know the changes have happened at hotmail.fr is because I’ve seen some folks mentioning problems with sending to those domains. So not everything is wonderful here. I don’t have many details on what the symptoms are, but if you’re seeing some level of problems to hotmail.fr it’s possible it’s them and not you.
There was also a report on the mailop list today that there is an issue with the TLS certificates on the new MX domains – the server names don’t match. This is minor, but may cause TLS to fail until they fix it.

What now?

Microsoft is undertaking a major infrastructure upgrade, without taking down any services. It’s not going to happen as smoothly or as transparently as anyone wants it to. The issues we’ve seen so far are all pretty minor given the scope of the project. If we can get through the rest of this changeover with this level of hiccups we should all be thankful.
Do be aware that things might change and delivery might be erratic until the changeover is finished. Just keep sending good mail and give them the space to work things out on their end. Even though in the short term this might drive up the consumption of antacids, I think long term, this is going to be a win.
 

Related Posts

Outlook 365 having a bad day

I’ve seen scattered reports today that some mail to the Outlook 365 servers is failing. This has been confirmed by ZDNet. Only folks with a Office 365 account can log in and see the status messages, but there are some folks on the mailop list posting updates from the website.
Attempts to mail to affected domains results in this response:

Read More

ISPs speak at M3AAWG

Last week at M3AAWG representatives from AOL, Yahoo, Gmail and Outlook spoke about their anti-spam technologies and what the organizations were looking for in email.
This session was question and answers, with the moderator asking the majority of the questions. These answers are paraphrased from my notes or the MAAWG twitter stream from the session.
What are your biggest frustrations?
AOL: When senders complain they can’t get mail in and we go look at their stats and complaints are high. Users just don’t love that mail. If complaints are high look at what you may have done differently, content does have an effect on complaints.
Outlook: When we tightened down filters 8 years ago we had to do it. Half of the mail in our users inbox was spam and we were losing a steady number of customers. The filter changes disrupted a lot of senders and caused a lot of pain. But these days only 0.5% of mail in the inbox is spam.  Things happen so fast, though, that the stress can frustrate the team.
Gmail: Good senders do email badly sometimes and their mail gets bulked. Senders have to get the basic email hygiene practices right. Love your users and they’ll love you back.
What’s your philosophy and approach towards mail?
AOL: There is a balance that needs to be struck between good and bad mail. The postmaster team reminds the blocking team that not all mail is bad or malicious. They are the sender advocates inside AOL. But the blocking team deals with so much bad mail, they sometimes forget that some mail is good.
Yahoo: User experience. The user always comes first. We strive to protect them from malicious mail and provide them with the emails they want to see. Everything else is secondary.
Gmail: The faster we stop spam the less spam that gets sent overall. We have highly adaptive filters that can react extremely quickly to spam. This frustrates the spammers and they will give up.
Outlook: The core customer is the mailbox user and they are a priority. We think we have most of the hardcore spam under control, and now we’re focused on personalizing the inbox for each user. Everyone online should hold partners accountable and they should expect to be held accountable in turn. This isn’t just a sender / ESP thing, ISPs block each other if there are spam problems.
What are some of your most outrageous requests?
We’ve been threatened with lawsuits because senders just don’t want to do the work to fix things. Some senders try to extort us. Other senders go to the advertising execs and get the execs to yell at the filtering team.
Coming to MAAWG and getting cornered to talk about a particular sender problem. Some senders have even offered money just to get mail to the spam folder.
Senders who escalate through the wrong channels. We spent all this money and time creating channels where you can contact us, and then senders don’t use them.
Confusing business interests with product interests. These are separate things and we can’t change the product to match your business interest.
What are your recommendations for changing behaviors?
Outlook: We provide lots of tools to let you see what your recipients are doing. USE THE TOOLS. Pay attention to your recipient interaction with mail. Re-opt-in recipients periodically. Think about that mail that is never opened. Monitor how people interact with your mail. When you have a problem, use our webpages and our forms. Standard delivery problems have a play book. We’re going to follow that playbook and if you try to get personal attention it’s going to slow things down. If there’s a process problem, we are reachable and can handle them personally. But use the postmaster page for most things.
Gmail: Get your hygiene right. If you get your hygiene right, deliverability just works. If you’re seeing blocking, that’s because users are marking your mail as spam. Pay attention to what the major receivers publish on their postmaster pages. Don’t just follow the letter of the law, follow the spirit as well. Our responsibility, as an ISP, is to detect spam and not spam. Good mailers make that harder on us because they do thinks that look like spammers. This doesn’t get spammer mail in more, it gets legitimate mail in less. Use a real opt-in system, don’t just rely on an implied opt-in because someone made a purchase or something.
Yahoo: ESPs are pretty good about screening their customers, so pay attention to what your ESPs are saying. Send mail people want. Verify that the email addresses given to you actually belong to people who want your mail. Have better sender practices.
What do you think about seed accounts?
The panel wasn’t very happy about the use of seed accounts. Seeds are not that useful any longer, as the ISPs move to more and more personalized delivery. Too much time and too many cycles are used debugging seed accounts. The dynamic delivery works all ways.
When things go wrong what should we do?
AOL: Open a ticket. We know we’ve been lax recently, but have worked out of our backlog and are caught up to date. Using the ticketing system also justifies us getting more headcount and makes everyone’s experience better. Also, don’t continue what you’re doing. Pausing sending while you’re troubleshooting the issue. We won’t adjust a rep for you, but we may be able to help you.
Gmail: Do not jump the gun and open a ticket on the first mail to the spam folder. Our filters are so dynamic, they update every few minutes in some cases. Be sure there is a problem. If you are sure you’re following the spirit and letter of the sender guidelines you can submit a ticket. We don’t respond to tickets, but we work every single one. When you’re opening a ticket provide complete information and full headers, and use the headers from your own email address not headers from a seed account. Give us a clear and concise description of the problem. Also, use the gmail product forum, it is monitored by employees and it’s our preferred way of getting information to the anti-abuse team. Common issues lots of senders are having will get addressed faster.
Outlook: Dig in and do your own troubleshooting, don’t rely on us to tell you what to fix. The support teams don’t have a lot of resources so use our public information. If you make our job harder, then it takes longer to get things done. But tell us what changes you’ve made. If you’ve fixed something, and tell us, our process is different than if you’re just asking for a delisting or asking for information. When you’ve fixed things we will respond faster.
How fast should users expect filters to respond after making changes?
Filters update continually so they should start seeing delivery changes almost immediately. What we find is people tell us they’ve made changes, but they haven’t made enough or made the right ones. If the filters don’t update, then you’ve not fixed the problem.

Read More

Thoughts on Hotmail filtering

One of the new bits of information to come out of the EEC15 deliverability discussions is how Hotmail is looking at engagement differently than other webmail providers.
Many webmail providers really do look at overall engagement with a mail when making delivery decisions. And this really impacts new subscribers the most. If there is a mailing where a lot of subscribers are engaged, then new subscribers will see the mail in their inbox. Based on what was said at the webinar earlier this week engagement has no effect at Hotmail outside of the individual user’s box.
I’ve certainly seen this with clients who’ve tried trimming subscriber lists but that doesn’t really help get mail moved from the Hotmail bulk folder to the inbox.
 
Instead of subscriber lists, Hotmail is really looking at bounces. They’re watching the number of nonexistent accounts senders are mailing to and they’re counting and a sender hits too many bad addresses and that is a major hit to their reputation.
All of this makes remediation at Hotmail challenging. Right now, we can remediate a bad reputation at a lot of ISPs and the filters catch up and mail starts flowing back to the inbox. Hotmail has set up a system that they say is “hard for spammers to game.” This seems to translate into hard for legitimate senders to fix their reputation.
Hotmail is, IMO, the current tough nut in terms of deliverability. Develop a bad reputation there and it’s difficult to fix it. I’m sure it’s possible, though.

Read More