Outlook

Authentication at Office365

This is a followup from a post a few weeks ago about authentication changes at Office365. We have some more clarity on what is going on there. This is all best information we have right now.

There’s been quite a bit of breakage and delivery failure to various Microsoft domains this month. It started with them changing the MX for hotmail.co.uk, then the MX for hotmail.fr… and both these things seem to have broken mail. I also saw a report this morning that some of the new MXs have TLS certificates that don’t match the hostnames.

One of these things is just like the other

Canonicalization is about comparing things to see if they’re the same. Sometimes you want to do a “fuzzy” comparison, to see if two things are interchangeable for your purposes, even if they’re not exactly identical.
As a concrete example, these two email addresses:

Microsoft deprecating SmartScreen filters

At the beginning of the month Microsoft announced that they were deprecating the SmartScreen filters used by the desktop Microsoft mail clients. These are the filters used in Exchange and various version of Outlook mail. This is yet further consolidation of spam filtering between the Microsoft free webmail domains, Office365 hosted domains and self hosted Exchange servers. The online services (hotmail.com, outlook.com, Office365, live.com, etc) have been using these filters for a while. The big change now is that they’re being pushed down to Exchange and Outlook users not hosted on the Microsoft site.
EOP was developed for Outlook.com (and friends) as well as Office365 users. From Microsoft’s description, it sounds like the type of machine learning engine that many providers are moving to.
Microsoft has published quite a bit of information about these filters and how they work on their website. One of the best places to start is the Anti-spam Protection FAQ. Something senders should pay attention to is the final question on that page: “What are a set of best outbound mailing practices that will ensure that my mail is delivered?” Those are all things deliverability folks recommend for good inbox delivery.
Poking around looking at the links and descriptions, there is a host of great information about spam filtering at Microsoft and how it works.
A page of note is their Exchange Online Protection Overview. This describes the EOP process and how the filters work.

August 2016: The Month in Email

August was a busy month for both Word to the Wise and the larger world of email infrastructure.

A significant subscription attack targeted .gov addresses, ESPs and over a hundred other industry targets. I wrote about it as it began, and Spamhaus chief executive Steve Linford weighed in in our comments thread. As it continued, we worked with M³AAWG and other industry leaders to share data and coordinate efforts to help senders recover from the attack.
In the aftermath, we wrote several posts about abuse, blocklists, how the industry handles these attacks currently, and how we might address these issues going forward. And obviously this has been on my mind before this attack — I posted about ongoing problems with internet security, how open subscription forms contribute to the problem, and other ways that companies inadvertently support phishing operations.
I posted about the history of email, and recounted some of my earliest experiences, when I had a .bitnet and a .gov address. Did you use email before SMTP? Before email clients? I’d be curious to hear your stories.
Speaking of email clients, I did two posts about how mail gets displayed to the end user: Gmail is displaying authentication results, which should provide end users with a bit more transparency about how authentication is used to deliver or block messages, and Microsoft is partnering with Litmus to improve some of the display issues people face using Outlook. These are both notable — if this is not your first time reading this blog, you know about my constant refrain that delivery is a function of sending people mail they want to engage with. If the mail is properly formatted and displayed, and people have a high degree of confidence that it’s been sent from someone they want to get mail from, that goes a long way towards improving engagement in the channel.
On that note, I spoke at length with Derek Harding about how marketers might change their thinking on deliverability, and he wrote that up for ClickZ. I also participated in the creation of Adobe’s excellent Teaching the Email Marketer How to Fish document (no, not phish…).
Steve was very busy behind the scenes this month thinking about abuse-related topics in light of the SBL issues, but he wrote up a quick post about the Traffic Light Protocol, which is used to denote sensitive information as it is shared.
Finally, for my Ask Laura column this month, I answered questions about delivery and engagement metrics and about permissions with purchased lists. As always, if you have a general question about email delivery, send it along and I’ll consider it for the column.

Improving Outlook Email Display

Today Litmus announced they had partnered with Microsoft to fix many of the rendering issues with Outlook. Congrats, Litmus! This is awesome. I know a lot of folks have tried to get MS to the table to fix some of the problems with Outlook. Take a bow for getting this off the ground.
According to Litmus, the partnership has two parts.

Getting unblocked at Outlook.com

It’s been a crazy week here at M³AAWG. I have a lot of stuff to blog about, but I think one of the really important things to get out is the new unblock request page at Outlook / Hotmail.
https://sender.office.com
Submit your IPs and it will be reviewed.
(Apologies for the repeated bad links. I’m blaming con crud, lack of sleep and MSN/Hotmail/Office/Outlook for having so many domains I can’t keep them straight. I have finally gotten it right and tested it.)

Filtering more than spam

The obvious application of machine learning for email is to send spam to the junk/bulk folder. Most services use some level of machine learning for filters. Places like Gmail have extensive machine learning filters to filter spam and unwanted mail away from their users.
Some organizations are taking the filtering process a step further. Almost every mail client more advanced than PINE has the ability for users to create rules to sort mail into folders. Late last year, Office 365 rolled out a feature, Clutter that tracks how a user interacts with mail and filters unimportant mail. This allows each user to have their own filters, but without the overhead of having to create the filters.
The Clutter engine looks at both how the user interacts with mail and things it knows about the organization. For example, if Exchange is tied into Active Directory, then mail from a manager will be prioritized while mail from a co-worker may end up in the clutter folder.
Email is a critical business tool. A significant number of companies rely on email for internal and external communication. Many users treat their inbox as a todo list, prioritizing what they work on based on what’s in their mail box. Despite the needs of users, the mail client hasn’t really changed.
Over the last few years, we’ve seen different online services attempt to build a more effective email client. Some of these features were things like tabs and priority inbox at Gmail. Microsoft created the “sweep” feature for Outlook/Hotmail users to manage inbox clutter. Third parties have created services to try and improve the mailbox experience for their users.
Many of the email filters, up to this point, have really been focused on protecting users from spam and malicious emails. Applying that filtering knowledge to more than just spam, but to the different kinds of emails makes sense to me. I’ve always had a fairly extensive set of filters, initially procmail but now sieve, to process and organize incoming mail. But I kinda like the idea that my mail client learns how I filter messages and do the right thing on its own.
I’d love to see some improvements in the mail client, that make it easier to manage and organize incoming email. It remains to be seen if this is a feature that takes off and makes its way to other clients or not.

Thoughts on Hotmail filtering

One of the new bits of information to come out of the EEC15 deliverability discussions is how Hotmail is looking at engagement differently than other webmail providers.
Many webmail providers really do look at overall engagement with a mail when making delivery decisions. And this really impacts new subscribers the most. If there is a mailing where a lot of subscribers are engaged, then new subscribers will see the mail in their inbox. Based on what was said at the webinar earlier this week engagement has no effect at Hotmail outside of the individual user’s box.
I’ve certainly seen this with clients who’ve tried trimming subscriber lists but that doesn’t really help get mail moved from the Hotmail bulk folder to the inbox.

Instead of subscriber lists, Hotmail is really looking at bounces. They’re watching the number of nonexistent accounts senders are mailing to and they’re counting and a sender hits too many bad addresses and that is a major hit to their reputation.
All of this makes remediation at Hotmail challenging. Right now, we can remediate a bad reputation at a lot of ISPs and the filters catch up and mail starts flowing back to the inbox. Hotmail has set up a system that they say is “hard for spammers to game.” This seems to translate into hard for legitimate senders to fix their reputation.
Hotmail is, IMO, the current tough nut in terms of deliverability. Develop a bad reputation there and it’s difficult to fix it. I’m sure it’s possible, though.

Outlook 365 having a bad day

I’ve seen scattered reports today that some mail to the Outlook 365 servers is failing. This has been confirmed by ZDNet. Only folks with a Office 365 account can log in and see the status messages, but there are some folks on the mailop list posting updates from the website.
Attempts to mail to affected domains results in this response:

ISPs speak at M3AAWG

Last week at M3AAWG representatives from AOL, Yahoo, Gmail and Outlook spoke about their anti-spam technologies and what the organizations were looking for in email.
This session was question and answers, with the moderator asking the majority of the questions. These answers are paraphrased from my notes or the MAAWG twitter stream from the session.
What are your biggest frustrations?
AOL: When senders complain they can’t get mail in and we go look at their stats and complaints are high. Users just don’t love that mail. If complaints are high look at what you may have done differently, content does have an effect on complaints.
Outlook: When we tightened down filters 8 years ago we had to do it. Half of the mail in our users inbox was spam and we were losing a steady number of customers. The filter changes disrupted a lot of senders and caused a lot of pain. But these days only 0.5% of mail in the inbox is spam. Things happen so fast, though, that the stress can frustrate the team.
Gmail: Good senders do email badly sometimes and their mail gets bulked. Senders have to get the basic email hygiene practices right. Love your users and they’ll love you back.
What’s your philosophy and approach towards mail?
AOL: There is a balance that needs to be struck between good and bad mail. The postmaster team reminds the blocking team that not all mail is bad or malicious. They are the sender advocates inside AOL. But the blocking team deals with so much bad mail, they sometimes forget that some mail is good.
Yahoo: User experience. The user always comes first. We strive to protect them from malicious mail and provide them with the emails they want to see. Everything else is secondary.
Gmail: The faster we stop spam the less spam that gets sent overall. We have highly adaptive filters that can react extremely quickly to spam. This frustrates the spammers and they will give up.
Outlook: The core customer is the mailbox user and they are a priority. We think we have most of the hardcore spam under control, and now we’re focused on personalizing the inbox for each user. Everyone online should hold partners accountable and they should expect to be held accountable in turn. This isn’t just a sender / ESP thing, ISPs block each other if there are spam problems.
What are some of your most outrageous requests?
We’ve been threatened with lawsuits because senders just don’t want to do the work to fix things. Some senders try to extort us. Other senders go to the advertising execs and get the execs to yell at the filtering team.
Coming to MAAWG and getting cornered to talk about a particular sender problem. Some senders have even offered money just to get mail to the spam folder.
Senders who escalate through the wrong channels. We spent all this money and time creating channels where you can contact us, and then senders don’t use them.
Confusing business interests with product interests. These are separate things and we can’t change the product to match your business interest.
What are your recommendations for changing behaviors?
Outlook: We provide lots of tools to let you see what your recipients are doing. USE THE TOOLS. Pay attention to your recipient interaction with mail. Re-opt-in recipients periodically. Think about that mail that is never opened. Monitor how people interact with your mail. When you have a problem, use our webpages and our forms. Standard delivery problems have a play book. We’re going to follow that playbook and if you try to get personal attention it’s going to slow things down. If there’s a process problem, we are reachable and can handle them personally. But use the postmaster page for most things.
Gmail: Get your hygiene right. If you get your hygiene right, deliverability just works. If you’re seeing blocking, that’s because users are marking your mail as spam. Pay attention to what the major receivers publish on their postmaster pages. Don’t just follow the letter of the law, follow the spirit as well. Our responsibility, as an ISP, is to detect spam and not spam. Good mailers make that harder on us because they do thinks that look like spammers. This doesn’t get spammer mail in more, it gets legitimate mail in less. Use a real opt-in system, don’t just rely on an implied opt-in because someone made a purchase or something.
Yahoo: ESPs are pretty good about screening their customers, so pay attention to what your ESPs are saying. Send mail people want. Verify that the email addresses given to you actually belong to people who want your mail. Have better sender practices.
What do you think about seed accounts?
The panel wasn’t very happy about the use of seed accounts. Seeds are not that useful any longer, as the ISPs move to more and more personalized delivery. Too much time and too many cycles are used debugging seed accounts. The dynamic delivery works all ways.
When things go wrong what should we do?
AOL: Open a ticket. We know we’ve been lax recently, but have worked out of our backlog and are caught up to date. Using the ticketing system also justifies us getting more headcount and makes everyone’s experience better. Also, don’t continue what you’re doing. Pausing sending while you’re troubleshooting the issue. We won’t adjust a rep for you, but we may be able to help you.
Gmail: Do not jump the gun and open a ticket on the first mail to the spam folder. Our filters are so dynamic, they update every few minutes in some cases. Be sure there is a problem. If you are sure you’re following the spirit and letter of the sender guidelines you can submit a ticket. We don’t respond to tickets, but we work every single one. When you’re opening a ticket provide complete information and full headers, and use the headers from your own email address not headers from a seed account. Give us a clear and concise description of the problem. Also, use the gmail product forum, it is monitored by employees and it’s our preferred way of getting information to the anti-abuse team. Common issues lots of senders are having will get addressed faster.
Outlook: Dig in and do your own troubleshooting, don’t rely on us to tell you what to fix. The support teams don’t have a lot of resources so use our public information. If you make our job harder, then it takes longer to get things done. But tell us what changes you’ve made. If you’ve fixed something, and tell us, our process is different than if you’re just asking for a delisting or asking for information. When you’ve fixed things we will respond faster.
How fast should users expect filters to respond after making changes?
Filters update continually so they should start seeing delivery changes almost immediately. What we find is people tell us they’ve made changes, but they haven’t made enough or made the right ones. If the filters don’t update, then you’ve not fixed the problem.

SNDS is back

For years now, Microsoft has maintained Smart Network Data Services (SNDS) for anyone sending mail to Hotmail/Outlook/Live.com. This is a great way for anyone responsible for an IP sending mail to hotmail to monitor what traffic Hotmail is seeing from that IP address.
This morning I got up to a number of people complaining that logins were failing on the website and the API was down. I contacted the person behind SNDS and they confirmed there was a problem and they were fixing it.
Sometime this afternoon it was possible to login to the SNDS interface again, so it looks like they did fix it.
A bit of a warning, though, don’t expect to see any of the data from the last few days. There seems to be something with SNDS that means that when the service is down data isn’t collected or available. In the past when there have been problems, older data was not populated when the service came back.

Outlook.com

The big news in email today is Microsoft’s announcement of the next version of Hotmail: Outlook.com. This does appear to be an attempt to compete with a host of Google’s offerings. Not only does Outlook.com include Skype and access to social media accounts, but it also includes web app versions of Word, Excel and Powerpoint with 7GB of storage space.
I’m not sure how actively people will be grabbing Outlook.com addresses, as you can use hotmail.com addresses with the Outlook.com interface. Only time will tell, though, how this affects email marketing and spam filtering.

Update on FixOutlook.org campaign

Last week I mentioned that the Email Standards Project has started a website (FixOutlook.org) and a twitter campaign to pressure Microsoft to use a HTML compliant rendering engine for Outlook. Currently Outlook uses the HTML engine in MS Word and that engine is not fully compliant with of the HTML standards as published by W3C.org.
Microsoft did reply to the FixOutlook.org campaign on the MSDN Developer blog. The money quote, which they bolded for emphasis in the original post: