We’ve known for a while that AOL email infrastructure is going to be merging with Yahoo’s, but apparently it’s happening sooner than anyone expected.
The MXes for aol.com will be migrated to Yahoo infrastructure around February 1st. Reading between the lines I expect that this isn’t a flag day, and much of the rest of the AOL email infrastructure will be in use for a while yet, but primary delivery decisions will be made on Yahoo infrastructure.
The AOL and Yahoo postmaster teams are pretty smart so I assume they’ll have made sure that their reputation data is consistent, and be doing everything else they can do to make the migration as painless as possible. But it’s a major change affecting a lot of email, and I wouldn’t be surprised to see some bumpiness.
If you’ve done anything … unwise … with delivery to AOL addresses, such as hard-wiring MXes for delivery to aol.com, you should probably look at undoing that in the next week or so. I’m guessing the changeover will happen at the DNS level, so if you’ve nailed down delivery IPs for aol.com you might end up trying – and probably failing – to deliver to the old AOL infrastructure.
When we say that you might just be sending too much email and fatiguing or annoying the recipient into unsubscribing or hitting spam, this is the sort of thing we mean.
Three emails (to the same email address) in four minutes might be a bit much.
If you can’t combine the content you want to send into a single personalized email, maybe spread deliveries out a bit? Or even not send all of it, perhaps.
Got an email this morning from a company advertising their newest webinar “The Two Pillars of Effective Large-Scale Email: Security and Deliverability.” The message came to a tagged address, so clearly I’d given them one at some point. But I didn’t recognize the name or company or anything. I did a search to seen when I may have interacted with this company in the past.
Looking through my old emails, it appears I contacted this company through their support form back in 2007. They were blocking a client’s newsletter. This is what I sent:
One of my clients has asked me to talk with you about your blocking schemes. They’re rather confused as their mail to a customer (one-to-one mail, not bulk) is ending up in the junk/spam folder. They’re not sure what they’re doing to get filtered.
Is there someone who can talk to me about your filtering schemes so I can explain to your mutual customer what is happening?
The response was pretty unhelpful.
I see that the email address insight@ESP is already in Michael’s allow list in his Email Defense filter settings — was this a recent addition? This should let any emails sent from that address through without being filtered first. Another suggestion would be to add the source IP address itself to his allow list if emails are still being caught. Let us know if this alleviates the situation.
That’s the last I heard from said company until this morning, when they sent me an ad.
A common question we’re asked is “How can I safely and securely utilize large-scale/mass emailing to communicate?”
Whether you’re sending newsletters, announcements, notifications, even sensitive or private information, there are two pillars you must have in place to ensure your communications are sent securely AND are delivered without being classified as spam.
One way to prevent communications from being classified as spam is to not grab addresses from a decade ago out of your support queue and use them for marketing out of the blue. Also, I’m much more likely to trust your opinion on delivery if you follow CAN SPAM. I mean, it’s nice you sent me a picture of the nice lady who sent the spam, but you forgot to put a postal address on the email.
Interestingly enough, the company actually has a pretty effective sounding AUP for their customers. They prohibit, among other things:
- Automatically opting visitors or purchasers into their subscriber list. This includes “pre-checking” an opt in box on forms.
- Automatically adding subscribers on one mailing list to unrelated mailing lists
- Sending emails to subscribers that are unrelated to the purpose to which they opted in
- Adding people to the mailing list without their permission
- Sending messages to people who have requested to be removed from the mailing list
- Using old lists without checking with the subscribers that their addresses are still valid and that they still wish to be subscribed.
Too bad they don’t apply their AUP to their own email program.
Things have been a little unsettled at Microsoft webmail properties over the last few months. A number of ESPs reported significantly increased deferrals from Microsoft properties starting sometime late in November. Others saw reduced open rates across their customer base starting in late October. More recently, people are noticing higher complaint rates as well as an increase in mail being dropped on the floor. Additionally, Return Path announced certification changes at the end of November lowering the Microsoft overall complaint rate to 0.2%, half of what is was previously.
Overall, sending mail to Microsoft is a challenge lately. This is all correlated with visible changes which may seem unrelated to deliverability, but actually are. What are the changes we know about?
- Towards the end of 2017, Microsoft moved their free webmail properties over to the same backend as Office365.
- Microsoft has updated their mobile interface to include this-is-spam buttons.
- FBL reports are generated now when users move mail to the bulk folder, even when using an IMAP client.
- Microsoft and Return Path decided to lower the acceptable complaint rate for certified IPs.
All of these changes are clear evidence Microsoft is investing in their email product and their filtering methodology.
Is it better?
I’m hearing mixed reports from folks. Many senders are seeing improvements in inboxing. However, it’s neither a consistent or linear progression. Things will improve for a while, then fall apart again. The overall trajectory is upwards, which is good; but there is frequent backsliding.
When will it be over?
I’m not sure. It may be that the new backend gives Microsoft some extra knobs and levers to dial in their filters and inboxing there will continue to be challenging. One thing I’m seeing on my clients is that Microsoft delivery is tracking closer to Gmail, or even a little worse. Senders that have traditionally struggled to reach the Gmail inbox are now starting to struggle to also reach the Microsoft inbox. This indicates that their filtering processes are focusing more on engagement and user interaction with email than before.
One bit of evidence for that is the move to more closely record feedback from mobile applications and generating FBL emails from IMAP clients. Sendgrid has a good blog post from last week discussing the increase in complaints. To me, this really confirms that Microsoft is looking at engagement and has improved their tools to better measure it.
A major problem with FBL reports is that they were only available when the ISP controlled the email interface. People who used a desktop or mobile mail client didn’t have a spam button, so even if they hated a particular mail, they couldn’t report it. Now, they have a report button in the mobile interface. On the desktop, reports are generated simply by moving messages to the spam folder. This means more complaints from recipients that couldn’t complain before.
At the same time they’re increasing the number of complaints generated, Microsoft has visibly lowered their complaint thresholds. This means more senders are going to struggle with inbox delivery, at least in the short term.
What does Microsoft say?
Many different folks attempted to discuss this with Microsoft employees, but there weren’t many answers forthcoming. One individual did report a response to one of their complaints that indicated Microsoft was seeing an overall increase in complaints. This may be due to the increase in reporting pathways, but it also might be due to the deluge of holiday mail. Many recipients react negatively to the holiday ramp up and hit spam or unsubscribe from more mail.
In all likelihood the increase in complaints is likely partially attributed to the increase in volume and the increase in reporting pathways.
As Microsoft has not published any specific information on their new filters, we’re left with seeing what has worked for senders to improve delivery. As always, send to users who want and expect your mail. Pay attention to engagement and remove recipients that show little interest in mail. Use other channels to encourage users to check their bulk folder and move the mail back to their inbox. If you’re doing all these things and not seeing improvement, talk to your deliverability team. They may see something you don’t. If your delivery team can’t help you, contact us and we’ll see what we can do.
Above all, don’t expect this to be resolved overnight. Microsoft has change how they’re doing things and it may take a few more weeks or months before their filters are dialed in exactly right. Expect changes to continue. I know it’s difficult to be patient, but sometimes there’s nothing else to do. Send good mail users want, and the filters will catch up.
For some reason otherwise legitimate ESPs have over the years picked up a habit of obfuscating who they are.
I don’t mean those cases where they use a customers subdomain for their infrastructure or bounce address. If the customer is Harper Collins then mail “from” @bounce.e.harpercollins.com sent from a server claiming to be mail3871.e.harpercollins.com isn’t unreasonable. (Though something in the headers that identified the ESP would be nice).
No, I mean random garbage domains created by an ESP to avoid using their real domains in the mail they send and in their network infrastructure. This isn’t exactly snowshoe behaviour. They’re not really hiding anything terribly effectively from someone determined to identify them – the domains are registered with real contact information, and the IP addresses the mail is sent from are mostly SWIPped accurately – but they do prevent a casual observer from identifying the sender.
Silverpop has registered over 9,000 domains in .com that are just “mkt” followed by some random digits that they use for infrastructure hostnames, bounce addresses and click-tracking links. Apart from anything else, it’s a terrible waste of domain name space to use links.mkt1572.com where they could just as well use links1572.silverpop.com or links.mkt1572.silverpop.com.
For what they’re paying just for domain name registration and management they could probably hire multiple full time employees.
And Marketo has registered over 17,000 domains in .com that are just “mkto-” followed by what looks like a location code.
(I’m not picking on Marketo and Silverpop in particular – several other notable ESPs do the exact same thing – they’re just relevant to the end of the story).
Using garbage domains like this makes you look more like a snowshoe spammer at first glance than a legitimate ESP.
It also makes it much harder for a human glancing at your headers to correctly identify a responsible party …
… which is probably why abuse@marketo are rather tired of receiving misdirected complaints about spam sent by Silverpop from machines called something like mkt1572.com.
If you follow any infosec sources you’ve probably already heard a lot about Meltdown and Spectre, Kaiser and KPTI. If not, you’ve probably seen headlines like Major flaw in millions of Intel chips revealed or Intel sells off for a second day as massive security exploit shakes the stock.
What is it?
These are all about a cluster of related security issues that exploit features shared by almost all modern, high performance processors. The technical details of how they work are fascinating if you have a background in CPU architecture but the impact is pretty simple: they allow programs to read from memory that they’re not supposed to be able to read.
That might mean that a program running as a normal user can read kernel memory, allowing a malicious program to steal passwords, authentication cookies or even the entire state of the kernels random number generator, potentially allowing it to compromise encryption.
Or it might mean a program running on a virtual machine being able to escape from the sandbox the virtual machine’s hypervisor keeps it in and reading memory of other virtual machines that are running on the same hardware. A malicious user could sign up for a cloud service, such as Amazon EC2 Google Code Engine or Microsoft Azure, repeatedly create temporary virtual machines and grovel through all the other virtual machines running on the same hardware to steal, login credentials or TLS private keys.
It’s pretty bad.
Meltdown and Spectre
One variant has been given the snappy name Meltdown. It (mostly) affects Intel CPUs, and is trivial to exploit reliably by unskilled skript kiddies. It can be mitigated at the operating system level, and all major operating system vendors are doing so, but that mitigation will have significant impact on performance – perhaps 20% slower for common workloads.
The other variant has been named Spectre. It’s more subtle, relying on measuring how long it takes to run carefully crafted code. Whether the code is fast or slow tells the malicious actor whether a particular bit of forbidden memory is zero or one, allowing them to step through reading everything they want. This is likely to be harder to exploit reliably, but is also going to be much harder to mitigate reliably in software (I’ve seen some speculation that it might be impossible to mitigate – I’m pretty sure that’s not true, but it is going to be difficult to do so reliably and will probably have significant performance impact). It affects pretty much everything, including AMD processors (despite what their PR flacks would like you to believe).
What should you do
As a typical end user you should apply your security patches as normal to mitigate Meltdown. macOS was patched on December 6th, the Windows kernel has mitigation in place. The latest release candidate of the Linux kernel has mitigation patches in place, which’ll presumably trickle out to various distributions over the next few days.
Keep updating your ‘phones. At least some of the ARM chips in iPhone and Android are vulnerable, and the more constrained ‘phone environment may make targeted attacks more likely.
If you’re using any virtual machines or cloud hosted services then your provider has probably already done rolling reboots so they can patch their hypervisors to mitigate Meltdown. You’ll still need to update your kernel yourself, to protect against attacks within your machine, even though your provider has patched their hypervisors.
Performance (and Email)
The operating system level mitigation for Meltdown works by having the CPU throw away a bunch of information every time the thread of execution goes from the kernel back to the application. Most common applications will switch between kernel code and application code a lot so this has a significant performance impact.
Initial tests with PostgreSQL show slowdowns as bad as 23%, but more realistic workloads look to be maybe 5-15% slower, depending on the workload and the hardware features available.
I wondered whether there’d be much impact on network service performance, so I set up a test network with a couple of mailservers running latest release candidates of the Linux kernel. I sent mail from one to the other, using postfix, smtp-source and smtp-sink – smtp-source and -sink are test tools distributed with postfix that make it easy to send mail or to receive and discard mail.
I wasn’t really expecting to find any performance impact for something that was likely network limited, but ran some tests anyway, slinging a few million emails from one machine to the other and turning mitigation on and off on the sender and receiver. There wasn’t any performance impact that I could measure – if it’s there it was well below the noise floor.
So you’ll probably see slight performance degradation for some things, especially disk-heavy workloads, but nothing to worry too much about.
One of the client projects I’m working on includes doing a lot of research on MXs, including some classification work. Part of the work involves identifying the company running the MX. Many of the times this is obvious; mail.protection.outlook.com is office365, for instance.
There are other cases where the connection between the MX and the host company is not as obvious. That’s where google comes into play. Take the domain canit.ca, it’s a MX for quite a few domains in this data set. Step one is to visit the website, but there’s no website there. Step 2 is drop the domain into google, who tells me it’s Roaring Penguin software.
In some cases, though, the domain wasn’t as obvious as the Roaring Penguin link. In those cases, Google would present me with seemingly irrelevant hosting pages. It didn’t make sense until I started digging through hosting documentation. Inevitably, whenever Google gave me results that didn’t make sense, they were right. The links were often buried in knowledge base pages telling users how to configure their setup and mentioning the domain I was searching for.
The interesting piece was that often it was the top level domain, not the support pages, that Google presented to me. I had to go find the actual pages. Based on that bit of research, it appears that Google has a comprehensive map of what domains are related to each other.
This is something we see in their handling of email as well. Gmail regularly makes connections between domains that senders don’t expect. I’ve been speaking for a while about how Gmail does this, based on observation of filtering behavior. Working through multiple searches looking at domain names was the first time I saw evidence of the connections I suspected. Gmail is able to connect seemingly disparate hostnames and relate them to one another.
For senders, it means that using different domains in an attempt to isolate different mainstreams doesn’t work. Gmail understands that domainA in acquisition mail is also the same as domainB in opt-in mail is the same as domainC in transactional mail. Companies can develop a reputation at Google which affects all email, not just a particular mail stream. This makes it harder for senders to compartmentalize their sends and requires compliance throughout the organization.
Acquisition programs do hurt all mail programs, at least at Gmail.
This is the time of year when everyone starts posting their predictions for the coming year. Despite over a decade of blogging and close to 2500 blog posts, I have’t consistently written prediction articles here. Many years I don’t see big changes on the horizon, so there’s not a lot to comment on. Incremental changes are status quo, nothing earth shattering there. But I’ve been thinking about what might be on the horizon in 2018 and how that will affect email marketing.
This is the biggie in 2018. The EU General Data Protection Rules come into effect starting May 18, 2018 and start being enforced May 25, 2018. These rules effect anyone holding personal data relating to an EU resident, no matter where the company resides. This is going to change how companies interact with EU residents. More and more guidance is starting to come out and I expect I’ll be writing about this extensively over this year.
2017 saw a number of changes at different ISPs, and I expect to see those changes moving forward.
The big change was Microsoft finished their back end migration of Hotmail/Live/Outlook to the same infrastructure as Office365. This unifies some of the filtering methodology across the two platforms, although I suspect we’ll continue to see different delivery between the consumer and business sides. There was a dip in open rates at Hotmail following the migration but many folks are now seeing open rates slowly recover (that’s a blog post I meant to write last month but didn’t get to).
We can expect to see some major changes at AOL in the early part of 2018. Reliable sources tell me that OATH will be consolidating their domains on the Yahoo infrastructure in the early part of 2018. This has implications for senders mailing to AOL, Verizon and Yahoo addresses. As yet, we don’t have many details, but I will share them as I get them.
2006 – 2008 brought about a lot of industry changes, many of which weren’t specifically visible to end users or marketers. The most visible of these changes was the proliferation of FBLs. The FBLs themselves were the result of investment in hardware and software as well as some attitude shifts in the people building the systems. Likewise, we’re seeing some major investment in hardware and software in the recent past. 3 of the 4 major consumer webmail providers are visibly changing their backends. Many of the cable companies are moving around and consolidating. These changes offer the opportunity for the ISPs to address technical debt and make changes they wanted to but couldn’t because they would be “too disruptive.” When you’re migrating to a whole new infrastructure, that’s major disruption right there. Why not take the opportunity to re-think the whole email delivery process and implement changes that may not have been possible on the old infrastructure.
The short version is that I expect to see some significant changes in how ISPs manage inbound mail and filtering over the next year or so. Specifically I expect to see more domains retired and more FBLs going away. This also means the “rules” we’ve been following for the last decade just may not be applicable.
Between GDPR rule changes, ISP infrastructure changes and the fact that it’s been 10 years since we’ve seen significant changes in email I think many email and delivery experts are going to have to step up our game. Changes are coming and we’re going to have to adapt. Even more, we’re going to have to step up our game in explaining delivery to customers, colleagues and management.
I just got some mail claiming to be from “Bank of America <firstname.lastname@example.org>”.
It passes SPF:
Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=22.214.171.124; helo=bofasecure.com;
It passes DKIM:
Authentication-Results: mx.wordtothewise.com (amavisd-new); dkim=pass (1024-bit key) header.d=bofasecure.com
The visible RFC 822 From address is strictly aligned with both the SPF domain and the DKIM domain. So if they’d published a DMARC record it would have passed DMARC.
The message branding is good, and looks like Bank of America (unsurprisingly, as it’s loading assets from bac-assets.com, which is Bank of America). The only visible giveaway is that it includes an attached Word file, one which will presumably try and install malware on my machine if I load it with Word.
The perfectly passing authentication tells me it’s from bofasecure.com. There’s nothing that tells me that bofasecure.com isn’t Bank of America, and isn’t someone I should trust.