The intent of DMARC is to cause emails to silently vanish.
Ideally deploying DMARC would cause all malicious email that uses your domain in the From address, but which has absolutely nothing to with you to vanish, while still allowing all email you send, including mail that was sent through third parties or forwarded, to be delivered.
For some organizations you can get really close to that ideal. If you control (and know about) all the points from which email is sent, if your recipients are individuals with normal consumer or business mailboxes, their mailbox providers don’t do internal forwarding in a way that breaks DKIM before DMARC is checked and, most importantly, if your recipients are a demographic that doesn’t do anything unusual with their email – no vanity domain forwarding, no automated forwarding to other recipients, no alumni domain forwarding, no forwarding to their “real” mailbox on another provider – then DMARC may work well. As long as you follow all the best practices during the DMARC deployment process it’ll all be fine.
What, though, if you’re not in that situation? What if your recipients have been happily forwarding the mail you send to them to internal mailing lists and alternate accounts and so on for decades? And that forwarding is the sort that’s likely to break DKIM signatures as well as break SPF? And while everyone would advise you not to deploy DMARC p=reject, or at least to roll it out very slowly and carefully with a long monitoring period where you watch what happens with p-none, you have to deploy p=reject real soon now?
What can you do that’s least likely to break things, while still letting you say “We have deployed DMARC with p=reject” with a straight face?
- Make sure that every bit of mail you send is sent with both valid SPF and a valid DKIM signature for the domain you use in the From address, so both are DMARC-aligned
- Use “relaxed/relaxed” canonicalization (not “relaxed” canonicalization, as that’s quite different).
- Don’t do anything “clever” with your SPF or DKIM configuration. Use simple configuration that’s likely to be well understood by recipients. Very simple.
- DKIM sign a minimal subset of headers – e.g. “h=From:Subject:Date”. Definitely don’t sign Reply-To:, To: or Message-ID: headers, as they’re the most likely to be modified during forwarding. (From: is also likely to be modified – but if it is, it’s no longer going to trigger your DMARC policy so you don’t care).
- If you’re generating bulk email, make sure that it’s not only entirely valid but that there’s nothing in the structure of the email that might trigger an intermediate server to rewrite it. Make sure that the headers aren’t excessively long, and are wrapped in a normal manner. Make sure that the mail doesn’t have any long lines in it (quoted-printable encoding is a good way to ensure that). Make sure there aren’t any non-ascii characters in the headers or body; that everything is encoded appropriately.
- If possible, consider separating your mailstreams. Put all your supposedly “security critical” email on a DMARC p=reject domain, and offload your mail that’s not likely to be impersonated onto it’s own domain, one that isn’t covered by the DMARC p=reject diktat. Do everything right on that new domain, make sure it’s all sent cleanly and with DMARC-aligned authentication … just don’t publish a DMARC p=reject record for it.
- Monitor how DMARC is affecting your mail. Make sure you’re receiving all the aggregate and forensic reports you can, and also that your postmaster and customer support teams notify you of any issue that looks at all like mail going missing (or being rejected, or going to a spam folder – as some mailbox providers won’t follow recommended practice for DMARC failures and will route the mail to a spam folder instead).
- Monitor things for as long as you can with your DMARC set to p=none, and fix as much as you can identify via DMARC forensic reports before switching it to p=reject.
- Track where mail that’s being impacted is being delivered to. If there’s low hanging fruit, maybe reach out to the forwarder and work with them to fix things so that their forwarding doesn’t break your DKIM signature. As a last resort, adding the forwarders IP address to your SPF record will allow DMARC to pass (at the cost of removing any pretense of DMARC protection for those recipients).
Have a relaxing picture of a forest. You’re going to need it.
But, unfortunately, some senders don’t actually think unsubscribe means stop sending mail.
Today, for instance, the nice folks at The Container Store sent me an email with an “important update to my POP! account”
Yes, that’s an address I gave them. But I don’t have any record of setting up an account. I was on their mailing list for all of 4 emails back in November 2016 before unsubscribing. But, they’ve decided they can email me despite my unsubscribe request.
They’ve cloaked this as an “Important Account Update” about some account I don’t have. In fact, when I go to their website and try and see what this oh so important account is about they tell me:
I understand legitimate account notifications might be an acceptable excuse to send mail even after the recipient opted out. This, however, was done extremely poorly. There is no record of the account that they are sending me information about. Neither the company nor I have any record of this account of mine.
At a minimum the emails should have only be sent to the folks that actually had an account. But, they weren’t.
I also have some issues with a company requiring recipients to accept email in order to continue using reward points. As a recipient, if I wanted what they were offering I might go ahead and continue receiving emails. But, I might not. It would all depend on how aggressive their email program is and how good the rewards are. As a deliverability consultant, this strikes me as a great way to create a mailing list full of unengaged users. Unengaged users lead to spam foldering and eventual failure of an email marketing program.
Whatever some executives think, and having been in this industry for a decade and I half I’m sure this is coming from the top down, this is not a good way to build an email program. You really can’t force folks to accept your email. ISPs are too protective of their users to make that a viable strategy.
In the deliverability space we talk about permission and consent a lot. All too often, though, consent is taken not given. Marketers and senders assume they have permission to send email, while the recipient is left expecting no email.
Other forms of taking permission include scanning badges at conferences and uploading address books to bulk sending programs. Neither of these things are actually permission. Just because someone corresponds with an employee does not mean they are giving permission to be added to mailing lists.
Modern laws are attempting to address this. Both CASL and GDPR make it clear that senders can’t simply assume they have permission to send mail. Permission must be explicitly granted and the terms of the permission must be clear. The new California privacy laws are also trying to make consent explicit.
Overall, laws requiring explicit permission are a direct result of too many marketers and senders assuming consent. The regulations weren’t created out of nothing, they’re a response to ongoing abuses by the marketing industry. The marketing industry had the ability to head this off by acting reasonably, but self regulation wasn’t on the table.
As I’m writing this, I’m watching Deputy Atty General Rod Rosenstein discuss the indictments of 12 Russian military officers for hacking activities during the 2016 election cycle. One of the methods used to gain access to systems was spearphishing.
I think most of us know what phishing is, sending lots of emails to a wide range of people in an attempt to collect some credentials. These credentials are usually passwords to bank or email accounts, but can also be things like amazon or other accounts.
Spearphishing is an attempt to collect credentials from a specific person. The net isn’t thrown wide, to collect any credentials, rather individuals are targeted and researched. These attacks are planned. The targets are carefully researched and observed. The emails are crafted specifically for that target. If one set of emails doesn’t work, then they try again.
In terms of email marketing and deliverability, phishing is something detectable by many anti-spam filters. They’re sent in bulk, and they all look similar or identical to the filters. Spearphising isn’t as simple to detect with standard tools. What many organizations have done is try and combat this with warnings in the client. Like this one from gmail:
Security is becoming a bigger and bigger part of email filtering. I expect that as filters start addressing security more, we’ll see increased warnings like the above.
What can senders do?
- Even if you can’t publish DMARC records make sure your domains are aligned.
- Expect and plan for filters crawling links at delivery time.
- Limit the number of redirects for any one click. (one is fine, 2 or 3 is probably OK, 7 or 8 is probably too much).
This is another example of the outside factors that are driving filtering and affecting email marketing.
The more I look at the industry, the more convinced I am that we’re in the middle of a fundamental shift in how email is filtered. This shift will change how we handle email deliverability and what tools we have and what information we can use as senders to address challenges to getting to the inbox.
This period is roughly between 2001 and 2006. Many of the email filters were focused on blocking unsolicited email. They were mostly IP based and were very all or nothing. Mail is either accepted or it’s not. It wasn’t until 2003 that Yahoo and AOL (now OATH) announced spam folders that they would put mail into. There were few formal communication channels between ISPs and mail senders.
We did, of course, have informal channels. Often delivery could be improved by asking the right people to make some adjustments to their internal filters. And, to be honest, there were some people who used this position of power to personally enrich themselves (the good news is most of them are no longer in the business).
As senders, there was very little information we had to work with. Most communication with ISPs went through gatekeepers. The primary gatekeepers were the companies providing certification services: Bonded Sender, Habeas, Goodmail.
First inflection point
The first inflection point happened in the middle of the 2000s. Over the course of a few years, there were some significant changes all of which served to create the email industry as we now know it. This post certainly isn’t going to cover the full range of changes, but there are a few notable things that happened.
Expansion of broadband. As broadband availability expanded and became more affordable, many of the old dialup ISPs simply folded. With it, folks stopped using ISP provided email addresses and moved to free email addresses provided by companies like Yahoo and Hotmail.
Gmail. Google launched the beta for Gmail in 2004. Most of what I remember about this was invites to the private beta being shared wildly. Everyone wanted to get in. Initial buzz aside, Gmail is consistently one of the top 3 domains for consumer and business (through hosted G Suite accounts) mailings alike.
Postmaster pages. ISPs started codifying what it took to get mail delivered to that ISP. One example is how pages published what they expected in terms of connections and messages per connection. Those pages also started explaining some of the SMTP return codes ISPs were using.
Feedback loops. In 2006 ISPs all over the US started announcing Feedback Loops. What a lot of folks don’t realize is this expansion in FBLs was driven in part by Return Path’s willingness to provide infrastructure to the ISPs to manage FBL registration.
These changes, along with some others, worked together to create a delivery environment that, while not always welcoming to bulk mail, was understandable.
For the next 5 to 8 years, this was the environment we worked in. Sure, there was some consolidation in the email space: ESPs were launched and bought, new deliverability companies arose, some accreditation companies folded. Overall, though, it was pretty solid and companies that followed a few simple rules were able to reach the inbox.
Today, though, I’m seeing a lot of those rules changing. Companies who keep their bounces and complaints low are still sometimes seeing delivery problems. They’re not able to reach the inbox, for at least some of their mail streams. Again, there are a number of things happening concurrently that is driving this change.
ISPs are changing. Two of the 4 major consumer ISPs Microsoft, AOL, Gmail, Yahoo (MAGY) providers have merged, to form what I’m fondly calling OMG (OATH, Microsoft, Gmail). We’re also seeing consolidation in the cable space with the Time Warner / ATT merger. Other cable companies are merging, too, and rethinking things like their postmaster pages. Additionally, FBLs are going away and no ISP offers whittling anymore.
Privacy changes. We’re also seeing a global focus on privacy and tracking and user data. A lot of this is translating into how the email providers and ISPs are handling incoming email. A lot of the changes we’ve seen in the last 12 months was driven, in part, by ISPs trying to upgrade their infrastructure to deal with the effects of GDPR. But California is passing its own GDPR style law, which will affect technology companies across the US.
Security Concerns. Email is a primary source of infections inside corporations. All of the major breeches we’ve heard about in the past few years have stared with email. Filters aren’t about spam any more, they’re about protecting end users and corporations from malicious traffic. This is going to affect how mail is processed and delivered and what can and cannot reach the inbox.
This is a very abbreviated list of the things that were happening at each stage in the process.
One of the bigger challenge is the ISPs are moving to less human involvement in filters and answering deliverability questions. They’re moving to the Hotmail “fill out the form and parse the boilerplate” model of deliverability support. Or they’re moving even further to the Gmail style “here’s a help page, have fun!” Even folks who know people inside the ISPs, are having problems getting the same helpfulness of answers that they had in the past. I’m not blaming the ISPs or the folks who work there. But, as I see it, providing support to senders was often a way for ISPs to tweak their filters. Now that the ISPs are using machine learning and AI as a way to filter mail, they need less in the way of direct feedback.
Overall, I think we’re in the early stages of another major shift in how email is managed at the ISPs. This will change how we have to send emails in order to reach the inbox. We have to think about the filters and what their goals are. Having good stats isn’t going to be enough. Almost anyone can have good stats, and there are large companies senders can pay to clean up their stats to be within the ISP thresholds. Once you can buy low bounce rates or low complaint rates, those stats are useless for determining the health of your email program.
Smart senders will start thinking about these issues sooner rather than later.
Mimecast is a filter primarily used by businesses. They’re fairly widely used. In some of the data analysis I’ve done for clients, they’re a top 10 or top 20 filter.
Earlier today someone asked on Facebook if mimecast may be blocking emails based on the TLD. The short answer is it’s unlikely. I’ve not seen huge issues with them blocking based on TLD of the domain. They’re generally more selective than that.
The good news is mimecast is really pretty good about giving you explanations for why they’re blocking. They’ll even tell you if it’s mimecast related or if it’s a specific user / user-company block.
Some example rejection messages from a recent dive into some bounce logs.
- Administrative prohibition – envelope blocked – https://community.mimecast.com/docs/DOC-1369#
- Email rejected due to security policies – https://community.mimecast.com/docs/DOC-1369#
- Envelope blocked – User Entry – https://community.mimecast.com/docs/DOC-1369#550
- Invalid Recipient – https://community.mimecast.com/docs/DOC-1369#
- Message expired -> Open relay not allowed – https://community.mimecast.com/docs/DOC-1369#451
- Rejected by header based Blocked Senders: firstname.lastname@example.org – https://community.mimecast.com/docs/DOC-1369#
- Rejected by header based manually Blocked Senders: email@example.com – https://community.mimecast.com/docs/DOC-1369#
- Remote server returned message denied by administrative policy -> Administrative prohibition – envelope blocked – https://community.mimecast.com/docs/DOC-1369#550
- spamcop.mimecast.org Blocked – see http://www.spamcop.net/bl.shtml?10.10.10.10. – https://community.mimecast.com/docs/DOC-1369#550
If you look at the page linked to you can see that there is a huge amount of flexibility in how and who can block mail using mimecast. Mimecast itself can push filters, local administrators can filter mail for the particular domain they manage, and individual users can set up filters. And, users seem to take advantage of that.
Dealing with a mimecast block involves figuring out who is responsible for the block. Luckily, the mimecast rejection messages and documentation give clues as to whether it’s the local administrator configuring the policy or if it’s the end user. In most cases it’s not actually mimecast blocking the mail.
Mimecast provides tools and an interface to manage incoming mail, but does not actually push out rules like many of the other appliances. That’s good and that’s bad. It’s good because you don’t have a 3rd party making delivery rules for different businesses. It’s bad because once a company administrator gets to the point of blocking specific mail it’s going to be very difficult to convince them to lift that block.
Why? Remember the discussion about productive mail?
Productive Mail: Mail that furthers a business’ goals and supports their underlying business model. Mail can be both solicited and wanted by specific endusers. But, a particular company can decide to block mail simply because they don’t see the mail as beneficial to the overall business. Thus the mail is blocked for being unproductive.
We can assume that employees who have access to create mail blocks in mimecast, and other business filters, have the authority to do so. Which means when you’re looking to get unblocked through mimecast, you’re likely having to convince the very person who blocked you to unblock you.
These types of blocks are distinctly different than negotiating with a consumer ISP or even a filtering company. There is no appealing to engagement or appealing to solicited. The business doesn’t really care about either, all they care about is their employees are working while they’re at work and using corporate resources.
I’m back in the office after a busy June. The 2 continent, 3 city tour was unexpectedly extended to a 4th city thus I was out most of last week as well.
What was I doing? We spent a week in Dublin, which is an awesome and amazing city and I love it a little bit more every time we visit. After Dublin I jetted off to Chicago, where I spoke at ActiveCampaign’s first user conference.
The talk I did for ActiveCampaign was about how we’re in the middle of a fundamental shift in how email is filtered, particularly at the consumer ISPs. In order reach the inbox. we need to think beyond IP or domain reputation. We need to stop thinking of filters as a way of sorting good mail from bad mail. I touched a little on these concepts in my What kind of mail do filters target? blog post.
The shift in filtering is changing how email reaches the inbox and what we can and should be monitoring. At the same time, the amount of data we can get back from the ISPs is decreasing. This means we’re looking at a situation when our primary delivery fixes can’t be based on feedback from the filters. This is, I think, going to be an ongoing theme of blog posts over the next few months.
The next trip was to spend 2 days onsite at a client’s office. These types of onsite training are intense but I do enjoy them. As this was mostly client specific, there isn’t much I can share. They did describe it as a masterclass in deliverability, so I think it was also intense for them.
That was the planned 2 continent, 3 city tour. The last city was a late addition of a more personal nature. We headed downstate to join my cousin and her family in saying goodbye to my uncle. He was an amazing man. A larger than life, literal hero (underwater EOD, awarded the silver star) whom I wish I had known better. Most of what I remember is how much he loved and adored my aunt.
I’ll be getting back into the swing of blogging over the next few days. It’s good to be back and not looking at traveling in the short term.
A c/p from an email I sent to a mailing list.
I think we’re seeing a new normal, or are still on the pathway to a new normal. Here’s my theory.
1) Hotmail made a lot of underlying code changes, learning from 2 decades of spam filtering. They had a chance to write a new codebase and they took it.
2) The changes had some interesting effects that they couldn’t test for and didn’t expect. They spent a month or two shaking out the effects and learning how to really use the new code.
3) They spent a month or two monitoring. Just watching. How are their users reacting? How are senders reacting? How are the systems handling everything?
3a) They also snagged test data along the way and started learning how their new code base worked and what it can do.
4) As they learned more about the code base they realized they can do different and much more sophisticated filtering.
5) The differences mean that some mail that was previously OK and making it to the inbox isn’t any longer.
5a) From Microsoft’s perspective, this is a feature not a bug. Some mail that was making it to the inbox previously isn’t mail MS thinks users want in their inbox. So they’re filtering it to bulk. I’ll also step out on a limb and say that most of the recipients aren’t noticing or caring about the missing mail, so MS sees no reason to make changes to the filters.
6) Expect at least another few rounds of tweak and monitor before things settle into something that changes more gradually.
Overall, I think delivery at Microsoft really is more difficult and given some of the statements coming out of MS (and some of the pointed silence) I don’t think they’re unhappy with this.
A quick post to say that posting will be light the next few weeks. I’m off later this week to visit Dublin. After I get back from that I’m headed to Chicago to speak at ACTIVATE hosted by Active Campaign. If you register by tomorrow you can use the code ACTIVATE and get in for $200. It’s looking like a good conference.
I’ll be speaking about deliverability, specifically how email filtering is all sorts of changing. My focus is on how the common “deliverability” techniques aren’t as effective in the new filtering environment. I’ll also be talking about further changes I see coming and how to address them.
After Chicago I’m onsite at a client’s for 2 days in Florida.
Basically, my June is booked. Both Steve and I will be blogging as we get inspired or have something to say. Overall, though, I’m giving myself time off from blogging through the end of the month.
We often say that a blacklist has “listed the world” when it shuts down ungracefully. What exactly does that mean, and why does it happen?
Blacklists are queried by sending a DNS lookup for an A record, just the same as you’d find the address of a domain for opening a webpage there. The IP address or domain name that’s being queried is encoded in the hostname that’s looked up.
For example, if you wanted to see whether the IP address 18.104.22.168 was listed on the SpamHaus SBL you’d ask DNS for an A record for the hostname 22.214.171.124.sbl.spamhaus.org. If that returns an answer, the IP address is listed. If it doesn’t, it isn’t.
If a blacklist returns an answer for any IP address (or domain) you ask it about it’s “listing the world” or “listing the internet”, saying that everyone you ask about is listed.
Sometimes this is done intentionally as an attempt to get people to stop using a blacklist. If it blocks all your mail, you’ll stop using it. Unfortunately, that never works. Most blacklists aren’t used to block mail, they’re used as part of a scoring based spam filter. And a blacklist that’s poorly run or unmaintained enough that it shuts down ungracefully probably wasn’t trusted much, so added a very small spamminess value to a spam filters score … so nobody notices when they start listing every address.
More often it’s done when a blacklist is abandoned, leaving it’s base domain name to expire.
When a domain expires it reverts to the control of the registrar and eventually is resold, typically to a domain squatter. (A domain squatter is someone who buys up domains when they become available and hopes to sell them on at vastly inflated prices).
Both the registrar and the squatter really want to resell the domain, for a lot of money. But while they control the domain they might as well make tiny amounts of money from it. The way they do that is to run advertising on the site, typically with low end banner or text ads (cheap to serve, low standards as to where they can be run) along with a link to “Buy This Domain For A Lot Of Money!”.
Every bit of traffic that went to websites in the expired domain is valuable to them – every misdirected open from someone looking for the expired content is now an advertising view. They don’t know what hostnames in the domain were actually in use. www.example.com and example.com are a safe bet, but there may also have been forums.example.com, webmail.example.com, chat.example.com and so on …
They don’t know, or care, what hostnames were in use. They just want as many page views as possible to inflate the tiny amount of money they’re getting from their text ads.
So they set up wildcard DNS for the domain, pointing it at a webserver that’s configured to show a domain-specific advertising page for any hostname pointed at it.
*.example.com -> 192.0.2.25
That means that forums.example.com will resolve to 192.0.2.25, as will www.example.com.
And so will 126.96.36.199.nfn.example.com – so anyone using nfn.example.com as a blacklist will get a valid A record response for any IP address the look up. It “listed the world”.