The Only Influencers list hosts a discussion about the value and use of open rates.
A potential client contacts me asking if I can get their open rates to a certain percentage.
A client shows me evidence of 100% inboxing but wants to improve their open rate.
An industry group runs sessions at multiple meetings discussing how inaccurate open rates are.
The industry needs to stop obsessing over open rates.
As measured by senders, an open means a particular image was loaded. This sometimes corresponds with an email being opened and read by a user.
There are a number of ways open rates can be wrong, though.
I mean, I get it, I use opens are easy to measure and easy to use. They’re a start for looking at a number of things. But we have to remember the data is, at best, an approximation. There are lots of folks opening and reading messages that never load a pixel (hi! is me!). There are also some people who show as opening the mail but have never looked at it.
At Gmail someone can open a mail, and then immediately mark it as spam. As I said recently, in some cases an open can hurt your reputation. “If they opened it they’re engaged with the mail” has always been an assumption. It’s become part of the delivery
They’re a data point. They’re not the be all and the end all of data points. In order to effectively use them you need to understand what they mean and what they don’t mean. They’re inaccurate at best and can be very misleading if you’re not paying attention.
We need to stop spending so much time obsessing about open rates and more time worrying about how accurate our data collection processes are.
In the 1970s, while the early drafts of the Internet were being developed, a competing model for networking was being put together by the ISO (International Organization for Standardization).
The OSI (Open Systems Interconnection) Model broke the work needed to implement a distributed network service into seven separate layers of abstraction, from the physical infrastructure at layer one all the way up to the application at layer seven. That was much more elegant and academically satisfying than the Internet’s TCP/IP, which was considered a bit ad-hoc in comparison.
It turns out that implementing a network often requires crossing the layers, so the seven layer model wasn’t a great basis for actually building a network on and the simpler TCP/IP four layer model won out.
Why do we care about an old, theoretical network model? Because even though it’s not really used in implementations it’s still an important part of the language engineers use to discuss networks. You need to be aware of it to understand references to “level 3” and “level 8” problems.
I’ve included rough mappings from bits of the email world onto each layer in this handy table:
That’s ten layers, not seven, you say. Yes, yes it is.
There’s always more to networking than just getting data in front of a user. There’ve been joking (not-joking) references to the “political” and even “religious” layers at the top of the stack. This shirt is one of the more notorious ones.
The three extra layers I gave above – individual, organization, government – are often useful abstractions, and are particularly relevant to email deliverability.
A “full stack” email expert needs to be intimately familiar with levels five through nine, and some situation will require more than a passing understanding of levels three, four and ten.
Right at the end of January, Microsoft appears to have made couple of changes to how they’re handling authentication. The interesting piece of this is that, in both cases, Microsoft is taking authentication protocols and using them in ways that are slightly outside the spec, but are logical extensions of the spec.
The first is an extension of DMARC. They’re rolling out inbox flags for Office365 users that identify some emails as Unverified Sender. It appears they’re basically doing DMARC verification whether or not the sender is publishing a DMARC record. They’re comparing the SPF and d= domains with the 5322.from to identify those senders that are unverified. Mail that doesn’t pass gets a mark in the inbox. Initial reports indicated that some messages were failing even if the authentication was in alignment as well as some increase in bulk foldering.
This is a very logical use of the concept of alignment. If the SPF domain or the d= domain are the same as the domain in the visible from, then it’s a clear sign the mail is actually from that company. And this isn’t new, companies were mentioning looking at the authentication and alignment for years. But it’s interesting that Microsoft is pushing that marker out to the user.
The other thing Microsoft is doing is around ARC. ARC is “authenticated received chain” and is one of the ways folks are trying to fix the breakage in forwarding introduced by DMARC. Essentially, when a company receives a message, they note whether or not the authentication passed, then sign the message with a ARC header (similar to a DKIM header). It’s a programatic way for a forwarder to say “hey, we received this and verified that it was authenticated and here’s our authenticating that fact.”
You may have seen ARC headers on some mail, Gmail’s been adding them for a while. At the end of January, I noticed that Microsoft was signing them fas well. But there was a bit of weirdness in it with regards to DKIM. Microsoft was asserting that they’d seen a DKIM signature that wasn’t available in the headers and a DNS lookup didn’t show any visible public key.
For Office365 tenants that have not implemented custom DKIM signing Microsoft is faking a DKIM signature and then wrapping that up in an ARC header and saying “yeah, this won’t authenticate for you, but we’re saying we authenticated it before sending it on.” The really clever bit about this is that there is no DKIM signature involved. Microsoft is using their login and customer authentication process to assign a d= to the message without forcing their customer to publish a DKIM key.
It does make for some messy headers (but ARC does that anyway). But it’s Microsoft saying “we authenticated that this person is legitimately using this domain to send mail even though they don’t have DKIM set up.” It’s outside the scope of the ARC protocol, but actually makes sense. Microsoft knows the user is legit and can just sidestep the work needed to publish custom DKIM.
It’s that time of year again where nearly all my client calls involve the question, “are you going to be at M3AAWG SF?” Up until last year, the answer was always yes. But now it’s not a brief drive up the peninsula and a BART ride into the city, it’s a transatlantic plane flight.
The clients I was talking with today said this was their first MAAWG and what did I recommend they should do or talk to. So here’s my list of things you should do if you’re new to MAAWG.
Attend the open round tables. These are where a lot of the documents and work starts so go and participate.
If you get into town on Monday attend the training sessions. Of particular note is the facilitation training that occurs on that day. MAAWG facilitation is a great way to get comfortable standing up in front of people, running a meeting and speaking out.
Go and meet folks at the member breakfast. It’s a great way to get to know folks who are there.
The working sessions are just that – sessions where folks get into a room to work on a document or some processes or a work product. Even if you just go and listen, it’s valuable to see how the group arrives at a consensus.
Introduce yourself and network. If there’s a group talking in the hallway, join them. Particularly if they’re in PacMan formation. And all of you who have been there before remember to be the PacMan.
Go to some of the non email sessions. A lot of the senders who go often focus on the sender sessions. But there’s a wealth of interesting information in the technical sessions as well. Join one and learn.
The vast majority of folks at MAAWG are awesome and welcoming and are more than happy to chat when they’re not running to another session or meeting. The best advice I can give is be there and jump right in.
Oh! And the hotel is back up on Nob Hill this year. You know those hills that San Francisco is famous for and the car chases? Yeah, one of them is Nob Hill. It’s a hike up and I do recommend a car up. You can BART from the airport, but grab an UBER or a cab for that last few blocks. Your leg muscles will thank you.
Have a great time! I’m definitely feeling the FOMO as I look at everyone making plans for the trip. But we’ll see everyone in London this summer (and, well, Dublin 2021 which is a short walk from our neighbourhood).
Podia has scraped the Word to the Wise blog and I’m currently receiving an ongoing drip campaign from them absolutely begging me to mention them in my blog post on cold emails.
I get maybe a dozen of this style of email a week. It’s pretty annoying but whatever. I delete them, blog about them or, very occasionally, share them with some folks who might have a big bigger of a stick to wave at them.
I have to admit, this time I spent about 30 seconds considering adding a note onto that article. A brilliant example of cold email that should go to the spam folder is from PodiaHQ, aka podia.com. My only hesitation is that gives them what they want, and antisocial behaviour should never be encouraged. Plus, then I’d have to actually think about something else to blog about today and it’s 4:30 already. Then I realised this is exactly the thing to illustrate how opens can hurt your delivery.
Wait. What? Opens don’t hurt delivery! An open is a positive signal, isn’t it? The user opened and read the mail this is good. Except…. when a user opens an email, gets half way through it and then immediately marks it as spam.
Using opens as a metric for who to continue mailing without also having FBLs to identify which of those opens resulted in a this is spam hit, leads to an increase in mail to folks who don’t want it and/or are receiving the message in their spamfolder because they marked the sender as spam. Longer term it can lead to reputation problems.
In the consumer context opens are a way for us to tell who is reading our mail. FBL messages are a way for us to remove anyone who doesn’t like that mail. By processing FBL complaints, folks who open the mail and then complain about it are removed from future mailings. We don’t end up with a build up of ‘engaged’ users who are not engaged at all.
The obvious exception to this is being Gmail as they don’t have an ARF style FBL. But even then, if you have a decent enough reputation Gmail will show the user a “do you want to unsubscribe, too” message. We can sorta wiggle around the lack of FBL data by treating unsubscribes through the List-Unsubscribe header as spam complaints.
That’s not how it works for B2B. There are no FBLs for business mail. Many business users do have a this-is-spam button, though. That this-is-spam button ties directly into their filtering system. What you end up with is an audience that opens a message and reports it as spam. That open is now a negative signal, not a positive one.
I think our spammer friends at Podia haven’t made the connection that their advice causes delivery problems. Or, maybe they have which is why they’re using podiahq.com in their emails instead of podia.com. Ironically, they’re not putting much effort into the subject line here. A few months ago I was toying with the idea of sharing all of the stupid B2B spam I get. I collected over 2 dozen cold outreach emails just by searching for the subject line ‘Quick Question’.
The broader picture here is that we can’t just look at metrics in isolation, particularly when we’re troubleshooting delivery. Only mailing engaged users, when you’re not also getting back complaint data, means some of the folks you’re mailing aren’t users who want your mail. At best they’re getting the mail in the spam folder due to ISP metrics. At worst the spam foldering isn’t working and they end up repeatedly reporting your mail as spam.
We make assumptions about what signals to measure and what they mean. In order to correctly model what’s happening with delivery we need to question those assumptions regularly.
As I continue to think about how people troubleshoot email delivery I keep finding other things to talk about. Today we’re going to talk about the question most folks start with when troubleshooting delivery. “Did ISP change something?”
At least once a week I check some delivery or email fora and some form of the question is sitting there.
“Did X change something? We haven’t done anything different and our delivery went way down overnight.”
Did Y change their filters? Our delivery is tanking and all our authentication is fine.”
Anyone hear of a change at Z? We have been having increasing difficulty reaching the inbox and we don’t understand why. Looking for suggestions.
In reality, the answer to this question Does Not Matter and asking it is only going to delay actually resolving your delivery issue.
When filters change
The reality is, filters are continually changing. ISPs and filtering companies are always tuning filters. These changes are roughly in 3 categories.
Ongoing tweaking and improvement to provide a better experience for their users
Specific changes to catch a type of spam they had previously been unable to effectively identify and filter.
Filters are not static. They are continually adjusting based on a number of things. We can always assume the answer to the question is yes. Something changed. Now what?
There are basically 3 situations here.
The filters did something unexpected and caught mail it wasn’t intended to catch, causing recipients to complain to the ISP.
The filter change was intentional but caught more mail than was intended, causing recipients to complain to the ISP.
The filter change was intentional and caught exactly the mail that was intended and the recipients didn’t care enough to notice that mail was missing.
In the first two cases, the ISP is going to fix things. They’re going to listen to their users and adjust the filters. In the first case, I expect to see changes and rollback within 24 – 48 hours. In the second, I expect to see changes in 24 – 96 hours.
The third case is the interesting one. Does anyone care about mail they don’t care about going to the bulk folder? Folks sending mail, even opt-in mail, that the users don’t complain about when it’s missing is the definition of grey mail. Filter maintainers listen to their users. If users complain they’ll change things, if users don’t complain they’ll assume the filters are working as intended.
The answer to the question did the filters changed tells you nothing. Of course the filters changed. Either they’re doing something that the maintainers don’t intend, which means they’ll be fixed or they’re catching mail they’re intended to catch.
Instead of asking if the filters changed, flip the question. Why are my users not interested enough in my mail to notice it when it’s gone? Start your troubleshooting from that perspective.
There are a lot of folks in the email industry that take issue with my stance that DMARC is not a viable solution to phishing. DMARC, at it’s absolute best, addresses one tiny, TINY piece of phishing.
Look at this message I received today. My mail client presents this as from Quickbooks and hides the actual from email address from me. Most mail clients do that by default. It is possible to change this in some clients, like desktop mail.app. But a lot of clients simply take the choice away from the user.
Mail clients are the biggest barrier to stopping phishing. As long as they hide the actual email address, users will be unable to tell when a message is actually phishing.
One of the things I do for clients is look at who is really handling mail for their subscribers. Steve’s written a nifty tool that does a MX lookup for a list of domains. Then I have a SQL script that takes the raw MX lookup and categorizes not by the domain or even the MX, but by the underlying mail filter.
Part of that script classifies domains hosted by Google apps as a separate filter from Gmail. Even though they’re actually all the same underlying system. I never had any real, definitive evidence that the filters were different. Just a lot of indirect evidence seeing mail delivered.
That changed today as I was checking delivery for a client. One of their mailstreams is getting 100% inboxing at Gmail, but 100% spam at Google Apps. That’s pretty clear evidence that Google Apps and Gmail are different filters.
I started looking at that mail in particular. Initially I noticed a feature of the subject line that looked like it may be something a business filter would trigger on. But, on looking deeper, there are other features that make it clear this is a different mail stream. What isn’t different is the From domain, the SPF domain or the DKIM signature.
In any case, this particular pattern makes it pretty clear that Google is specifically depositing this mail stream in the bulk folder of Google Apps users. Meanwhile the messages are going to the inbox at Gmail and all the other messages from this sender are going to the inbox at both places.
Google filters are specific and sensitive. They can identify different mail streams and target messages separately between Gmail and Google Apps.
In my previous career I was a molecular biologist. Much of my work was done on bacteria but after I left grad school, I ended up working in a developmental biology lab. Bacteria were (mostly) simple: just about every trait was controlled by a single gene. We could study what that gene did by removing it from the bacteria or adding it to a well characterised bacteria.
When I moved to developmental biology, the world got more complex. In higher organisms many traits are controlled by a whole bunch of genes, and there is a lot of redundancy and overlap and duplication. But, there was still quite a bit of removing a gene to see what happens. The lab I was in was specifically studying teratogens – chemicals that interfere with development. The most well known teratogen is thalidomide. In fact, a lot of the work we were doing with vitamin A and alcohol involved many of the same pathways that were disrupted by thalidomide.
One of the important parts of development is controlled by a complex of genes called Hox genes. These do a lot of things, but one of the most important things they do is define what parts of the embryo will become the front and back, the top and bottom and the near and far.
OK, now that we have 3 paragraphs of background, here’s the story. There was one seminar we went about Hox genes. The research being done was trying to assign specific activities to Hox genes by knocking them out. But, because Hox genes are so redundant, knocking one of them out doesn’t actually change much. There was nothing really wrong with the single knockouts this lab was studying. So, they ended up knocking out two Hox genes. At that point most things still worked, except… 2 vertebra switched places.
That story has always stuck with me, because, you have these genes that are so important they exist in everything from worms to humans. And they’re so vital that higher vertebrates like humans have the same set of genes duplicated across 4 different chromosomes. You knock out two of these vital developmental genes… and the only real evidence of anything happening is two vertebra switch places.
Recently I’ve been blogging about how to troubleshootdelivery problems. And I realised that a lot of how I treat delivery problems is influenced by my time in research. Much of how I troubleshoot starts with the premise that the things we’re testing aren’t independent variables. Everything, or almost everything, is conditional.
Email filtering, particularly that driven by machine learning, is closer to molecular biology than I realised. We can imagine each individual rule like it’s a gene. And these genes all work together and, in some cases, modify each other. Some rules don’t get activated unless another rule is active, or inactive. In some cases, one rule is so dominant none of the other rules matter. For instance, if an IP is listed on the SBL, your mail is blocked, no questions asked. But, if the sending IP isn’t listed, then hundreds of rules act on the message. Or, on the other end, if a user has a rule that says “always deliver this to my inbox” none of the rules matter, that message will always go to the inbox.
Filtering variables aren’t independent. In order to troubleshoot delivery problems, we need to start looking at the whole picture and the whole system. We can’t troubleshoot things in a vacuum.