It’s a near silent St. Paddy’s day here in Ireland. We took a walk along the canal and took in the silence. On the way back, our neighbour’s kids had decorated their front window and I had to take a picture.
Things are weird. Be kind to yourself and to others. Wash your hands. Don’t touch your face. Don’t send that email to your entire database.
Gartner has some really good recommendations for companies considering mailing about the coronavirus pandemic.
Launch your COVID-19-themed marketing email campaign only if you can answer yes to four questions:
- Am I telling customers something different from other brands versus saying the same thing as everyone else?
- Am I telling customers something they don’t already expect of my company or brand?
- Is the WIIFM (what’s in it for me) conspicuous in the subject line and opening paragraph?
- And, most importantly, is the WIIFM attuned to your customers needs right now?
Things are scary right now. But many of the companies who are sending emails DO NOT NEED TO DO SO. The insurance company I deal with solely by email didn’t need to send me email telling me their office was closed. I’ve never been to their office.
The vast majority of what I’m hearing from recipients and consumers is that this mail is all useless and they’re deleting without reading. Too much irrelevant or annoying mail will drive unsubscribes and this is spam hit. The first means you can’t mail that person again. The second means your reputation will take a hit.
Think twice before sending that mail. Most of you don’t need to be sending it.
This is a followup from a post a few weeks ago about authentication changes at Office365. We have some more clarity on what is going on there. This is all best information we have right now.
Microsoft is now requiring authentication to match the visible from address in order to reach the inbox at Office365. That means, either the SPF domain or the DKIM domain must align (in the DMARC sense) to the visible from domain. Simply, that means that the visible from and the signing domain must be identical or one must be a subdomain of the other.
The reason they’re doing this is to protect their users from forged emails. I can’t fault them for this at all. Many of their customers are SMBs. These businesses are targets for wire fraud, to the tune of tens of billions of dollars. In fact, one of the other companies my bookkeeper worked for in CA almost got roped in by this fraud back in 2016 or so.
Microsoft has always been looking for ways to validate the visible from address. That’s a big part of their push for SenderID, which suffered really poor uptake. This is leveraging the philosophy of DMARC and the improvement in support for authentication technology that’s developed over the last 15 years.
Adapting to this will be challenging for some ESPs, particularly those that service the SMB market. At many of these companies, handling technical issues is often handled by employees who manage technology as a small part of their job. Thus, there is a steep learning curve when trying to deploy new technology. Others have consultants or outsourced technology, many of whom are great at handling internal Windows networks and hardware, but don’t really get the intricacies of email authentication.
I see this as somewhat akin to Yahoo deploying DMARC p=reject. That was a significant and email breaking change implemented by Yahoo in response to specific security issues. This made it clear to other consumer mail providers, email intermediaries and receivers that DMARC was something they’d have to adapt to. That adaptation was neither easy nor cost free. But it did force a change in how ESPs were doing business.
Here, we have Office365 making a decision that is significant and email breaking, even for some of their customers. It may be that longer term we see other consumer webmail providers starting to tighten down their requirements for alignment even in the absence of a DMARC record. I don’t think it’s that unreasonable, ESPs have had 6 years to build the infrastructure to manage this.
The takeaway here is that if your customers are having problems getting mail into Office365, one of your first troubleshooting steps should be to ensure that authentication aligns with the visible from address. If it doesn’t fix that first. Of course, alignment is not magic wand into the inbox. If your content is spammy or your reputation is poor, your mail will go to the bulk folder.
Recently had the pleasure of sitting down with Jillian Bowen and talking about deliverability for her podcast.
Back at the office after traveling to visit a bunch of our US friends recently. A lot of news, both in and out of the email space, happened while we were gone. The biggest stories are outside the email space and I will admit to following the coronavirus news probably closer than I should. (My graduate work was done across the hall from one of the major avian epidemic monitoring labs. This is the kind of thing we discussed at lunch and over beers.)
The next few paragraphs are general musings about the state of the world. Scroll down to “IN EMAIL NEWS” if you just care about that.
As a consequence of the spread of COVID-19, many conferences and company gatherings are being cancelled. I know it’s frustrating to not get to go to a conference you’ve been looking forward to, but minimising contact really is the best way to slow the spread of the infection. There’s also the developing financial crisis and oil wars. Fun times.
A number of conferences are being cancelled and postponed. Many folks in the industry are on orders to work from home. The good news is that the data from Hong Kong shows that social distancing works to curb the spread of disease (and even worked to lower seasonal flu there this year). If your conference isn’t cancelled, don’t shake hands, wash your hands and don’t touch your face.
While we were gone the judge dismissed Tulsi Gabbard’s case against Google. The whole ruling is short and worth a read. But the crux is here:
Plaintiff’s essential allegation is that Google violated Plaintiff’s First Amendment rights by temporarily suspending its verified political advertising account for several hours shortly after a Democratic primary debate. Plaintiff’s claim, however, “runs headfirst into two insurmountable barriers—the First Amendment and Supreme Court precedent.” Prager Univ. v. Google LLC, No. 18- 15712, 2020 WL 913661, at *1 (9th Cir. Feb. 26, 2020).
I know there were some folks in the email space who were hoping her claims for biased email filtering would get adjudicated, but it wasn’t going to happen. Not only is there extensive case law reinforcing that ISPs can filter any mail they want, but some data shows that Google isn’t biased against her email.
The Markup have been working on a story related to filtering political email where they created a mailbox and signed it up for mail from lots of candidates and then looked at where the mail went. Much of the mail from all candidates is going to promotions. Tulsi’s mail is certainly not being treated any harsher than other candidates in this test.
As I’ve talked about, there are a lot of things that go into delivery at Gmail and I spent quite a bit of time talking with the folks researching this article about 6 months ago. The concern is that ISPs are influencing elections by how they filter. I don’t think this is true for a number of reasons.
While free consumer email addresses are ad supported, there’s not the same expectation that email is a cash cow. One reason is simply email predates the ad supported internet. And aggressively inserting ads into email accounts has not been well accepted by the users. Another reason is that economically, it is cheaper to keep a existing email user happy and on the platform than it is to acquire a new one. Social media doesn’t care about keeping users happy, in fact, they make money from outrage.
Email filtering culture is another thing that keeps me from thinking there are rogue operators blocking emails for candidates they don’t like. The culture started back in the late 90s with individuals working out ways to keep spam out of their inboxes and sysadmins trying to keep spam off their USENET spools. During that time, there was a significant amount of energy and thought directed to the idea that the Internet was a meritocracy and that they weren’t shutting down ideas, rather they were shutting down behavior. Some of the common phrases used by folks handling these early filters were “consent not content” – ie, if you wanted that message, no one was going to block it. Another was “abuse of the net, not on the net” a little more problematic, but again, it was about causing harm to the underlying infrastructure rather than hurting individual people. Many of the individuals around during this time went on to found filtering companies, work for major ISPs in their filtering creation departments and become thought leaders in the space. This underlying culture is still a part of the filtering in email.
Email was built into the fabric of the internet. Folks have been wrestling with the question of how to stop spam and “unwanted” mail for almost a quarter century now. Email filters, at the consumer ISPs in particular but also at the various business appliances, are monsters of machine learning. It’s nearly impossible to “whitelist” any particular sender and exempt them from the filters acting on their mail. The flip side is also true, it’s difficult to secretly blacklist a group of senders based on their political leanings. The underlying machine learning engines just doesn’t have those kinds of switches built into them.
In any case, the data from The Markup’s article was going to demonstrate that Tulsi is not being treated unfairly by Gmail’s filters.
It’s been a light blogging month. We’ve been dancing around getting the final plans, financing, and contractors set up for the work we’re doing on the Dublin house and then heading off for our first actual vacation in almost 5 years. But, I wrote half of this answering a question on mailop, so I may as well polish and publish.
FCrDNS stands for Full Circle reverse DNS or Forward-Confirmed reverse DNS. It means that if you do a DNS lookup on the domain in a reverse DNS lookup than that domain will point back to the original IP. The name actually comes from the fact that if you start with the IP address and go through the hostname, you get a full circle.
The reason FCrDNS is a thing is because any IP address owner can assign any domain to the rDNS of an IP address. They are in complete control and there are no technical checks that the hostname be a domain they own. Anyone could assign their IP a rDNS of angrygoose.google.com, or flowerchild.facebook.com or jupiter.spamhaus.com to their IPs. And, in fact, lots of spammers did just this, assigning domains to their IPs that they didn’t own.
Spammers lie, a lot. The did all sorts of things to avoid being blocked. Stealing legitimate domain names in their rDNS was one of those. They’d set up their IPs forging known domains as a way to try and get around some filters. Receiving systems figured this out pretty quickly. They started doing FCrDNS checks to verify that the person managing DNS for that IP space also manages DNS for the domain space. The underlying idea, is that if the IP points to a hostname and that hostname points back to the same IP, then everything is under control of the same entity.
FCrDNS is a method of deciding whether or not the IP address is legitimately being used by the domain in the rDNS entry. FCrDNS is a way to verify the identity of the connecting IP. If the rDNS doesn’t match, then it’s much more likely that the mail is coming from an illegitimate source.
Basically, any time you set up rDNS on an IP address it’s good practice to give the corresponding hostname an A record. For IP sending outgoing mail, this is one of those expected best practices. There’s an IP address with a rDNS of a single hostname and the hostname points back to the IP address. That IP uses the same hostname to introduce itself during the SMTP transaction. Certainly when I’m looking at IP addresses and domains and EHLO values I do check to see if everything matches.
But. Not every hostname has to have a single A/AAAA record. A single hostname can point to multiple IPs:
A single IP can also point to many different hostnames or no hostnames at all. In fact spot checks show me that none of the IP addresses in the example above actually have a rDNS set up.
|
1 2 |
;; QUESTION SECTION: ;2.160.97.40.in-addr.arpa. IN PTR |
The ability of an IP to point to many hostnames and a hostname to point to many IPs complicates completing the circle. Anyone verifying FCrDNS on an IP with multiple PTR records needs to do multiple DNS lookups for the verification step. Lookups can quickly get out of hand if each of the domains in the PTR has multiple IPs then there’s even more DNS work.
These technical and practical realities are why we can only recommend that an IP sending mail have FCrDNS, we can’t require it. And, in fact, not all outgoing mail servers do have it.
FCrDNS is a hack to link an IP address to a domain. That’s all it’s there for. You set it up if you can, and should probably expend some effort to do so for dedicated outbound servers, particularly those sending bulk mail. But, no, your 5321.from domain doesn’t need to point to an IP simply so you can check this box
According to the UK DMA, marketers report improvements in deliverability after GDPR went into effect.
I mean, thank you UK DMA for doing the research. Of course deliverability is going to improve when you send mail to those people who have expressly opted into your mail.
76% report an increase in open rates in the past 12 months; 75% say click- throughs are higher; 51% say ROI has risen. Opt-out rates have decreased according to 41%, and 55% say spam complaints are also down — further evidence of the GDPR effect. DMA UK Marketer Email Tracker 2019
We often gloss over the most important best practice: send mail to people who ask for it. Many of the data hygiene recommendations are all about making sure that the recipients still want your mail.
Recipients, and ISPs, expect senders to have permission before they start blasting out emails. Most best practices are a way to confirm permission. The others, things like authentication and the like, they’re ways for ISPs to identify your mail streams so they can assign reputation and effectively deliver mail.
Would you trust an address verification company that used twitter spam to advertise their product?
Would you trust an address verification company that sent email spam to advertise their product?
Would you trust an address verification company that sent email advertising their product to made up addresses?
Would you trust an address verification company that sent email advertising their product to spamtraps?
Would you trust an address verification company that sent email advertising their product to role accounts?
Would you trust an address verification company that uses a botnet as part of their business?
Would you trust an address verification company that sends spam to verify addresses?
Would you trust an address verification company that resells customer address data to other customers?
Would you trust an address verification company that lies about their relationship with blocklists like Spamhaus?
If you’re using an email address verification company, chances are they’re doing at least one of the above.
Address verification is primarily a way to identify which addresses are deliverable. The simple way to verify an address is to start a normal mail transaction, tell the MX you are sending mail to a particular address and then wait for the MX to tell you if that email address is active or not. Once the MX has said the address exists or not, you can quit the email transaction and go on to the next MX.
This technique was pioneered by spammers back in the early 2000s. Known as dictionary attacks they were a way for spammers to identify which addresses might exist on a domain without having to spend any effort sending mail to non existent ones. This technique was also used by companies who sold addresses to ensure ‘high deliverability’ for their lists.
Dictionary attacks caused a number of problems for the ISPs, including tying up SMTP resources for hours or days at a time. ISPs quickly developed automated defences against these types of attacks. In fact, I suspect some of these defences are why spammers pivoted to selling address verification services.
I can hear the protests now, just because they use the same underlying technology and intend to do the same thing, address verification isn’t the same as a dictionary attack. It’s not a problem! People aren’t doing it for the same reason. Well, except they are. The fundamental idea behind both processes is the same: to remove a non-existent address before sending mail to it. There is not even a real difference in intention here.
If ISPs wanted senders to verify addresses before sending to them, they would have never turned off VRFY. That’s right, there’s verification built right into the SMTP protocol. This was quickly abused and disabled at most ISPs. It’s not a far stretch to say ISPs don’t support this activity since they disabled VRFY and they implemented active measures to stop 3rd parties from probing their MX looking for active addresses.
Some ISPs, like Yahoo, have gone a step further. They moved all of their rejections, even ones for non-existent addresses, until after the entire email has been sent to them. It’s impossible for dictionary attacks to identify deliverable addresses at domains they manage.
My biggest issue, though, is not that address verification is what spammers do, or that the ISPs don’t like it or that it’s borderline abusive when done in bulk. The reason I hate address verification is that is erases one of the most valuable pieces of data we have about how accurate our address collection process is.
Bounce rates give us an indication of how inaccurate our address collection process is. We know that every bounced address is an address that doesn’t belong to the person who entered it into the form. Whether it was a typo or a deliberately faked address it is a fact someone gave us an address that didn’t belong to them. Bad addresses are a way of monitoring data accuracy.
This doesn’t mean that every deliverable address is accurate. We have to assume that some of the deliverable addresses also don’t belong to the person who submitted it. Senders who use address verification are lulled into a false sense of security. They assume that a deliverable address is a correct address. This simply isn’t true.
To address the Yahoo problem and the deliverable problem, many address verification companies have moved on to new techniques. One of the things they’re doing now is collecting PII on recipients for resale. I’m not sure how honest they’re being with their customers about this. The words proprietary technology hide a multitude of sins and business practices.
Let’s be honest, though, no company actually has permission from the address owner to collect this data. Senders typically don’t have permission to share email addresses with 3rd parties for commercial gain. The third parties don’t have permission to collect data about the address. We all kind of hand wave around the issue, but the reality is, if consumers knew their activity was being tracked in this manner, many would object.
There’s a woman in Florida who thinks the gmail address I acquired in 2004 belongs to her. I get lots of things on her behalf: event tickets, forwards from her church groups, invites for Sunday brunch. A couple years ago an insurance company sent me all her documents. Do you know how much PII is in insurance documents? Address, date of birth, social security number, home value, next of kin… the list goes on and on.
The only thing typical address verification does is identify if an address is deliverable or not. It cannot say an address belongs to the person the sender thinks it does. If the insurance company did true data verification rather than simply identifying an address was deliverable, then K.S. who lives on the Loxahatchee River in Jupiter, Florida would have her insurance documents and I wouldn’t know anything about her.
Address verification doesn’t actually fix the real problem, it hides it. The real problem is data accuracy. With verification we can pretend our data is accurate, simply by removing the very visibly non-accurate data. But the much bigger issue is sending mail to addresses that don’t belong to our customers. The most problematic of those is where the email address belongs to a completely different person. Address verification does nothing to affect that, other than hide that our address collection process is broken.
Address verification has been aggressively sold as a way to make data more accurate, but it doesn’t. All standard verification does is hide very obvious data errors. So what can we do?
I know my anti-spammer friends are jumping to tell me that confirmed opt-in is the solution. They can just sit back down because COI has it’s own problems. As senders we can’t always trust that just because a link is clicked that we are connecting with the right person. One ESP tracked thousands of clicks on a confirmation link sent to a spamtrap. Various security appliances and filters follow links in emails. Even some spamtraps will follow links in messages, including confirmation links. COI is not the solution.
At some point, the email industry has conflated the issues of deliverability and data accuracy. They’ve assumed if their delivery is good their data must also be good. It’s never been true, though,
The first step of solving a problem is admitting there is one. Address verification lets senders of all sizes pretend there is no problem with their data collection. It hides the most visible examples of bad data, and lets companies ignore the true scope of the problem.
We need to measure data accuracy by the metric of data being accurate not by the metric of being deliverable. How we do that depends on how important or vital the data is. Banks, health care and insurance providers should incorporate much more verification into the data processes. But even receipts and tickets and ‘low value’ email gets misdirected all the time.
I wish I had a pre-packaged, easily saleable product that I could tell you would fix the issue. But there isn’t going to be one solution, nor is it going to apply to every situation. I certainly can’t sell it as a fix to deliverability problems. Let’s be honest, even perfectly accurate data is going to have delivery problems.
The challenge I pose to email marketers is to incorporate real data validation into your collection processes. Start paying attention to the accuracy of your data instead of focusing on deliverability. Stop designing address acquisition processes that focus on how to collect the most email addresses with the least amount of friction. Design processes starting with the question: how can we link the email address to the person associated with this transaction, account or inquiry? What are the ways to close the circle so we know our data is accurate?
In 2020:
The industry needs to stop obsessing over open rates.
As measured by senders, an open means a particular image was loaded. This sometimes corresponds with an email being opened and read by a user.
There are a number of ways open rates can be wrong, though.
I mean, I get it, I use opens are easy to measure and easy to use. They’re a start for looking at a number of things. But we have to remember the data is, at best, an approximation. There are lots of folks opening and reading messages that never load a pixel (hi! is me!). There are also some people who show as opening the mail but have never looked at it.
At Gmail someone can open a mail, and then immediately mark it as spam. As I said recently, in some cases an open can hurt your reputation. “If they opened it they’re engaged with the mail” has always been an assumption. It’s become part of the delivery
They’re a data point. They’re not the be all and the end all of data points. In order to effectively use them you need to understand what they mean and what they don’t mean. They’re inaccurate at best and can be very misleading if you’re not paying attention.
We need to stop spending so much time obsessing about open rates and more time worrying about how accurate our data collection processes are.
In the 1970s, while the early drafts of the Internet were being developed, a competing model for networking was being put together by the ISO (International Organization for Standardization).
The OSI (Open Systems Interconnection) Model broke the work needed to implement a distributed network service into seven separate layers of abstraction, from the physical infrastructure at layer one all the way up to the application at layer seven. That was much more elegant and academically satisfying than the Internet’s TCP/IP, which was considered a bit ad-hoc in comparison.
It turns out that implementing a network often requires crossing the layers, so the seven layer model wasn’t a great basis for actually building a network on and the simpler TCP/IP four layer model won out.
Why do we care about an old, theoretical network model? Because even though it’s not really used in implementations it’s still an important part of the language engineers use to discuss networks. You need to be aware of it to understand references to “level 3” and “level 8” problems.
I’ve included rough mappings from bits of the email world onto each layer in this handy table:
That’s ten layers, not seven, you say. Yes, yes it is.
There’s always more to networking than just getting data in front of a user. There’ve been joking (not-joking) references to the “political” and even “religious” layers at the top of the stack. This shirt is one of the more notorious ones.
The three extra layers I gave above – individual, organization, government – are often useful abstractions, and are particularly relevant to email deliverability.
A “full stack” email expert needs to be intimately familiar with levels five through nine, and some situation will require more than a passing understanding of levels three, four and ten.
Recent Comments