BLOG

Industry News & Analysis

Misinformation on filters

I’ve seen reports that someone is asserting that utm=COVID19 in URLs results in all mail going to bulk at multiple ISPs. This is the type of thing that someone says is true and dozens of folks believe it and thus a “deliverability phact” is born. For a plethora of reasons, this doesn’t pass the sniff test. Don’t believe everything you read on the internet.

It’s very tempting to identify this One Simple Trick to get your email into the inbox. Change this font. Take out this UTM. Change this hostname. And, in some cases it may even work for a time.

But, look, if filters really were that simple they’d be wholly ineffective. Not just slightly ineffective but wholly ineffective. Anything that is easy to test can be defeated, and spammers test as much or even more than marketers do.

Don’t believe me? Over a decade ago I was invited to a meeting with a “marketing company” based out of San Francisco. After I got there and signed the NDA, they explained their strategy to get mail into Hotmail. Starting at 5pm they would have their content staff start writing emails and sending them to Hotmail. They’d test and test and test until one of them got into the inbox. Once they found content that would get through the filters, they’d turn on the floodgates and send as much mail as they could until the filters caught up. They’d do this all night, every night. (They were shut down by the FTC not long after I declined to work with them.)

It’s naive to believe that filters would be so transparent and think they’d still work. Anything so simple is going to be discovered and exploited by the spammers. Don’t fall prey to this kind of deliverability nonsense. Think about what the bad guys would do if this were true. And then remember that the bad guys have a lot of practice exploiting naive filters.

2 Comments

Discussion Session

More than 30 people joined our delivery discussion from last Wednesday evening (Irish time). Thanks to all who joined and participated.

I had a number of different topic suggestions, but the thing on everyone’s mind was the current pandemic and how much email many of our customers, clients, and companies are sending. We had what I think was a productive discussion sharing information and experiences. Some highlights

  • One ESP has seen a significant uptick in volume that has stayed high over the course of multiple weeks.
  • Another ESP has seen a slight uptick in volume but that too has remained consistent.
  • There’s no clear consensus on deliverability effects. Given the changes in volume and changes in audience it’s difficult for folks to identify if delivery problems are due to the changes or to the ISPs filtering changes.

Overall, I think this was a successful experiment. There were some technical kinks, but I think I’ll be hosting calls in the future. I’ll be announcing the next one in a few days.

(And, well, it seems I wrote this and then failed to actually hit “publish” because that’s just how distracted I am right now.)

No Comments

Deliverability discussion

Lots of folks are socialising distantly these days, so I thought I’d try a scheduled deliverability discussion over video. Given the time difference, I’ll log on in the evening Irish time which makes this daytime for most of the US folks.

The first call will be Wednesday, March 25, 2020 at 5pm Ireland time. That’s 1pm Eastern and 10am Pacific. If you’re interested in joining, drop me an email laura-ddiscuss@ the obvious domain and I’ll send you an invite.

If there’s a particular thing about delivery you want me to talk about, send it in the email.

Talk to you wednesday.

3 Comments

Moment of Zen

Things are very unsettled right now. Completely and totally unsettled. Even for those of us who are well geared up for and used to working from home are struggling in our current situation.

We went for a walk down the canal on Tuesday, and it was very quiet, with almost no traffic on the street. There were various couples and families walking, but most were doing a good job at social distancing.

Stay safe. Wash your hands. Don’t touch your face.

1 Comment

Happy St. Patrick’s Day

It’s a near silent St. Paddy’s day here in Ireland. We took a walk along the canal and took in the silence. On the way back, our neighbour’s kids had decorated their front window and I had to take a picture.

Things are weird. Be kind to yourself and to others. Wash your hands. Don’t touch your face. Don’t send that email to your entire database.

No Comments

Advice on coronavirus emails

Gartner has some really good recommendations for companies considering mailing about the coronavirus pandemic.

Launch your COVID-19-themed marketing email campaign only if you can answer yes to four questions:

  • Am I telling customers something different from other brands versus saying the same thing as everyone else?
  • Am I telling customers something they don’t already expect of my company or brand?
  • Is the WIIFM (what’s in it for me) conspicuous in the subject line and opening paragraph?
  • And, most importantly, is the WIIFM attuned to your customers needs right now?

Things are scary right now. But many of the companies who are sending emails DO NOT NEED TO DO SO. The insurance company I deal with solely by email didn’t need to send me email telling me their office was closed. I’ve never been to their office.

The vast majority of what I’m hearing from recipients and consumers is that this mail is all useless and they’re deleting without reading. Too much irrelevant or annoying mail will drive unsubscribes and this is spam hit. The first means you can’t mail that person again. The second means your reputation will take a hit.

Think twice before sending that mail. Most of you don’t need to be sending it.

No Comments

Authentication at Office365

This is a followup from a post a few weeks ago about authentication changes at Office365. We have some more clarity on what is going on there. This is all best information we have right now.

Outlook Logo

Microsoft is now requiring authentication to match the visible from address in order to reach the inbox at Office365. That means, either the SPF domain or the DKIM domain must align (in the DMARC sense) to the visible from domain. Simply, that means that the visible from and the signing domain must be identical or one must be a subdomain of the other.

The reason they’re doing this is to protect their users from forged emails. I can’t fault them for this at all. Many of their customers are SMBs. These businesses are targets for wire fraud, to the tune of tens of billions of dollars. In fact, one of the other companies my bookkeeper worked for in CA almost got roped in by this fraud back in 2016 or so.

Microsoft has always been looking for ways to validate the visible from address. That’s a big part of their push for SenderID, which suffered really poor uptake. This is leveraging the philosophy of DMARC and the improvement in support for authentication technology that’s developed over the last 15 years.

Adapting to this will be challenging for some ESPs, particularly those that service the SMB market. At many of these companies, handling technical issues is often handled by employees who manage technology as a small part of their job. Thus, there is a steep learning curve when trying to deploy new technology. Others have consultants or outsourced technology, many of whom are great at handling internal Windows networks and hardware, but don’t really get the intricacies of email authentication.

I see this as somewhat akin to Yahoo deploying DMARC p=reject. That was a significant and email breaking change implemented by Yahoo in response to specific security issues. This made it clear to other consumer mail providers, email intermediaries and receivers that DMARC was something they’d have to adapt to. That adaptation was neither easy nor cost free. But it did force a change in how ESPs were doing business.

Here, we have Office365 making a decision that is significant and email breaking, even for some of their customers. It may be that longer term we see other consumer webmail providers starting to tighten down their requirements for alignment even in the absence of a DMARC record. I don’t think it’s that unreasonable, ESPs have had 6 years to build the infrastructure to manage this.

The takeaway here is that if your customers are having problems getting mail into Office365, one of your first troubleshooting steps should be to ensure that authentication aligns with the visible from address. If it doesn’t fix that first. Of course, alignment is not magic wand into the inbox. If your content is spammy or your reputation is poor, your mail will go to the bulk folder.

3 Comments

Deliverability mythbusting

Recently had the pleasure of sitting down with Jillian Bowen and talking about deliverability for her podcast.

2 Comments

Back at it

Back at the office after traveling to visit a bunch of our US friends recently. A lot of news, both in and out of the email space, happened while we were gone. The biggest stories are outside the email space and I will admit to following the coronavirus news probably closer than I should. (My graduate work was done across the hall from one of the major avian epidemic monitoring labs. This is the kind of thing we discussed at lunch and over beers.)

The next few paragraphs are general musings about the state of the world. Scroll down to “IN EMAIL NEWS” if you just care about that.

As a consequence of the spread of COVID-19, many conferences and company gatherings are being cancelled. I know it’s frustrating to not get to go to a conference you’ve been looking forward to, but minimising contact really is the best way to slow the spread of the infection. There’s also the developing financial crisis and oil wars. Fun times.

A number of conferences are being cancelled and postponed. Many folks in the industry are on orders to work from home. The good news is that the data from Hong Kong shows that social distancing works to curb the spread of disease (and even worked to lower seasonal flu there this year). If your conference isn’t cancelled, don’t shake hands, wash your hands and don’t touch your face.

IN EMAIL NEWS

While we were gone the judge dismissed Tulsi Gabbard’s case against Google. The whole ruling is short and worth a read. But the crux is here:

Plaintiff’s essential allegation is that Google violated Plaintiff’s First Amendment rights by temporarily suspending its verified political advertising account for several hours shortly after a Democratic primary debate. Plaintiff’s claim, however, “runs headfirst into two insurmountable barriers—the First Amendment and Supreme Court precedent.” Prager Univ. v. Google LLC, No. 18- 15712, 2020 WL 913661, at *1 (9th Cir. Feb. 26, 2020). 

I know there were some folks in the email space who were hoping her claims for biased email filtering would get adjudicated, but it wasn’t going to happen. Not only is there extensive case law reinforcing that ISPs can filter any mail they want, but some data shows that Google isn’t biased against her email.

The Markup have been working on a story related to filtering political email where they created a mailbox and signed it up for mail from lots of candidates and then looked at where the mail went. Much of the mail from all candidates is going to promotions. Tulsi’s mail is certainly not being treated any harsher than other candidates in this test.

As I’ve talked about, there are a lot of things that go into delivery at Gmail and I spent quite a bit of time talking with the folks researching this article about 6 months ago. The concern is that ISPs are influencing elections by how they filter. I don’t think this is true for a number of reasons.

While free consumer email addresses are ad supported, there’s not the same expectation that email is a cash cow. One reason is simply email predates the ad supported internet. And aggressively inserting ads into email accounts has not been well accepted by the users. Another reason is that economically, it is cheaper to keep a existing email user happy and on the platform than it is to acquire a new one. Social media doesn’t care about keeping users happy, in fact, they make money from outrage. 

Email filtering culture is another thing that keeps me from thinking there are rogue operators blocking emails for candidates they don’t like. The culture started back in the late 90s with individuals working out ways to keep spam out of their inboxes and sysadmins trying to keep spam off their USENET spools. During that time, there was a significant amount of energy and thought directed to the idea that the Internet was a meritocracy and that they weren’t shutting down ideas, rather they were shutting down behavior. Some of the common phrases used by folks handling these early filters were “consent not content” – ie, if you wanted that message, no one was going to block it. Another was “abuse of the net, not on the net” a little more problematic, but again, it was about causing harm to the underlying infrastructure rather than hurting individual people. Many of the individuals around during this time went on to found filtering companies, work for major ISPs in their filtering creation departments and become thought leaders in the space. This underlying culture is still a part of the filtering in email. 

Email was built into the fabric of the internet. Folks have been wrestling with the question of how to stop spam and “unwanted” mail for almost a quarter century now. Email filters, at the consumer ISPs in particular but also at the various business appliances, are monsters of machine learning. It’s nearly impossible to “whitelist” any particular sender and exempt them from the filters acting on their mail. The flip side is also true, it’s difficult to secretly blacklist a group of senders based on their political leanings. The underlying machine learning engines just doesn’t have those kinds of switches built into them. 

In any case, the data from The Markup’s article was going to demonstrate that Tulsi is not being treated unfairly by Gmail’s filters.

No Comments

What is FCrDNS and why do we care

It’s been a light blogging month. We’ve been dancing around getting the final plans, financing, and contractors set up for the work we’re doing on the Dublin house and then heading off for our first actual vacation in almost 5 years. But, I wrote half of this answering a question on mailop, so I may as well polish and publish.

What is FCrDNS

FCrDNS stands for Full Circle reverse DNS or Forward-Confirmed reverse DNS. It means that if you do a DNS lookup on the domain in a reverse DNS lookup than that domain will point back to the original IP. The name actually comes from the fact that if you start with the IP address and go through the hostname, you get a full circle.

Image illustrating the full circle from connecting IP to hostname and back to the connecting IP using rDNS and DNS queries

The reason FCrDNS is a thing is because any IP address owner can assign any domain to the rDNS of an IP address. They are in complete control and there are no technical checks that the hostname be a domain they own. Anyone could assign their IP a rDNS of angrygoose.google.com, or flowerchild.facebook.com or jupiter.spamhaus.com to their IPs. And, in fact, lots of spammers did just this, assigning domains to their IPs that they didn’t own.

Why do we care about FCrDNS?

Spammers lie, a lot. The did all sorts of things to avoid being blocked. Stealing legitimate domain names in their rDNS was one of those. They’d set up their IPs forging known domains as a way to try and get around some filters. Receiving systems figured this out pretty quickly. They started doing FCrDNS checks to verify that the person managing DNS for that IP space also manages DNS for the domain space. The underlying idea, is that if the IP points to a hostname and that hostname points back to the same IP, then everything is under control of the same entity.

FCrDNS is a method of deciding whether or not the IP address is legitimately being used by the domain in the rDNS entry. FCrDNS is a way to verify the identity of the connecting IP. If the rDNS doesn’t match, then it’s much more likely that the mail is coming from an illegitimate source. 

What should have a FCrDNS?

Basically, any time you set up rDNS on an IP address it’s good practice to give the corresponding hostname an A record. For IP sending outgoing mail, this is one of those expected best practices. There’s an IP address with a rDNS of a single hostname and the hostname points back to the IP address. That IP uses the same hostname to introduce itself during the SMTP transaction. Certainly when I’m looking at IP addresses and domains and EHLO values I do check to see if everything matches.

But. Not every hostname has to have a single A/AAAA record. A single hostname can point to multiple IPs:

DNS output showing outlook.com pointing to 8 different IP addresses in 40.97.0.0/16

A single IP can also point to many different hostnames or no hostnames at all. In fact spot checks show me that none of the IP addresses in the example above actually have a rDNS set up.

The ability of an IP to point to many hostnames and a hostname to point to many IPs complicates completing the circle. Anyone verifying FCrDNS on an IP with multiple PTR records needs to do multiple DNS lookups for the verification step. Lookups can quickly get out of hand if each of the domains in the PTR has multiple IPs then there’s even more DNS work.

These technical and practical realities are why we can only recommend that an IP sending mail have FCrDNS, we can’t require it. And, in fact, not all outgoing mail servers do have it.

image showing one of outlook.com's outgoing IP addresses (52.101.142.83) does not have FCrDNS.

FCrDNS is a hack to link an IP address to a domain. That’s all it’s there for. You set it up if you can, and should probably expend some effort to do so for dedicated outbound servers, particularly those sending bulk mail. But, no, your 5321.from domain doesn’t need to point to an IP simply so you can check this box

No Comments