Much of the equipment and wires that the internet runs on is privately owned, nor is it a public utility in the traditional sense. The owners of the property have a lot of leeway to do what they like with that property. Yes, there are standards, but the standards are about interoperability. They describe things you have to do in order to exchange traffic with other entities. They do not dictate internal policies or processes.
As the owners of the equipment, companies have a lot of discretion about what they allow on their network, hence their network, their rules. As an example, both Twitter and Facebook are well within their rights to deny or allow traffic on their networks, no matter what the rest of us think about it. As they are not interoperating with other social networks, they make the rules.
This lack of interoperability extends to inbound email filtering as well. The filters can block any mail for any reason, and the sender has no real recourse. There are, of course, folks who can make changes to filters, but they are recipients, customers and business priorities of the filter maintainer.
Recipients are the final arbiters of what mail they want or don’t want. Many of the consumer mail filters are tuned to parse whether a mail is wanted or unwanted based on signals from the recipients. These aren’t the only signals used, mail has to be safe and come from a well behaved MTA. But most of the consumer ISPs care about keeping their users happy.
This is why so much advice, from myself and others, relies on getting the users to interact with the message. Most of the providers want users to be happy and so they will listen when users start complaining. Some providers, like Microsoft, even have formal processes to gather feedback from users on the accuracy of their email filters.
For business filters, customers are the primary driver. Most business filters, even those maintained by consumer ISPs, have an extra layer of filtering. This layer sits on top of the filters sent out to all customers, allowing each individual company to control their own incoming mail. Filtering priorities are set by the company.
Filters do what they’re told to do. Ultimately, business needs and priorities drive what filters do. The reason they can is because mail servers are private property and the owners can manage them the way they want to.
Your server. Your rules.
At WWDC 2019 Apple announced “Sign in with Apple.” This is a service that allows iOS users to log into different applications with private, dedicated email address. When developers send mail to that address, Apple will forward it to the email address associated with the users AppleID. App developers that offer any third party log in will be required to also offer AppleID log in.
Apple has set up a private email relay service for this program. Program users must register their sending email domain and addresses and publish SPF records for that domain.
In order to send email messages through the relay service to the users’ personal inboxes, you will need to register your outbound email domains. All registered domains must create Sender Policy Framework (SPF) DNS TXT records in order to transit Apple’s private mail relay. You can register up to 10 domains and communication emails. Configure Private Relay
Not only are Apple protecting their user’s email addresses, but they’re also denying access to anyone who is not preregistered. This means any stolen apple addresses are likely to be invalid after they’re stolen from the initial sender.
I do have to wonder what deliverability will be like. This is just a forwarding service so there are questions about how this will affect marketers.
- When registering addresses, do you need to register the 5321.from, 5322.from or both?
- Will the relay server rewrite the 5321.from?
- If the relay server rewrites the 5321.from, how will that interact with companies using only SPF authentication for DMARC?
- If the relay server doesn’t rewrite the 5321.from, how will that interact with companies who use only SPF authentication for DMARC?
- Will the relay server make any changes that break DKIM?
- When forwarding to domains that have DKIM based FBLs will FBL mails reveal the recipient address to the marketer?
- What happens to mail coming from an unregistered email address?
- How do users unsubscribe from emails? Will Apple include the private email address in emails?
- How is Apple going to maintain the reputation of their relay IP addresses?
I’ve got mail into Apple asking if they’ll answer some technical questions about this. We’ll see if they answer.
Over the last few years I’ve reduced the complaints I send to ESPs about their customers to almost nothing. The only companies I send complaints to are ones where I actually know folks inside the compliance desk, and I almost never expect action, I just send them as professional courtesy.
The sad fact is, many ESPs are really horrible about dealing with spam coming from their networks. The older, larger companies are often a jumble of poorly integrated technologies resulting from a decade of acquisition. More than a decade ago I sat at a MAAWG conference with the director of deliverability at one of the oldest ESPs. We were talking about their recent Spamhaus listings that I’d been hired to help address and their overall complaint processes. One of the issues was, due to multiple mergers and acquisitions, half of their abuse mail went to the wrong place and some of it was being thrown away.
This is an old story, but only as an example of how long this problem has been going on. Even now, companies retire domain names from receiving mail, but still have them littered throughout their email headers. They miss complaints, they miss notices and then they discover they and most of their customers have extensive delivery problems.
The newer companies are lean and agile and don’t think about investing in actual compliance work until they run face first into an escalated Spamhaus listing. Their solution to the problem is to throw machine learning at it, and try and come up with a way to programatically identify bad customers. The problem is this is a moving target and there’s nothing set and forget about it. Algorithms like this need to be constantly maintained and trained. May as well invest in the human element.
Of course, this is all about the customers sending mail through ESPs. But that’s not the only problem. There are any number of ESPs whose own marketing teams use spam. I cannot tell you the number of companies in the space who’ve decided to add me to their marketing list without bothering to ask me if I want to hear from them.
Just last month I started receiving mail from an ESP. “We’ve made an acquisition! We’re growing!” was the first message I received from them. I wasn’t sure what was going on so I contacted their abuse desk asking what opt-in data they have for me. The person I contacted was apologetic and said she’d chase it down. She also informed me I’d be removed from future emails.
A few days later I received an email telling me that they weren’t really sure where I opted-in, but that it was probably a page on their website that they no longer had up. This doesn’t sound right as the address was one I don’t enter into forms. If a form doesn’t take a tagged address, I use a gmail account. But, I want to give the company the benefit of the doubt so I treat it as solved and move on.
Three weeks later I get another email from the same ESP advertising an upcoming webinar. Again, I send mail pointing out that I was assured I’d been unsubscribed. This time my colleague responds and tells me that I signed up for their mailing list because I attended a conference with them in 2016.
I don’t even have words for how grossly inadequate this response is. If it’s true, which I don’t even know any more, it’s horrible marketing to wait 3 years to start mailing someone after acquiring their email address. But the incompetence doesn’t stop there. This was a conference I attended to speak on two different panels, both regarding deliverability and how not to send spam. As a speaker I don’t always visit the trade floor and if I do, I don’t hand out cards or ask for more information. In any case, I can say with quite a bit of certainty this company wasn’t at the trade show, as they announced this version of their name about 6 months after the conference.
Of course, this isn’t as unusual as it should be, one reason I’m not naming names. ESPs hire aggressive marketers who often send spam… er… “cold emails.” It still amounts to the same thing – an unending bombardment of unsolicited emails from companies who then turn around and ask to be added to my list of “good ESPs” that don’t allow purchased lists.
ESPs need to step up and stop allowing spam on their networks. This goes for customer mail and for their own mail. It’s long past time for them to invest in actual compliance desks and start actually requiring customers to send better mail.
A number of folks in the sender space are reporting intermittent “This link may be suspicious” warnings on their emails. I first heard about it a few weeks ago from some clients. One wasn’t sure what was going on, the other found a bunch of malware uploaded into their customer accounts.
At least 3 people have mentioned it today. One of them asked on Mailop, and the couple Google employees over there are generally pretty helpful when issues like this come up.
What we currently know:
- Multiple senders are seeing warnings;
- The warnings are intermittent and not reproducible;
- They’re affecting lots of links, including some ESP click domains.
From the outside, it looks like Google is being overly aggressive with their suspicious link detector and are generating a significant number of false positives. This is something they’ve got to fix. I’ll update when I know something.
Back in 2017 Techdirt wrote a series of articles about Shiva Ayyadura. Shiva claims he invented email. (narrator voice: he didn’t). I wrote about the lawsuit when it was dismissed on First Amendment grounds. The parties cross appealed, and have been in settlement talks for 18 months.
According to Techdirt, the non-monetary settlement they agreed to is that all the articles in dispute will have a link to a statement published by Shiva.
You may wonder how it could possibly take 18 months to negotiate a settlement about adding links to old articles — and, indeed, I wonder that myself. The entire process has been quite a pain for us. I cannot and would not describe this result as a victory, because this has been nearly two and a half years of wasted time, effort, resources, attention and money just to defend our right to report on a public figure and explain to the world that we do not believe his claims to have invented email are correct, based on reams of evidence.
Increasingly over the last few months I’ve been seeing questions from folks struggling with reputation at Gmail and inbox delivery. It seems like everything exploded in the beginning for 2019 and everything changed. I’ve been avoiding blaming it all on TensorFlow, but maybe the addition of the new ML engine really did fundamentally change how things were working at gmail.
What folks are seeing
- Cutting back to engaged only users is not effectively improving reputation.
- Inboxing is no longer directly tracking with reputation on GPT (high reputation mail is going to bulk, low reputation mail is going to inbox).
- Recipients are complaining about mail being misfiltered.
What can we do?
Right now, we’re mostly falling back on the things that worked 6 months ago: cut back sending to the most engaged recipients and then gradually add back in other addresses once you’re back in the inbox. The challenge is we’re not seeing the improvements we’ve become accustomed to seeing when using this strategy.
With one of my clients their reputation, as reported on GPT, actually fell during the period of cutbacks. Based on consistently high open rates and various inbox tests, including my own and those by one of the commercial vendors, we determined that recipients were getting mail in the inbox despite the low reputation.
Other delivery experts have reported similar patterns. Horrible domain and IP reputation (sometime in the deepest, darkest red) but reaching the inbox and seeing open rates in the 20 – 30% range.
I’ve also seen the flip side, green IPs and domain rep, but the mail is going to bulk.
Yup. Sorry. Wish I had better news. Right now we’re falling back to what worked 4 months ago, because it’s what we had and it did work.
One thing that is new information to me is that, according to folks inside Google, the domain and IP reputation showing on google postmaster tools includes all domains handled by gmail, including GSuite hosted domains. Maybe these are having a disproportional effect on reputation, I don’t know.
My current thoughts are:
- Pay attention to your engagement and open rates at Gmail
- Don’t worry about domain and IP rep too much, if your marketing is performing.
- Maybe we need to start including G Suite hosted domains in engagement restrictions, as painful as that’s going to be.
Anyone got any insight? Is there something we’re missing here?
Recently, a significant majority of discussions of email delivery problems mention that neither the IPs or domains in use are on any of the public blocklists. I was thinking about this recently and realised that, sometime in the past, I stopped using blocklists as a source of useful information about reputation.
I’m not even sure exactly when it happened. I just stopped checking most of the websites for information about blocks. Part of that is likely due to the change in my client base. Over the years I’ve transitioned away from handling immediate, crisis level blocking issues. These days I’m spending the majority of my time providing strategy advice.
It used to be that the public lists could provide some types of insight into what might be wrong. Even the mix of lists an IP or domain was on could lead to useful activity.
These days, though, I’m finding the vast majority of senders I talk to are not on any lists. Their IPs and domains are totally clean, even when putting them into lists that check over a hundred lists.
One conclusion this leads me to is that modern filtering at the consumer ISPs and many of the major corporate gateways has moved well beyond blocking IPs and domains. The filters look for much more subtle clues about mail than whether or not the sender is hitting spamtraps. Filters are able to make nuanced decisions about what to do with an email.
This is such progress! We’ve gotten to a place where we have nuanced filters that can separate out different mail streams and deliver the mail to the place where they believe the recipient wants it.
Blocklists do still have their place and I do sit up and take notice when a client or potential client mentions they’re on a blocklist. Fundamentally, the widely used lists deal with very ugly, problematic senders. They are still valuable simply because they list the very bad sources of email. This means the filters on the other side don’t need to be quite so strict.
All in all, the nature of filtering is changing. In parallel, deliverability is changing. There are sub-specialisations developing in the industry. it’s an interesting time, one where no on has all the answers. I think it’s important to not these types of milestones when we see them.
This is a milestone. Filtering and blocklists have diverged and are addressing different types of mail.
There’s an ongoing discussion, one I normally avoid, regarding how much impact an ESP has on deliverability. Overall, my opinion is that as long as you have a half way decent ESP they have no impact on deliverability. Then I started writing an email and realised that my thoughts are more complex than that.
Here are some excerpts from the email, because in other circumstances I would have just written it as a blog post.
I have dealt with ESP clients in the past who had a collection of customers that were so bad everything mentioning that ESP (even tests from my wttw account to my gmail account that simply contained the domain name of the client) went to bulk. That was a few years ago, though, and the gmail filters have improved and are in some ways even more discerning. I still occasionally find some domain reputations so bad it breaks delivery from one account to another. This is unusual, though, and it never happens overnight.
Of course there are things the ESPs can do that affect all of their client mail. Most of those things involve letting customers get away with bad address collection practices in enough volume that all the customers are considered problematic. If the ESP doesn’t make their customers behave and lets them send whatever they want to whomever they want, then yes, the ESP is going to have problems.
The ESPs with good delivery have extensive and active deliverability and compliance desks. One desk catches customers at the early end of problems, where it’s not enough to actually hurt their delivery but they’re clearly on a path to bad delivery. The other deals with customers who have not taken the initial advice and have continued down the path. What they’ve done is unacceptable and they have to either fix it and get back up to snuff or find a new ESP.
The bulk of my clients right now are ESPs, or SaaS providers that are ESPs but don’t realize they are. They generally come to me because they’ve not been handling deliverability at all and now much of their customer mail is going to bulk. They didn’t see themselves as ESPs so they didn’t pay any attention to what customers were doing and there as enough grey mail to ruin delivery. The thing is, these folks are often using one of the commercial SMTP by API vendors (all the ones I can think of right now start with S) so I know that all of the actual technical stuff is correctly managed. I also know their overall complaint rates and bounce rates and all the surface stuff are within acceptable parameters, otherwise they’d be turfed off their SMTP provider.
A lot of senders, and even some of the deliverability folks, haven’t really kept up with how the ISPs are tracking and filtering mail today. In the B2C space IP address is almost irrelevant. In the B2C space IP address gets you through the SMTP transaction. After that it’s (almost) irrelevant for inbox delivery.
This has been true for ages – it’s been 7 or 8 years since I had a Return Path certified client showing me data that their content mail and their advertising mail was delivered differently at Yahoo, despite identical authentication, IP and everything technical. ISP technology has only gotten better in that time. The content, history and mailstream reputation drives where the mail gets delivered much more than most technical factors, assuming a marginal competence in setting up the mailserver or using one of the commercial bulk MTAs.
In my own work don’t really look at IP rep any more – the publicly available IP reputations don’t reflect on delivery like they used to. I mean, I gave up joking about folks confused by delivery problems senderscore was 99 years ago because it was so overdone. Even now, good IP reputation gets you in the front door, it doesn’t get you to the inbox.
Right now, delivery is challenging. The filtering technology we’ve been modelling for the last decade has changed significantly over the last 18 – 24 months in ways that confound those models. I think we’re in for another year or more of fine tuning before the filters themselves are stable enough to create accurate models. It is a gross oversimplification to blame any one factor for poor delivery.
The Commission finds that nCrowd, Inc. committed one violation of paragraph 6(1)(a) and one violation of paragraph 6(2)(c) of Canada’s Anti-Spam Legislation (the Act) in relation to commercial electronic messages sent to recipients in Canada. The Commission also finds that Brian Conley is liable, under section 31 of the Act, for those violations. Accordingly, the Commission imposes an administrative monetary penalty of $100,000 on Brian Conley. CRTC
The commission’s report is well worth a read as it discusses many of the things I’ve noticed from spamming operations over the years. It’s pretty standard business practice for spammers to have a complex set of sorta but not really different businesses. They all interact and share data, but not legal liability. They’re mostly treated as one business by the principles and there’s no real dedication to any one brand name.
I don’t think I can describe it much better than the commission did:
The investigation uncovered a complex corporate network and a business pattern, characterized by acquisitions, foreclosures, and bankruptcies. During this chain of acquisitions, the customer email distribution list grew exponentially to reach more than 2 million email addresses. The common threads amid this corporate network and string of acquisitions of email distribution lists are the key players involved, namely Ghassan Halazon and Brian Conley Footnote3 , as shown in the chart below and detailed in the corresponding description.
Besides this chain of acquisitions, the investigation uncovered a whole network of corporations in various lines of businesses, throughout several countries, and characterized by (a) high frequency of business registrations, (b) high frequency of business acquisitions, (c) high frequency of corporate changes, including names and addresses, and (d) dealings with companies specialized in dividend routing and tax optimization services.
Through the years I’ve dealt with folks in this space as they regularly have delivery problems. Seeing the image of the myriad companies and bankruptcies and list transfers from the CRTC solidified in my mind that this really was a nest of spammers. Just look at how many businesses they went through without losing any email addresses.
It’s not coincidental that there are so many business name changes. In this case, there was some fraud going on for the customers, but frequent domain changes are a hallmark of spammers. If you talk to them, and I have, they will tell you they have to change names otherwise the ISPs block their mail. They’ll tell you they keep bounces low and remove complainers and follow all the best practices, but the ISPs hate them so they have to change business names.
Business models like this are nothing new, of course. One of my earliest clients asked me to help them set up a fake ESP. The idea would be to set up a whole bunch of different entities each with their own domains and business information. When one of the entities would get blocked, they’d tell the blocklist or ISP that it was a customer who was now terminated. After the block was lifted, they’d spin up a new customer and keep sending.
The ease at which spammers set up companies and sending domains and Its is the reason why ISPs treat mail with new domains and IPs as spam unless proven otherwise. 20 years of history indicates spammers do go so far as to create new companies to send spam. Given the recent action by the CRTC, it’s pretty clear that this isn’t ancient history, it’s continuing to this very day.