Back in 2017 Techdirt wrote a series of articles about Shiva Ayyadura. Shiva claims he invented email. (narrator voice: he didn’t). I wrote about the lawsuit when it was dismissed on First Amendment grounds. The parties cross appealed, and have been in settlement talks for 18 months.
According to Techdirt, the non-monetary settlement they agreed to is that all the articles in dispute will have a link to a statement published by Shiva.
You may wonder how it could possibly take 18 months to negotiate a settlement about adding links to old articles — and, indeed, I wonder that myself. The entire process has been quite a pain for us. I cannot and would not describe this result as a victory, because this has been nearly two and a half years of wasted time, effort, resources, attention and money just to defend our right to report on a public figure and explain to the world that we do not believe his claims to have invented email are correct, based on reams of evidence.
Increasingly over the last few months I’ve been seeing questions from folks struggling with reputation at Gmail and inbox delivery. It seems like everything exploded in the beginning for 2019 and everything changed. I’ve been avoiding blaming it all on TensorFlow, but maybe the addition of the new ML engine really did fundamentally change how things were working at gmail.
What folks are seeing
- Cutting back to engaged only users is not effectively improving reputation.
- Inboxing is no longer directly tracking with reputation on GPT (high reputation mail is going to bulk, low reputation mail is going to inbox).
- Recipients are complaining about mail being misfiltered.
What can we do?
Right now, we’re mostly falling back on the things that worked 6 months ago: cut back sending to the most engaged recipients and then gradually add back in other addresses once you’re back in the inbox. The challenge is we’re not seeing the improvements we’ve become accustomed to seeing when using this strategy.
With one of my clients their reputation, as reported on GPT, actually fell during the period of cutbacks. Based on consistently high open rates and various inbox tests, including my own and those by one of the commercial vendors, we determined that recipients were getting mail in the inbox despite the low reputation.
Other delivery experts have reported similar patterns. Horrible domain and IP reputation (sometime in the deepest, darkest red) but reaching the inbox and seeing open rates in the 20 – 30% range.
I’ve also seen the flip side, green IPs and domain rep, but the mail is going to bulk.
Yup. Sorry. Wish I had better news. Right now we’re falling back to what worked 4 months ago, because it’s what we had and it did work.
One thing that is new information to me is that, according to folks inside Google, the domain and IP reputation showing on google postmaster tools includes all domains handled by gmail, including GSuite hosted domains. Maybe these are having a disproportional effect on reputation, I don’t know.
My current thoughts are:
- Pay attention to your engagement and open rates at Gmail
- Don’t worry about domain and IP rep too much, if your marketing is performing.
- Maybe we need to start including G Suite hosted domains in engagement restrictions, as painful as that’s going to be.
Anyone got any insight? Is there something we’re missing here?
Recently, a significant majority of discussions of email delivery problems mention that neither the IPs or domains in use are on any of the public blocklists. I was thinking about this recently and realised that, sometime in the past, I stopped using blocklists as a source of useful information about reputation.
I’m not even sure exactly when it happened. I just stopped checking most of the websites for information about blocks. Part of that is likely due to the change in my client base. Over the years I’ve transitioned away from handling immediate, crisis level blocking issues. These days I’m spending the majority of my time providing strategy advice.
It used to be that the public lists could provide some types of insight into what might be wrong. Even the mix of lists an IP or domain was on could lead to useful activity.
These days, though, I’m finding the vast majority of senders I talk to are not on any lists. Their IPs and domains are totally clean, even when putting them into lists that check over a hundred lists.
One conclusion this leads me to is that modern filtering at the consumer ISPs and many of the major corporate gateways has moved well beyond blocking IPs and domains. The filters look for much more subtle clues about mail than whether or not the sender is hitting spamtraps. Filters are able to make nuanced decisions about what to do with an email.
This is such progress! We’ve gotten to a place where we have nuanced filters that can separate out different mail streams and deliver the mail to the place where they believe the recipient wants it.
Blocklists do still have their place and I do sit up and take notice when a client or potential client mentions they’re on a blocklist. Fundamentally, the widely used lists deal with very ugly, problematic senders. They are still valuable simply because they list the very bad sources of email. This means the filters on the other side don’t need to be quite so strict.
All in all, the nature of filtering is changing. In parallel, deliverability is changing. There are sub-specialisations developing in the industry. it’s an interesting time, one where no on has all the answers. I think it’s important to not these types of milestones when we see them.
This is a milestone. Filtering and blocklists have diverged and are addressing different types of mail.
There’s an ongoing discussion, one I normally avoid, regarding how much impact an ESP has on deliverability. Overall, my opinion is that as long as you have a half way decent ESP they have no impact on deliverability. Then I started writing an email and realised that my thoughts are more complex than that.
Here are some excerpts from the email, because in other circumstances I would have just written it as a blog post.
I have dealt with ESP clients in the past who had a collection of customers that were so bad everything mentioning that ESP (even tests from my wttw account to my gmail account that simply contained the domain name of the client) went to bulk. That was a few years ago, though, and the gmail filters have improved and are in some ways even more discerning. I still occasionally find some domain reputations so bad it breaks delivery from one account to another. This is unusual, though, and it never happens overnight.
Of course there are things the ESPs can do that affect all of their client mail. Most of those things involve letting customers get away with bad address collection practices in enough volume that all the customers are considered problematic. If the ESP doesn’t make their customers behave and lets them send whatever they want to whomever they want, then yes, the ESP is going to have problems.
The ESPs with good delivery have extensive and active deliverability and compliance desks. One desk catches customers at the early end of problems, where it’s not enough to actually hurt their delivery but they’re clearly on a path to bad delivery. The other deals with customers who have not taken the initial advice and have continued down the path. What they’ve done is unacceptable and they have to either fix it and get back up to snuff or find a new ESP.
The bulk of my clients right now are ESPs, or SaaS providers that are ESPs but don’t realize they are. They generally come to me because they’ve not been handling deliverability at all and now much of their customer mail is going to bulk. They didn’t see themselves as ESPs so they didn’t pay any attention to what customers were doing and there as enough grey mail to ruin delivery. The thing is, these folks are often using one of the commercial SMTP by API vendors (all the ones I can think of right now start with S) so I know that all of the actual technical stuff is correctly managed. I also know their overall complaint rates and bounce rates and all the surface stuff are within acceptable parameters, otherwise they’d be turfed off their SMTP provider.
A lot of senders, and even some of the deliverability folks, haven’t really kept up with how the ISPs are tracking and filtering mail today. In the B2C space IP address is almost irrelevant. In the B2C space IP address gets you through the SMTP transaction. After that it’s (almost) irrelevant for inbox delivery.
This has been true for ages – it’s been 7 or 8 years since I had a Return Path certified client showing me data that their content mail and their advertising mail was delivered differently at Yahoo, despite identical authentication, IP and everything technical. ISP technology has only gotten better in that time. The content, history and mailstream reputation drives where the mail gets delivered much more than most technical factors, assuming a marginal competence in setting up the mailserver or using one of the commercial bulk MTAs.
In my own work don’t really look at IP rep any more – the publicly available IP reputations don’t reflect on delivery like they used to. I mean, I gave up joking about folks confused by delivery problems senderscore was 99 years ago because it was so overdone. Even now, good IP reputation gets you in the front door, it doesn’t get you to the inbox.
Right now, delivery is challenging. The filtering technology we’ve been modelling for the last decade has changed significantly over the last 18 – 24 months in ways that confound those models. I think we’re in for another year or more of fine tuning before the filters themselves are stable enough to create accurate models. It is a gross oversimplification to blame any one factor for poor delivery.
The Commission finds that nCrowd, Inc. committed one violation of paragraph 6(1)(a) and one violation of paragraph 6(2)(c) of Canada’s Anti-Spam Legislation (the Act) in relation to commercial electronic messages sent to recipients in Canada. The Commission also finds that Brian Conley is liable, under section 31 of the Act, for those violations. Accordingly, the Commission imposes an administrative monetary penalty of $100,000 on Brian Conley. CRTC
The commission’s report is well worth a read as it discusses many of the things I’ve noticed from spamming operations over the years. It’s pretty standard business practice for spammers to have a complex set of sorta but not really different businesses. They all interact and share data, but not legal liability. They’re mostly treated as one business by the principles and there’s no real dedication to any one brand name.
I don’t think I can describe it much better than the commission did:
The investigation uncovered a complex corporate network and a business pattern, characterized by acquisitions, foreclosures, and bankruptcies. During this chain of acquisitions, the customer email distribution list grew exponentially to reach more than 2 million email addresses. The common threads amid this corporate network and string of acquisitions of email distribution lists are the key players involved, namely Ghassan Halazon and Brian Conley Footnote3 , as shown in the chart below and detailed in the corresponding description.
Besides this chain of acquisitions, the investigation uncovered a whole network of corporations in various lines of businesses, throughout several countries, and characterized by (a) high frequency of business registrations, (b) high frequency of business acquisitions, (c) high frequency of corporate changes, including names and addresses, and (d) dealings with companies specialized in dividend routing and tax optimization services.
Through the years I’ve dealt with folks in this space as they regularly have delivery problems. Seeing the image of the myriad companies and bankruptcies and list transfers from the CRTC solidified in my mind that this really was a nest of spammers. Just look at how many businesses they went through without losing any email addresses.
It’s not coincidental that there are so many business name changes. In this case, there was some fraud going on for the customers, but frequent domain changes are a hallmark of spammers. If you talk to them, and I have, they will tell you they have to change names otherwise the ISPs block their mail. They’ll tell you they keep bounces low and remove complainers and follow all the best practices, but the ISPs hate them so they have to change business names.
Business models like this are nothing new, of course. One of my earliest clients asked me to help them set up a fake ESP. The idea would be to set up a whole bunch of different entities each with their own domains and business information. When one of the entities would get blocked, they’d tell the blocklist or ISP that it was a customer who was now terminated. After the block was lifted, they’d spin up a new customer and keep sending.
The ease at which spammers set up companies and sending domains and Its is the reason why ISPs treat mail with new domains and IPs as spam unless proven otherwise. 20 years of history indicates spammers do go so far as to create new companies to send spam. Given the recent action by the CRTC, it’s pretty clear that this isn’t ancient history, it’s continuing to this very day.
One of the fascinating parts of my job is seeing how different groups in email have radically disparate points of view. A current example is how much value senders put on spamtraps compared to ISPs and filtering companies.
I understand why this is. In all too many cases, when a sender asks why they’re mail is going to bulk or being blocked, the answer is “you’re hitting spamtraps.” The thing is, spamtraps are almost never the only reason mail is being blocked.
In many circumstances mail is blocked because the recipients don’t like it. They’re complaining to whomever is running the filter or they’re acting in ways that signal the mail is unwanted. Spamtraps are often used to confirm blocking decisions, rather than driving them.
When I talk to clients and colleagues and mention that spamtraps don’t drive most filtering decisions they’re shocked. They are so conditioned that spamtraps are bad. And, it’s totally understandable where that conditioning comes from.
When asked why mail is blocked many filter maintainers will answer “because you’re hitting spamtraps.” The reasons they say this are mostly because they’re tired of arguing with senders about their mail. Spamtraps are hard to argue with. The mail is arriving at addresses that didn’t opt-in to receive it, therefore the mail is spam.
Now we have companies selling list hygiene services which offer to remove spamtraps. We have other companies selling “sensor networks” that most of their customers refer to as spamtrap feeds. We’ve monetized spamtraps. Great for the companies who are selling the services, but what does it get their customers?
Spamtraps are a signal. They are a sign that there is a problem with an address collection process or a problem with list maintenance. Removing every spamtrap on a list will not fix the problem. It won’t even stop blocks.
Blocks aren’t primarily based on spamtraps in most cases. Spamtraps aren’t the problem. Don’t spend time or money on removing spamtraps from a list. Instead, focus on on sending mail recipients ask for and want.
The US CAN-SPAM act is the primary US legislation covering commercial email. It’s been around since 2003, but I still see a steady stream of questions about it, and the folkloric answers to some of them are all over the place.
What does CAN-SPAM require?
The important requirements are
- Don’t use false or misleading header information
- Don’t use deceptive subject lines
- Make it clear the message is an advert
- Provide a valid physical postal address
- Tell recipients how to opt out of receiving future email
- Do not require anything other than the recipients email address to opt-out
- Honor those opt out requests promptly
You can’t contract away your legal responsibility. If you contract with another company to handle your email marketing both you and they may be held liable.
What doesn’t CAN-SPAM require?
CAN-SPAM doesn’t forbid anything other than deceptive and criminal behaviour in any sort of email that has no commercial content.
CAN-SPAM doesn’t require that recipients opt-in.
CAN-SPAM only applies to email, not social media and not SMS (other than SMS sent by sending email to an email-to-SMS gateway).
Does CAN-SPAM apply only to commercial email?
Mostly. CAN-SPAM divides email content into three categories:
- Transactional / Relationship
The vast majority of CAN-SPAM requirements apply only to mail which has a “primary purpose” that is commercial, not to transactional / relationship mail (which I’m going to just call “transactional” from now on).
You’re not allowed to use false or misleading routing information for transactional emails, but otherwise CAN-SPAM puts no limits on them.
Political email, including fundraising email, is considered “Other”, though relying on that for email that is selling something concrete is sketchy and likely a bad idea.
There’s no specific mention of mail sent by charities or other non-profits in the act – their mail and compliance requirements are judged exactly the same as mail from commercial companies.
Is my mail transactional?
The primary purpose of an email is transactional or relationship if it consists only of content that:
1. facilitates or confirms a commercial transaction that the recipient already has agreed to;
2. gives warranty, recall, safety, or security information about a product or service;
3. gives information about a change in terms or features or account balance information regarding a membership, subscription, account, loan or other ongoing commercial relationship;
4. provides information about an employment relationship or employee benefits; or
FTC Compliance Guide for Businesses
5. delivers goods or services as part of a transaction that the recipient already has agreed to.
What if I mix transactional and commercial content in the same message?
When an email contains both kinds of content, the primary purpose of the message is the deciding factor. Here’s how to make that determination: If a recipient reasonably interpreting the subject line would likely conclude that the message contains an advertisement or promotion for a commercial product or service or if the message’s transactional or relationship content does not appear mainly at the beginning of the message, the primary purpose of the message is commercial. FTC Compliance Guide for Businesses
So for your mail to have a primary purpose that is transactional, and so to be exempt from most CAN-SPAM requirements, it must be clear – to a reasonable recipient – from the subject line that it’s intended to be transactional and the beginning of the message must contain primarily transactional content.
What if I mix commercial and “other” content in the same message?
In that case, the primary purpose of the message is commercial and the provisions of the CAN-SPAM Act apply if:
1. A recipient reasonably interpreting the subject line would likely conclude that the message advertises or promotes a commercial product or service; and
2. A recipient reasonably interpreting the body of the message would likely conclude that the primary purpose of the message is to advertise or promote a product or service.
FTC Compliance Guide for Businesses
Factors relevant to that interpretation include the location of the commercial content (for example, is it at the beginning of the message?); how much of the message is dedicated to commercial content; and how color, graphics, type size, style, etc., are used to highlight the commercial content.
Does CAN-SPAM apply to B2B email?
Nothing in CAN-SPAM requires that recipients be consumers. B2B and B2C mail are both covered.
What are the penalties for violation?
Theoretically, civil penalties of up to $42,530. Per email.
Realistically, behaviour has to be pretty egregious for the FTC to consider enforcement beyond a warning letter and an agreement to comply in the future.
But most reputation services and email service providers have much higher standards than CAN-SPAM requires and will likely consider wilful violations of CAN-SPAM as a priori evidence of bad intent.
Legal standards in other jurisdictions, notably Canada and Europe, are higher and enforcement is more likely.
Where can I get more advice?
The FTC has an excellent compliance guide that provides understandable, quotable summaries of the act requirements. I’ve quoted liberally from it here, and it’s what I point people at when they ask “But does CAN-SPAM say …?”.
Or you can read the act itself (or, more readably, at Cornell), if you enjoy legalese. It’s been modified and clarified by he FTC since it was passed, so read some of the rule making and FTC commentary too. But don’t try and rules-lawyer to justify your doing something that’s a dark shade of grey.
Friday I attempted to make a purchase online. I go through the selection and checkout process … up through the payment choices. When I pick pay by credit card I get an error message that says “credit card expiration date wrong.” All very strange because I’ve not put in a credit card number or expiration date.
I try and call, but it’s a holiday weekend and no one is around. Fine, I’ll make the purchase Tuesday (Easter monday is a national holiday here) when things are working again or I can call the store and order by phone.
This particular store has some abandoned cart automation. Which means that for the last 3 days I’ve gotten morning reminders that there is something left in my cart. Yes. I know. I’d really like to purchase said items. But I can’t, because your website is currently broken.
No one is really at fault here. I’m pretty sure even the most advanced marketing automation doesn’t have a “pause because payment system broken” button. The daily messages are annoying, but that may be because I went and edited the cart (more stuff!) on Sunday.
Maybe marketing automation does need a pause and an interrupt button. This situation is pretty minor in the grand scheme of things. But sometimes big events that rock our world happen and marketing reminders seem so trivial and unimportant. In any case, set and forget is never the answer.
Is 10,000 emails sent a few times a week a high enough volume to improve a Gmail reputation?