Anyone actively handling deliverability issues has had the experience of submitting a ticket or email and receiving no response. Alternatively, we get a boilerplate response that seems to not address the question. It happens to me, it happens to colleagues, it happens to everyone. One of the biggest challenges we face is taking that lack of response and channeling it into action items for our customers and clients.
This is where the experience and insight of a good deliverability consultant comes into play. We translate both the random boilerplate answers and the silence into actionable recommendations for clients. Good deliverability experts can read between the lines and use their experience to extrapolate the silence into actionable information.
Silence is an answer
Many people believe that lack of a reply means that no one saw their ticket or email. Sometimes, it is true, the contact method goes into a blackhole and no one sees the message. In the cases I can think of, this isn’t personal. The person (and it’s almost always an individual person) doesn’t want any feedback, so they don’t check the one channel they have for feedback. Or, the request is going to the wrong place. For example, I get a lot of mail (and even phone calls) from people who think I’m support for any number of ISPs. I answer some, but don’t answer others, particularly when I’m busy.
In the case of a list that ignores every inquiry, silence tells us this is likely not relevant to blocking problems. A filter or blocklist that doesn’t speak with listees typically doesn’t have many users. This is especially true in the case where we just found the listing by using an online blacklist checker like mxtoolbox or multirbl.valli.org. There are hundreds of blocklists out there, but in my experience the number that have a wide enough footprint to affect mail delivery is less than 10.
There are other cases, however, where we don’t receive answers even though the message is read and the person reading it could help. These are a little harder to interpret, but it’s not impossible. There are different reasons a volunteer or employee might choose to not respond to a message. For instance, some places have a policy of not responding to any message that contains a threat of legal action. Any mention of lawyers in a message and the only response is silence. In other cases, the issue may be one where there is nothing to say because it’s all been said before.
Every once in a while a client will hire me to mediate a blocklisting and when I reach out I either get silence or a brief “we’ve told the client what they need to do” from the list. Inevitably this has been an issue that has been going on for a while. I was contacted because the filter or blocklist stopped answering. They stopped answering because the issue was going in circles. Sometimes it’s a communication disconnect, and I can help. Sometimes it’s simply the listee doesn’t want to do anything different and refuse to make the necessary changes to get delisted.
Silence can be a no
Sometimes the reason we don’t get an answer is because the answer is no.
For instance, some senders send long emails requesting significant amounts of data from the ISP or the filter company. These emails are more likely to be ignored than others. Sometimes the company simply doesn’t have the data being requested. Other times it’s too much data and will take too long to compile. While some ISPs provide official support channels for senders, the vast majority don’t. Any work done on behalf of senders, is at the discretion and leisure of the employee.
Why doesn’t the person just send back a no? Sometimes it’s just due to overwork. Stuff gets dropped on the floor by accident. There is no intention to ignore it, but we’re all busy and overworked. When an email is a favor it ends up at the bottom of the todo pile. And some of us never find the bottom of the pile. Or by the time someone gets to the question it’s 10 days later and the filters have all changed anyway so an answer is no longer timely.
Another reason is because there is a perception that saying no will cause conflict. Unfortunately, history and experience tell us that some percentage of senders will take a no answer as the opening to a negotiation. They send back email after email trying to negotiate some answer or get some information. This is frustrating for the person who said no. After more than a couple experiences like this it turns into too much work to say no, so email is just not answered.
The issue has been fixed
For a lot of folks, responding to messages is not as important as making things work. Most of the time in deliverability we’re not the customer. None of the normal customer interaction rules apply. If there are dozens or hundreds of messages about a mistake affecting multiple listings, the priority will be fixing the mistake. Answering those dozens or hundreds of emails simply may not happen, even after things are fixed. This is more likely to happen when, for instance SNDS or GPT has been down for days. No one is going to respond to all the direct inquiries. They may post a note to a mailing list, or answer one person who they know will spread the word.
Silence speaks volumes
Most deliverability people can interpret the silence pretty well; if they can’t, get a better deliverability team.
There’s a whole belief system built around the idea that the best way to get good deliverability is to have your own dedicated IPs. In fact, senders regularly approach me to ask when is the right time for them to get a dedicated IP. They assume all their deliverability problems will disappear if they get a dedicated IP.
Generally they’ve not asked the most important question: should they get a dedicated IP? They don’t consider the benefits and the strengths of being on a shared IPs. One of the biggest issues is we’ve mostly run out of free IPv4 space. Even though some very large networks are consolidating and selling off their IPs to others, IPv4 addresses are still a limited resource. For this reason, among others, Many ESPs, in fact most of them, offer dedicated IPs only to their biggest clients. The majority of their customers are on shared IPs.
The good news is that with modern filters dedicated IPs are not critical for good delivery.
Yes. I said it, and I’ll say it again. Dedicated IPs are not required for good delivery. Reputation, particularly at the webmail providers but increasingly elsewhere, is not solely based on the connecting IP address.
Google, in particular, has made it very clear that they use a matrix of domain and IP reputation. If domains need to be warmed up, that means Google is able to separate out mail from the same IP using different domains. OATH, too, focuses a lot on content and domains, rather than IP addresses. They’ve been able to selectively filter mail, even from dedicated IPs, for years now. Microsoft does put a little more emphasis on IP addresses, but some of the evidence I’ve seen says they look at reputation of the range of IPs not just the connecting one. Even there, a dedicated IP doesn’t buy you that much if your neighbours are not clean.
Of course, nothing about email is one size fits all. There are legitimate reasons to use dedicated IPs. The two big ones are certification and email programs sending more than 1 million emails a day. For certain senders, certification is important and those senders have to have a dedicated IP to get certified. Maintaining a dedicated IP take work. Generally the folks who do it right have a dedicated deliverability person on staff, or are paying a consultant to help them maintain their reputation.
On the other hand folks on shared IPs don’t have to worry about sending enough mail. They don’t have to manage their volume carefully and watch out for spiky traffic. If they’ve chosen a good provider, then the companies they share the IP with are meeting minimum standards and the overall IP reputation is high.
Some of the largest email service providers are built on shared IP addresses: Sendgrid, Mailchimp, and Sparkpost are all primarily shared IP services. Even ESPs that service some of the largest companies, like ExactTarget, have shared ranges for customers.
Over the last 3 or 4 years ISPs approach to IP reputation has changed dramatically. It’s just not as critical as it was a few years ago.
In 2014 we, for various reasons, had to move our mail server to new hardware. At the time we had a full cabinet of servers and a /25 SWIPed to us from our provider. We spun up the new hardware and assigned it an unused IP address in our range. We immediately started having deliverability problems. The problems were despite the IP being next to our old mail server IP, and having the same domain authentication. Rather than try to do anything fancy, we simply moved back to our old IP.
This experience had me slightly concerned as we moved from our colo space early in 2018. We moved our mail server from our own hardware to a dedicated IP address “in the cloud.” But, the transition went smoothly. We’ve had not a whiff of deliverability problems, even though we’re in a range we don’t control.
Today’s reality is that dedicated IPs are often not worth the effort to maintain them. Shared IPs can get even medium size senders the same level of delivery that dedicated IPs can. In fact, some of the messier senders out there actually ask their providers for shared IPs, rather than dedicated IPs because they can get better delivery off the shared IPs.
Don’t worry if you can’t get on a dedicated IP. Just focus on sending mail your recipients want and expect and you’ll reach the inbox just fine.
It’s probably one of the most consequential midterm elections in the US ever.
I was on the phone with a colleague recently. They were talking about collecting a bit of data over the weekend and mentioned how great it was they had the tools to be able to do this. Coincidentally, another colleague mentioned that when the subscription bombing happened they were able to react quickly because they had a decent tool chain. I’ve also been working with some clients who are dealing with compliance issues but don’t have the tools they need.
Internal tools, particularly those for deliverability and compliance, are seen as luxuries. They’re not that necessary and they’ll get done when there is time. This attitude is wrong.
Look at how ESPs responded to the subscription bombing problem. Some of them had good tools in place and were able to address the problem and limit the damage within hours of understanding the problem. Others needed to cobble together tools and access to even get a handle on the issue. It took them much longer to get delisted.
One of the bigger complaints I hear from colleagues is they don’t have tools they need. Now, I’m not saying that every deliverability and compliance team needs their own developer, although some do. But they do need access to internal dev resources so they can build up a suite of tools they can use to address emergent threats.
Captchas – those twisty distorted words you have to decipher and type in to access a website – have been around since the 1990s. Their original purpose was to tell the difference between a human user and an automated system, by requiring the user to answer a challenge – one that was supposedly hard for computers to solve, but easy for humans. A few years later they acquired the name CAPTCHA, an acronym for “Completely Automated Public Turing test to tell Computers and Humans Apart”.
Optical character recognition was pretty inaccurate in the 90s, especially with blurry or misaligned text. Text that was clearly legible to a human was completely inscrutable to state of the art OCR software. The first developers of CAPTCHA took all the advice for getting accurate OCR scans, and did the opposite – intentionally creating text that would be readable, but impossible for OCR to parse. This worked fairly well to differentiate humans and robots for a while, but eventually technology began to catch up. Off the shelf OCR got better, and mechanical attacks specific to commonly used CAPTCHAs were getting good enough to answer a significant fraction correctly.
Attempting to combat that by making CAPTCHAs harder to solve mechanically also made them much harder for humans to solve, and was terrible for accessibility. That arms race had gone about as far as it could.
Meanwhile, a group at CMU realized that the millions of hours of human time wasted by solving CAPTCHAs could be applied to do useful work instead. They had a lot of scanned documents they wanted to digiitize, but the quality of the images was too poor to OCR. So they mechanically extracted single words from the documents and showed them, two at a time, to users as a CAPTCHA and asked them to enter the two words. They knew the right answer for one word, so if the user entered that word right they’d assume they probably got the unknown word right too (and, almost as an aside, allowed the user access). That was “reCAPTCHA”.
By making reCAPTCHA a hosted service it became much simpler for website owners to use CAPTCHAs, so they began to be used more widely. reCAPTCHA was acquired by Google in 2009 and they kept developing it. They used it to digitize street numbers for Google street view, and added “pick matching images” as an alternative puzzle. Increasingly, though, they didn’t actually need the humans to solve puzzles in the common case – they could tell from the history of the connecting IP address and the behaviour of the web browser that a user was likely legitimate, and let them through without making them do anything other than check a box. It was tracking reputation instead.
If you’ve ever used TOR – the secure browser that hides any identifying cookies and mixes your web traffic untraceably in with other TOR users you’ll have seen what the web looks like to users with a poor reputation. Every time you see a reCAPTCHA it’s not just a simple checkbox, it’s answering dozens of visual puzzles. That’s what most bots will see with reCAPTCHA.
reCAPTCHA is no longer really based on separating humans from bots so much as it’s differentiating between “normal”, “good” users – probably human – and “bad” users – probably bots – based on many characteristics of the users session. It’s really pretty good at that, and with the “just check a box” version of reCAPTCHA most users will see most of the time it’s pretty low friction. It’s probably the most effective tool against subscription bombing at the moment.
Google have just released version 3 of reCAPTCHA. This isn’t really a CAPTCHA at all, rather it’s a way to track the behaviour and reputation of users as visible to Google. It watches your users as they interact with your site, sends all that data to Google and they provide a quality score for each user that you can combine with your own information about them to make decisions.
It’s all very low friction, and probably very effective at detecting malicious bots. But it’s also a very intrusive user tracking technology that’ll send user history to Google whenever they’re on a site that uses it. The list of information it captures is definitely enough to uniquely fingerprint and track a user. It’ll be interesting to see what happens when that collides with the move towards web browsers being more privacy-focused and hostile to tracking.
Much of the current deliverability advice focuses on a few key ideas:
- Authenticate your mail with SPF, DKIM and DMARC
- Use a dedicated IP.
- Monitor delivery.
- Clean your data.
All of these things are absolutely things you should be doing, but senders can do all these things and still have cruddy delivery. These things are great and can help your mail deliver better. But they’re not enough to get mail into the inbox.
As I think about email filters I can put them in some different broad categories based on the email types they target.
- Safety filters – block phishing and malware
- Unsolicited filters – block mail that looks like it’s unsolicited
- Unwanted filters – block mail that is unwanted by the recipient
Safety filters are pretty self explanatory. They use different signals like source IP, links, and virus signatures to block mail. Some safety filters are about protecting the infrastructure, so they act on senders that open too many connections, or send non-RFC compliant mail or act in ways that are not well-behaved.
Unsolicited filters have their own signals they use to determine if a message is unsolicited. These include a lot of the things we talk about in delivery. Things like spam complaints, bounces, and spamtrap hits are all things that indicate recipients never opted in to receive a particular email.
Unwanted filters use some of the same signals as unsolicited filters, the difference between unsolicited email and unwanted email is somewhat subtle. Signals for unwanted filters are the things we describe as engagement metrics. When emails are read and saved and moved between folders that tells the ISP these mails are wanted. Emails that are deleted without opening are likely unwanted.
Understanding what kind of mail filters are targeting helps drive what types of fixes we do. If the problem is our users don’t want the mail then removing bounces isn’t going to affect the signals driving the filter.
On the flip side, if the problem is the mail looks unsolicited because it has too many dead addresses and hits spamtraps, we can sometimes fix them using the same techniques we use for addressing unwanted filters. When we limit sending to people who have opened and engaged with the messages, we’re removing the addresses that signal the email is unwanted and the email is unsolicited.
Filters like Spamhaus focus on unsolicited emails. That’s why they focus on making sure that senders have permission rather than focusing on engagement metrics.
Understanding what type a mail a filter is attempting to protect users from is crucial for solving delivery problems.
There are probably hundreds of thousands of really awesome SaaS products out there. They provide a framework to do all sorts of stuff that used to be really hard to do. Almost all of them include some email component. They dutifully build the email piece into their platform and, because they’re smart, they outsource the actual sending to one of SMTP providers. They’re happy, their customers are happy, and spammers are happy.
SaaS providers focus on their core competencies, which is their platform. Their focus is building a product that meets the needs of their customers. They’re not an email service provider, so they think, and they don’t really pay much attention to email. They send mail by handing it off to their provider and assume all will be well with delivery because their customers are small businesses and are not sending lots of mail and aren’t spammers.
The problem is, spammers have recognized these SaaS companies are a way to access high powered sending infrastructure that have banned them from sending directly. Many of these bad guys take advantage of freemium models and simply send low volumes of email through multiple accounts. Because they’re hiding in the middle of real customers, they can often go undetected for months or years.
Eventually, though, someone notices and the SaaS provider experiences a blocklisting or other delivery problem. At that point, there’s a scramble to figure out why delivery is horrible or they’re listed on Spamhaus or their provider is sending them compliance notices. Often responding to these problems can take months and require some business processes to be rethought from the ground up.
There’s no easy fix for this problem. But just as SaaS providers need to think about application and data security, they also need to think about email security. How can they detect and prevent spammers from abusing their system and hurting them and their customers.
Marketers have a thing about transactional mail. In the US, transactional mail is exempt from many of the CAN SPAM regulations. If they label a mail transactional, then they can send it even when the recipient has opted-out! The smart marketer looks for opportunities to send transactional mail so they can
bother spam get their brand in front of people who’ve opted out.
Enter the anniversary, birthday or holiday message. If a message is sent once per year and is based around some event, then it’s classified as transactional. But these messages aren’t really transactional. There’s nothing the recipient did. The only “transaction” is the passage of time.
These kind of fake transactional messages are just legal spam, particularly if the recipient has opted out of marketing mail.
I mentioned this on the email geeks slack channel, along with the comment that if companies were going to ignore preferences they should at least offer recipients something for reading their spam. One smart person pointed out that if the messages contained rewards, like a percentage off, then it’s not transactional mail any more and it’s a clear violation of CAN SPAM and other laws.
There are, of course, legitimate transactional messages. My favourite definition of transactional message is a message confirming or completing an action initiated by the recipient. The whole point of carving out transactional messages from CAN SPAM is for the recipient, not for the sender. Any truly transactional mail, therefore, must benefit the recipient more than the sender.
There are also cases where one time emails benefit the recipient, but are not generated due to a direct interaction by the recipient. One common case of this is messages alerting users that a particular product is back in stock. These messages are wanted by the user, and they benefit the user, but the timing of their send isn’t related to a user’s action. Timing is dictated by restocking timelines. In discussions with clients, I describe these types of messages as triggered rather than transactional.
Transactional mail must center the recipient. If a recipient has opted out from mail from a company, then they probably don’t want to receive marketing mail, even if the subject line is Happy Birthday or Happy Anniversary.
The MAAWG conference was held in Brooklyn a few weeks ago. Many positive discussions and sessions happened at the conference. But there was an incident of harassment during the conference where one participant assaulted multiple other attendees during late evening activities. I’m not going to speak too much to what happened as I wasn’t there. What I will say is that I am proud of my friends and colleagues who stepped up to make sure that the targets of the harassment made it safely to their rooms. I’m also pleased that the conference pulled the harasser’s badge and banned him from the conference in short order.
This incident did expose some issues with how the conference is collecting and handling code of conduct violations. But that’s nothing unusual. In fact, most good policies include the after-action report to look at ways to improve the process in the future. MAAWG is looking at how to handle things better, moving forward, and that’s what they should be doing.
There’s been much discussion about this across the industry since this happened and a number of people have asked for information and resources on how to handle incidents. As I’ve been participating in these discussions, I’ve found a number of sites that are useful resources. I’m using this post, primarily, as a way to document those resources and make it such that I have a single link I can send to men who ask me how they can help.
Wikis and FAQs
Conference anti-harassment from Geek Feminism. It collects policies from different conferences and talks about what works and what doesn’t.
Code of Conduct 101 from Ashe Dryden
Men who advocate for safer conferences.
The Code of Conduct Jess Noller from Python
Jim C. Hines, and John Scalzi both SF Authors
Other good resources
Actions to take
Why women don’t speak up: HBR and Ashe Dryden
For myself, a lot of what I’m doing is sharing information with people in different fora. Some of them are public, like this blog post. Others are semi-private fora. Still others are one on one (or one on few) discussions. The group I co-founded, Women of Email is also starting to look into this problem, and I am the sponsoring board member. Given how much success we’ve had getting women speaking at conferences, I’m confident we will make a difference.
Through the course of these discussions, I’ve had a number of men ask what they can do. They want to help but they don’t know how to fix things. I deeply respect this position, but, women don’t know how to fix this either. We don’t have the answers. What I think would really help is for men to start educating themselves and other men. Stop asking women to shoulder the burden of telling you what to do on top of of being targets, of navigating reporting, of dealing with the personal and professional fallout and of figuring out how not to be targeted again.
I know there are other resources out there, I’ve seen them. But I don’t always bookmark what I should. These pages are ones I’ve found helpful over the last few weeks and others I remember. What resources have you found to be helpful and would like to share?
I wear a number of hats and have a lot of different email addresses. I like to keep the different email addresses separate from each other, “don’t cross the streams” as it were.
Recently I’ve been getting spam to my womenofemail.org address asking about the wordtothewise.com website.
I’m not sure where Ms. Catherine Metcalf bought my Women of Email address or how she connected it with a post written by Steve here on the WttW website. But it did bring up an interesting question I don’t necessarily have an answer to.
A little background. Women of Email hosts email on G Suite. Unlike most of my accounts, my “this is spam” reports actually are measured for reputation purposes. I can report this as spam, Google will take action and I won’t have to see mail from Ms. Metcalf (and ideally bluelabellabs.net) ever again. But then I started thinking. The only URL in the message posts to my website.
My question is: if I click “this is spam” on the above message will Google knock my reputation? How much mail is sent to Google on any given day mentioning WttW links and websites?
This is, of course, a question that proves I’m a little too focused on reputation. No normal person would consider whether or not to report this as spam. They’d just hit this is spam and assume it will only hurt the sender’s reputation, not affect the reputation of anything in the message. But I know that every link in a message has its own reputation.
I suspect there wouldn’t be a huge amount of fallout. Gmail looks at every “resource combination” (to use their term) when making decisions. Yes, a t-i-s click is likely to be a minor tick against “wordtothewise.com” in general. But only a minor one. It’s likely to be a bigger tick against mail from email@example.com that mentions wordtothewise.com.
Looking at the even bigger picture. Google knows that Catherine Metcalf is sending out a lot of mail mentioning different URLs. They have to, she’s using Google to send them. But they can also see that a subset of the recipients delete the mail without opening, or report the message as spam. They probably can even tell that she’s using some sort of automated software to send the mail, rather than typing each message herself. If enough people report the message as spam, Google is smart enough to spam folder her mail. Given they’re handling her outgoing messages, it would be nice if they could block it on the outbound, but we can’t have everything.