The great folks over at Zettasphere and Emailmonday have released their Email Addiction Survey. Nothing surprising in the data that I can see, although I suspect one particular data point is going to surprise folks.
Yup, more than 70% of people don’t really care about a do not reply address in a message. Honestly, I’m not surprised. Most users don’t really care. In all honesty it probably doesn’t affect delivery that much any longer, either. I still think it’s generally a bad idea but as long as you’re providing a communications channel for recipients to connect with your company it’s probably not going to be a giant problem.
You take a turn, I take a turn
At the SMTP level email is very much a simple line-by-line text based protocol. The client sends a command on a single line, the server responds with one or more lines (the last one marked by having a space in the fourth column), and then the client sends another command.
The main exception to that is when the client sends the payload of the email. Once the server has said it’s ready to receive the email the client sends the whole thing, then tells the server it’s finished by sending a line consisting of just a single period “.”.
But what if you have a line in your email that’s just a single period? The developers of SMTP thought of that! Whenever the SMTP client sees a line that begins with a period, it has to add another one. And whenever the SMTP server sees a line that begins with a period, it has to remove it.
This is called “dot-stuffing”. If an email has the line “.”, it’ll be sent on the wire as “..”. Email with the line “.yakko.wakko.dot.” will be sent as “..yakko.wakko.dot.”.
If everything works properly, nobody will ever notice the dot-stuffing. Often it doesn’t, though. One way to break things is for a sender to use a message composition API that produces mail that’s been dot-stuffed and is ready to send, then to send it via an API that expects non-dot-stuffed content, and will stuff it again before sending. Another is for the receiving server or intermediate server to just not do un-stuffing properly.
Understuffed, overstuffed, wombling free
The symptoms of this vary, depending on whether the mail is over-stuffed or under-stuffed. If it’s not stuffed enough, and there’s a line in the email which consists of a single dot then the receiving server will see that line as the end of the message, and accept the (truncated) message for delivery at that point – and then throw an error for every line the client sends after that, as it’s expecting SMTP keywords, not a message body.
If it’s over-stuffed then the problem is a bit less obvious. Any line that begins with a single dot when it’s sent will begin with two dots when it’s received.
You might think that it’d be pretty rare for a line to begin with a dot, and even rarer for anyone to notice a problem if that dot is doubled. But there’s one case where it’s quite likely.
Quoted-Printable encoded HTML
HTML sent quoted-printable encoded is the normal format for commercial emails. The HTML gives us all the rich content we like, while the quoted-printable encoding lets us not have to worry so much about sending non-ascii characters or violating SMTP specs by accidentally sending lines longer than 998 characters.
Quoted-printable encoding is pretty simple. Some characters are encoded as three ascii characters – an equals sign, followed by two hex digits representing the character. So “=” is quoted-printable encoded as “=3D”. And long lines are wrapped at 76 characters, with an = sign to mark the wrapping. There are a few other rules, but that’s about it.
(We have a tool that’ll decode quoted-printable, if you ever need that, on tools.wordtothewise.com).
If you have a line that looks like …
about forty-five characters of text <a href=”http://whatever.example.com”>click here</a>
… it will quoted-printable encode to look like …
about forty-five characters of text <a href=3D”http://whatever.example=
… then overstuffing will mean it ends up at the recipient as …
about forty-five characters of text <a href=3D”http://whatever.example=
… and the mail client will quoted-printable decode it to be …
about forty-five characters of text <a href=”http://whatever.example..com”>click here</a>
That looks identical, but the link won’t work because of the doubled dot in the URL.
The symptoms of this will be that some links in the mail you send won’t work when it’s received. If the stuffing bug is at your end then it’ll mean that some links, depending on the content and layout of your email, won’t work. If the stuffing bug is at the recipient mailserver or a forwarding middlebox it means that some links, depending on the content and layout of your email, won’t work for some recipients.
If your mail generation and delivery pipeline isn’t handling dot-stuffing correctly you should fix that. You can test it by sending yourself a plain text email with a line that starts with a dot – if it’s doubled, there’s a problem.
If the issue is at the receiver it’s harder to identify that it’s happening, let alone get it fixed. One way to mitigate in that case would be to configure your quoted-printable encoder to encode “.” as “=2E”. That way no raw “.” will appear in your messages, so there’s no way it can get overstuffed.
As of 6pm UTC the fbl.returnpath.com website is down. Return Path are aware of the issue and are working to fix it. I haven’t seen any estimated time to fix.
But, it’s not just you and they are aware.
EDIT: And 30 minutes after I posted this, it’s back. All fixed! Go and submit your FBL changes.
Anyone actively handling deliverability issues has had the experience of submitting a ticket or email and receiving no response. Alternatively, we get a boilerplate response that seems to not address the question. It happens to me, it happens to colleagues, it happens to everyone. One of the biggest challenges we face is taking that lack of response and channeling it into action items for our customers and clients.
This is where the experience and insight of a good deliverability consultant comes into play. We translate both the random boilerplate answers and the silence into actionable recommendations for clients. Good deliverability experts can read between the lines and use their experience to extrapolate the silence into actionable information.
Silence is an answer
Many people believe that lack of a reply means that no one saw their ticket or email. Sometimes, it is true, the contact method goes into a blackhole and no one sees the message. In the cases I can think of, this isn’t personal. The person (and it’s almost always an individual person) doesn’t want any feedback, so they don’t check the one channel they have for feedback. Or, the request is going to the wrong place. For example, I get a lot of mail (and even phone calls) from people who think I’m support for any number of ISPs. I answer some, but don’t answer others, particularly when I’m busy.
In the case of a list that ignores every inquiry, silence tells us this is likely not relevant to blocking problems. A filter or blocklist that doesn’t speak with listees typically doesn’t have many users. This is especially true in the case where we just found the listing by using an online blacklist checker like mxtoolbox or multirbl.valli.org. There are hundreds of blocklists out there, but in my experience the number that have a wide enough footprint to affect mail delivery is less than 10.
There are other cases, however, where we don’t receive answers even though the message is read and the person reading it could help. These are a little harder to interpret, but it’s not impossible. There are different reasons a volunteer or employee might choose to not respond to a message. For instance, some places have a policy of not responding to any message that contains a threat of legal action. Any mention of lawyers in a message and the only response is silence. In other cases, the issue may be one where there is nothing to say because it’s all been said before.
Every once in a while a client will hire me to mediate a blocklisting and when I reach out I either get silence or a brief “we’ve told the client what they need to do” from the list. Inevitably this has been an issue that has been going on for a while. I was contacted because the filter or blocklist stopped answering. They stopped answering because the issue was going in circles. Sometimes it’s a communication disconnect, and I can help. Sometimes it’s simply the listee doesn’t want to do anything different and refuse to make the necessary changes to get delisted.
Silence can be a no
Sometimes the reason we don’t get an answer is because the answer is no.
For instance, some senders send long emails requesting significant amounts of data from the ISP or the filter company. These emails are more likely to be ignored than others. Sometimes the company simply doesn’t have the data being requested. Other times it’s too much data and will take too long to compile. While some ISPs provide official support channels for senders, the vast majority don’t. Any work done on behalf of senders, is at the discretion and leisure of the employee.
Why doesn’t the person just send back a no? Sometimes it’s just due to overwork. Stuff gets dropped on the floor by accident. There is no intention to ignore it, but we’re all busy and overworked. When an email is a favor it ends up at the bottom of the todo pile. And some of us never find the bottom of the pile. Or by the time someone gets to the question it’s 10 days later and the filters have all changed anyway so an answer is no longer timely.
Another reason is because there is a perception that saying no will cause conflict. Unfortunately, history and experience tell us that some percentage of senders will take a no answer as the opening to a negotiation. They send back email after email trying to negotiate some answer or get some information. This is frustrating for the person who said no. After more than a couple experiences like this it turns into too much work to say no, so email is just not answered.
The issue has been fixed
For a lot of folks, responding to messages is not as important as making things work. Most of the time in deliverability we’re not the customer. None of the normal customer interaction rules apply. If there are dozens or hundreds of messages about a mistake affecting multiple listings, the priority will be fixing the mistake. Answering those dozens or hundreds of emails simply may not happen, even after things are fixed. This is more likely to happen when, for instance SNDS or GPT has been down for days. No one is going to respond to all the direct inquiries. They may post a note to a mailing list, or answer one person who they know will spread the word.
Silence speaks volumes
Most deliverability people can interpret the silence pretty well; if they can’t, get a better deliverability team.
There’s a whole belief system built around the idea that the best way to get good deliverability is to have your own dedicated IPs. In fact, senders regularly approach me to ask when is the right time for them to get a dedicated IP. They assume all their deliverability problems will disappear if they get a dedicated IP.
Generally they’ve not asked the most important question: should they get a dedicated IP? They don’t consider the benefits and the strengths of being on a shared IPs. One of the biggest issues is we’ve mostly run out of free IPv4 space. Even though some very large networks are consolidating and selling off their IPs to others, IPv4 addresses are still a limited resource. For this reason, among others, Many ESPs, in fact most of them, offer dedicated IPs only to their biggest clients. The majority of their customers are on shared IPs.
The good news is that with modern filters dedicated IPs are not critical for good delivery.
Yes. I said it, and I’ll say it again. Dedicated IPs are not required for good delivery. Reputation, particularly at the webmail providers but increasingly elsewhere, is not solely based on the connecting IP address.
Google, in particular, has made it very clear that they use a matrix of domain and IP reputation. If domains need to be warmed up, that means Google is able to separate out mail from the same IP using different domains. OATH, too, focuses a lot on content and domains, rather than IP addresses. They’ve been able to selectively filter mail, even from dedicated IPs, for years now. Microsoft does put a little more emphasis on IP addresses, but some of the evidence I’ve seen says they look at reputation of the range of IPs not just the connecting one. Even there, a dedicated IP doesn’t buy you that much if your neighbours are not clean.
Of course, nothing about email is one size fits all. There are legitimate reasons to use dedicated IPs. The two big ones are certification and email programs sending more than 1 million emails a day. For certain senders, certification is important and those senders have to have a dedicated IP to get certified. Maintaining a dedicated IP take work. Generally the folks who do it right have a dedicated deliverability person on staff, or are paying a consultant to help them maintain their reputation.
On the other hand folks on shared IPs don’t have to worry about sending enough mail. They don’t have to manage their volume carefully and watch out for spiky traffic. If they’ve chosen a good provider, then the companies they share the IP with are meeting minimum standards and the overall IP reputation is high.
Some of the largest email service providers are built on shared IP addresses: Sendgrid, Mailchimp, and Sparkpost are all primarily shared IP services. Even ESPs that service some of the largest companies, like ExactTarget, have shared ranges for customers.
Over the last 3 or 4 years ISPs approach to IP reputation has changed dramatically. It’s just not as critical as it was a few years ago.
In 2014 we, for various reasons, had to move our mail server to new hardware. At the time we had a full cabinet of servers and a /25 SWIPed to us from our provider. We spun up the new hardware and assigned it an unused IP address in our range. We immediately started having deliverability problems. The problems were despite the IP being next to our old mail server IP, and having the same domain authentication. Rather than try to do anything fancy, we simply moved back to our old IP.
This experience had me slightly concerned as we moved from our colo space early in 2018. We moved our mail server from our own hardware to a dedicated IP address “in the cloud.” But, the transition went smoothly. We’ve had not a whiff of deliverability problems, even though we’re in a range we don’t control.
Today’s reality is that dedicated IPs are often not worth the effort to maintain them. Shared IPs can get even medium size senders the same level of delivery that dedicated IPs can. In fact, some of the messier senders out there actually ask their providers for shared IPs, rather than dedicated IPs because they can get better delivery off the shared IPs.
Don’t worry if you can’t get on a dedicated IP. Just focus on sending mail your recipients want and expect and you’ll reach the inbox just fine.
It’s probably one of the most consequential midterm elections in the US ever.
I was on the phone with a colleague recently. They were talking about collecting a bit of data over the weekend and mentioned how great it was they had the tools to be able to do this. Coincidentally, another colleague mentioned that when the subscription bombing happened they were able to react quickly because they had a decent tool chain. I’ve also been working with some clients who are dealing with compliance issues but don’t have the tools they need.
Internal tools, particularly those for deliverability and compliance, are seen as luxuries. They’re not that necessary and they’ll get done when there is time. This attitude is wrong.
Look at how ESPs responded to the subscription bombing problem. Some of them had good tools in place and were able to address the problem and limit the damage within hours of understanding the problem. Others needed to cobble together tools and access to even get a handle on the issue. It took them much longer to get delisted.
One of the bigger complaints I hear from colleagues is they don’t have tools they need. Now, I’m not saying that every deliverability and compliance team needs their own developer, although some do. But they do need access to internal dev resources so they can build up a suite of tools they can use to address emergent threats.
Captchas – those twisty distorted words you have to decipher and type in to access a website – have been around since the 1990s. Their original purpose was to tell the difference between a human user and an automated system, by requiring the user to answer a challenge – one that was supposedly hard for computers to solve, but easy for humans. A few years later they acquired the name CAPTCHA, an acronym for “Completely Automated Public Turing test to tell Computers and Humans Apart”.
Optical character recognition was pretty inaccurate in the 90s, especially with blurry or misaligned text. Text that was clearly legible to a human was completely inscrutable to state of the art OCR software. The first developers of CAPTCHA took all the advice for getting accurate OCR scans, and did the opposite – intentionally creating text that would be readable, but impossible for OCR to parse. This worked fairly well to differentiate humans and robots for a while, but eventually technology began to catch up. Off the shelf OCR got better, and mechanical attacks specific to commonly used CAPTCHAs were getting good enough to answer a significant fraction correctly.
Attempting to combat that by making CAPTCHAs harder to solve mechanically also made them much harder for humans to solve, and was terrible for accessibility. That arms race had gone about as far as it could.
Meanwhile, a group at CMU realized that the millions of hours of human time wasted by solving CAPTCHAs could be applied to do useful work instead. They had a lot of scanned documents they wanted to digiitize, but the quality of the images was too poor to OCR. So they mechanically extracted single words from the documents and showed them, two at a time, to users as a CAPTCHA and asked them to enter the two words. They knew the right answer for one word, so if the user entered that word right they’d assume they probably got the unknown word right too (and, almost as an aside, allowed the user access). That was “reCAPTCHA”.
By making reCAPTCHA a hosted service it became much simpler for website owners to use CAPTCHAs, so they began to be used more widely. reCAPTCHA was acquired by Google in 2009 and they kept developing it. They used it to digitize street numbers for Google street view, and added “pick matching images” as an alternative puzzle. Increasingly, though, they didn’t actually need the humans to solve puzzles in the common case – they could tell from the history of the connecting IP address and the behaviour of the web browser that a user was likely legitimate, and let them through without making them do anything other than check a box. It was tracking reputation instead.
If you’ve ever used TOR – the secure browser that hides any identifying cookies and mixes your web traffic untraceably in with other TOR users you’ll have seen what the web looks like to users with a poor reputation. Every time you see a reCAPTCHA it’s not just a simple checkbox, it’s answering dozens of visual puzzles. That’s what most bots will see with reCAPTCHA.
reCAPTCHA is no longer really based on separating humans from bots so much as it’s differentiating between “normal”, “good” users – probably human – and “bad” users – probably bots – based on many characteristics of the users session. It’s really pretty good at that, and with the “just check a box” version of reCAPTCHA most users will see most of the time it’s pretty low friction. It’s probably the most effective tool against subscription bombing at the moment.
Google have just released version 3 of reCAPTCHA. This isn’t really a CAPTCHA at all, rather it’s a way to track the behaviour and reputation of users as visible to Google. It watches your users as they interact with your site, sends all that data to Google and they provide a quality score for each user that you can combine with your own information about them to make decisions.
It’s all very low friction, and probably very effective at detecting malicious bots. But it’s also a very intrusive user tracking technology that’ll send user history to Google whenever they’re on a site that uses it. The list of information it captures is definitely enough to uniquely fingerprint and track a user. It’ll be interesting to see what happens when that collides with the move towards web browsers being more privacy-focused and hostile to tracking.
Much of the current deliverability advice focuses on a few key ideas:
- Authenticate your mail with SPF, DKIM and DMARC
- Use a dedicated IP.
- Monitor delivery.
- Clean your data.
All of these things are absolutely things you should be doing, but senders can do all these things and still have cruddy delivery. These things are great and can help your mail deliver better. But they’re not enough to get mail into the inbox.
As I think about email filters I can put them in some different broad categories based on the email types they target.
- Safety filters – block phishing and malware
- Unsolicited filters – block mail that looks like it’s unsolicited
- Unwanted filters – block mail that is unwanted by the recipient
Safety filters are pretty self explanatory. They use different signals like source IP, links, and virus signatures to block mail. Some safety filters are about protecting the infrastructure, so they act on senders that open too many connections, or send non-RFC compliant mail or act in ways that are not well-behaved.
Unsolicited filters have their own signals they use to determine if a message is unsolicited. These include a lot of the things we talk about in delivery. Things like spam complaints, bounces, and spamtrap hits are all things that indicate recipients never opted in to receive a particular email.
Unwanted filters use some of the same signals as unsolicited filters, the difference between unsolicited email and unwanted email is somewhat subtle. Signals for unwanted filters are the things we describe as engagement metrics. When emails are read and saved and moved between folders that tells the ISP these mails are wanted. Emails that are deleted without opening are likely unwanted.
Understanding what kind of mail filters are targeting helps drive what types of fixes we do. If the problem is our users don’t want the mail then removing bounces isn’t going to affect the signals driving the filter.
On the flip side, if the problem is the mail looks unsolicited because it has too many dead addresses and hits spamtraps, we can sometimes fix them using the same techniques we use for addressing unwanted filters. When we limit sending to people who have opened and engaged with the messages, we’re removing the addresses that signal the email is unwanted and the email is unsolicited.
Filters like Spamhaus focus on unsolicited emails. That’s why they focus on making sure that senders have permission rather than focusing on engagement metrics.
Understanding what type a mail a filter is attempting to protect users from is crucial for solving delivery problems.
There are probably hundreds of thousands of really awesome SaaS products out there. They provide a framework to do all sorts of stuff that used to be really hard to do. Almost all of them include some email component. They dutifully build the email piece into their platform and, because they’re smart, they outsource the actual sending to one of SMTP providers. They’re happy, their customers are happy, and spammers are happy.
SaaS providers focus on their core competencies, which is their platform. Their focus is building a product that meets the needs of their customers. They’re not an email service provider, so they think, and they don’t really pay much attention to email. They send mail by handing it off to their provider and assume all will be well with delivery because their customers are small businesses and are not sending lots of mail and aren’t spammers.
The problem is, spammers have recognized these SaaS companies are a way to access high powered sending infrastructure that have banned them from sending directly. Many of these bad guys take advantage of freemium models and simply send low volumes of email through multiple accounts. Because they’re hiding in the middle of real customers, they can often go undetected for months or years.
Eventually, though, someone notices and the SaaS provider experiences a blocklisting or other delivery problem. At that point, there’s a scramble to figure out why delivery is horrible or they’re listed on Spamhaus or their provider is sending them compliance notices. Often responding to these problems can take months and require some business processes to be rethought from the ground up.
There’s no easy fix for this problem. But just as SaaS providers need to think about application and data security, they also need to think about email security. How can they detect and prevent spammers from abusing their system and hurting them and their customers.