Delivery Improvement

How to choose an ESP based on deliverability

Despite what a lot of SEO slop will try and tell you there’s no way to measure deliverability performance across multiple ESPs in any way that’s meaningful.

Diagnosing Hard Bounces

A very short post about diagnosing hard bounces, because I’ve had to give the same advice to a dozen folks over the past few months.

Your bounce classification is a bit rubbish

When a mailbox provider rejects or defers an email it sends back a message explaining why.

If you’re using Entrust for your BIMI VMC …

BIMI is the protocol you can use to publish an icon to display in the inbox of some large consumer webmail providers. To have it displayed at Gmail you need more than just clean DMARC and a published BIMI record, you also need to have a Verified Mark Certificate, a VMC.

I always want to say “Emails, and Opens, and Clicks… Oh My!” when I’m talking about them.

A still from *Cargo Cult*, by Bastien Dubois

During the war they saw airplanes land with lots of good materials, and they want the same thing to happen now. So they’ve arranged to make things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head like headphones and bars of bamboo sticking out like antennas—he’s the controller—and they wait for the airplanes to land. They’re doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn’t work. No airplanes land. — Richard Feynman

It’s deliverability week, so everyone is talking about deliverability. But I’d like to take a moment to mention deliverability’s evil twin from the mirror universe – filter evasion.

Mailbox providers want happy recipients

Mailbox providers want their users to be happy with the mail they receive and the service they get. That’s driven by stark business reasons: acquiring new users is costly, happy users bring in revenue – whether directly, or indirectly via advertising – and their word of mouth helps bring in more users, and hence more revenue. That’s still true when the email service is bundled as part of a larger package, such as broadband service or domain registration.

What is Deliverability Week? Al Iverson decided it should happen, and asked a bunch of deliverability folks to share some of their thoughts about the deliverability industry – why do we do this? where did we come from? what’s next?

Deliverability Matters. Deliverability Week 2024.

Next week everyone will be talking Deliverability.

DNS for white label authentication with SproutDNS

I wrote last year about using “stunt” nameservers for customer subdomain authentication – i.e. dynamically generating all the authentication records needed in DNS for each customer as needed.

About My Email

Happy 2024, everyone!

We’ve released a shiny new tool to let folks self-check a lot of common questions we see about email requirements.

Yahoogle Requirements Update

Since I wrote about it last month the requirements for bulk senders to Yahoo and Google have changed a little.

Customer subdomain authentication

EDIT: Now with a production-ready implementation I talk about more here.

They Must Have Changed Something…

One of the most common refrains I hear from folks with delivery problems is that the filters must have changed because their mail suddenly started to go to the bulk folder. A few years ago, I posted about how even when there is no change in the sender’s behavior, reputation can slowly erode until mail suddenly goes to the Gmail bulk folder. Much of that still applies – although the comments on pixel loads (what other folks call ‘open rates’) are a bit outdated due to changes in Gmail behavior.

Unresolvable RFC.5321 domain at Yahoo

Seen this recently?

451 Message temporarily deferred due to unresolvable RFC.5321 from domain; see https://postmaster.yahooinc.com/error-codes

“Friendly From” addresses

When we’re looking at the technical details of email addresses there are two quite different contexts we talk about.

Don’t break the (RFC) rules

It looks like Microsoft are getting pickier about email address syntax, rejecting mail that uses illegal address formats. That might be what’s causing that “550 5.6.0 CAT.InvalidContent.Exception: DataSourceOperationException, proxyAddress: prefix not supported – ; cannot handle content of message” rejection.

Life of an Email

I’m repeating the presentation I gave at M3AAWG in London for the Certified Senders Alliance.

It’s all about how to send an email by hand, and how knowing the mechanics of how an email is sent can help us diagnose email delivery issues.

We’re starting in about five hours from when I post this.

Register at https://register.gotowebinar.com/register/2268789893122531343

Sending email

I did a class at M3AAWG teaching the basic mechanics of sending an email, both really by hand using dig and netcat, and using SWAKS. No slides, but if you’re interested in the script I’ve posted a very rough copy of my working notes here.

Step by Step guide to fixing Gmail delivery

I regularly see folks asking how to fix their Gmail delivery. This is a perennial question (see my 2019 post and the discussions from various industry experts in the comments). Since that discussion I haven’t seen as much complaining about problems.

When opens hurt reputation

Podia has scraped the Word to the Wise blog and I’m currently receiving an ongoing drip campaign from them absolutely begging me to mention them in my blog post on cold emails.

Purging to prevent spamtraps

Someone recently asked when they should purge addresses to remove spamtraps. To my mind this is actually the wrong question. Purging addresses that don’t engage is rarely about spamtraps, it’s about your overall communication processes.

An open is not permission

A decade or so ago I was helping a client troubleshoot a Spamhaus listing. They, as many companies do, had a database with addresses from a number of different sources. Spamhaus was asking for them to reconfirm the entire database, which they didn’t want to do. I came up with the idea that if we had some sign of activity on the email address, like an open or a click and some other corresponding activity related to that open or click then we could assume that the address was likely a real user and was interested in the emails.

Measurements

One of the things I’ve been spending a lot of time thinking about lately is how we measure deliverability. Standard deliverability measurements include: opens, bounces, complaints, and clicks. There are also other tools like probe accounts, panel data, and public blocklists. Taken together these measurements and metrics give us an overall view of how our mail is doing.

Gmail suddenly puts mail in the bulk folder

One of the delivery challenges that regularly comes up in various delivery discussion spaces is the “Gmail suddenly put my mail in spam.” From my perspective, there is rarely a “suddenly” about Gmail’s decision making process.

Dead addresses tell us things

There was confirmation this week that the increase in “user unknown” messages from Yahoo is actually Yahoo cleaning out abandoned accounts. At the same time a Yahoo is sending out notifications to folks to log into mail.

My domain reputation is bad, should I get a new domain?

Many companies have the occasional “oops” where they send email they probably shouldn’t have. This can often cause a decrease in reputation and subsequent delivery problems. Some companies rush to fix things by changing domains.

What’s a suspicious domain?

The question came up on slack and I started bullet pointing what would make a domain suspicious. Seemed like a reasonable blog post. In no particular order, some features that make a domain suspicious to spam filters.

Marking mail as spam says what?

I wear a number of hats and have a lot of different email addresses. I like to keep the different email addresses separate from each other, “don’t cross the streams” as it were.

Who are mimecast?

Mimecast is a filter primarily used by businesses. They’re fairly widely used. In some of the data analysis I’ve done for clients, they’re a top 10 or top 20 filter.
Earlier today someone asked on Facebook if mimecast may be blocking emails based on the TLD. The short answer is it’s unlikely. I’ve not seen huge issues with them blocking based on TLD of the domain. They’re generally more selective than that.

The good news is mimecast is really pretty good about giving you explanations for why they’re blocking. They’ll even tell you if it’s mimecast related or if it’s a specific user / user-company block.
Some example rejection messages from a recent dive into some bounce logs.

What does mitigation really mean?

It is a regular occurrence that senders ask filters and ISPs for mitigation. But there seems to be some confusion as to what mitigation really means. I regularly hear from senders who seem to think that once they’ve asked for mitigation that they don’t have to worry about filtering or blocking at that ISP for a while. They’re surprised when a few weeks or even days after they asked for mitigation their mail is, one again, blocked or in the bulk folder.

What kind of mail do filters target?

All to often we think of filters as a linear scale. There’s blocking on one end, and there’s an inbox on the other. Every email falls somewhere on that line.
Makes sense, right? Bad mail is blocked, good mail goes to the inbox. The bulk folder exists for mail that’s not bad enough to block, but isn’t good enough to go to the inbox.
Once we get to that model, we can think of filters as just different tolerances for what is bad and good. Using the same model, we can see aggressive filters block more mail and send more mail to bulk, while letting less into the inbox. There are also permissive filters that block very little mail and send most mail to the inbox.
That’s a somewhat useful model, but it doesn’t really capture the full complexity of filters. There isn’t just good mail and bad mail. Mail isn’t simply solicited or unsolicited. Filters take into account any number of factors before deciding what to do with mail.

Tempo

When we say that you might just be sending too much email and fatiguing or annoying the recipient into unsubscribing or hitting spam, this is the sort of thing we mean.
Three emails (to the same email address) in four minutes might be a bit much.

If you can’t combine the content you want to send into a single personalized email, maybe spread deliveries out a bit? Or even not send all of it, perhaps.

Filters evolving

I started writing this blog post while sitting on a conference call with a bunch of senders discussing some industry wide problems folks are having with delivery. Of course the issue of Microsoft comes up. A lot of senders are struggling with reaching the inbox there and no one has any real, clear guidance on how to resolve it. And the MS employees who regularly answer questions and help folks have been quiet during this time.

In some ways the current situation with Microsoft reminds me of what most deliverability was like a decade ago. Receivers were consistently making changes and they weren’t interacting with senders. There weren’t FBLs really. There weren’t postmaster pages. The reason knowing someone at an ISP was so important was because there was no other way to get information about blocking.
These days, we have a lot more institutional knowledge in the industry. The ISPs realized it was better to invest in infrastructure so senders could resolve issues without having to know the right person. Thus we ended up with postmaster pages and a proliferation of FBLs and best practices and collaboration between senders and receivers and the whole industry benefited.
It is challenging to attempt to troubleshoot deliverability without the benefit of having a contact inside ISPs. But it is absolutely possible. Many ISP folks have moved on over the years; in many cases due to layoffs or having their positions eliminated. The result is ISPs where there often isn’t anyone to talk to about filters.
The lack of contacts doesn’t mean there’s no one there and working. For instance, in the conference call one person asked if we thought Microsoft was going to fix their systems or if this is the new normal. I think both things are actually true. I think Microsoft is discovering all sorts of interesting things about their mail system code now that it’s under full load. I think they’re addressing issues as they come up and as fast as they can. I also think this is some level of a new normal. These are modern filters that implement the lessons learned over the past 20 years of spam filtering without the corresponding cruft.
Overall, I do think we’re in a period of accelerating filter evolution. Address filtering problems has always been a moving target, but we’ve usually been building on known information. Now, we’re kinda starting over. I don’t have a crystal ball and I don’t know exactly what the future will bring. But I think the world of deliverability is going to get challenging again.

Interacting in professional fora

There are a bunch of online communities – mailing lists, Slack channels, etc. – where “people who do email” interact.
Some of them are open to anyone to subscribe, some of them are semi-private and require an invitation, others are closed and only available by invitation and yet others are associated with trade associations and only open to their members.
Many of them include representatives from ISPs, ESPs, reputation providers and technical specialists. They also – especially the open lists – have participants with no particular role in the industry, but very strong opinions on what others should do.
They’re a useful place to keep up to date on current issues and industry trends, and to get help when you need it. But … quite a lot of people reduce their chance of getting timely help by the way they behave there. Don’t be like those people.
Some of the things you should and shouldn’t do are specific to mailing lists. Some are specific to professional fora. Some are specific to entreating others for help. Here, in no particular order, are some suggestions:

DO: Be friendly. Be patient. Be welcoming. Be considerate. Be respectful.
DO: Be careful in the words that you choose.
DON’T: Be a dick.

DON’T: Be wildly unprofessional. If you think sexist or racist behaviour isn’t wildly unprofessional, leave the email industry. Ditto for unwanted sexual attention, personal insults, sexualized language or imagery.
DON’T: Harass people. If someone wants you to stop, then stop.

DO: Follow the community norms. Different communities have different styles and traditions – try and pick up on what they are, and avoid violating them.
DO: Follow the community norms for replying to messages, quoting them and trimming threads. If you’re not sure what they are then snipping out parts that aren’t relevant and replying in-line isn’t likely to offend anyone.
DO: Follow the level of formality of the community. Some are very formal, and should be treated much the same as a business meeting. Others much less so, and blend professional discussion with blowing off steam, ranting about idiot clients and social banter between friends.
DO: Lurk on the list for a day or three before posting to get a feel for how the community works (unless there’s a “welcome to the new person” thread). If you’ve joined because you have an immediate emergency you’re looking for help on, say so and be polite – maybe even a little apologetic – about it. Maybe spend five minutes checking the list archives first.
DON’T: Lurk except when you have a problem. Interacting with others when you’re not asking for help builds up relationships and karma. If you only appear when you’re looking for help, people are less likely to be helpful.

DO: Be clear about what company or organization, you’re affiliated with. That might mean using a corporate email address, mentioning it in a sig file or in a “Hi, I’ve just joined the group” message. Or it might mean including the relevant company name when asking for help. If, for political reasons, you absolutely cannot admit to your affiliations it’s still useful to know that you work for an unnamed major US cable company or an email provider based in Switzerland – particularly when you’re offering help or advice where your insight is coming from your experience in that role.
DO: Remember that the vast majority of the people you’re interacting with aren’t being paid to be there. They’re sharing their time and expertise in return for benefiting from others. Try to both give and take.
DO: Remember that a representative from a large ISP probably doesn’t have answering your questions or helping with your problem in their job description.

DON’T: Aggressively demand help. Nobody owes you anything.

DO: Read responses carefully. Someone may not be able to publicly join the dots on an issue for you, but may point out which dots you might want to look at.
DO: Understand limits. If someone says “our lawyers say this is the process you must follow” then follow that process. And don’t push that person to do things that their lawyers say they can’t do.
DO: Be aware that you’re interacting with people, not company representatives. They almost certainly have opinions that don’t reflect those of their organizations.
DO: Remember that nobody owes you support. Be nice. And if someone doesn’t volunteer help or stops responding, don’t badger them.

DO: Follow the community style for how you present your message. But … in general, mostly plain text won’t offend anyone, heavy use of rich text will annoy some people.
DON’T: Rely on rich text for meaning. It may not be visible to some people or not visible when quoted. “Look at the log lines highlighted in yellow” isn’t a good approach.
DON’T: Warlord. There’s no need for long legal disclaimers on your mail. Nor for more than four lines of signature – we don’t need to know your life history. Graphics are cheesy, even if they’re your employers professionally drawn logo. Even colour can be distracting if it’s not used carefully.

DON’T: Assume that you’re the best representative of your organization to interact with a community. If you’re a senior manager and you have a smart employee who is actively working in the area – they may be a better rep than you are.

DO: Be aware of how public a community is. Does it have a public archive that’s indexed by Google? Is it open subscription? Be aware of how public things you say are.
DO: Be aware of what is expected from you in terms of information distribution. Can things you learn from the community be shared elsewhere? With attribution, or not? If you’re not sure, don’t share information unless the person providing it OKs that – it’s always OK to ask if you’re not sure. Terms you might see are Traffic Light Protocol or Chatham House Rule.

DO: Assume good faith.

DO: Provide relevant information when looking for help or asking “has anyone else seen this?”.
DO: Check unread mail to a list before posting. If someone else is already talking about an issue, join that thread rather than starting your own.
DO: Check the archives first, if you can. The answer to your problem might be in there. And if it’s not, including a mention of “this looks similar to what Yahoo was doing in October” signals that you’ve done a little work before asking for help and might trigger someone’s memory of what happened last time.
DO: Include relevant IP addresses and hostnames, if you’re asking about a delivery issue.
DO: Include exact error or rejection messages – “blocked at AOL” isn’t particularly useful, “554 RLY:B1” is much more so.
DO: Mention what sort of email it is, especially if you think the problems may be content related.
DON’T: Obfuscate.
DO: If you’re asking about a problem, say how long it’s been going on and what you’ve already tried to fix it.
DO: Respond promptly if someone asks for more details.
DON’T: Expect help if you’re not prepared to share data.
DON’T: Vanish once you resolve the problem. Share what you did, even if it’s just “it cleared up around 3pm”.

DO: Be prepared to take conversations that only you and one other person, out of hundreds, are interested in to direct message or private email.

DO: Stick around and help others. Share what you know.
DON’T: Post off-topic stuff people aren’t going to be interested in. It’s great that your kid is selling girl scout cookies or you’re doing a charity 5k, but unless you’re absolutely sure that this is a good place to fundraise, it almost certainly isn’t.
DO: Keep conversation on a mailing list, on the mailing list. There’s no need to Cc everyone involved – they’re on the mailing list too.

DON’T: Email angry. If someone has made you mad, wait before responding.

Improving Gmail Delivery

Lately I’m hearing a lot of people talk about delivery problems at Gmail. I’ve written quite a bit about Gmail (Another way Gmail is different, Gmail filtering in a nutshell, Poor delivery at Gmail but no where else, Insight into Gmail filtering) over the last year and a half or so. But those articles all focus on different parts of Gmail delivery and it’s probably time for a summary type post.

5 steps for addressing deliverability issues

Following on from my reading between the lines post I want to talk a little bit about using the channels. From my perspective the right way to deal with 99% of issues is through the front door.
Last week I found myself talking to multiple folks in multiple fora (emailgeeks slack channel, mailop, IRC) about how to resolve blocking issues or questions. All too often, folks come into these spaces and start by asking “does anyone know someone at…” Fundamentally, that’s the wrong first question. Even if the answer is yes. It’s even the wrong question if a representative of the company is on the list where you’re asking for help.
If that’s the wrong question, what is the right question? Where can we start to get help with issues when we’re stuck trying to fix a delivery problem we don’t understand?

Filtering by gestalt

One of those $5.00 words I learned in the lab was gestalt. We were studying fetal alcohol syndrome (FAS) and, at the time, there were no consistent measurements or numbers that would drive a diagnosis of FAS. Diagnosis was by gestalt – that is by the patient looking like someone who had FAS.
It’s a funny word to say, it’s a funny word to hear. But it’s a useful term to describe the future of spam filtering. And I think we need to get used to thinking about filtering acting on more than just the individual parts of an email.

Filtering is not just IP reputation or domain reputation. It’s about the whole message. It’s mail from this IP with this authentication containing these URLs. Earlier this year, I wrote an article about Gmail filtering. The quote demonstrates the sum of the parts, but I didn’t really call it out at the time.

Permission trumps good metrics

Most companies and senders will tell you they follow all the best practices. My experience says they follow the easy best practices. They’ll comply with technical best practices, they’ll tick all the boxes for content and formatting, they’ll make a nod to permission. Then they’re surprised that their mail delivery isn’t great.

Filters do what we tell them

In the email space we talk about filters as if they were sentient beings. “The filters decided…” “The filters said…” This is convenient shorthand, but tends to mask that filters aren’t actually deciding or saying anything. Filters are software processes that follow rules dictated by the people who create and maintain them. The rules flow from the goals set by the mailbox provider. The mailbox provider sets goals based on what their users tell them. Users communicate what they want by how they interact with email.

What we end up with is a model where a set of people make decisions about what mail should be let in. They pass that decision on to the people who write the filters. The people who write the filters create software that evaluates email based on those goals using information collected from many places, including the endusers.
What mail should be let in is an interesting question, with answers that differ depending on the environment the filter is deployed in.
Consumer ISPs typically want to keep their users happy and safe. Their goals are to stop harmful mail like phishing, or mail containing viruses or malware. They also want to deliver mail that makes their users happy. As one ISP employee put it, “We want our users to be delighted with your mail.”
Businesses have a few other goals when it comes to filters. They, too, need filters to protect their network from malicious actors. As businesses are often directly targeted by bad actors, this is even more important. They also want to get business related email, whether that be from customers or vendors. They may want to ensure that certain records are kept and laws are followed.
Governments have another set of goals. Universities and schools have yet another set of goals. And, of course, there are folks who run their own systems for their own use.
Complicating the whole thing is that some groups have different tolerances for mistakes. For instance, many of our customers are folks dealing with being blocked by commercial filters. Therefore, we don’t run commercial filters. That does mean we see a lot of viruses and malware and rely on other strategies to stop a compromise, strategies that wouldn’t be as viable in a different environment.
Filters are built to meet specific user needs. What they do isn’t random, it’s not unknowable. They are designed to accomplished certain goals and generally they’re pretty good at what they do. Understanding the underlying goals of filters can help drive solutions to poor delivery.
Use the shorthand, talk about what filters are doing. But remember that there are people behind the filters. Those filters are constantly maintained in order to keep up with ever changing mail streams. They aren’t static and they aren’t forgotten. They are updated regularly. They are fluid, just like the mail they act on.

The perfect email

More and more I’m moving away from consulting on technical setup issues as the solution to delivery problems. Delivery is not about the technical perfection of a message. Spammers get the technical right all the time. No, instead, delivery is about sending messages the user wants. While looking for something on the blog I found an old post from 2011 that’s still relevant today. In fact, I’d say it’s even more relevant today than it was when I wrote it 5 years ago.

Email is a fluid and ever changing landscape of things to do and not do.
Over the years my clients have frequently asked me to look at their technical setup and make sure that how they send mail complies with best practices. Previously, this was a good way to improve delivery. Spamware was pretty sloppy and blocking for somewhat minor technical problems was a great way to block a lot of spam.
More recently filter maintainers have been able to look at more than simple technical issues. They can identify how a recipient interacts with the mail. They can look at broad patterns, including scanning the webpages an email links to.
In short, email filters are very sophisticated and really do measure “wanted” versus “unwanted” down to the individual subscriber levels.
I will happily do technology audits for clients. But getting the technology right isn’t sufficient to get good delivery. What you really need to consider is: am I sending email that the recipient wants? You can absolutely get away with sloppy technology and have great inbox delivery as long as you are actually sending mail your recipients want to receive.
The perfect email is no longer measured in how perfectly correct the technology is. The perfect email is now measured by how perfect it is for the recipient.

Open subscription forms going away?

A few weeks ago, I got a call from a potential client. He was all angry and yelling because his ESP had kicked him off for spamming. “Only one person complained!! Do you know him? His name is Name. And I have signup data for him! He opted in! How can they kick me off for one complaint where I have opt-in data? Now they’re talking Spamhaus listings, Spamhaus can’t list me! I have opt-in data and IP addresses and everything.”
We talked briefly but decided that my involvement in this was not beneficial to either party. Not only do I know the complainant personally, I’ve also consulted with the ESP in question specifically to help them sort out their Spamhaus listings. I also know that if you run an open subscription form you are at risk for being a conduit for abuse.
This abuse is generally low level. A person might sign up someone else’s address in an effort to harass them. This is a problem for the victim, but doesn’t often result in any consequences for the sender. Last week’s SBL listings were a response to subscription abuse happening on a large scale.

Ask Laura: What should we be measuring?

Who owns the inbox

One of the questions asked of my panel during Connections 16 last month was who owns the inbox.

The source of deliverability problems

Most deliverability problems don’t start where many people think they do. So very often people call looking for deliverability help and tell me all about the things they’re doing to reach the inbox. They’ll tell me about content, they’ll tell me about bounces, they’ll talk about complaints, engagement, opens and clicks. Rarely will they bring up their list source without some prompting on my part.

The reality is, though, that list source is to root of deliverability success and deliverability problems. Where did those addresses come from and what do the people who gave them think you’re going to do with them?
Outsourcing collection to a third party can cause significant issues with delivery. Letting other people collect addresses on your behalf means you lack control over the process. And if you’re paying per address, then there monetary incentive for that company to pad the list with bogus addresses.
Sometimes there are even issues with having your own employees collect addresses from customers. For instance, a retailer requires sales associates collect a minimum percentage of addresses from customers. The company even ties the associates’ evaluations to that percentage. Associates have an incentive to submit addresses from other customers. Or a retailer will offer a discount for an address and customers want the discount but not the mail, so they give a fake address.
All of these things can affect deliverability.
Address collection is the key to delivery, but too many companies just don’t put enough attention to how they’re collecting addresses and entering into the relationship with subscribers. This is OK for a while, and delivery of small lists collected like this can be great. But as lists grow in size, they come under greater scrutiny at the ISPs and what used to work doesn’t anymore.
The first step to diagnosing any delivery problem is to look at the list. All of the things ISP use to measure reputation measure how well you’re collecting addresses. Changing IPs or domains or content doesn’t change the reason mail is being filtered. It just means the filters have to figure out something new to key on.
Want great deliverability? Start with how you’re collecting addresses.
Want to fix deliverability? Start with how you’ve collected addresses, how you’ve stored them and how you’ve maintained them.

Dueling data

One of the things I miss about being in science is the regular discussions (sometimes heated) about data and experimental results. To be fair, I get some of that when talking about email stuff with Steve. We each have some strong view points and aren’t afraid to share them with each other and with other people. In fact, one of the things we hear most when meeting folks for the first time is, “I love it when you two disagree with each other on that mailing list!” Both of us have engineering and science backgrounds, so we can argue in that vein.

One of the challenges of seemingly contradictory data is figuring out why it seems to disagree. Of course, in science the first step is always to look at your experimental design and data collection. Did I do the experiment right? (Do it again. Always do it again.) Did I record the data correctly? Is the design right? So what did I do differently from what you did? For instance, at one of my labs we discovered that mixing a reagent in plastic tubes created a different outcome from mixing the reagent in glass vials. So many variables that you don’t even think of being variables that affect the outcome of an experiment.

The 10 worst …

Spamhaus gave a bunch of us a preview of their new “Top 10 worst” (or should that be bottom 10?) lists at M³AAWG. These lists have now been released to the public.

The categories they’re measuring are:

Ask Laura

An Advice Column on Email Delivery
When we work with brands and senders to improve email delivery, there are many questions that come up again and again. For 2016, we thought it might be interesting to answer some of those questions here on the blog so others can benefit from the information.
Confused about delivery in general? Trying to keep up on changing policies and terminology? Need some Email 101 basics? This is the place to ask. We can’t answer specific questions about your server configuration or look at your message structure for the column (please get in touch if you’d like our help with more technical or forensic investigations!), but we’d love to answer your questions about how email works, trends in the industry, or the joys and challenges of cohabiting with felines.
Your pal,
Laura

Dear Laura,
I’m having a hard time explaining to our marketing team why we shouldn’t send email to addresses on our lists with very low read rates, that are dormant but not bouncing, or that spend less than 2 seconds reading our mail. I’m also struggling to convince them that it’s not a good idea to dramatically increase email volume during the holidays (i.e. going from one send/day to 2-3 sends/day).
We already segment based on recency, engagement, and purchase behavior, and we also have some triggered messaging based on user behavior.
Can you help me find a way to help explain why sometimes less is more?
Thanks,
The Floodgates Are Open

Dear Floodgates,
ISPs ask two fundamental questions about email when it comes in:

Is it safe?
Is it wanted?

If the answer to both those questions is yes, the mail is delivered to the inbox.

Troubleshooting delivery is hard, but doable

Even for those of us who’ve been around for a while, and who have a lot of experience troubleshooting delivery problems things are getting harder. It used to be we could identify some thing about an email and if that thing was removed then the email would get to the inbox. Often this was a domain or a URL in the message that was triggering bulk foldering.
Filters aren’t so simple now. And we can’t just randomly send a list of URLs to a test account and discover which URL is causing the problem. Sure, one of the URLs could be the issue, but that’s typically in context with other things. It’s rare that I can identify the bad URLs sending mail through my own server these days.
There are also a lot more “hey, help” questions on some of the deliverability mailing lists. Most of these questions are sticky problems that don’t map well onto IP or domain reputation.
One of my long term clients recently had a bad mail that caused some warnings at Gmail.
We tried a couple of different things to try and isolate the problem, but never could discover what was triggering the warnings. Even more importantly, we weren’t getting the same results for identical tests done hours apart. After about 3 days, all the warnings went away and all their mail was back in the inbox.
It seemed that one mailing was really bad and resulted in a bad reputation, temporarily. But as the client fixed the problem and kept mailing their reputation recovered.
Deliverability troubleshooting is complicated and this flowchart sums up what it’s like.

Here at Word to the Wise, we get a lot of clients who have gone through the troubleshooting available through their ESPs and sometimes even other deliverability consultants. We get the tough cases that aren’t easy to figure out.
What we do is start from the beginning. First thing is to confirm that there aren’t technical problems, and generally we’ll find some minor problems that should be fixed, but aren’t enough to cause delivery problems. Then we look at the client’s data. How do they collect it? How do they maintain it? What are they doing that allows false addresses on their list?
Once we have a feel for their data processes, we move on to how do we fix those processes. What can we do to collect better, cleaner data in the future? How can we improve their processes so all their recipients tell the ISP that this is wanted mail?
The challenging part is what to do with existing data, but we work with clients individually to make sure that bad addresses are expunged and good addresses are kept.
Our solutions aren’t simple. They’re not easy. But for clients who listen to us and implement our recommendations it’s worth it. Their mail gets into the inbox and deliverability becomes a solved problem.

CASL botnet take down

biohazardmail The CRTC served its first ever warrant as part of an international botnet takedown. The warrant was to take down a C&C (command and control) server for Win32/Dorkbot. International efforts to take down C&C servers take a lot of effort and work and coordination. I’ve only ever heard stories from folks involved but the scale and work that goes into these take downs is amazing.
Bots are still a problem. Even if we manage to block 99% of the botnet mail out there people are still getting infected. Those infections spread and many of the newer bots steal passwords, banking credentials and other confidential information.
This kind of crime is hard to stop, though, because the internet makes it so easy to live in one country, have a business in a third, have a shell corp in a fourth, and have victims in none of those places. Law enforcement across the globe has had to work together and develop new protocols and new processes to make these kinds of takedowns work.

Tell me about your business model

I posted Friday about how most deliverability folks roll their eyes when a sender starts talking about their business model.
The irony is that one of the first things I do with a client is ask them to tell me about their business model and how email fits into their business plan. Once I know that, I can help them improve their email sending to meet the requirements of ISPs, blocklists and recipients.
While most deliverability people don’t care about your business model, for me it’s essential that I understand it. I want to hear about it, all the details. Tell me about what you’re doing and together we’ll craft a strategy to make email work for you in your unique situation.
We have one goal for every client: their email gets to the inbox. But no two clients have the same problems so we tailor our advice specifically for their unique situation. We don’t have a 3-ring binder that we read a standard answer from when clients ask for recommendations for their email strategy. We use our own knowledge of email and our history in the industry to craft unique solutions to deliverability problems.
Your business model is disruptive? Great! We can help you get those disruptive emails into their inbox.
You have a niche social platform that uses email as part of your growth strategy? We’ll make sure users and future users see your email in their inboxes.
You have a SaaS platform and you want customers to be able to use email to communicate with their customers? We’ll help you craft the right policy for your business.
You’re a retail company and struggle to reach the inbox consistently? We’ve helped dozens of companies navigate email challenges. We’ve helped clients figure out how to effectively capture addresses at point of sale in brick and mortars. We’ve helped clients restructure their entire data flow.
We can help you too.
You bring us your business model and we’ll create a comprehensive strategy that gets your email into the inbox. What’s more, we’ll help you understand what factors relate to inbox delivery and train you how to handle most issues on your own. Once we’ve got you set up, a process that takes 3 – 6 months, you have everything you need to run an email program. Even better, when those rare, complicated issues come up we’ve got your back and can get your emails delivering to the inbox again.

SPF debugging

Someone mentioned on a mailing list that mail “from” intuit.com was being filed in the gmail spam folder, with the warning “Our systems couldn’t verify that this message was really sent by intuit.com“. That warning means that Gmail thinks it may be phishing mail. Given they’re a well-known financial services organization, I’m sure there is a lot of phishing mail claiming to be from them.
But I’d expect that a company the size of Intuit would be authenticating their mail, and that Gmail should be able to use that authentication to know that the mail wasn’t a phish.
Clearly something is broken somewhere. Lets take a look.
Looking at the headers, the mail was being sent from Salesforce, and (despite Salesforce offering DKIM) it wasn’t DKIM signed by anyone. So … look at SPF.
SPF passes:

We have multiple measures of deliverability. Ones that we don’t even let in the door, and then we have ones that customers indicated that they don’t want to be delivered.
Read More

IPv6 and authentication

I just saw a post over on the mailop mailing list where someone had been bitten by some of the IPv6 email issues I discussed a couple of months ago.
They have dual-stack smarthosts – meaning that their smarthosts have both IPv4 and IPv6 addresses, and will choose one or the other to send mail over. Some domains they send to use Office 365 and opted-in to receiving mail over IPv6, so their smarthosts decided to send that mail preferentially over IPv6.
The mail wasn’t authenticated, so it started bouncing. This is probably going to happen more and more over the next year or so as domain owners increasingly accept mail over IPv6.
If your smarthosts are dual stack, make sure that your workflow authenticates all the mail you send to avoid this sort of delivery issue.
One mistake I’ve seen several companies make is to have solid SPF authentication for all the domains they send – but not for their IPv6 address space. Check that all your SPF records include your IPv6 ranges. While you’re doing that keep in mind that having too many DNS records for SPF can cause problems, and try not too bloat the SPF records you have your customers include.

Outrunning the Bear

bear
You’ve started to notice that your campaigns aren’t working as well as they used to. Your metrics suggest fewer people are clicking through, perhaps because more of your mail is ending up in junk folders. Maybe your outbound queues are bigger than they used to be.
You’ve not changed anything – you’re doing what’s worked well for years – and it’s not like you’ve suddenly had an influx of spamming customers (or, if you have, you’ve dealt with them much the same as you have in the past).
So what changed?
Everything else did. The email ecosystem is in a perpetual state of change.
There’s not a bright line that says “email must be this good to be delivered“.
ride Instead, most email filtering practice is based on trying to identify mail that users want, or don’t want, and delivering based on that. There’s some easy stuff – mail that can be easily identified as unwanted (malware, phishing, botnet spew) and mail that can easily be identified as wanted (SPF/DKIM authenticated mail from senders with clean content and a consistent history of sending mail that customers interact with and never mark as spam).
The hard bit is the greyer mail in the middle. Quite a lot of it may be wanted, but not easily identified as wanted mail. And a lot of it isn’t wanted, but not easily identified as spam. That’s where postmasters, filter vendors and reputation providers spend a lot of their effort on mitigation, monitoring recipient response to that mail and adapting their mail filtering to improve it.
Postmasters, and other filter operators, don’t really care about your political views or the products you’re trying to sell, nor do they make moral judgements about your legal content (some of the earliest adopters of best practices have been in the gambling and pornography space…). What they care about is making their recipients happy, making the best predictions they can about each incoming mail, based on the information they have. And one of the the most efficient ways to do that is to look at the grey area to see what mail is at the back of the pack, the least wanted, and focusing on blocking “mail like that”.
If you’re sending mail in that grey area – and as an ESP you probably are – you want to stay near the front or at least the middle of the grey area mailers, and definitely out of that “least wanted” back of the pack. Even if your mail isn’t great, competitors who are sending worse mail than you will probably feel more filtering pain and feel it sooner.
Some of those competitors are updating their practices for 2015, buying in to authentication, responding rapidly to complaints and feedback loop data, and preemptively terminating spammy customers – and by doing so they’re both sending mail that recipients want and making it easy for ISPs (and their postmasters and their machine learning systems) to recognize that they’re doing that.
Other competitors aren’t following this years best practices, have been lazy about providing customer-specific authentication, are letting new customers send spam with little oversight, and aren’t monitoring feedback and delivery to make sure they’re a good mail stream. They end up in the spam folder, their good customers migrate elsewhere because of “delivery issues” and bad actors move to them because they have a reputation for “not being picky about acquisition practices“. They risk spiraling into wholesale bulk foldering and becoming just a “bulletproof spam-friendly ESP”.
If you’re not improving your practices you’re probably being passed by your competitors who are, and you risk falling behind to the back of the pack.
And your competitors don’t need to outrun the bear, they just need to outrun you.

The holiday mailing season

We’re half way through September and it seems way too early to start thinking about the holidays. But for marketers, even email marketers, planning should be starting now. This planning shouldn’t just be about content and targeting and segmentation, but should also cover deliverability.
Most retailers use email marketing to drive traffic to their websites during the holidays. Experian reported that in 2014 email was the second largest driver of traffic, behind search, to the Hitwise Retail 500. In recent years, though, some retailers have run afoul of filters during the holiday season, losing precious opportunities to reach potential buyers due to delivery problems.
Retailers should consider deliverability as a factor in their marketing strategy.
Choices about who, how, how much and when to email can and do significantly affect marketing. The good news is that smart marketers can use their understanding of filters as part of their strategic planning and avoid some of the bigger problems that have plagued retailers in the past.
In December 2012, retailers Gap and Gilt were listed on the Spamhaus Block List. Since then, other retailers have also had delivery and blocking problems during the holiday season, although none have been quite so public.
Delivery problems can have a significant impact on a retailer’s bottom line. Mark Zadon, the chairman of Zulily, blamed his company’s lower profits in Q3 2014 on changes at their unspecified email service provider. After that announcement, Zulily’s stock value dropped 15%. Zulily isn’t the only company to have email delivery problems affect business growth enough to be mentioned in SEC filings. “Various private spam blacklists have in the past reduced, and may in the future reduce, the effectiveness of our solutions and our ability to conduct our business, which may cause demand for our solutions to decline.”
Deliverability rules don’t change.
Some people argue that the increase in blocking during the holiday season is because the folks running the filters are attempting to sabotage retail marketing. The available evidence doesn’t support this conclusion. For webmail providers and consumer ISPs, the overarching rule for filters is to give users email they want and filter email users don’t want. The processes and techniques the ISPs and filter companies use don’t change during the holidays. A few years ago Return Path interviewed people at a number of providers and all agreed that the receivers don’t change during the holidays.
It is true that during the holiday season some retailers see an increase in delivery problems. These are mostly self-inflicted. The good news is that given the changes are happening at the sending end, there are things senders can do to minimize the impact of filters. It’s all in their control.
Mail volume increases for multiple reasons.
The volume of transactional email goes up because brick-and-mortar retailers collect addresses in the store and email receipts to shoppers. This often involves the shopper spelling out the address for a harried sales associate in the middle of a store blasting holiday music. Typos can, and do, happen. Even when shopping online, from the comfort of the couch, there is a risk of a mis-typed email address.
These typos hurt deliverability a few different ways. The receipt can go to the wrong person, causing a complaint and hurting the reputation of the sender. The receipt can go to a non-existent account, causing a bounce and hurting the reputation of the sender. Both of these things happen, and can hurt delivery if they happen in significant enough numbers. Of even more concern is when a receipt goes to a spamtrap. Enough trap hits or complaints and the sender risks blocking and delivery failures at one or more ISPs.
Many of the larger brick-and-mortar retailers have implemented processes to reduce the chance of bad addresses. Some ask the shopper to input their email address right into the credit card pad. Others show the address to the user on the register and have the user confirm it. These things do help lower the risk of problems and incorrect addresses. But they don’t resolve it completely. Verification services can weed out undeliverable addresses, but can’t really do anything to make sure a deliverable address is the right one.
Transactional email isn’t the only reason volume increases during the holiday season. The volume of marketing email goes up as well. Marketers increase their frequency, sometimes to ridiculous amounts. A few years ago, I was on a list for a cooking store. They increased their volume from 2x a week to 3x a day in the 3 weeks leading up to Thanksgiving. This may make perfect sense from their point of view, but some recipients just don’t want that much email.
In addition to increasing volume to current and engaged customers, retailers often look to older, unengaged lists during the holidays. This has a double negative effect. First, addresses that have gone dormant, whether they bounce or not, can drive reputation down. Second, sending to people after a long period of no email can result in increased complaint rates. Increased complaints, increased bounces, and increased email to abandoned addresses all drive reputation down.
Taken together it’s no wonder some retailers see an increase in deliverability problems during the holiday emailing season. The good news is that mailers have the ability to control and manage their deliverability, even as they manage the holiday volume.

Reputation is about behavior

Reputation is calculated based on actions. Send mail people want and like and interact with and get a good reputation. Send mail people don’t want and don’t like and don’t interact with and get a bad reputation.

Reputation is not
… about who the sender is.
… about legitimacy.
… about speech.
… about message.
Reputation is
… about sender behavior.
… about recipient behavior.
… about how wanted a particular mail is forecast to be.
… based on facts.
Reputation isn’t really that complicated, but there are a lot of different beliefs about reputation that seem to make it complicated.
The reputation of a sender can be different at different receivers.
Senders sometimes target domains differently. That means one receiver may see acceptable behavior but another receiver may see a completely different behavior.
Receivers sometimes have different standards. These include standards for what bad behavior is and how it is measured. They may also have different thresholds for things like complaints and bounces.
What this means is that delivery at one receiver has no impact on delivery at another. Just because ISP A delivers a particular mail to the inbox doesn’t mean that ISP B will accept the same mail. Each receiver has their own standards and sometimes senders need to tune mail for a specific receiver. One of my clients, for instance, tunes engagement filters based on the webmail domain in the email address. Webmail domain A needs a different level of engagement than webmail domain B.
Public reputation measures are based on data feeds.
There are multiple public sources where senders can check their reputation. Most of these sources depend on data feeds from receiver partners. Sometimes they curate and maintain their own data sources, often in the form of spamtrap feeds. But these public sources are only as good as their data analysis. Sometimes, they can show a good reputation where there isn’t one, or a bad reputation where there isn’t one.
Email reputation is composed of lots of different reputations.
Email reputation determines delivery. Getting to the inbox doesn’t mean sending from an IP with a good reputation. IP reputation is combined with domain reputation and content reputation to get the email reputation. IP reputation is often treated as the only valuable reputation because of the prevalence of IP based blocking. But there are SMTP level blocks against domains as well, often for phishing or virus links. Good IP reputation is necessary but not sufficient for good email delivery.
Reputation is about what a sender does, not about who a sender is.
Just because a company is a household name doesn’t mean their practices are good enough to make it to the inbox. Email is a meritocracy. Send mail that merits the inbox and it will get to recipients. Send email that doesn’t, and suffer the repercussions.

Deliverability strategy to reach the inbox

I wrote a piece for the Only Influencers special Holiday Preparations edition about deliverability and the holiday email rush. One thing I like about the chance to write for other publications is the process often leads me down thought pathways and generate some new ideas.

Utilizing all of your data

Email marketing continues to be a great way to reach out to prospects and customers and many companies utilize multiple mail streams. Companies often have the following systems sending mail:

Continuous Testing

HubSpot recently posted an blog article comparing which was better for engagement, plain text emails or HTML emails. In a survey they sent out in 2014, 64% of the responses said they preferred the HTML and image-based emails. It seems pretty straight forward, recipients say they want HTML emails over text based emails but through their A/B testing, the text versions had a higher open rate.
They also reported:

Give Recipients Options

A few years ago I subscribed to a financial website that emails out articles about investing as well as a recap of your investments. For the first few months I enjoyed reading these emails but as time went on, I found them less valuable and receiving them every other day they turned into a burden to clean up and deal with.
My options were to either unsubscribe or I could create a rule in Outlook to file away the emails to possibly read them later.
options What I would really like is the option to define how often I would receive the updates. If I’m actively looking to change my investments, I would want to receive the emails daily. I would also like to have the option for either a weekly or monthly email.
The frequency of mailings should be tailored to the subscriber. Buying a new car? I may want to see emails and reviews daily. Just bought a new blender? I want to receive emails for the first few days learning about the different features and recipes. The idea is to present options to each subscriber on what they prefer. It’s better to treat subscribers as individuals rather than sending the same message to your entire list.
The newsletter I was receiving does not provide me with any type of control over how many times I receive the updates. The newsletter is also lacking a working unsubscribe link leaving me no alternative to clicking “this is junk”.
Senders should consider providing recipients with options:

Yes, Virginia, there is list churn

Yesterday I talked about how data collection, management, and maintenance play a crucial role in deliverability. I mentioned, briefly, the idea that bad data can accumulate on a list that isn’t well managed. Today I’d like to dig into that a little more and talk about the non-permanence of email addresses.
A common statistic used to describe list churn is that 30% of addresses become invalid in a year. This was research done by Return Path back in the early 2000’s. The actual research report is hard to find, but I found a couple articles and press releases discussing the info.

What We Do

Occasionally when we meet longtime readers of the blog at conferences and industry events, they are surprised to learn that we are not just bloggers. We actually spend most of our time consulting with companies and service providers to optimize their email delivery. Though we try to avoid using the blog as a WttW sales pitch, we thought it might be useful to devote a short post to explaining a bit more about what we do.

Bad SPF can hurt your reputation

Can a bad SPF record ruin your delivery, even though all your mail still passes SPF?
Yes, it can.
One of our clients had issues with poor delivery rates to the inbox at gmail and came to us with the theory that it was due to other people using their domain to send spam to gmail. This theory was based on ReturnPath instrumentation showing mail “from” their domain coming from other IP addresses, and a plausible looking correlation between that mail being sent and their problems at gmail.
Checking their bounce handler, we see a lot of bounces coming in suggesting that someone is sending poor quality mail using their bounce domain from quite a few IP addresses, including a suspicious number scattered in small blocks across 69.64.0.0/8.
Their question they had was whether they should publish DMARC records to fix the problem, and whether they should use a DMARC policy of p=reject or p=none. They’re a good candidate for DMARC – their domains are used purely for bulk or transactional mail, they have a tightly controlled mail infrastructure for their marketing domains, and they’re already publishing SPF records and signing all the mail they send with DKIM.
I was half way through writing up my normal answer about DMARC deployment for customers with this sort of mail infrastructure – “It won’t help with delivery problems directly, but publishing with p=none and analyzing the reports you get back will give you insight into your mail flows, and provide the data you need to decide whether using DMARC p=reject is appropriate for your business model and mail flows.” – when I realized that something just didn’t make sense.
Gmail, perhaps more than most other mailbox providers, base their delivery decisions on data they gather mechanically from all their mailboxes. And they really understand domain-based reputation and the difference between authenticated and non-authenticated email. Why on earth would non-authenticated email from an unrelated IP address be damaging the domain reputation, and hence the delivery of authenticated legitimate email? That makes no sense.
Meanwhile, over in our slack channel, Josh was double-checking their infrastructure…

slack
Oops. They have a small block of 8 IP addresses from which they send most of their email. When setting up their SPF records they inadvertently used ip4:69.20.20.48/8 instead of ip4:69.20.20.48/29 for that block of addresses. A /8 isn’t eight IP addresses – it’s every one of the 16,777,216 IP addresses that begins with “69.”.
Suddenly everything makes sense.
The SPF thinko means that all mail claiming to be from the client domain that’s sent from any IP address beginning with “69.” passes SPF – including the deluge of spam coming from the snowshoe spammers in 69.64.*.
Gmail (and other ISPs) don’t see a difference between the legitimate email and the SPF authenticated spam – they’re just seeing a high volume of authenticated email from the client domain, a large fraction of which is spam. That’s damaged the reputation of the client domain, causing their legitimate email to end up in the spam folder.
(The reality of filtering is more than just domain reputation, of course, but a terrible domain reputation is definitely going to cause you problems.)
The immediate action to take is simple – fix the SPF record so only legitimate mail will be authenticated. That’ll take effect within a couple of hours, as the old SPF record has a short TTL, and ISPs will start seeing the correct SPF record and begin rejigging their reputation.
We’ll keep monitoring delivery rates, check how long ISPs take to notice reputation changes, potentially reach out to some ISPs to see if it’s appropriate for them to do a one-time reputation reset for the affected domains, but we’re hoping things will begin to improve in the next few days.
What can you do to avoid or mitigate this sort of problem?

How to send better emails: engagement

Today Direct Marketing News hosted a webinar: ISP Mythbusters: How to Send Better Emails. The speakers were Matt Moleski, the Executive Director of Compliance Operations from Comcast and Autumn Tyr-Salvia, the Director Of Standards And Best Practices from Message Systems.
The webinar went through a series of myths. After Autumn introduced the myth, Matt commented on it and explained why the statement was, or was not, a myth. Throughout the webinar, Matt clearly explained what does, and does not, get mail delivered. Don’t let the Comcast after Matt’s name fool you. He is very active in different fora and discusses filtering strategies with experts across the ISP industry. His insight and knowledge is broadly applicable. In fact, many of the things Matt said today were things I’ve heard other ISPs say over and over again.
One of the very first things he said was that ISPs want to deliver mail their customers want. They want to give customers the best inbox experience possible and that means delivering mails customers want and keeping out mails customers don’t. He also pointed out that recipients complain to the ISPs when they lose wanted mail, perhaps even more than they complain about spam.
He also touched on the topic of engagement. His message was that absolutely engagement does matter for inbox delivery and that engagement is going to matter more and more as filtering continues to evolve. There has been some discussion recently about whether or not engagement is an issue, with some people claiming that some ISP representatives said engagement doesn’t matter. The reality is, that engagement does matter and Matt’s words today only reinforce and clarify that message.
Matt did say is that ISPs and senders have a bit of a disconnect when they are speaking about engagement. ISPs look at engagement on the “macro” level. They’re looking to see if users delete a mail without reading it, file it into a folder, mark it spam or mark it not spam. Senders and marketers look at engagement on a much more finite level and look at interactions with the specific emails and links in the email.
When discussing the relationship between senders and ISPs, he pointed out that both senders and ISPs have the same goal: to personalize the customer experience and to give customers a great experience. As part of this, ISPs are mostly aligned when it comes to blocking principles, but each ISP responds slightly differently. ISPs do adhere to best practices for handling incoming email, but those practices are implemented based on the individual company and handles incoming mail in ways that better supports their company specifically.
Matt talked about Comcast’s Postmaster pages and says they try to give feedback to senders before putting a block in place. He mentions that invalid recipients and poor list hygiene as the fastest way to be blocked or throttled when sending to Comcast. He also said that the core filtering rules at Comcast are static. Changes are mostly “tweaks around the edges.”
During the Q&A portion, Matt took a number of questions from the audience.

ISPs speak at the EEC conference

Massimo Arrigoni has a great blog post up summarizing the final session of the EEC conference with representatives from major mailbox providers. This session a number of representatives from major mailbox providers spoke about what it takes to get to the inbox. They discussed what engagement really was, why you need to warmup and what the mailbox providers are measuring.
The short version is delivery is becoming more and more personalized. It’s not about if a mailbox provider thinks mail is spam, it’s about if a mailbox provider thinks this recipient thinks the mail will be spam. It’s all about connecting with each individual recipient.
None of this should be news to any of our regular readers. We’ve long talked about how ISPs measure things differently than senders. We’ve also talked about personalized delivery and how IP reputation isn’t the most important part of delivery.
Know your recipients. Make them want your mail and it will end up in the inbox. ‘
Edit: EmailCopilot also has more information on the ISP session at EEC.

Spam is not a moral judgement

Mention an email is spam to some senders and watch them dance around trying to explain all the ways they aren’t spammers. At some point, calling an email spam seems to have gone from a statement of fact into some sort of moral judgement on the sender. But calling an email spam is not a moral judgement. It’s just a statement of what a particular recipient thinks of an email.
There are lots of reasons mail can be blocked and not all those reasons are spam related. Sometimes it’s a policy based rejection. Mailbox providers publishing a DMARC record with a reject policy caused a lot of mail to bounce, but none of that was because that user (or that mailing list) was sending spam. Most cable companies prohibit customers from running mail servers on their cable connection and mail from those companies is widely rejected, but that doesn’t mean the mail is spam.
Sometimes a block is because some of the mail is being sent to people who didn’t ask for it or are complaining about it. This doesn’t make the sender a bad person. It doesn’t make the sending company bad. It just means that there is some issue with a part of the marketing program that need to be addressed.
The biggest problem I see is some senders get so invested in convincing receivers, delivery experts and filtering companies that they’re not spammers, that they miss actually fixing the problem. They are so worried that someone might think they’re spammers, they don’t actually listen to what’s being said by the blocking organization, or by their ISP or by their ESP.
Calling email spam isn’t a moral judgement. But, if too many people call a particular email spam, it’s going to be challenging to get that mail to the inbox. Instead of arguing with those people, and the filters that listen to them, a better use of time and energy is fixing the reasons people aren’t liking your email.

More from Gmail

Campaign Monitor has an interview with Gmail looking at how to get mail to the Gmail inbox. It’s a great article and I think everyone should go read it.
One of the most important things it talks about is how complex filters are.

Is volume a problem?

Volume in an of itself is not a problem. Companies sending mail people want can send multiple emails a day to every user. The volume isn’t a problem because the mail is wanted.
Many senders are confused and think volume is a filtering criteria. It’s not. Send all you want; just send it to people who actually want the mail.
A lot of companies in their growth phase find they do have delivery problems as their volume ramps up. But the problem isn’t the volume, the problem is that mail programs don’t scale. Companies mailing lower volumes can get away with sloppier practices. One because the chances of hitting bad addresses increases with the number of addresses you have. But the other is that filters do take volume into account. It’s not that the volume directly causes the filters to trigger, but volume causes the filters to look harder at mail. If the reputation and metrics are good, the mail is fine and hits the inbox. If they are poor, then mail hits the bulk folder or is filtered.
Overall, volume isn’t a problem, but increasing volume can expose fundamental problems in a mail program that result in delivery issues.

What goes into successful email campaigns?

Campaign Monitor analyzed over 2.2 million campaigns and came up with some rules of thumb for effective email marketing.

Thoughts on "ISP relations"

I’ve been thinking a lot about the field of ISP relations and what it means and what it actually is. A few years ago the answer was pretty simple. ISP relations is about knowing the right people at ISPs in order to get blocks lifted.
The fact that ISPs had staff just to deal with senders was actually a side effect of their anti-spam efforts. In many places blocking was at least partially manual, so there had to be smart, technical, talented people to handle both the blocking and unblocking. That meant there were people to handle sender requests for unblocking.
Spam filters have gotten better and more sophisticated. Thus, the ISPs don’t need smart, technical, talented and expensive people in the loop. Most ISPs have greatly scaled back their postmaster desks and rely on software to handle much of the blocking.
Another issue is that some people on the sender side rely too heavily on the ISPs for their data. This makes the ISP reps, and even some spam filtering company reps, reluctant to provide to much help to senders. I’ve had at least 3 cases in the last 6 months where a sender contacted me to tell me they had spoken with someone at an ISP or filtering company and were told they would get no more help on a particular issue. In talking with those reps it was usually because they were drowning under sender requests and had to put some limits on senders.
All of this means ISP Relations is totally different today than it was 5 years ago. It’s no longer about knowing the exact right person to contact. Rather it’s about being able to identify problems without ISP help. Instead of being able to ask someone for information, ISP Relations specialists need to know how to find data from different sources and use that data to identify blocking problems. Sure, knowing the right person does help in some cases when there’s an obscure and unusual issue. But mostly it’s about putting together any available evidence and then creating a solution.
We still call it “ISP Relations” but at a lot of ISPs there is no one to contact these days. I think the term is a little misleading, but it seems to be what we’re stuck with.

Uploading your address book to social media

I am one of the moderators of a discussion list working on a document about getting off blocklists. If anyone not on the list attempts to post to the list I get a moderation request. One came through while I was gone.
Now, I don’t really think Jim Mills wants to be friends with a mailing list. I think he probably gave LinkedIn his email password and LinkedIn went through and scraped addresses out of his address book and sent invitations to all those addresses.
I don’t have any problem with connecting to people on social media. I do even understand that some people have no problem giving their passwords over to let social media sites plunder their address books and find connections. What I do have a problem with is social media sites that don’t do any pruning or editing of the scraped addresses before sending invitations.
In this case, the email address, like many mailing lists, has in the email address “mailman.” While it’s probably impossible to weed out every mailing list, support address and commercial sender, it doesn’t seem like it would be too difficult to run some minor word matching and filtering. It’s not even like those addresses have to be removed from invites. Instead they could be presented to the user for confirmation that these are real people and addresses.
Yes, it’s friction in the transaction and it costs money to do and do well. But those costs and friction are currently offloaded onto uninvolved third parties.

Auto-opt-in?

Bronto’s Chris Kolbenschlag frames the discussion well: He purchased from an online retailer, they assumed he wanted to receive followup emails, and thus, those emails did eventually commence.
This is something I’ve had a lot of experience with. Working for an e-commerce service provider from later 2000 through mid 2006, I was the guy setting permission policy, dealing with spam complaints and advising on deliverability issues, primarily regarding email lists built over time from online store purchasers. There was an opt-in checkbox on the platform’s checkout pages, and it was up to the client as to whether or not it was pre-checked (“opted-in”) by default. Most clients pre-checked it by default.
My experience was, from a deliverability perspective, this kind of auto opt-in didn’t really present issues. People didn’t tend to forge addresses when purchasing, and people tended not to report mail as spam when it’s coming from somebody they just did business with.
I’m not saying it’s the wisest way to do things, by any means. If you have any other deliverability challenges at all, this kind of thing could likely add to them. And is it the most consumer friendly way to run things? I don’t think so. In my humble opinion, it’s always better to wait for the consumer to sign up on their own. But I’m not one of those aggressive marketer types.
And of course, the laws governing email permission vary by locale.

AOL Updates Spam Filtering

Over on the AOL Postmaster blog, Lili Crowley announced yesterday that AOL has made changes to their spam filtering system. Specifically, more senders may be subject to blocking with CON:B1 errors. AOL’s website explains that CON:B1 errors indicate that an IP address is being blocked “due to a spike in unfavorable e-mail statistics.” This strongly suggests that a sender blocked with a CON:B1 error message has a negative sending reputation. This is yet another data point as to how ISPs have been tightening up spam filtering and reputation requirements over the past few years. What you might have been able to deliver five years ago, you might not be able to get delivered today.

Q3 Email intelligence report from Return Path

Return Path released their 3rd quarter email intelligence report this week. And the numbers aren’t looking that great for marketers.
Complaints are a major problem for commercial mailers. In the data Return Path examined, commercial mail made up 18% of the total inbox volume. That same mail accounted for 70% of all email complaints.
Additionally, 60% of the email sent to spamtraps was commercial email.
The combination of complaints and spamtrap hits mean 16% of commercial email doesn’t make it to the inbox.
And, as no surprise to anyone, Postini is the worst performing B2B filter out there. Folks behind postini filters only get 23% of the email they’re sent in their inbox. And 44% of that mail is just outright lost.
Ken Magill article.
DMNews article.

You can't technical yourself out of delivery problems

In many cases these days, many more cases than a lot of senders want to admit, delivery problems at the big ISPs are a result of sending mail recipients just don’t care about. The reason your mail is going to bulk? It’s not because you have minor problems in your headers. It’s not because you have some formatting issues. The reason is because your recipients just don’t care if the ISP delivers your mail or not.
A few years ago the bulk of my clients hired me to do technical audits for their mail. I fixed a lot of delivery problems that way. They’d send me their email and I’d run it through tools here and identify things they were doing that were likely to be causing problems. I’d give them some suggestions of things to change. Believe it or not, minor tweaks to headers and configuration actually did make a lot of difference in delivery.
Over time, though those tweaks less effective to fix delivery problems. Some of it is due to the MTA vendors, they’re a lot better at sending technically correct mail than they were before. There are also a lot more people giving good advice on the underlying structure and format of emails so senders can send technically clean email. I started seeing technically perfect emails from clients who were seeing major delivery problems.
There are a number of reasons that technical fixes don’t work like they used to. The short version, though, is that ISPs have dealt with much of the really blatant spam and they can focus more time and energy on the “grey mail”.
This makes my job a little harder. I can no longer just look at an email, maybe run it through some of our tools and provide a few suggestions that fix delivery problems. Delivery isn’t that simple any longer. Filters are really more focused on how the recipients react to mail. That means I need to know a lot more about a clients email program before I can even start to identify what might be causing the delivery issues.
I wish it were still so simple I could give minor technical tweaks that would appear to magically improve a client’s delivery. It was a lot simpler process then. But filters have evolved, and senders must evolve, too.

Broken record…

The Return Path In the Know blog listed 4 reasons mailing those old addresses is a bad idea.
Ashley, the author, is completely right and I endorse everything she said. (Although I’d really like to hear what happened to the customer that added back all those addresses. What was the effect on that campaign and future email marketing?) As I was reading the article though, I realized how many times this has been said and how depressing it is that we have to say it again. And again. And again.
A number of folks have told me that the reason they don’t pay any attention to delivery professionals is because we don’t provide enough real data. They can show that sending mail to old addresses costs them nothing, and makes them real money.
That’s not really true, though. We do provide data, they just don’t like it so they don’t listen to it. Return Path publishes lots of numbers showing that mailing unengaged recipients lowers overall delivery. I can provide case studies and data but companies that are committed to sending as much mail as possible throw up many reasons why our data isn’t good or valid.
The biggest argument is that they want hard numbers. I do understand this. Numbers are great. Direct and clear answers are wonderful. But delivery is a squishy science. There are a lot of inputs and a lot of modifiers and sometimes we can’t get exactly one answer. The data is noisy, and difficult to replicate. One of the reasons is that filtering is a moving target. Filters are not, and cannot be, fixed. They are adaptive and are changing even between one hour and the next.
Delivery experts are about risk management. They are the parents requiring everyone in the car wear seat belts, even though the driver has never had an accident. They are the fire department enforcing fire codes, even though it’s the rainy season.
Risk management isn’t about the idea that bad things will absolutely happen but rather that it is more likely that a bad thing will happen in some cases.
In this case, it’s more likely that delivery problems will happen when mailing old addresses. And if those addresses aren’t actively contributing to revenue, it’s hard to argue that their presence on a list is more beneficial than their absence.
But I repeat myself. Again.

Reputation is more complex than a single number

I checked our SenderScore earlier this month, as quite a few people mentioned that they’d seen SenderScore changes – likely due to changed algorithms and new data sources.

It sure looks like something changed. Our SenderScore was, for a while, zero out of a hundred. That’s as bad as it’s possible to get. I didn’t get a screenshot of the zero score, but I grabbed this a couple of days later:

Are ReturnPath wrong? No. Given what I know about the traffic from our server (very low traffic, particularly to major consumer domains, and a negligible amount of unavoidable backscatter due to our forwarding role addresses for a non-profit to final recipients on AOL) that’s not an unreasonable rating. And I’m fairly sure that as they get their new algorithms dialed in, and get more history, it’ll get closer. (Though I’m a bit surprised that less than 60 mails a day is considered a moderate volume.)
But all our mail is delivered fine. I’ve seen none of my mail bounce. It’s very rare someone mentions that our mail has ended up in a bulk folder. I’ve received the replies I’ve expected from all the mail I’ve sent. Recipient ISPs don’t seem to see any problems with our mail stream.
A low reputation number doesn’t mean you actually have a problem, it’s just one data point. And a metric that’s geared to model one particular sort of sender (very high-volume senders, for example) isn’t going to be quite as useful in modeling very different senders. You need to understand where a particular measure is coming from, and use it in combination with all the other information you have rather than focusing solely on one particular number.

Delivery reflects recipient desires

Ken has an article today about how Pro Flowers managed to get their mail out of the bulk folder at Gmail by asking their recipients for help.

The 500 mile email

This is a great story from Trey Harris about a real email delivery issue from the mid 1990s.
Here’s a problem that sounded impossible… I almost regret posting the story to a wide audience, because it makes a great tale over drinks at a conference. 🙂 The story is slightly altered in order to protect the guilty, elide over irrelevant and boring details, and generally make the whole thing more entertaining.
I was working in a job running the campus email system some years ago when I got a call from the chairman of the statistics department.
“We’re having a problem sending email out of the department.”
“What’s the problem?” I asked.
“We can’t send mail more than 500 miles,” the chairman explained.
I choked on my latte. “Come again?”
“We can’t send mail farther than 500 miles from here,” he repeated. “A little bit more, actually. Call it 520 miles. But no farther.”
“Um… Email really doesn’t work that way, generally,” I said, trying to keep panic out of my voice. One doesn’t display panic when speaking to a department chairman, even of a relatively impoverished department like statistics. “What makes you think you can’t send mail more than 500 miles?”
“It’s not what I think,” the chairman replied testily. “You see, when we first noticed this happening, a few days ago–”
“You waited a few DAYS?” I interrupted, a tremor tinging my voice. “And you couldn’t send email this whole time?”
“We could send email. Just not more than–”
“–500 miles, yes,” I finished for him, “I got that. But why didn’t you call earlier?”
“Well, we hadn’t collected enough data to be sure of what was going on until just now.” Right. This is the chairman of *statistics*. “Anyway, I asked one of the geostatisticians to look into it–”
“Geostatisticians…”
“–yes, and she’s produced a map showing the radius within which we can send email to be slightly more than 500 miles. There are a number of destinations within that radius that we can’t reach, either, or reach sporadically, but we can never email farther than this radius.”
“I see,” I said, and put my head in my hands. “When did this start? A few days ago, you said, but did anything change in your systems at that time?”
“Well, the consultant came in and patched our server and rebooted it. But I called him, and he said he didn’t touch the mail system.”
“Okay, let me take a look, and I’ll call you back,” I said, scarcely believing that I was playing along. It wasn’t April Fool’s Day. I tried to remember if someone owed me a practical joke.
I logged into their department’s server, and sent a few test mails. This was in the Research Triangle of North Carolina, and a test mail to my own account was delivered without a hitch. Ditto for one sent to Richmond, and Atlanta, and Washington. Another to Princeton (400 miles) worked.
But then I tried to send an email to Memphis (600 miles). It failed. Boston, failed. Detroit, failed. I got out my address book and started trying to narrow this down. New York (420 miles) worked, but Providence
(580 miles) failed.
I was beginning to wonder if I had lost my sanity. I tried emailing a friend who lived in North Carolina, but whose ISP was in Seattle. Thankfully, it failed. If the problem had had to do with the geography of the human recipient and not his mail server, I think I would have broken down in tears.
Having established that–unbelievably–the problem as reported was true, and repeatable, I took a look at the sendmail.cf file. It looked fairly normal. In fact, it looked familiar.
I diffed it against the sendmail.cf in my home directory. It hadn’t been altered–it was a sendmail.cf I had written. And I was fairly certain I hadn’t enabled the “FAIL_MAIL_OVER_500_MILES” option. At a loss, I telnetted into the SMTP port. The server happily responded with a SunOS sendmail banner.
Wait a minute… a SunOS sendmail banner? At the time, Sun was still shipping Sendmail 5 with its operating system, even though Sendmail 8 was fairly mature. Being a good system administrator, I had standardized on Sendmail 8. And also being a good system administrator, I had written a sendmail.cf that used the nice long self-documenting option and variable names available in Sendmail 8 rather than the cryptic punctuation-mark codes that had been used in Sendmail 5.
The pieces fell into place, all at once, and I again choked on the dregs of my now-cold latte. When the consultant had “patched the server,” he had apparently upgraded the version of SunOS, and in so doing downgraded Sendmail. The upgrade helpfully left the sendmail.cf alone, even though it was now the wrong version.
It so happens that Sendmail 5–at least, the version that Sun shipped, which had some tweaks–could deal with the Sendmail 8 sendmail.cf, as most of the rules had at that point remained unaltered. But the new long configuration options–those it saw as junk, and skipped. And the sendmail binary had no defaults compiled in for most of these, so, finding no suitable settings in the sendmail.cf file, they were set to zero.
One of the settings that was set to zero was the timeout to connect to the remote SMTP server. Some experimentation established that on this particular machine with its typical load, a zero timeout would abort a connect call in slightly over three milliseconds.
An odd feature of our campus network at the time was that it was 100% switched. An outgoing packet wouldn’t incur a router delay until hitting the POP and reaching a router on the far side. So time to connect to a lightly-loaded remote host on a nearby network would actually largely be governed by the speed of light distance to the destination rather than by incidental router delays.
Feeling slightly giddy, I typed into my shell:

Gathering data at subscription time

I recently received a survey from my Congressional Representative. She wanted to know what I wanted her to focus on in the coming year. I decided to go ahead and answer the survey, as I have some rather strong opinions on some of the stuff happening in Congress these days.
The email itself was pretty unremarkable, although quite well done. I was as much interested in answering the survey because it’s one of the few emails I’ve seen with an embedded survey.

DKIM deployment challenges

Cloudmark has an interesting blog post pointing out some of the challenges of signing mail with DKIM in a large company with a diverse mail system.

About that Junk Folder

I use a pretty standard mail filtering setup – a fairly vanilla SpamAssassin setup on the front end, combined with naive bayesian content filters in my mail client. So I don’t reject any mail, it just ends up in one of my inboxes or a junk folder. And I have a mix of normal consumer mail – facebook, twitter, lots of commercial newsletters, mail from friends and colleagues and spam. (As well as that I have a lot of high traffic industry mailing lists, but overall it’s a fairly normal mix.)
My bayesian filter gets trained mostly by me hitting “this is spam” when spam makes it to my inbox. If I’m expecting an email “immediately” – something like a mailing list COI confirmation or email as part of buying something online – I’ll check my spam filter and move the mail to my inbox in the rare case it ended up there. Other than that I let it and spamassassin chug along with no tweaking.
I’m starting a data analysis project, based on my own inboxes, and as part of that I’m using some tools to look for false positives in my junk folders, and manually fixing anything that’s misclassified. I’ve been doing this for a couple of hours now, and I’ve found some interesting things.

IP reputation and the bulk folder

I’ve spent much of today talking to various people about IP reputation and bulk foldering. It’s an interesting topic, and one that has changed quite a bit in the past few months. Here are a few of the things I said on the topic.
Generally IPs that the ISP has not seen traffic from before starts out with a slight negative reputation. If you think about all the new IPs that an ISP will see mail from on a daily basis, 99 out of 100 of those will be bot infected windows boxes. So they’re going to treat that mail very suspiciously. And, in the grand scheme of things, that mail is going to be spam a lot more than it’s not going to be spam.
Some ISPs put mail in the inbox and bulk foldering during the whitelisting process. Basically they’re looking to see if your recipients care enough about your mail to look for it in the bulk folder. This then feeds back to create the reputation of the IP address. There is another fairly major ISP that told me that when they’re seeing erratic data for an particular sender they will put some mail in bulk and some mail in the inbox and let the recipients tell the system which is more correct.
That’s what happens while you’re establishing a reputation on an IP. Once there is some history on the IP, things get a little different. At that point, IP reputation becomes unimportant in terms of bulk foldering. The ISP knows an IP has a certain level of reputation, and *all* their mail has that level of reputation. So bulk foldering is more related to content and reputation of the domains and URLs in the message.
The other reason IP reputation isn’t trumping domain / content reputation as much as it did in the past is that spammers stomped all over that. Affiliates, snowshoers, botnets, all those methods of sending spam made IP reputation less important and the ISPs had to find new ways to determine spam / not spam.
So if you’re seeing a lot of bulk foldering of mail, it’s unlikely there’s anything IP reputation based to do. Instead of worrying about IP reputation, focus instead on the content of the mail and see what you may need to do to improve the reputation of the domains and URLs (or landing pages) in the emails. While the content may not appear that different, the mere mention of “domain.com” where domain.com is seen in a lot of spam can trigger bulking.

How do I know you're spamming?

There are a number of reasons I know that mail coming into my mailbox is spam.

URL Shortening and Email

Any time you put a URL in mail you send out, you’re sharing the reputation of everyone who uses URLs with that hostname. So if other people send unwanted email that has the same URL in it that can cause your mail to be blocked or sent to the bulk folder.
That has a bunch of implications. If you run an affiliate programme where your affiliates use your URLs then spam sent by your affiliates can cause your (clean, opt-in, transactional) email to be treated as spam. If you send a newsletter with advertisers URLs in it then bad behaviour by other senders with the same advertisers can cause your email to be spam foldered. And, as we discussed yesterday, if spammers use the same URL shortener you do, that can cause your mail to be marked as spam.
Even if the hostname you use for your URLs is unique to you, if it resolves to the same IP address as a URL that’s being used in spam, that can cause delivery problems for you.
What does this mean when it comes to using URL shorteners (such as bit.ly, tinyurl.com, etc.) in email you send out? That depends on why you’re using those URL shorteners.
The URLs in the text/html parts of my message are big and ugly
Unless the URL you’re using is, itself, part of your brand identity then you really don’t need to make the URL in the HTML part of the message visible at all. Instead of using ‘<a href=”long_ugly_url”> long_ugly_url </a>’ or ‘<a href=”shortened_url”> shortened_url </a>’ use ‘<a href=”long_ugly_url”> friendly phrase </a>’.
(Whatever you do, don’t use ‘<a href=”long_ugly_url”> different_url </a>’, though – that leads to you falling foul of phishing filters).
The URLs in the text/plain parts of my message are big and ugly
The best solution is to fix your web application so that the URLs are smaller and prettier. That will make you seem less dated and clunky both when you send email, and when your users copy and paste links to your site via email or IM or twitter or whatever. “Cool” or “friendly” URLs are great for a lot of reasons, and this is just one. Tim Berners-Lee has some good thoughts on this, and AListApart has two good articles on how to implement them.
If you can’t do that, then using your own, branded URL shortener is the next best thing. Your domain is part of your brand – you don’t want to hide it.
I want to use a catchy URL shortener to enhance my brand
That’s quite a good reason. But if you’re doing that, you’re probably planning to use your own domain for your URL shortener (Google uses goo.gl, Word to the Wise use wttw.me, etc). That will avoid many of the problems with using a generic URL shortener, whether you implement it yourself or use a third party service to run it.
I want to hide the destination URL from recipients and spam filters
Then you’re probably spamming. Stop doing that.
I want to be able to track clicks on the link, using bit.ly’s neat click track reporting
Bit.ly does have pretty slick reporting. But it’s very weak compared to even the most basic clickthrough reporting an ESP offers. An ESP can tell you not just how many clicks you got on a link, but also which recipients clicked and how many clicks there were for all the links in a particular email or email campaign, and how that correlates with “opens” (however you define that).
So bit.ly’s tracking is great if you’re doing ad-hoc posts to twitter, but if you’re sending bulk email you (or your ESP) can do so much better.
I want people to have a short URL to share on twitter
Almost all twitter clients will abbreviate a URL using some URL shortener automatically if it’s long. Unless you’re planning on using your own branded URL shortener, using someone else’s will just hide your brand. It’s all probably going to get rewritten as t.co/UgLy in the tweet itself anyway.
If your ESP offers their own URL shortener, integrating into their reporting system for URLs in email or on twitter that’s great – they’ll be policing users of that just the same as users of their email service, so you’re unlikely to be sharing it with bad spammers for long enough to matter.
All the cool kids are using bit.ly, so I need to to look cool
This one I can’t help with. You’ll need to decide whether bit.ly links really look cool to your recipient demographic (Spoiler: probably not) and, if so, whether it’s worth the delivery problems they risk causing.
And, remember, your domain is part of your brand. If you’re hiding your domain, you’re hiding your branding.
So… I really do need a URL shortener. Now what?
It’s cheap and easy to register a domain for just your own use as a URL shortener. Simply by having your own domain, you avoid most of the problems. You can run a URL shortener yourself – there are a bunch of freely available packages to do it, or it’s only a few hours work for a developer to create from scratch.
Or you can use a third-party provider to run it for you. (Using a third-party provider does mean that you’re sharing the same IP address as other URL shorteners – but everyone you’re sharing with are probably people like you, running a private URL shortener, so the risk is much, much smaller than using a freely available public URL shortener service.)
These are fairly simple fixes for a problem that’s here today, and is going to get worse in the future.
(9/18/17: Closing comments because this post attracts spam comments)

Bit.ly gets you Blocked

URL shorteners, like bit.ly, moby.to and tinyurl.com, do three things:

It's Wednesday – do you know where your sales staff are?

I received an email yesterday with the subject “Please confirm your lunch reservation”. It didn’t look like a typical spam subject line, but wasn’t from anywhere I recognized.
I take a look.

Permission-ish based marketing

My Mum flew in to visit last week, and over dinner one evening the talk turned to email.

Why is shared hosting like phishing?

A client of a friend was getting rejection messages when they tried to send mail

Is your mail being bulk foldered?

Daisy at Signup.to posted a list of 11 things to do when mail is going to the bulk folder. Her suggestions are a good start to troubleshooting and fixing persistent bulk foldering of mail.
One thing she doesn’t mention is that while bulk foldering can sometimes be the result of poor content, more often it’s the result of unengaged recipients. Think of bulk foldering this way: the ISP has some subscribers they’re pretty sure want your mail, so they’re not going to block your mail. But they’re pretty sure a lot of subscribers don’t want your mail so they’re not going to deliver it to the inbox.
The trick to getting mail moved out of the bulk folder is to get more people engaged with your email marketing. This is tough to do if they’re not actively checking their bulk folder for mail but there are some ways I’ve helped clients get mail into the inbox.

Size isn't the only metric

MarketingSherpa has a case study up today about a company that took an aggressive stance on re-engagement that reduced their house list size by over 95%. While the size of the list went down, online sales doubled.
The whole article is a lesson in how to do email right. They are sending relevant and engaging mail to their subscribers. They kept the addresses of people who wanted the mail, but designed a new program from the ground up. All of the key points I, and others, keep talking about is present in their new program.

Standing in the stadium

If you distill marketing down to its very essence what you find is everyone battling for a targets attention. Everything marketers do is to get “mindshare” or, in normal people terms, attention. The goal is to get people to remember your product over all the other products out there.
Many email marketers seem to think that increasing the frequency of mail is the most successful way to get attention from their recipients. What would happen, then, if every email marketer started sending more email? Would this really get more attention from recipients?
I don’t think so.
Increasing mail frequency is like standing up to see better in a stadium. One person stands up (increases frequency) and that person sees better than they did before. But if everyone stands up (increases frequency), then everyone is back to where they were and everyone is back to not being able to see.

The dark side of email marketing

Everyone I talk to when dealing with issues inevitably has to tell me they are legitimate email marketers. They’re not spammers, they’re just business people. I often find it difficult to fathom why they need to tell me this. It’s not like email marketers are criminals or anything.
Two recent stories reminded me how evil some folks are. While I’ve not had any direct contact (that I know of) with any of the players on this end of things I have zero doubt that if they called me they would tell me that they were legitimate email marketers.
In one case, a members of a spam gang kidnapped the teenage daughter of someone investigating their activities. The gang held her for more than 5 years in horrific conditions. Yesterday Joseph Menn, author of “Fatal System Error” posted on Boing Boing that his friend got his daughter back. It is a heartbreaking story and incredibly sobering.
In another case, the Russian police arrested a man who ran spammit.com, a clearinghouse for viagra sellers to find spammers to send their mail. Reports say that mail volumes dropped by a fifth after the site was taken offline.
There is real evil in the email marketing industry. Sure, they’re spammers and we can all stand up and say they’re not legitimate. But, this is what the ISPs and Spamhaus and law enforcement are dealing with on a regular basis.

Content based filters

Content based filters are incredibly complex and entire books could be written about how they work and what they look at. Of course, by the time the book was written it would be entirely obsolete. Because of their complexity, though, I am always looking for new ways to explain them to folks.
Content based filters look at a whole range of things, from the actual text in the message, to the domains, to the IP addresses those domains and URLs point to. They look at the hidden structure of an email. They look at what’s in the body of the message and what’s in the headers. There isn’t a single bit of a message that content filters ignore.
Clients usually ask me what words they should change to avoid the filters. But this isn’t the right question to ask. Usually it’s not a word that causes the problem. Let me give you a few examples of what I mean.
James H. has an example over on the Cloudmark blog of how a single missing space in an email caused delivery problems for a large company. That missing space changed a domain name in the message sufficiently to be caught by a number of filters. This is one type of content filter, that focuses on what the message is advertising or who the beneficiary of the message is. Some of my better clients get caught by these types of filters occasionally. A website they’re linking to or a domain name they’re using in the text of the message has a bad reputation. The mail gets bulked or blocked because of that domain in the message.
One of my clients went from 100% inbox every day to random failures at different domains. Their overall inbox was still in the 96 – 98% range, but there was a definite change. The actual content of their mail hadn’t changed, but we kept looking for underlying causes. At one point we were on the phone and they mentioned their new content management system. Sure enough, the content management company had a poor reputation and the delivery problems started exactly when they started using the content management. The tricky part of this was that the actual domains and URLs in the messages never changed, they were still clickthrough.clientdomain.example.com. But those URLs now pointed to an IP address that a lot of spammers were abusing. So there were delivery problems. We made some changes to their setup and the delivery problems went away.
The third example is one from quite a long time ago, but illustrates a key point. A client was testing email sends through a new ESP. They were sending one-line mail through the ESPs platform to their own email account. Their corporate spamfilter was blocking the mail. After much investigation and a bit of string pulling, I finally got to talk to an engineer at the spamfiltering company. He told me that they were blocking the mail because it “looked like spam.” When pressed, he told me they blocked anything that had a single line of text and an unsubscribe link. Once the client added a second line of text, the filtering issue went away.
These are just some of the examples of how complex content based filters are. Content is almost a misnomer for them, as they look at so many other things including layout, URLs, domains and links.

Don't be the tomcat

Our tomcat, Grover, wants to go outside. He wants to go out the side door, so he’s been sitting in front of it, looking at me, then staring at the door. He’s been doing this for about half an hour, intermixed with occasional sad yowling.
The back door is open, and he can get from outside the back door to outside the side door in a minute, tops. But he wants to go out the side door.
(I’d get up and open the side door for him anyway, but I have another cat asleep on my lap and it’s not fair to move her just because Grover is being difficult).
I’ve been having a discussion with him:
Grover, the back door is open.
But I want to go out the side door.
You could go out the back door and then go round to the side door.
But I want to go out the side door.

Content based filtering

A spam filter looks at many things when it’s deciding whether or not to deliver a message to the recipients inbox, usually divided into two broad categories – the behaviour of the sender and the content of the message.
When we talk about sender behaviour we’ll often dive headfirst into the technical details of how that’s monitored and tracked – history of mail from the same IP address, SPF records, good reverse DNS, send rates and ramping, polite SMTP level behaviour, DKIM and domain-based reputation and so on. If all of those are OK and the mail still doesn’t get delivered then you might throw up your hands, fall back on “it’s content-based filtering” and not leave it at that.
There’s just as much detail and scope for diagnosis in content-based filtering, though, it’s just a bit more complex, so some delivery folks tend to gloss over it. If you’re sending mail that people want to receive, you’re sure you’re sending the mail technically correctly and you have a decent reputation as a sender then it’s time to look at the content.
You want your mail to look just like wanted mail from reputable, competent senders and to look different to unwanted mail, viruses, phishing emails, botnet spoor and so on. And not just to mechanical spam filters – if a postmaster looks at your email, you want it to look clean, honest and competently put together to them too.
Some of the distinctive content differences between wanted and unwanted email are due to the content as written by the sender, some of them are due to senders of unwanted email trying to hide their identity or their content, but many of them are due to the different quality software used to send each sort of mail. Mail clients used by individuals, and content composition software used by high quality ESPs tends to be well written and complies with both the email and MIME RFCs, and the unwritten best common practices for email composition. The software used by spammers, botnets, viruses and low quality ESPs tends not to do so well.
Here’s a (partial) list of some of the things to consider:

Don't always believe the statistics

Mark Brownlow has a great roundup of how statistics and data can mislead marketers if they’re not really paying attention.

Thursday mini-audit part 2

A week ago you signed up for your mailing list using a virgin email address. (You didn’t? Maybe you should do that today – there’s no time like Thursday for a quick sanity check!)
Check the mailbox for the account you signed up

What you should do Right Now – Thursday Mini-Audit

… if your company runs any sort of email marketing, anyway.
Right now is the best time to do a mini-audit of your mail campaign. It’ll just take ten minutes, and if you put off doing it until tomorrow it’ll probably never get done.

ESPs, Non-portable Reputation and Vendor Lock-in

I’ve seen some mentions recently of ESPs suggesting that if you use your own domain in the From: of mail you send through an ESP then that ESP can’t “do email authentication” properly unless they require you to edit your domains DNS settings. That’s not really so, but there is a kernel of truth in there.
The real situation is, unsurprisingly, a bit more complicated.
What authentication features should you look for in an ESP?

Basic email delivery using telnet

Whenever we’re working with someone to diagnose some obscure delivery issue one of the things we usually have them try is to “run a transaction by hand”. Being able to do that is a trick that everyone working with email should be able to do. I was drafting a blog post today and wanted to refer to running a transaction by hand and I realized that we hadn’t actually explained it anywhere. So here we are.
When you’re running a transaction by hand you’re doing everything your mailserver would do to deliver an email, but you’re doing it yourself. That means that you get to see all the responses from the mailserver you’re sending the mail to, and also any delays or errors in much more detail than you can usually get from mailserver delivery logs.
I want to send some email to playingwithtelnet@gmail.com. There are two main steps to doing this – first I need to find out which mailserver I need to talk to to send mail to gmail, then I need to actually send the mail.
To find the mailserver I have to look up the MX record for gmail.com. From a unix / linux / mac command prompt you can do that like this (the bits you type are in orange):

The rules of delivery success

Senders with delivery problems ask about “the rules.” “Just tell us what the rules are!” “If the ISPs would just tell us what to do we’d do it!” There is only one rule anyone needs to pay attention to for good mail delivery: Respect the recipient.
Not good enough for you? Want more specific rules? OK.
The two rules everyone must follow for good mail delivery.

The secret to fixing delivery problems

There is a persistent belief among some senders that the technical part of sending email is the most important part of delivery. They think that by tweaking things around the edges, like changing their rate limiting and refining bounce handling, their email will magically end up in the inbox.
This is a gross misunderstanding of the reasons for bulk foldering and blocking by the ISPs. Yes, technical behaviour does count and senders will find it harder to deliver mail if they are doing something grossly wrong. In my experience, though, most technical issues are not sufficient to cause major delivery problems.
On the other hand, senders can do everything technically perfect, from rate limiting to bounce handling to handling feedback loops through authentication and offer wording and still have delivery problems. Why? Sending unwanted mail trumps technical perfection. If no one wants the email mail then there will be delivery problems.
Now, I’ve certainly dealt with clients who had some minor engagement issues and the bulk of their delivery problems were technical in nature. Fix the technical problems and make some adjustments to the email and mail gets to the inbox. But with senders who are sending unwanted email the only way to fix delivery problems is to figure out what recipients want and then send mail meeting those needs.
Persistent delivery problems cannot be fixed by tweaking technical settings.

Reputation

It used to be that every potential client that called me up to ask me to help them with their delivery issues would tell me they weren’t a spammer. Over the last year or so that’s changed to telling me that they have a good reputation and don’t understand why they’re having delivery problems.
This leads me to believe that there is some confusion about what reputation is and what reputation is not.
Reputation is a shorthand term for a complex formula measuring the history of email from an IP address. Some reputation schemes measure the history of email containing a particular URL or domain.
Recipient domains measure a lot of things and use them at various points during the email transaction. Some measurements are integrated into a single value that is queried during the SMTP transaction. If the measurement is too bad, the email is rejected or rate limited. Other measurements are queried after the email is accepted by the ISP, and those values determine if an email is delivered to the inbox or the bulk folder.
There are a couple important things to remember about reputation.

Spam is in the eye of the beholder

But only the opinion of the recipient counts. So says a blog post on All Spammed Up.

This is why the ISPs throw up their hands at senders

I recently saw a question from an ESP rep asking if anyone had a personal contact at a particular ISP. The problem was that they had a rejection from the ISP saying: 571 5.7.1 too many recipients this session. The ESP was looking for someone at the ISP in order to ask what the problem was.
This is exactly the kind of behaviour that drives ISPs bonkers about senders. The ISP has sent a perfectly understandable rejection: “5.7.1: too many recipients this session.” And instead of spending some time and energy on the sender side troubleshooting, instead of spending some of their own money to work out what’s going on, they fall back on asking the ISPs to explain what they should do differently.
What, exactly, should you do differently? Stop sending so many recipients in a single session. This is not rocket science. The ISP tells you exactly what you need to do differently, and your first reaction is to attempt to mail postmaster@ the ISP and then, when that bounces, your next step is to look for a personal contact?
No. No. No.
Look, connections and addresses per connections is one of the absolute easiest things to troubleshoot. Fire up a shell, telnet to port 25 on the recipient server, and do a hand SMTP session, count the number of receipts. Sure, in some corporate situations it can be a PITA to do, sometimes you’re going to need to get it done from a particular IP which may be an interface on an appliance and doesn’t have telnet or whatever. But, y’know what? That Is Your Job. If your company isn’t able to do it, well, please tell me so I can stop recommending that as an ESP. Companies have to be able to test and troubleshoot their own networks.
Senders have been begging ISPs for years “just tell us what you want and we’ll bother you less.” In this case the ISP was extremely clear about what they want: they want fewer recipients per connection. But the ESP delivery person is still looking for a contact so they can talk to the ISP to understand it better.
This is why the ISPs get so annoyed with senders. They’re tired of having to do the sender’s job.

Taking permission

Permission is always a hot topic in email marketing. Permission is key! the experts tell us. Get permission to send email! the ISPs tell us.

Permission versus forgiveness

Stephanie at Return Path has a great blog post on permission and how permission is an ongoing process not a one time thing. There were a couple statements that really grabbed my attention.

Delivery reference site

Over the years I’ve picked up a lot of useful and relevant information about email delivery. I’ve shared a lot of information here on the blog, and while that’s great, a blog is not a great format for a reference. The ISP information page was an initial pass at creating a reference. I realized that just linking to the ISP provided information didn’t communicate very much about how to deliver email even to those ISPs that were explicitly mentioned.
Enter the Word to the Wise Delivery Wiki to fill the need for a publicly accessible reference on email delivery. The cornerstone of the site is the ISP Information page. This page contains summary information about a number of ISPs, including known connection and sending limits. Each ISP mentioned on that page also has a individual page with more detailed information on delivering to that ISP. The information is as accurate as I could make it, and in many cases have been reviewed by representatives of the ISPs.
I welcome contributions from the general community. I will also be continuing to add content. My goal is to have a community resource for people handling email and delivery issues.

It doesn't matter what you say

“What should we tell the ISP?” is a frequent question from my customers. The answer is pretty simple. It doesn’t usually matter what you tell the ISP. What matters are your actions.
If a sender is having delivery problems then the solution is not to call the ISP and talk to them about why the sender’s mail should not be delivered to the bulk folder. Instead, the solution is to evaluate the email and the address acquisition process and the list hygiene process. Identify where potential problems are and then resolve those problems.
Typically, the ISPs won’t need to be contacted. The changes to the email will register and delivery will improve. In some cases, particularly when there’s been some major mistake, contacting the ISP and explaining the mistake and what steps have been taken to stop the mistake from happening in the future may help resolve the issue faster. But if nothing has changed, then there’s no reason for the ISP to expect anything to change.
It doesn’t matter what you say. It matters what you do.

Keep subscribers happy

Mark Brownlow writes about engagement. “…the people we really need to keep happy are the subscribers.” Go read the whole thing.

SpamAssassin Problems

The default SpamAssassin configuration considers any date far in the future to be extremely suspicious, which is pretty reasonable.
However, as @schampeo points out, it also seems to consider any date later than 2009 to be “far in the future”.
That means that until the SpamAssassin folks roll out a fix, and that gets deployed by SpamAssassin users pretty much all email will get an additional 2-3.5 spamminess points. That’s likely to cause a lot of content-based blocking over the next few weeks, until fixed rules are deployed both by SpamAssassin users and by all the various spam filtering appliances that use SpamAssassin rulesets.
(If you’re a SpamAssassin user, add “score FH_DATE_PAST_20XX 0.0” to your local.cf file to disable that rule).
EDIT: Mike has some more background on the bug.
EDIT: Fix it out on the spamassassin homepage.

Are you listening?

One of the more common complaints from senders is that ISPs won’t tell them what to do. Al Iverson takes this on in his post “Did you catch that?” He says:

Controlling delivery

How much control over delivery do senders have? I have repeatedly said that senders control their delivery. This is mostly true. Senders control their side of the delivery chain, but there is a point where the recipient takes over and controls things.
As a recipient I can

Internationalisation (part 2)

In part 1 I talked about internationalised domain names, and how they were mapped onto ASCII strings.
For sending email there are four bits of the message where internationalisation might need to be considered.

Internationalisation (part 1)

There’s been a gentle bit of uproar recently about ICANN finally beginning the process of rolling out support for internationalized domain names (IDN) at the DNS root and the effect that may have on email senders. Even if you haven’t noticed the uproar, it’s still a subject you probably want to be familiar with if you’re sending email.
What are internationalised domain names?
An internationalised domain name is simply a domain name that uses non-ascii characters – most anything other than a-z, 0-9 and ‘-‘ – such as those used in these URLs: http://пример.испытание/ or http://例子.測試/ (If those links are unreadable or don’t work, it means that your browser isn’t handling IDN well or doesn’t have the appropriate fonts installed yet).
They’re an obvious thing to want, especially if you’re from anywhere other than an anglophone country, but the Internet was originally built as an ascii-only network, and under the covers it still is entirely ascii-only, so layering non-ascii characters on top has taken a lot of work and time to roll out. IDN development dates back to at least 1996 and it has been supported by some top level domains since 2003. So the recent announcement to support non-ascii top level domains is just the latest step in a long and careful process.
Almost all of the underlying internet protocols are still ASCII based though, including DNS and SMTP, so a lot of the internationalisation work involves mapping non-ASCII words onto ASCII strings before they’re passed to the network, and mapping them back again before they’re displayed to the user. This is done in a fairly ad-hoc way, different in different protocols.
If you were to visit the cyrillic URL I mentioned above then the first thing your web browser would do would be to take the cyrillic string “пример.испытание” and translate it to the ASCII hostname “xn--e1afmkfd.xn--80akhbyknj4f” then look that up in the DNS to find the server handling that URL.
If you were to display that on a webpage or in an HTML email it might be converted to ASCII as”http://приме р.и с п ы тание/”.
If you were to send it as part of a plain text email, encoded as UTF8/quoted-printable, it would look like “http://%D0%BF%D1%80%D0%B8%D0%BC%D0%B5%D1%80.%D0%B8%D1%81%D0%BF%D1%8B%D1%82%D0%B0%D0%BD%D0%B8%D0%B5/”. If there’s a lot of non-ASCII characters in the message then it’s more likely to be encoded as UTF8/base64: “aHR0cDovL9C/0YDQuNC80LXRgC7QuNGB0L/Ri9GC0LDQvdC40LUvCg==”.
And all of those will (or at least should) be displayed to the end user identically.
Confused yet? That’s fine. Internationalisation on the Internet is a very complex and inconsistent subject. In my next post I’ll try and narrow down which bits of it you need to worry about when it comes to sending email and to not upsetting phishing or spam filters at the recipients ISP.

Dealing with blacklists

Al has a good post listing the top 5 things senders should remember when dealing with blacklists.
One of the critical things to remember about blocklists is that they are an early warning sign. Sure, some of them are one crank and his cat and will not hurt your overall delivery. A sender may be listed for totally spurious reasons . On the other hand, many of the widely used public lists and the private lists at the big ISPs, list IPs that they see as doing something wrong.
The challenge for anyone listed on any IP based blocklist is to look inside and determine what it is that they’re doing that caused the listing. The first step is to look at the technical issues, does your mail look like something coming out of infected bots? Is there a configuration problem? If the answer is no, then senders have to look at their practices. Are they sending mail to people who don’t expect it? Are they sending mail to people who didn’t ask for it? Most listings that will affect large numbers of recipients fall into the above 2 categories: technical or practices.
Technical problems can be fixed easily, once they’re identified. Permission or practice problems can also be fixed, but may require a sender reassess how they are using email and what value email brings to the business.

How reputation and content interact

Recently, one of my clients had a new employee make a mistake and ended up sending newsletters to people in their database that had not subscribed to those particular newsletters. This resulted in their recipients getting 3 extra emails from them. These things happen, people fat-finger database queries or aren’t as careful with segmentation as they should be.
My clients were predictably unhappy about sending mail their users hadn’t signed up for and asked me what to do to fix their reputation. I advised they not do anything other than make sure they don’t do that again. The first send after their screw-up had their standard 100% inbox delivery. The second send had a significant problem with bulk foldering at Hotmail and Yahoo. The third send had their standard 100% inbox delivery.
So what happened on the second send? It appears that on that send they had a link or other content that “filled the bucket.” Generally, their IP reputation is high enough that content isn’t sufficient to send their mail into the bulk folder. However, their reputation dipped based on the mistake last week, and thus the marginal content caused the bulk foldering.
Overall, these are senders with a good reputation. Their screw up wasn’t enough to damage their delivery itself, but may have contributed to all their mail going into the bulk folder the other day. I expect that their reputation will rebound quickly and they will be able to send the same content they did and see it in the inbox.

Technology does not trump policy when it comes to delivery

Recently Ken Magill wrote an article looking at how an ESP was attempting to sell him services based on the ESPs ‘high deliverability rates.’ I commented that Ken was right, and I still think he is.
Ken has a followup article today. In the first part he thanks Matt Blumberg from Return Path for posting a thoughtful blog post on the piece. Matt did have a very thoughtful article, pointing out that the vast majority of things affecting delivery are under the control of the list owner, not under the control of the ESP. As they are both right, I clearly agree with them. I’ve also posted about reputation and delivery regularly.

Hidden cost of email blasts

Seth Godin has a post up today talking about how friction, that is the cost of sending marketing, is good for marketing. With more friction, marketers make choices about sending instead of sending to everyone.
The post touches on a point I’ve certainly tried to explain to clients and senders in general.

Yahoo and Verizon

Mickey at Spamtacular has information about Verizon’s email system that will have relevance for anyone working in delivery.

Failed delivery of permission based email

A few weeks ago, ReturnPath published a study showing that 20% of permission based email was blocked. I previously discussed the definition of permission based email and that not all the mail described as permission based is actually sent with the permission of the recipient. However, I only consider this a small fraction of the mail RP is measuring, somewhere in the 3 – 5% range. What happens with the other 17 – 15% of that mail? Why is it being blocked?
There are 3 primary things I see that cause asked for and wanted email to be blocked.

New Delivery tools

A couple nifty new delivery tools were published over the weekend.
Mickey published Bounce P.I. where senders can paste in an error message or bounce and it will tell you what filter generated it. If the rejection is unrecognized, it will flag the message internally and it will be researched to see if the filter can be identified.
Steve has a new tool at the DKIMCore site. The key generating tool and the record checking tool have been up for a while. This weekend, though, he published a tool to check the validity the DKIM record published in DNS. Tool output shows if the record is valid, the version and the public key.

Permission Based Emails? Are you sure?

Yesterday I wrote about the ReturnPath study showing 21% of permission based email does not make it to the inbox. There are a number of reasons I can think of for this result, but I think one of the major ones is that not all the mail they are monitoring is permission based. I have no doubt that all of the RP customers say that the mail they’re sending is permission based, I also have no doubt that not all of the mail is.
Everyone who sends mail sends permission based email. Really! Just ask them!
In 10 years of professionally working with senders I have yet to find a marketer that says anything other than all their email is permission based. Every email marketer, from those who buy email addresses to those who do fully confirmed verified opt-in with a cherry on top will claim all their email is permission based. And some of the mailers I’ve worked with in the past have been listed on ROKSO. None of these mailers will ever admit that they are not sending permission based email.
Going back to ReturnPath’s data we don’t really know what permission based email means in this context and so we don’t know if the mail is legitimately or illegitimately blocked. My guess is that some significant percentage of the 20% of email to the probe accounts that doesn’t make it to the inbox is missing because the sender does not have clear recipient permission.
When even spammers describe their email as permission based email marketing, what value does the term have?

Delivery Metrics

Last week ReturnPath published a study that shows 20% of permission based email fails to be delivered to the inbox. For this study, ReturnPath looked at the mail sent by their mailbox monitor customers and counted the number of deliveries to the inbox, the number of deliveries to the bulk folder and the number of emails that were not delivered.
At US ISPs 21% of the permission based emails sent to the ReturnPath probe network did not make it to the inbox. 3% of the emails sent went to the bulk folder and 17% did not make it to the mailbox at all. MSN/Hotmail and Gmail were the worst ISPs to get mail to. They each failed to deliver more than 20% of the mail that was sent to them. At Canadian ISPs, even less of the mail made it to the inbox, primarily because primus.ca is such a large portion of the Canadian market and they use Postini as a filter. Postini is a quite aggressive filter and takes no feedback from senders.
ReturnPath’s take home message on the survey is that one set of metrics is not enough to effectively evaluate a marketing program. Senders need to know more about their mailings than they can discover from just the bounce rate or the revenue rate or response rate or open rate.
There are a lot of reasons an email doesn’t get to the recipient’s inbox or bulk folder. Mail can be hard blocked at the MTA, and rejected by the ISP outright. Mail can be soft blocked at the MTA and the ISP can slow down sending. Sometimes this is enough to cause the sending MTA to stop attempting to deliver the mail, thus causing mail to not show up. Both of these types of blocks are usually visible when looking at the bounce rate.
Some ISPs accept mail but then fail to deliver it to the recipient. Everything on the sender end says the ISP accepted it for delivery but the ISP just drops it on the floor. This is the type of block that a mailbox monitoring program is best able to identify.
Despite all the discussions of numbers, many marketers are still not measuring the variables in their email campaigns. Ken Magill wrote today about a study released by eROI that indicates more than a third of marketers are not doing any testing on their mailings.
Now, both of these studies are done in an attempt to sell products, however, the numbers discussed should be making smart senders think about what they are measuring in regards to their email campaign, how they are measuring those factors and what the measurements mean.

AOL changes bounce behaviour

A couple other bloggers have commented on the recent AOL blog post talking about changes to the MAILER-DAEMON string on bounce messages.

Fixing high complaint rates and improving reputation

Why do recipients complain about my email?
This question is asked over and over again and there is no one answer. There are a number of reasons and all of them interact with one another.
What factors have recipients mentioned?

Best time to send email: analysis and discussion

Mark Brownlow (who I don’t think is here in Ams, much to my disappointment) wrote a long assessment of how to determine what is the best time to send email. He walks through the questions and the data that a sender should evaluate when making the decision when to best send email.
I have previously posted about my views on the best time to send email. There is no one best time to send email. In fact, my experience leads me to believe if someone said the best time to send email is at 4pm on Tuesday afternoon then 4pm on Tuesday afternoon would rapidly become the absolute worst time to send email.
It should come as no surprise, then, that I really like Mark’s #4 recommendation.

Delivery advice from Politico

Politico published an article Sunday looking at the best e-mail lists in politics. Their criteria for choosing the winner focused on list size and recipient engagement, measured by amount of money raised and recipient response to issues. Despite not being a delivery focused article or even mentioning delivery at all, this article is all about delivery.
How can an article be about delivery without ever mentioning the word? By actually looking at the effectiveness of the overall campaign and measuring how the lists actually perform. In the article, Politico used a number of criteria to evaluate different email lists and programs.

TWSD: Dumb and dumber

I recently received a spam offering to get one of my personal websites listed in foreign search engines. Harvesting addresses off websites is dumb. Even dumber is sending a followup a week later with a notice at the top.

Best time to send email: redux

Last week I wrote about a study classifying different types of email users. My point is that senders should be very aware of how their users interact with email, in order to provide the best user experience and the most revenue for the sender. If, for instance, the bulk of recipients are daytime (9 – 5 M-F) users, then the best time to email is different than if the bulk of recipients are all the time users of email.
At least 2 different people commented on when the “best” time to send email was, completely missing the entire point of my post. When you send email should be related to when your users are active in their email client. Senders know this, because they can track times when people open and click on links in the email. The data is all there, it just needs to be mined.
Plus, if every sender sent mail at the exact same time, that being the best time to send mail, then it will immediately become the absolute worst time to send email.
Pay attention to your recipients, and not to the internet experts. Listen to what your customers and recipients are telling you. Do what’s best for them, not what’s best for Joe’s Bait and Tackle Shop.

The Weekend Effect

Sending mail only Monday through Friday can cause reputation and delivery problems at some ISPs, even when senders are doing everything right. This “weekend effect” is a consequence of how ISPs measure reputation over time.
Most ISPs calculating complaint rate use a simple calculation. They measure how many “this is spam” clicks a source IP generates in a 24 hour period. Then they divide that number by how many emails were delivered to the inbox in the same 24 hour period.
The weekend effect happens when a sender sends on weekdays and not on the weekend thus lowering the number of emails delivered to the to the inbox. Recipients, however, still read mail on the weekend, and they still hit the “this is spam” button on the email. Even if the number of “this is spam” clicks is lower than a normal weekday, with no incoming email the rate of spam complaints goes above ISP thresholds. Even a very well run mailing program may see spikes in complaint rate on the weekends.
Now, when the ISPs are measuring complaint rates over time, they take the average of the average complaint rates. If the rates spike high enough on the weekend (and they can spike to the 1 – 3% range, even for a well run list), that can hurt the senders’ reputation.
The good news is that ISPs are aware of the weekend effect and take this into account when manually looking at complaints. The bad news is that not all of the major ISPs take this into account when programatically calculating reputation.
There isn’t very much senders can do to combat the weekend effect, except be aware this can happen and may be responsible for poor mailing performances on Monday. If you are seeing delivery problems you think may be a result of the weekend effect you can contact the ISPs and ask for manual review of your reputation. Some ISPs can provide manual mitigation for senders with otherwise clean stats. d

How Spamfilters Work

AllSpammedUp has a post describing the primary techniques anti-spam filters use to identify mail as spam or not spam. While is this not sender or delivery focused knowledge, it is important for people sending mail to have a basic understanding of filtering mechanisms. Without that base knowledge, it’s difficult to troubleshoot problems and resolve issues.

Confirmed opt-in

I spent the morning in multiple venues correcting mis-understandings of confirmed opt-in. The misunderstandings weren’t so much that people didn’t understand how COI works, but more they didn’t understand all the implications.
In one venue, the conversation centered around how small a portion of deliverability the initial subscription process affects. Sure, sending unwanted, unexpected email can and does cause reputation problems, but merely using COI as a subscription methodolgy doesn’t automatically give a sender a good reputation or good delivery. Senders using COI as a subscription practice need to also need to send relevant and engaging mail that their recipients expect to receive. They need to handle their bounces well and purge or re-engage inactive subscribers. They need to keep their complaints low and their responses high.
How you manage subscriptions is only one factor in reputation schemes, and even if the subscription method is COI other factors can negate any bonus involved.
The second conversation involved Ken challenging me on the comment I left on his quiz yesterday. I said COI wasn’t foolproof and he challenged me to explain how. I did, and he’ll be following up next week.

Reputation as measured by the ISPs

Part 3 in an ongoing series on campaign stats and measurements. In this installment, I will look a little closer at what other people are measuring about your email and how that affects your reputation at the ISPs.
Part 1: Campaign Stats and Measurements
Part 2: Measuring Open Rate
Reputation at the ISPs is an overall measure of how responsive recipients are to your email. ISPs also look at how much valid email you are sending. Anything the ISP can measure and use to distinguish good mail from bad is used in calculating reputation.
Some of the major metrics ISPs use include the following.
Invalid Address Rates
The ISPs count how much mail from any particular IP address is hitting non-existent addresses. If you are mailing a large number of email addresses that do not exist (550 user unknown), this is a suggestion that your address collection techniques are not very good. Responsible mailers do have the occasional bad address, including typos, expired/abandoned addresses, but the percentage in comparison to the number of real email addresses is low. How low is low? Public numbers suggest problems start at 10% user unknowns, but conversations with ISP employees show they consider lower levels a hint there may be a problem.
To calculate bounce rate ISPs take the total number of addresses that were for invalid accounts and divide that by the total number of addresses that the sender attempted to send mail to. Rates above 10% may cause significant delivery issues on their own, rates lower that 10% may still contribute to poor delivery through poor reputation scores.
Spamtraps
ISPs pay a lot of attention to how much mail is hitting their “trap” or “bait” accounts. There are a number of different sources of these trap accounts: old abandoned email addresses, addresses that never existed or even role accounts. Hits to a trap account tells the ISP there are addresses on your list that did not opt-in to receive mail. And if there are some addresses they know about that did not opt-in, it is likely that there are other addresses that did not opt in.
Spamtraps tend to be treated as an absolute number, not as a percentage of emails. Even a single spamtrap on a list can significantly harm delivery. According to the ReturnPath Benchmark report lists with a single spamtrap had nearly 20% worse delivery than lists without spamtraps.
This is spam clicks (FBL complaints)
Complaints from users are heavily used by ISPs. This tells them directly how many people are objecting to your email. In this case, permission is removed from the equation. Even if a sender has permission to send email, the recipient can say “no, I don’t want this, it is spam.” The ISPs put more weight on what their users tell them than on what the senders tell them.

TWSD: Lying and Hiding

Another installment in my ongoing series: That’s What Spammers Do. In today’s installment we take a look at a company deceiving recipients and hiding their real identity.
One of my disposable addresses has been getting heavily spammed from mylife.com. The subject lines are not just deceptive, they are provably lies. The mail is coming from random domains like urlprotect.com or choosefrequency.com or winnernotice.com advertising links at safetyurl.com or childsafeblogging.com or usakidprotect.com.
The spam all claims someone is “searching for…” at their website. The only thing is, the email address is associated with a fake name I gave while testing a website on behalf of a client. I know what website received the data and I know what other data was provided during the signup process. I also know that the privacy policy at the time said that my data would not be shared and that only the company I gave the information to would be sending me email.
Just more proof that privacy policies aren’t worth the paper they’re written on. But that’s not my real issue here.
The real issue is that I am receiving mail that is clearly deceptive. The subject lines of the emails up until yesterday were “(1) New Message – Someone Searching for You, Find Out…” Yesterday, I actually clicked through one of the messages to confirm that the emails were ending up at mylife.com. After that, the subject lines of the emails changed to “(1) New Person is Searching for You.” I don’t know for sure that my click has caused the change in subject lines, but the timing seems a bit coincidental.
It’s not that someone, somewhere gave mylife.com bad data, or that someone typed a name into the mylife.com search engine and the mylife.com database showed that name and my email address were the same. Neither this name or this email address show up in a google search and I can say with certainty that this is a unique address and name combination given to a specific website. Therefore, the subject lines are clearly and demonstrably lies.
The spams are also coming from different domains and advertising links in different domains. The content is identical, the CAN SPAM addresses are identical. While the court may not rule this is deceptive under the rules of CAN SPAM, it certainly is an attempt to avoid domain level spam filters.
Who are mylife.com? Well, their website and the CAN SPAM address on their spam claims they are the company formerly known as reunion.com. I’ve talked about reunion.com here before. They have a history of harvesting addresses from users address books. They were sued for deceptive email practices under California law, but won the case just recently. They seem to think that the court case was permission to send deceptive email and have thus ramped up their deceptive practices.
If you are a legitimate email marketer, there are a couple take home messages here.
1) Spammers send mail with different domains, from different IP addresses, that contain identical content, landing pages and CAN SPAM addresses. Legitimate marketers should not rotate content and sends through different domains or different IP addresses. Pick your domain, pick your IP and stick with it.
1a) Spammers use randomly chosen domain names and cycle through domains frequently. Legitimate marketers must not use unrelated domains in marketing. Use a domain name that relates to your product, your industry or you.
2) Spammers send mail with deceptive subject lines. Legitimate marketers should make sure their subject lines are clear and truthful.
3) Spammers send mail in violation of the privacy policy under which information was collected. Legitimate marketers should be very careful to handle data in accordance with their privacy policies.
That’s what spammers do. Is that what you do?

Subscribers notice what marketers do

Stephanie Miller from RP blogs at EEC about a UK consumer survey with the take home message: “consumers notice what email marketers do. When we send something interesting and relevant at a good pace, they are happy to stay active with our programs. When we don’t… well, then we’ve lost them, perhaps for good.”

Measuring open rate

In this part of my series on Campaign Stats and Measurements I will be examining open rates, how they are used, where they fail and how the can be effectively used.
There has been an lot written about open rates recently, but there are two posts that stand out to me. One was the EEC’s post on renaming open rate to render rate and Mark Brownlow’s excellent post on what open rate does and does not measure. I’ve also weighed in on the subject.
Overall, I find open rates to be a very frustrating metric. Some senders, particularly those relatively new to email marketing, are so sure they know what open rate is and what it means, that they don’t take any time to actually understand the number. While the name “open rate” seems self explanatory, it’s actually not. Open rate is actually not a measure of how many recipients open an email. However, there are times where open rate is a useful metric for measuring a marketing program over time.
What is an open?
If asked, most people will tell you that open rate is the number of emails that were opened by the recipients. The problem is that this isn’t actually true. An open is counted when a tagged image in an email is rendered by the recipient’s email client. Not all mail clients render images by default, but the emails are still available for the recipient to read. If a user clicks on a link in an email that has not had an image rendered, some ESPs count that as an open as well as a click. In other cases, visiting a link in an email with no image rendered is just a click, no open is recorded.
What is the open rate?
Open rate is generally the percentage of email opens divided by some number representing the number of emails sent. Many senders use the number of emails sent minus the number of bounced emails, others use just the number of emails sent without factoring in the number of emails bounced.
Open rate is a secondary metric. While it does not measure the success, or failure, of a campaign directly, it can be used as a indicator for campaigns. Many people use open rate as a metric because it’s easy to measure. Direct metrics, such as clicks or average purchase or total purchase, may take days or even weeks to collect and analyze. Open rates can be calculated quickly and easily.
What the open rate isn’t
Open rate is not a measure of how many people opened a mail. It is not a measure of how many people read a mail. It really only records that an image in a particular email is loaded and, sometimes, that a link was clicked on. Open rates can be wildly different depending on how the sender measures opens and how the sender measures sends.
What senders use open rates for
To compare their open rates with industry averages
As I talked about above, this use of open rates is problematic at best. You cannot compare numbers, even when they have the same name, if the numbers were arrived at using different calculations. Open rate is not open rate and unless you know the underlying algorithm used you cannot compare two open rates. This is a poor use of open rate.
As a metric for advertising rates
Since a sender can manipulate the open rate by using different calculation methods, this is a good metric for the advertiser to use. It is not so great for the purchaser though, who is at the mercy of the sender’s metrics. There are contractual ways a purchaser can protect herself from an unscrupulous marketer, but only if she understands how open rate can be manipulated and takes steps to define what open rate is in use.
To judge the success of campaigns over time
A single open data point doesn’t mean very much, however, using consistently measured open rates a sender can measure trends. Open trends over time are one area that open rates can help senders judge the success, or failure, of a marketing campaign.
As one metric in A/B testing
Comparing open rates in A/B testing gives some indication of which campaigns recipients may be more interested in. As with trends over time, the lone measurement isn’t useful, but as a comparative metric, it may provide senders with insight into a particular mailing.
To judge the engagement of recipients
Over the long term, recipients who do not interact with a mailing become dead weight on the list. Too many non responders can hurt a sender’s reputation at an ISP. List hygiene, in the form of removing people who never open or click on an email, is an important part of reputation management.
As metrics for email campaigns go, open rate is limited in what it measures about an email campaign. However, as a quick way to measure trending or do head to head comparisons it is a useful metric.

Campaign stats and measurements

Do you know what your campaign stats mean? Do you know what it is that you’re measuring? I think there are a lot of emailers out there who have no idea what they are measuring and what those measurements mean.
The most common measurement used is “open rate.” There’s been quite a bit of discussion recently about open rates, how they’re calculated, and is there a better way. In my own opinion, open rate can be useful, but only in some circumstances. More often it is a distraction from real measurements
Not only has there been the recent discussions about “open rate” versus “render rate” and a lot of confusion among people about what the underlying issues are, but I’ve also been working through some campaign stats questions with other people that indicate maybe they don’t actually understand the numbers they’re using.
For instance, what do the delivery statistics reported by the various mailbox monitoring companies mean? If you have 100% inbox delivery as measured by the program, does that mean all your mail has reached the recipient’s inbox?
What about bounce rates? Everyone says “keep them low” but what does low mean? How do you measure them?
Over the next few posts, I’ll talk about the different stats and measurements in common use and what they do and don’t mean.

Deliverability versus delivery

Deliverability is a term so many people use every day, but what do we really mean when we use it? Is there an accepted definition of deliverability? Is the concept different than delivery?
At a recent conference I was running a session talking about email delivery, senders and the roles senders play in the email industry and at that particular organization. The discussion went on for a while, and the subject of deliverability versus delivery came up. J.D. Falk had a comment about the difference that resonated with me. Paraphrased, he said:

Delivery can be counter-intuitive

We all know that receiving ISPs rate limit incoming email. With the volumes of mail that they’re currently dealing with they must do that in order to keep their servers from falling over.
A client was dealing with rate limits recently. These were not typical rate limits, in that the recipient ISP was 4xxing mail. Instead, the recipient ISP was not accepting any incoming connections. The client was having a bit of a difficult time understanding what was happening and why the problem wouldn’t be solved by increasing the rate at which they were trying to send to the ISP.
Imagine if you will, that at every ISP there is a reception desk that manages the incoming calls. The receptionist is under orders from the to limit the number of calls coming in. When the phone rings, the receptionist can do any of the following:
1) answer the phone and put the call through (250 message accepted)
2) answer the phone and put the caller on hold (connection hangs or delivery is slow)
3) answer the phone and tell the caller to call back later (4xx message deferred)
4) fail to answer the phone (no connection at all)
The delay the client was seeing was #4, in that they were attempting so many connections at once that the ISP was just not answering.
In this case, reducing the number of connections attempted worked. The “receptionist” was not so overwhelmed by the number of ringing lines that she was able to actually answer all the calls and put them right through.
While lowering the rate at which the client was attempting to delivery seems counter intuitive to getting improved delivery, because we understood the mechanism we could lower rate and get an increase in delivery.

Email is store and forward

Many of us are so used to email appearing instantaneous, we forget that the underlying protocol was never designed for instant messaging. When the SMTP protocol was originally proposed it was designed to support servers that may have had intermittent connectivity. The protocol allowed for email to be spooled to disk and then sent when resources were available. In fact, almost everyone who was around more than 10 years ago knows of a case where an email took weeks, months or even years to deliver.
These days we’re spoiled. We expect the email we send to friends and relatives to show up in their mailbox within moments of sending it. We expect that sales receipt or e-ticket to show up in our mailbox within instants of a purchase. We expect that our ISPs will get us email immediately, if not sooner.
But there are a lot of things that can slow down email delivery. At several points in the process an email may be spooled to disk. It stays on the spool until the next part of the delivery process can happen. Other points of slowdown include the various anti-spam, anti-virus and anti-phishing protections that ISPs must implement. Then add in the extreme volume of email (around 10 billion messages a day) and all of a sudden email delivery is slower than many senders and recipients expect it to be. This delay is not ideal, but the system is designed so that mail is not silently discarded.
While individual emails may be delayed, most users will rarely see that delay in the email that they send. Bulk senders, who may be sending thousands or hundreds of thousands of emails a day, may see more delays in a single send than the average user sees in years of sending one-to-one email.
Email is store and forward, not instant. Sometimes that means there is a delay in getting email into the recipients inbox. And, sometimes there isn’t anything anyone can do to speed up delivery, except to adjust expectations of how email works.

Delivery lore

(Image from Bad Astronomy)
Almost every delivery consultant, delivery expert or deliverability blog offers their secrets to understanding spam filters. As a reader, though, how do you know if the author knows what they’re talking about? For instance, on one of the major delivery blogs had an article today saying that emails with a specific subject line will not get past spam filters.
This type of statement is nothing new. The lore around spam filters and what they do and do not do permeates our industry. Most of the has achieved the status of urban legend, and yet is still repeated as gospel. Proof? I sent an email with the subject line quoted in the above blog post to my aol, yahoo, gmail and hotmail accounts. Within 3 minutes of sending the email it was in the inbox of all 4 accounts
I can come up with any number of reasons why the email ended up in my inbox, rather than being caught by spam filters as the delivery expert originally claimed. But none of those reasons really matter. The expert in question is spreading delivery lore that is demonstrably false. Emails with that subject line will get through spam filters. I even added an extra 4 exclamation points in the subject line.
Not all delivery lore is true. In fact, most lore involving “always” “all” “never” or “none” is not going to be true. Just because you read it on the internet, and because it came from someone claiming to know what they’re talking about does not absolve individual senders from critically thinking about the information.

Soft bounces and rate limiting

What is your policy for handling soft bounces? What do you consider a soft bounce? What is the right thing to do about soft bounces?
The first step in talking about soft bounces is to define them. When I talk about soft bounces, I mean mail that has been rejected with a 4xx response during the SMTP transaction. As described by RFC5321, when a recipient MTA responds with a 4xx it is telling the sending MTA “Wait! I can’t take this mail right now. Come back a little later and try again.” The sending MTA will then continue to attempt to deliver the message until either it is delivered or until it hits the max delivery time, usually 3 – 5 days.
In a well behaved and RFC compliant MTA, messages that have reached the maximum time without delivery due to 4xx rejections will be converted to permanent rejections (5xx). With a correct MTA, this means too many emails in a row timing out shoud result in an email address being removed from future mailings.
For a number of reasons some ISPs, notably Yahoo, are using 4xx responses to slow down mail from some senders. Many senders treat this as a inconvenience and a frustration and try to figure out how to get around the rate limiting. The UK DMA published an article on soft bounces with the following words of wisdom.

Palpable ennui

Put any group of senders together and the conversation invariably turns to discussions of how to get email delivered to the Inbox. There is an underlying flavor to most of these conversations that is quite sad. Many senders seem to believe that the delivery of their email is outside of their control and that since the ISPs are difficult to reach that senders are stuck. The ennui is palpable.
I am here to tell you that nothing could be further from the truth!
Senders are not passive victims of the evil ISPs. In 99% of cases, delivery problems are fully under the control of the sender.
Mail being deferred? Mail being blocked? Mail being delivered to the bulk folder? Senders do NOT NEED TO CALL THE ISP to fix most of these. Tickets do not need to be opened nor do personal contacts need to be employed. You can resolve the vast majority of problems with data you already have.

Negative branding, part 2

Last week I commented on negative branding in email. One of the comments on that post was an advertisement for a company called WrapMail. In the course of attempting to determine if this was spam or a real comment, I checked out their website. While the comment itself may not be spam, and it may not be providing services to spammers, the entire business model strikes me as a delivery nightmare.
Briefly, once you sign up with this company, you set your mail client to use their SMTP server. As all of your mail goes through their server is it “wrapped” with a HTML template of your choosing. All of your email is now branded with that template, allowing you to formally advertise your business even during the course of standard business communications.
There are multiple ways this can negatively impact a specific brand.

List hygiene

Bronto blog has step by step directions on how to run a successful re-engagement campaign.

Personal contacts at ISPs

A lot of senders seem to think that the secret to good delivery is having personal contacts at the ISPs. That way, when there is a delivery problem you can call up your friend at the ISP and inform them that they have made a mistake. In this little sender fantasy world, the ISP rep then apologizes profusely, unblocks the sender’s mail and perform magic to prevent a block from ever happening again.
Like many fantasies, it doesn’t usually happen that way.
The big ISPs are moving more and more to automated systems that prevent individual employees from interfering. This isn’t actually anything new. I was at a party once and sharing a drink with a representative of one of the big three ISPs. We were talking about a delivery problem one of my clients was having. The rep told me that the ISP did not have any way to actually whitelist around the filter that this client was getting trapped in. The reputation based filtering systems that some ISPs are building are much more performance based and will probably result in those ISPs who can make exceptions now not being able to do so in the future.
When looking for a good delivery person, the real question senders should be asking relate to the skills of the people doing the troubleshooting not who do they know. Does the delivery person have experience troubleshooting delivery? Can they actually resolve problems without having to rely on information from the ISPs? Given the response times at many ISPs, even for personal contacts, it’s often faster to listen to your delivery person than find the ISP rep who will apologize for the mistake.

Following the script

Yesterday I talked about breaking through the script in order to escalate an issue. I briefly mentioned that I always start out following the script and using the channels ISPs have provided. There are a number of reasons to do this all of which benefit you, the sender.
First off, when you use the designated communication pathway at an ISP there is a record of your contact. There are procedures in place to make sure your communication is addressed and you get a response. When you’re escalating to an individual, you’re using their communication channel. IMs get lost, email ends up buried in the pile, other things come up and a week later you’re still waiting for your answer.
Secondly, when you use the designated communication pathway at an ISP your contact is logged and tracked. This means that if the person you’re used to dealing with gets another job, moves on or otherwise isn’t able to communicate with you any longer you have a history with that ISP. The next person to move into the position and deal with issues can see that you’re a legitimate sender with a history of dealing fairly and professionally with ISPs.
Thirdly, handling direct and personal escalations are often outside the official job description the people directly contacted. This means that when they come up for review, the work they’re doing for people who won’t use channels is not as important as the other work they do. Sure, they may get some credit for helping people with problems, but they may not get the review they should get. This hurts not just the senders who believe they shouldn’t have to follow channels but also those of us who do follow channels, particularly in the current business climate. Do you really want to lose that awesome person you use because some dork thought they were too good, too important to use the provided form and that awesome person lost their job because they didn’t meet their official work goals?
Fourth, you’re not the only one escalating. I had the opportunity to visit my friend Anna from AOL a few years ago. One morning both of us had to actually get some work done, so we were parked in her living room on laptops. I was astonished at the number of IM windows she was juggling constantly. We’re talking 20 – 30 separate windows open at once, many of them troubleshooting sender issues. After seeing that I do as much as possible through the official channels that AOL has provided. She is my friend, and a very good one, and I still avoid using her as a contact point unless there is some emergency.
Remember this next time you are searching for that email address of the person from that ISP that’s currently blocking your mail. Use the official communication channels where possible, and always use them first. Using back channels for issues where the intended workflow works causes a lot of overhead and doesn’t scale at all well.

Breaking through the script

In handling day to day issues I use the ISP designated channels. This means I frequently get dragged into long conversations with people, probably outsourced to the far east, who can do nothing beyond send me a boilerplate.
This can be a frustrating experience when the issue you’re trying to deal with is not handled by the script. Generally, by the time someone has come to me for help, they are “off script” and I do need to actually talk to a human to get resolution.
With Hotmail, I’ve found that persistent repeating of very simple phrases will eventually get the issue kicked up to someone who can respond with something beyond another boilerplate. This can take days, but it is possible.
I’ve recently run into a Yahoo issue where I am trying to punch through the script, but have so far been unable to.
One of the services Word to the Wise offers is whitelisting. I collect info from customers, verify that what they’re doing will get them whitelisted at the ISPs that offer it, and then submit the information to the ISPs. Yahoo has recently moved to an online submission form for their whitelisting process, which is great for me. No more creating a giant document and then cutting and pasting the document into an email and then mailing it off.
The problem is, there seems to be a minor problem with the Yahoo Whitelisting submission form. When submitting an online application to Yahoo, they respond with a message that says “this application is not complete.”
I’ve been attempting to break through the script in order to find out what about the application is not complete. The webform has data checking, and you cannot submit a form while leaving any of the questions blank. Asking “what is wrong” when the application is kicked back has resulted in me having multiple copies of the whitelisting submission form.
It’s gotten so frustrating that I’ve escalated to personal contacts, but they can’t explain what’s not complete about the application as submitted online, either.
Has anyone had any success breaking through the Yahoo script? Has anyone managed to get IP addresses whitelisted through Yahoo using the online form?

Opt-in is dead! Long live opt-in.

This week there has been an ongoing discussion on one of my mailing lists about spam, definitions, use of the this-is-spam button and permission. One small part of the discussion centers around the definition of spam. Is spam unwanted email or is it unasked for email? If a sender doesn’t have permission from recipients to send mail, but the sender manages to keep their reputation at major ISPs clean enough to get good delivery, is the sender doing anything wrong?
My own answers really sidestep the question. I believe spam as a term is so ill-defined as to be meaningless. I try not to use “spam” when describing any mail. Even mail that comes to my own personal spamtraps and spammer-created email addresses, if I’m describing it to people I try to be more descriptive than just “spam.”
Coincidentally, Ken had an article in his newsletter today pointing out that the term opt-in has been rendered meaningless. First, he points out that all lists should be opt-in. Marketers shouldn’t have to describe a list as opt-in, it should just be opt-in. He then goes on to tell a story about an email marketer, who bought or purchased a list with Ken’s email address on it. The marketer claims the mail is opt-in, but it’s not. Ken ends the article with some advice to marketers.

Google Apps – where's my abuse@

Most ISP feedback loops require you to demonstrate that you’re really responsible for your domain before they’ll start forwarding reports to you. The usual way that works is pretty similar to a closed-loop opt-in signup for a mailing list – the ISP sends an email with a link in it to the abuse@ and postmaster@ aliases for your domain, and you need to click the link in one or both of the emails to continue with the feedback loop signup process.
That’s mostly there to protect you, by making sure that someone else can’t get feedback loop messages for your domain. And it’s not too difficult to do, as you should already have an abuse@ and postmaster@ alias set up, and have someone reading the abuse@ alias.
But maybe you’re using Google Apps to host your corporate email, and that’s the domain you need to use for your feedback loops. So you go to create abuse and postmaster users, but it won’t let you – you just get the error Username is reserved for email list only. Uhm, what?
Google want to police use of domains hosted on their service, so they automatically set up abuse and postmaster aliases for your domain, and any mail sent to them is handled by Google support staff. You may well be happy with Google snooping on your abuse role account, but you really need to be able to read the mail sent to it yourself too.
So what to do? Well, the way Google set things up they actually create invisible mailing lists for the two role accounts, and subscribe Google Support to the lists. In older versions of Google Apps you could make those mailing lists visible through the user interface by trying to create a new mailing list with the same name, then simply add yourself to the mailing list and be able to read your abuse@ email.
But Google broke that functionality in the latest version of the Google Apps control panel, when they renamed email lists to “groups”. If you try and create a new group with the email address abuse@ your domain you’ll get the error Email already exists in this domain, and no way to make that list visible.
So, what to do?
Well, there’s a workaround for now. If you go to Domain Settings you can select the “Current Version” of the control panel, rather than the “Next Generation” version. That gives you the old version of the control panel, where all this worked. Then you can go to User Accounts, create a new email list delivering to abuse@ and add one of your users to the mailing list. You can then set the control panel back to “Next Generation” and have access to the mailing lists via Service Settings → Email → Email Addresses.
Hopefully Google will fix this bug, but until they do here’s the step-by-step workaround:

Feedback loops: net benefit or net harm?

There has been a very long, ongoing discussion on one of my mailing lists about whether or not feedback loops are a net good or a net harm. I believe, overall, they are a net good, but there are people who believe they are not. The biggest objection is that the lawyer mandated redaction of the To: address combined with the fact that some users use the “this is spam” button to delete unwanted email, makes it difficult for some FBL recipients to sort out the real issues from the cruft.
Redaction can be a problem for some senders, particularly for the small mailing list hosted as a hobby or contribution to the community. In order to effectively deal with FBL emails, a sender needs to have tools on the email sending side and on the FBL receiving side. This is often more overhead than the volunteer list maintainer wants to handle. Unfortunately, these senders are a minority and therefore their issues are often not addressed by the ISPs.
Some of the objections and complaints about “broken” or “useless” FBLs come from people who do not really have any history for the FBLs, where they are, what they were designed for and who their target audience is. A bit of history may help explain why things are how they are.
The First FBL
The “this is spam” button evolved from the “notify AOL” button. This button was a way email recipients could notify AOL staff about any number of problems, including threats, viruses and other unwanted emails. As time went on, this was changed to “this is spam” to encourage users to report more spam so the AOL would have the data to make delivery decisions. Eventually, AOL made the decision to share that data with some senders and ISPs. The lawyers made the decision to redact the “To:” address, but not make any other changes to the message because they believe they should not be sharing subscriber email addresses with third parties. As some people correctly point out, the lawyers are not interested in hearing from non lawyers about changing this. It is possible that another lawyer may be able to put together a position paper and convince them this stance is overly cautious. I am pretty sure, though, that no one without a legal degree will be given any audience from them.
Given the success of the AOL FBL and the demand from both ESPs and ISPs for FBLs, other ISPs started offering FBLs as well. Many of them also redacted the To: address, either just following AOL’s lead or under advice of their own counsel.
That means, as senders, we are in a situation where we really cannot make the ISPs change what they’re doing. We can either adapt our own mailing practices to cope with them or we can forego the data provided by the FBL. One of the challenges in choosing to shun the whitelist at AOL that in order to qualify for whitelisting, you have to accept a FBL. For ISPs, who want to whitelist their outgoing MTAs, but have customers sending mail, maybe running small mailing lists, or who are forwarding mail to their ISP account, this can be a problem. However, any ISP needs some sort of abuse desk automation, and this automation should be able to handle FBLs. This can also be a problem for small ESPs or companies doing in-house email marketing. They buy something off the shelf to handle mail (or install mailman) that does not do VERP or otherwise enter the specific address in the email. When faced with a redacted email they cannot do anything with the complaint.
What does the FBL email tell the FBL recipient?
This really depends on what role the FBL recipient plays in the mail transport system. Bandwidth and network service providers use the FBL as an aggregate tool. They really only deal with FBL complaints if there is a change in complaint volume about an IP, they don’t treat each complaint as a valuable source of information. Typically what happens is that an ISP abuse desk notices a spike in complaints. After investigation, they may discover that a customer machine is compromised. They then notify the customer, the customer patches or disconnects the machine and the problem is fixed.
ESPs tend treat the FBL as an unsubscribe mechanism as well as a way to monitor customers. A few FBL complaints are not necessarily a sign that the sender is spamming, but once a threshold is reached the ESP delivery / abuse team addresses the issue. Spammers can get FBLs and often use them as a way to clean lists of complainants. Some really dirty spammers even suppress those complainants from all their lists.
Is a FBL useful?
This is really something that someone else cannot tell you. Some companies find FBLs to be extremely useful, even after they have had to make investments in software (either off the shelf or custom) to send mail that will survive the FBL redaction process and to handle the actual FBL email. Some companies find the FBLs to be more trouble than they are worth. The question, however, is really one only the sender can answer.
Overall, I think FBLs are more helpful than they are harmful. They do require investment on both sides of the transaction, but does encourage senders and receivers to cooperate with one another.

McColo goes offline

Last week a major player in the botnet arena was taken offline when they were shutdown by their upstream provider. With the demise of McColo, there has been a 30 – 50% drop in the amount of spam as measured by any number of different techniques. The CBL team has posted an article about their view of the McColo disconnection, which includes links to press articles about the shutdown. Spamhaus has their own take on the shutdown and another collection of links to articles about the shutdown.
In my own mailbox, I have noticed a drastic decrease in the amount of spam over the last week. I am too jaded to expect that the change is permanent, but it is nice while it lasts.

Don't do this

I recently received an email from someone I do not know. This email was welcoming me to the friends and family beta of a new website. This email got under my skin a bit and it has been one of those weeks and so I decided to reply to the email.
“Whomever sold you this email address lied to you,” says I. I did not point out all the reasons I know this, including the two @home.com addresses in the To: line next to mine, just stated that as a fact.
The sender replied telling me he did not purchase any email addresses, he just mailed the contents of his address book. At that point, I did a little poking around the web to see if I recognized the sender or we had worked together in the past or if there was a clear join between him and me. I could not find anything that triggered a memory in my mind, so I replied again. “Do you know where we met? I am not placing you.”
He finally replied, “Oh, you’re in my address book under Spamhaus. We must have interacted while you were working there.”
Please, please, dear readers, when you next launch your online business do not mail everyone in the address book you have been maintaining for the last 10 years. If you want to mail your friends and family, then do so. But just because you have an email address does not mean that the recipient wants to hear from you. And, really, mailing the folks you think work for Spamhaus? Not smart. Had I really been a Spamhaus employee, chances are his bright, shiny new company would be blocklisted before it ever had a chance.

AOL and DKIM

Yesterday, on an ESPC call, Mike Adkins of AOL announced upcoming changes to the AOL reputation system. As part of these changes, AOL will be checking DKIM on the inbound. Best estimates are that this will be deployed in the first half of 2009, possibly in Q1. This is something AOL has been hinting at for most of 2008.
As part of this, AOL has deployed an address where any sender can check the validity of a DKIM signature against the AOL DKIM implementation. To check a signature, send an email to any address at dkimtest.aol.com.
I have done a couple of tests, from a domain not signing with either DK or DKIM, from a domain signing with DK and from a domain signing with both DK and DKIM. In all cases, the mail is rejected by AOL. The specific rejection messages are different, however.
Unsighng domain: host dkimtest-d01.mx.aol.com[205.188.103.106] said: 554-ERROR: No DKIM header found 554 TRANSACTION FAILED (in reply to
end of DATA command)
DK signing domain: “205.188.103.106 failed after I sent the message.
Remote host said: 554-ERROR: No DKIM header found
554 TRANSACTION FAILED”
DK/DKIM signing domain: “We tried to delivery your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 554 554-PASS: DKIM authentication verified
554 TRANSACTION FAILED (state 18).”
As you can see, in all cases mail is rejected from that address. However, when there is a valid DKIM signature, the failure message is “554-PASS.”
As I have been recommending for months now, all senders should be planning to sign with DKIM early in 2009. AOL’s announcement that they will be using DKIM signatures as part of their reputation scoring system is just one more reason to do so.

Two reasons not to buy a list

Ken Magill, celebrity, has two articles today that highlight the issue with buying lists from vendors. The first is yet another article about EmailAppenders selling bad data. In this case, it is not the buyers who are complaining. According to Ken EmailAppenders are sending out email advertising they can sell Internet Retailer’s list of 2008 conference attendees. Internet Retailer is disputing this and has sent EmailAppenders a cease and desist. EmailAppenders is currently dodging Ken’s attempts to get their side of the story.
The second is an article about Zoominfo, a new group in the list selling business. Zoominfo has long been harvesting information from other sites. Now, they are offering to sell their scraped and harvested list. Their only requirement is that the buyer sign an agreement to comply with CAN SPAM. And, yes, if someone is dumb enough to buy this harvested list, they should comply with CAN SPAM as sending mail to a harvested list triggers additional penalties if or when the FTC decides to go after the sender.
Not only are Zoominfo harvesting data, they are harvesting from ancient and obscure sources. They have no current information for me, but they managed to find an email address for a job I left in 1993. They have Steve listed as an employee of “postgreSQL INC” because they harvested the postgres mailing list archives. Mickey pointed out one of Zoominfo’s sources is http://free-personal-ads-wanted-sex-partner-near-hays-kansas.themasterwithin.ca/arch/4/. You do not even need to visit that site, just look at the URL!
Zoominfo’s VP and general manager claims they send emails to people regularly, offering them the chance to opt-out. First of all, I have never received one of these, have any of you? Secondly, some of the addresses are so old opt-outs are not relevant. Finally, unless they are monitoring their delivery, which I strongly doubt given their business model, anyone buying addresses from them is going to buy lots of dead addresses. And spamtraps. Lots of spamtraps.
I am sure that people who buy and sell lists regularly will tell me that these are outliers and that most companies who sell lists have higher data collection standards. My experience suggests that these are middle of the road list brokers. They are companies who are willing to sell anything with an @ sign in it and do not care about how sending to that data affects their customers.

Bad Idea

My mailbox and IM windows have been swamped with messages about an ISP sending out mail to participants in their FBL program. It seems this particular ISP could use some delivery consulting.
See, this ISP sent out emails with blocks of 50 – 75 email addresses in the To: line. Bad idea. Delivery wise, I do not expect that they had many delivery problems. In the copy I saw, most of the addresses started with “support” and those addresses often have fewer blocks on them than other addresses. I also do not think this ISP will see subscriber emails blocked because of this.
However, it is very, very bad practice to do what they did. And the fact that the ISP can get away with it does not mean that any other mailer can get away with it.

Reputation: part 2

Yesterday, I posted about reputation as a combination of measurable statistics, like bounce rates and complaint rates and spamtrap hits. But some mailers who meet those reputation numbers are still seeing some delivery problems. When they ask places, like AOL, why their mail is being put into the bulk folder or blocked they are told that the issue is their reputation. This leads to confusion on the part of those senders because, to them, their reputation is fine. Their numbers are exactly where they were a few weeks ago when their delivery was fine.
What appears to have changed is how reputation is being calculated. AOL has actually been hinting for a while that they are looking at reputation, and even published a best practices document back in April. Based on what people are saying some of that change has started to become sender visible.
We know that AOL and other ISPs look at engagement, and that they can actually measure engagement a lot more accurately than sender can. Senders rely on clicks and image loading to determine if a user opened an email. ISPs, particularly those who manage the email interface, can measure the user actively opening the email.
We also know that ISPs measure clicks. Not just “this is spam” or “this is not spam” clicks in the interface, but they know when a link in an email has been clicked as well.
I expect that both these measures are now a more formal and important part of the AOL reputation magic.
In addition to the clicks, I would speculate that AOL is now also looking at the number of dead addresses on a list. It is even possible they are doing something tricky like looking at the number of people who have a particular from address in their address book.
All ISPs know what percentage of a list is delivered to inactive accounts. After a long enough period of time of inactivity, mail to those accounts will be rejected. However for some period of time the accounts will be accepting mail. Sending a lot of mail to a lot of dead accounts is a sign of a mailer who is not paying attention to recipient engagement.
All ISPs with bulk folders have to know how many people have the from address in their address book. Otherwise, the mail would get delivered incorrectly. In this way, ISPs can monitor the “generic” recipient’s view of the email. Think of it as a similar to hitting the “this is not spam” button preemptively.
This change in reputation at the ISPs is going to force senders to change how they think of reputation, too. No longer is reputation all about complaints, it is about sending engaging and relevant email. The ISPs are now measuring engagement. They are measuring relevancy. They are measuring better than many senders are.
Senders cannot continue to accrete addresses on lists and continue sending email into the empty hole of an abandoned account while not taking a hit on their reputation. That empty hole is starting to hurt reputation much more than it helps reputation.

Reputation

Reputation is the buzzword in delivery these days. Everyone talks about building a good reputation and how to do it. Makes sense, the ISPs are always hammering on reputation and how critical reputation is. The more I talk with delivery folks on the ESP side of thing, the move I realize that there is a fundamental disconnect between what the ESPs mean when they say reputation and what the ISPs mean when they say reputation.
Many people handling delivery think that the bulk of reputation is wrapped up in complaint rates and bounce rates. I think they know the ISPs measure more than just complaints and bounces (spamtraps!) but really believe that most of developing a good reputation is all about keeping those complaints low.
This perspective may have been true in the past, but is becoming less true as time goes on. There are a lot of very smart people managing incoming mail at the ISPs and they are constantly looking for ways to better meet the desires of their customers. Lest we forget, their customers are not the senders, their customers are the end users. Their customers are not senders.
Part of meeting the needs of end users means actually giving them a way to provide feedback. AOL started the trend with the this-is-spam button, and other ISPs (ones that controlled the user interface at least) followed suit. For a very long time, reputation was dominated by complaint percentages, with modifiers for number of spamtrap addresses and number of non-existent users.
The problem is, these numbers were easy to game. Spammers could modify their metrics such that their email would end up in the inbox. In response, the ISPs started measuring things other than complaints, bounces and spamtraps. These other measurements are strong modifiers to complaints, such that mailers with what used to be acceptable complaint rates are seeing their mail end up bulked or even rejected.
Recently, AOL seems to have made some subtle modifications to their reputation scores. The result is mailers who have previously acceptable complaint rates are seeing delivery problems. When asked, AOL is only saying that it is a reputation issue. Lots of senders are trying to figure out what it is that is more important than complaints.
Tomorrow, I will talk about what I think AOL could be measuring.

Evaluating email

DJ posts the top 4 reasons an email campaign fails.

Constituents clog lawmaker mail servers

With the recent credit market turmoil and the proposed 700 billion dollar bail out bill many, many Americans are taking the opportunity to contact their congressional representatives. This increase in traffic has resulted in the house.gov website being slow or unresponsive, the mailservers being clogged and the phone system straining.
In response to the increased load, the CAO has put some limits on incoming email and is restricting the number of e-mails sent via the “Write Your Representative” function of the House website.

The Question

Mark Brownlow has a list of 12 questions every email marketer should ask about their marketing program. Buried in the middle is the most important question for delivery.

The overlooked secret of marketing

Seth Godin posted recently about the overlooked secret of marketing: time

Lashback tackles opt-in fraud

Last week Lashback posted a three part series on opt-in fraud.
One of the issues they commented on is that suppression lists are being passed around and some mailers are actually spamming them. This is something that used to be common, where spammers were harvesting email addresses from opt-out forms and then spamming the addresses or selling them to other mailers. This is why some ISPs and anti-spammers recommend recipients not unsubscribe from mail that they never subscribed to.
In the last few years there has been conflicting data on the prevalence of harvesting unsub or suppression lists. The FTC determined that there was no risk to recipients from unsubscribing. Lashback is now seeing some spam coming into their test addresses.
Overall, there are people who will continue to be suspicious of unsubscribing from mail they do not expect. This will drive up spam complaints and lower delivery. While responsible mailers are not the cause of the negative perception of email, they are competing with the spammers and scammers and sometimes recipients may not draw distinctions. This is why building relationships and trust over email marketing campaigns is such a critical part of delivery.

Results based email marketing

Two articles showed up in my RSS feed in the last 24 articles that touched on different aspects of the same issue. Senders should improve their email marketing program even when they are working well.
Stephanie Miller over at ReturnPath addresses the lost revenue from current programs.

Opt-in Reconfirmation in the Wild

What’s an opt-in reconfirmation email? Also called, as fellow blogger Al
Iverson mentioned lately, a re-engagement email, or a permission pass email.
Al links to DJ Waldow’s write up on Shop.org’s recent re-engagement
strategy, and today I see that Janine Popick, CEO of VerticalResponse,
talking about Coach’s turn at culling their list through this process. What’s interesting here is that, according to Janine, Coach didn’t target this reconfirmation email only at recipients who never open or click. She says she does both, regularly, and received this email message anyway. Another friend of mine, who is also a Coach subscriber, reports to me that she receives regular emails from them (most recently as just about
ten days ago), but that she did not receive this reconfirmation email message.

Getting whitelisted by endusers

One of the best ways to ensure mail is delivered to a recipients inbox is to encourage the recipient to add the senders from: address to their address book. In cases where an ISP might otherwise bulk folder the email, they will instead put the email into the inbox.
Senders are changing their practices to get recipients to add from addresses to address books. There are a number of companies reminding users to add addresses on the webpage at the time of signup. Most emails have recommendations in each email. Recently, there have been multiple reports of companies who send specific email campaigns to encourage recipients to whitelist the sender.
Cool Email Idea: Customized Whitelisting Instructions from ReturnPath.
How & Why You Need to be Added to Your Recipient’s Address Book from VerticalResponse.
In addition to the direct benefit to the recipient that whitelists the individual sender, there are some hints that ISPs are looking at individual whitelisting as part of their internal sender reputation scoring.

Letting Go

Derek Harding has an article over at ClickZ, discussing the importance of letting subscribers go.

Delivery thoughts

I have spent quite a bit of time over the last few days working with new clients who are asking a lot of basic, but insightful questions about delivery problems. I have been trying to come up with a model of how spam filtering works, to help them visualize the issue and understand how the different choices they make change, either for better or worse, their delivery.
One model that I have been using is the bucket model. Spamfilters are like buckets. Everything a sender does that looks like what spammers do adds water to the bucket. Reputation is a hole in the bottom of the bucket that lets some of the water flow out. When the bucket overflows, delivery is impacted.
I think this analogy helps explain why fixing any technical issues is so important. The less water that is added to the bucket by technical problems, the less likely it is that the water from content filters will overflow the bucket.

Strategic email marketing

Mark Brownlow has an interview with Simms Jenkins the author of The Truth about Email Marketing. Well worth a read for anyone who is sending email.

Engaging recipients critical for delivery

One of the issues I have touched on repeatedly is the changing face of blocking and filtering at ISPs. Over the last 12 – 18 months, large, end-user ISPs have started rolling out more and more sophisticated filters. These filters look at a lot of things about an email, not just the content or the sending IP reputation or URLs in the message but also the recipient profile. Yes, ISPs really are measuring how engaged recipients are with a sender and, they are using that information to help them make blocking decisions.
There were two separate posts on Friday related to this.
Mark Brownlow has a great blog post speculating about a number of things ISPs might be looking at when making decisions about what to do with an incoming email. He lists a number of potential measurements, some of which I can definitively confirm are being measured by ISPs.

Preventing subscriber remorse

Mark Brownlow has a great article up about how senders can stop subscribers from regretting they signed up for mail. He starts off saying

Information you should know

MailChimp is using microformats technology to allow recipients to add senders to their address book from the subscription page. All senders should tell recipients what address mail is coming from at the point of subscription and encourage recipients to add the senders to their address books. This new technology simplifies that for the recipient.
Denise Cox posts about a recent conference she attended in London looking at what makes email valuable. She has many good suggestions on how to improve your ROI, but captures the essence of getting a good ROI on mail in 3 sentences.

Images in email

There have been a number of blog posts recently discussing designing emails that look good and inspire clicks even when images are suppressed. Much of this is related to the EEC whitepaper showing that the majority of email users suppress images in emails and the 2008 email design guidelines published by Campaign Monitor.
I happen to be one of those cranky users who suppresses images by default, even in the emails with the cute bunny pictures. Up until about 3 years ago, I was using a mail reader that did not render images (or attachments). It is always interesting to me to watch how senders in general are reacting to the marketplace. Some senders have mail completely incomprehensible without images turned on. Their weekly (or more often!) email is a network of empty boxes on my screen.
For the most part, the emails are useless to me without images, and while I have occasionally loaded images or check the website, the emails are not that much better with images. I stay on the lists now as an informal study of how long it will take the sender to notice I have not opened, clicked or purchased from their emails.

Engaging subscribers

On Friday I talked about using clicks and opens as a way to monitor recipient engagement and dropping recipients who do not interact after a certain period of time. One of the critical parts of this is that a sender must send mail that encourages recipients to click and then actually tracks clicks. If a sender does not send mail that encourages recipients to interact, then using interactions as a way to measure engagement does not work.
For some types of emails this is more difficult than others. One example is newsletters. Newsletters do not always encourage recipients to click through to read the full article and many readers will not click on ads. There are a couple ways around this.
One way is the way used by Ken Magill in his Magilla Marketing Newsletter. In his newsletter there are article slugs to 4 articles, but to see the whole article a reader must click through the link. Ken can quickly see which recipients are actively reading his newsletter by tracking clicks. By writing timely and interesting articles, Ken engages his readers into interacting with his newsletter in a way he can track.
Another way newsletter recipients can be encouraged to interact with the newsletter is demonstrated by the Minnesota Companion Rabbit Society. Their newsletters contain full articles but there have lots of links. These links encourage people to click through to the website to view pictures of cute bunnies, see more stories about the cute bunnies and discover more ways the reader can help save the cute bunnies. Joanna, President of the Minnesota Companion Rabbit Society, says that adding a fun link to emails encourages people to click. She tries to put one fun link (like Howard’s Big Dig) in every email. Give the readers something to pique their interest and they will click through.
For marketing and advertising messages, there are different pressures and partial slugs may not work. Generally, market email must stand on its own and should not rely on readers having to click through to get the full offer. Most marketers also do not have lots of pictures of cute bunnies to encourage readers to click through. Buy our product or the bunny gets it will probably generate a lot of unwanted negative publicity for a company. But there are ways to provide interaction opportunities with recipients in ways that allow senders to track the interaction. Discounts can work, or links to recommended products based on the recipient’s previous purchases. In this case it is not always about getting users to make a purchase, although that is a nice bonus, but just to click on something to indicate they are reading your mail.

Sender complaints about spamfiltering

JD posed a question in my post about Postini and trying to sort out a customer getting marked as spam by their filtering mechanism and I think it bears more discussion than can be done in comments.

Troubleshooting a Postini block

Mail from one of my clients is being filtered at Postini and they asked me to look into this. Not that there is anything that can be done, of course. Even before they were bought out by Google, they were the poster child for a spam filtering company that believed they could do no wrong. It was difficult, if not impossible to get a straight answer from Postini about filtering, and the only statement they would ever make in regards to blocking problems was ‘have the recipient whitelist your mail.’
It is not just that Postini will not talk with people who are blocked, they will not talk to their own customers, either. Many years ago, I was dealing with another Postini issue for a customer. This customer was a Postini customer and was sending mail to themselves to test their new ESP. Postini was blocking the mail and the customer wanted me to find out why. After a couple days of digging I did actually find a really-o truly-o human at Postini. [1] He explained to me that a single line of text, followed by an unsubscribe link was spam, always spam and nothing but spam. He also explained that the only way for that mail to be let through, was for my customer to turn off his Postini filters.
Fast forward 4 years and I once again have a customer blocked by Postini. Usually, I tell customers there is nothing to be done for Postini blocks and that no one can find any information about them, but this customer is insistent. This particular customer has extremely clean mailing practices, sends highly relevant and wanted mail and consistently gets 95+% inbox delivery. They are not spammers, not even a little bit. Because I know this customer is so clean, I poked around a little to find some information about them. They do use the ReturnPath Mailbox Monitor so I have a copy of the headers Postini is adding. I also discovered that Postini is now providing a decoder service for their headers at https://www.postini.com/support/header_analyzer.php
The response you get back from pasting in a header is not that useful if you have found any of the numerous explanations of Postini headers, but it does show some willing. Note, there is no way to ask a question or provide feedback to Postini on the listing.
There is not much that can be done to deal with Postini filtering your email. The best you can do is have your recipients whitelist you.
[1] I believe I am the only person on the delivery end that has ever been able to actually talk to a live human at Postini, and I think that is only because I called them from the same area code they are in and some engineer decided to return the message I left on their corporate voicemail.

Categories of email

The question came up on a mailing list about how senders classify email. Steve came up with the following list of email types from the recipient (not sender) perspective

More about FBLs and unsubscribes

In the comments of the last post, Gary DJ asked an insightful questions and I think my answer probably deserves a broader audience.

FBLs, complaints and unsubscribes

On one of my mailing lists there was a long discussion about the Q Interactive survey. Some of the senders on the list were complaining that unless ISPs provide FBLs they should not use complaints to make filtering decisions. The sender perspective is that it isn’t fair for the ISPs to have data and use it without sharing it back so that the senders could remove complainers.
This deeply, deeply misses the point.
The ISPs are in the business of keeping their users happy. Part of that is measuring how users react to mail. This includes providing “report spam” or similar buttons when they control the interface. Some ISPs have chosen to share that data back with senders. Some ISPs have made the choice not to share that information back.
But even the ISPs that share FBL data with senders do not expect that the only thing a sender will do is remove the email address. ISPs expect senders to actually pay attention, to not send mail that their recipients do not want. They expect that ESPs are going to notice that one customer has consistently high complaint rates and actually force their customer to stop sending mail that recipients think is spam.
Senders should keep track of complaint rates. Measure them per send. Do not waste time whining that this ISP or that ISP will not set you up with a FBL. Take the data from those ISPs that do have FBLs and measure it. It is extremely unlikely that a mailing will have grossly different complaint rates between ISPs. You have all the data you need in order to evaluate how your recipients are perceiving your email.
ESPs and senders who think that their only response to FBL complaints should be to remove that email are the ones most likely to have filtering and blocking problems. The ISPs are giving them valuable data that they can use to evaluate how their emails are being received. Instead of being ungrateful, wagging fingers and blaming the ISPs for not giving them the data they want, senders should spend more time focusing on what they can discover from the data that is shared with them.
A FBL email is more than an unsubscribe request, senders should stop focusing on the unsubscribe portion of the FBL process and focus more on the recipient feedback portion of it. What can you learn about your mail from a FBL?

Report spam button broken

Q Interactive and Marketing Sherpa published a press release today about how fundamentally broken the “report spam” button is. They call for ISPs to make changes to fix the problem. I think the study on recipient perceptions is useful and timely. There is an ongoing fundamental paradigm shift in how ISPs are handling email filters. ISPs are learning how to measure a senders collective reputation with end users, and, more importantly integrate that reputation into the equation used to determine how to filter and deliver incoming email.
Q Interactive and Marketing Sherpa acknowledge this change in the report:

How much mail?

Yesterday I had a call with a potential new client. She told me she had a list of 4M Yahoo addresses and she wanted to mail them twice a day. Her biggest concern was that this volume would be too much for Yahoo and her mail would be block solely on volume. As we went through the conversation, she commented that this list is also being used by someone else she knows and they were getting inbox delivery at Yahoo on every mailing.
From other bits of the conversation, I suspect that these are not the only two people using this list, but I have no feel for the volume. But how much email is each person on that list receiving a day?
I have a current client who is in a similar field to the above potential client. I signed up for their list back in December. Since then I have received 1728 emails to the address I used on their site. 4 of those emails have actually been from my clients, the rest were stolen by a partner of theirs and sold off to all sorts of mailers. Yesterday I received 40 emails.
I just cannot see how this is a valid, long term business model. The bulk of these mails are advertising payday and other kinds of loans. Some of them are duplicate offers from the same senders (judged by CAN SPAM addresses) using different From: lines. The mailbox these mails are filtered into is completely useless, it has been swamped by loan offers. I cannot imagine that anyone, even someone looking for a loan, is receptive to this much email. The only thing I can figure is that the mailers believe that if their email is the one at the top of the mailbox at the exact moment the recipient gets most desperate for money in their bank account tomorrow they will make the sale and get paid.
This model is going to be less and less viable as time goes on.
On the permission level, there really is no permission associated with that email address. Sure, I could call up the former client of mine who mailed that address today and challenge them to show me where they got the address and they would probably tell me they bought it from that company over there. But when I submitted my email address to my client’s site, I did not expect to receive offers for Mickey Mouse Collectible Watches. It certainly is not what I signed up for.
Not only is the permission tenuous, but ISPs are moving away from a permission based model for access to their subscribers. What they really care about now is how recipients react to email. An email marketing model based on getting as much email in front of the recipient as possible will be harder and harder to be profitable as ISPs get better at measuring how much their subscribers want email. The mailers who get good delivery are those are able to make the mail interesting, wanted and relevant to recipients.
It is difficult for me to imagine a case where you can make 2 emails a day relevant to 4 million recipients.

What to expect from your delivery consultant

Every once in a while I get a phone call asking me what delivery consultant do. What can I do for them? How can I help them? Delivery consulting is a very new field and it is understandable a lot of people do not know what we do.
The overall delivery consulting process here a Word to the Wise involves collecting detailed information about your mailing program and your technical setup, like:

Unauthenticated email

A few weeks ago, NetworkWorld posted an interview with Mark Risher of Yahoo. In it, Mark talked about how Yahoo had no plans to outright block or refuse any unauthenticated email. Of course authentication will be a large part of their decision making for incoming emails but they cannot just refuse to accept mail that is unauthenticated, because there are times when unauthenticated email is the most important mail to their recipients.
A lot of marketers often seem to forget that they are competing for time and space with other, non-marketing, types of email. Email from friends and family and discussion lists are both more important to most people that the latest and greatest email advertisement. These are the emails people want to receive, the ones they open, read and respond to.
In terms of authentication, right now the majority of wanted emails are unsigned with DK or DKIM. Sure there are the early adopters who are using DK/DKIM to sign their emails, and a few large ISPs have started signing outgoing email. But until the vast majority of wanted email is actually signed, recipient ISPs are going to have to accept unsigned email.
Looking forward, even if all of the ISPs sign email sent through their SMTP servers, there will still be some fraction of desired email that will be unsigned. Individuals and small businesses who choose to run their own mailservers may not sign email. Even though these servers make up a tiny fraction of total email, they make up a much larger fraction of wanted email. ISPs cannot block this email without angering their customer base.
Marketers should not be concerned about ISPs blocking unauthenticated email, as it is extremely unlikely that any major ISP will do that. Marketers should focus, instead on making their email relevant and wanted by the recipients. I have been recommending clients plan to have all their outgoing emails DK/DKIM signed by the end of 2008.

Updating SenderID records for Microsoft

In the past, bulk senders who wanted Microsoft to check SenderID had to email information to a special Microsoft address. Microsoft would then cache the senderID data from the sender’s DNS records and verify incoming email.
Microsoft has simplified the process and now has a webform to submit the data.
http://support.msn.com/…
In order to submit your information you will need a contact email address, the domains that you want to add and the SPF records of those domains.

Do you know where your addresses go?

Being a deliverability consultant, I end up signing up for a lot of lists and providing email addresses to a lot of different websites I may not normally trust with my email address. The only way to manage the resulting volume of email is using a disposable address system. There are a number of commercial versions, but we built our own system.
Any time I need to sign up with a client, I create a new email address. Part of the address creation process involves making notes about where and when the address was used. When mail is received at any of the email addresses I have used, that email is appended with the data I provided at the time I signed up and forwarded to a mailbox on my main system. If an address ends up compromised or sold and getting too much mail, I can just turn it off. This system allows me to freely hand out addresses, without a large amount of mail ending up in my primary mail box.
Disposable addresses great way to monitor what my clients are doing with my email address. I have found, in at least 2 cases, that my clients are doing nothing wrong, but there are leaks in their process that lets email addresses get out to spammers. My reports of data leaking were the first they knew about any problems with their vendors or customers.
I strongly recommend any marketer who shares any data, include in that data test or seed accounts. Sign up for your own lists, using unique addresses, so that you can see what kind of mail your subscribers are receiving once they sign up at your site. If you are providing data to customers or vendors, include unique test data in each list. If you start getting unexpected mail to those addresses, you can track back to the specific vendor with the data problem.
Your email address list is one of the biggest assets your company has. Protect that asset by monitoring what others are doing with it.

Success!

A few weeks back I posted about changes a client of mine was making in order to improve their IP reputation and their delivery.
They are now getting inbox delivery at the major ISPs. Their actions are up and they are seeing great results.
They did not implement all of my specific suggestions. Instead, they took the underlying advice, and modified it to fit their needs. They took my advice to increase relevancy and engage their recipients, then applied their own knowledge of their recipients and made changes to their mailings. The result is lower complaints, better delivery and more responsiveness.
Clients are happy. Recipients are happy. Everything is good.

When to send mail

I had a call with a potential client recently asking me what was the best day to send mail. It’s a question that I did not have a good answer to. Email Insider does have an answer to that question: there is no one day to mail to get the best response.

Spam in the workplace

In comments on my last post Lux says:

It seems to me that in regard to PR people sending press releases to a professional journalist, you’ve got a very specific use case with slightly different rules of engagement from the norm.
Read More

Do it yourself mail systems

Through my position here at Word to the Wise I’ve interacted with dozens of companies over the years. Some companies outsource the mechanics of email sending to email service providers, others buy a software or MTA solution from one of the many vendors out there. For both these groups delivery problems are usually issues with permission or user expectations. Technically there are few problems with sending, bounce handling, unsubscriptions and rate limiting. The commercial software, either as created by an ESP or a vendor, typically does these things well.
The last group, those who use a home built system, are a whole different story. They often do no bounce processing relying on the underlying mail transport agent (typically qmail) to do all that work. The problem is that a general mail transport agent handles bounces for a particular email send, but does not have any functionality to handle future emails to addresses that bounce. Consequently the list does not get bounce handled, dead addresses pile up and their delivery rates plummet.
A few weeks ago Derek talked about senders using homegrown email systems and the pitfalls therein. He has a good list of things companies should think of before deciding a home grown system is right for them.

More on Relevancy

Al Iverson comments on information from Craig Spiezel at the Exacttarget customer conference this week. Craig confirms that MSN/Hotmail is also looking at user engagement, opens and moving mail out of the spam folder as part of their delivery metrics.

Relevance: don't underestimate it, measure it.

Ken Magill has an article today about a new service from e-Dialog called the Relevancy Trajectory. This product

Visiting customers through email

A few months ago I was working on a deliverability problem with a new client. They are a social networking site heavily branded with black background and pink text. One of the questions they asked me during the course of troubleshooting their delivery problems was if sending pink and black branded emails to match their site branding would decrease their deliverability.
That was actually a more interesting question than many I have received and led to the following analogy. A website is your showroom on the web. It is the equivalent of a brick and mortar store where people visit you and come to see what you have to offer for sale. Heavily branding the store is the right thing to do.
An email, be it marketing, transactional or relationship, is the equivalent of sending a traveling salesperson to someone’s house. That sales person is entering the customer’s space. In this case overly branding your presence in the customer’s space which can annoy or completely turn off your customers.
Branding emails to customers is a good thing; it builds brand recognition and customer relationships. Just remember, though, that you’re entering the customer’s space. Be respectful of that space.
As an aside, I did actually ask AOL about the color of email would decrease delivery. The nice folks over there did reply “AOL SAYS NO PINK!” But I’m fairly sure they weren’t serious.