We’re back at work after a trip to M3AAWG. This conference was a little different for me than previous ones. I spent a lot of time just talking with people – about email, about abuse, about the industry, about the ecosystem. Sometimes when you’re in a position like mine, you get focused way too much on the trees.
Of course, it’s the focusing on the trees that makes me good for my clients. I follow what’s going on closely, so they don’t have to. I pay attention so I can distill things into useable chunks for them to implement. Sometimes, though, I need to remember to look around and appreciate the forest. That’s what I got to do last week. I got to talk with so many great people. I got to hear what they think about email. The different perspectives are invaluable. They serve to deepen my understanding of delivery, email and where the industry is going.
One of the things that really came into focus for me is how critical protecting messaging infrastructure is. I haven’t spoken very much here about the election and the consequences and the changes and challenges we’re facing. That doesn’t mean I’m not worried about them or I don’t have some significant reservations about the new administration. It just means I don’t know how to articulate it or even if there is a solution.
The conference gave me hope. Because there are people at a lot of places who are in a place to protect users and protect privacy and protect individuals. Many of those folks were at the conference. The collaboration is still there. The concern for how we can stop or minimize bad behavior and what the implications are. Some of the most difficult conversations around policy involve the question who will this affect. In big systems, simple policies that seem like a no-brainer… aren’t. We’re seeing the effects of this with some of the realities the new administration and the Republican leaders of congress are realizing. Health care is hard, and complex. Banning an entire religion may not be a great idea. Governing is not like running a business.
Talking with smart people, especially with smart people who disagree with me, is one of the things that lets me see the forest. And I am so grateful for the time I spend with them.
One of the things that regularly happens at email conferences is a bunch of representatives from various ISPs and sometimes deliverability companies get up on stage and entertain questions from the audience about how to get email to the inbox. I’ve sat in many of these sessions – on both sides of the stage. The questions are completely predictable.
Almost invariably, someone asks if they can quote the ISP representative, because there is this belief that if you connect a statement with an employee name that will give the statement more weight. Except it doesn’t really. People who aren’t going to listen to the advice won’t listen to it even if there are names attached.
A lot of what I publish here is based on things the ISP reps have said. In some cases the reps actually review and comment on the post before I publish it. I don’t really believe attaching names to these posts will make them any more accurate. In fact, it will decrease the amount of information I can share and will increase the amount of time it takes to get posts out.
Last night I was joking with some folks that I should just make up names for attribution. Al did that many years ago, coining the pseudonym Barry for ISP reps. Even better, many of the ISP employees adopted Barry personas and used them to participate in different online spaces. Barry A. says X. Barry B. says Y. Barry C. says W. Barry D.
It doesn’t matter what names I attach.
I think I’m going to start adding this disclaimer to the appropriate blog posts:
Any resemblance to persons living or dead should be plainly apparent to them and those that know them. All events described herein actually happened, though on occasion the author has taken certain, very small, liberties with narrative.
Many years ago, back when huge levels of spam involved hundreds of thousands of emails, there was a group of people who spent a lot of time talking about what to do about abuse. One of the distinctions we made was abuse of the net as opposed to abuse on the net. We were looking at abuse of the network, that is activity that made the internet less useable. At the time abuse of the network was primarily spam; sure, there were worms and some malicious traffic, but we were focused on email abuse.
In the last 20 years, multiple industries have arisen around network abuse. I’m sitting at a conference with hundreds of people discussing how to address and mitigate abuse online. In the context of the early discussions, we’re mostly focused on abuse of the network, not abuse on the network.
But abuse on the network is an issue. It’s a growing issue, IMO. The internet has contributed to the rise and normalization of the alt-right. Social media is a medium used for abuse on the net. Incidents range from bullying of school kids to harassment of celebrities to sharing of child abuse material. All of these things are abuse on the net. They are an issue. They need to be addressed.
Today M3AAWG gave the 2017 Mary Litynski Award to Mick Moran from Interpol for his work in fighting child exploitation and abuse on the net. As I tweeted during the session, I have a phenomenal amount of respect for Mick and people like him who work tirelessly to protect children online. I don’t talk much about child abuse materials*, but I know the problem is there and it’s bad.
One of the discussions I’ve had with some folks lately is how we can better fight abuse on the net. Many of the tools we’ve built over the years are focused on volume – more complaints mean a more serious incident. But in the case of abuse on the net, or who is wrong. volume isn’t really an issue. It’s a hard problem to solve. It’s easy to create a system that lets the good guys get information, but it’s hard to create a system that also keeps the bad guys out and prevents gaming and is effective and values single complaints of problems.
Folks like Mick, and the abuse teams at ISPs all over the world, are integral to finding and rescuing abused and exploited children. Their work is so important, and most people have no idea they exist. On top of that, the work is emotionally difficult. Some of my friends work in that space, dealing with child abuse materials, and all of them have the untold story of the one that haunts them. They don’t talk about it, but you can see it in their eyes and faces.
We can do better. We should do better. We must do better.
*Note: Throughout this post I use the term “child abuse materials” to describe what is commonly called child pornography. This is because porn isn’t necessarily bad nor abusive and the term child porn minimizes the issue. It’s important to make it clear that children are abused, sometimes for years, in order to make this material.
That time of year when my friends and colleagues join the annual migration to San Francisco for 3 days and 4 nights of messaging, mobile, malware, and midnight meetings. We’re headed up to the conference later today. Do stop by and say hi!
I recently looked at a popular ESP’s IPv4 space and I was astounded. How does an ESP get an IP allocation of 20,480 IPs? ARIN guidelines do not allow “MX/Mailing” IPs to count towards a valid justification especially in the case when each and every IP is being used for this purpose. That’s 80 /24’s…and at a time when we are out of IPv4 space….Would love to see a blog post with your insight about this issue….
I’ve been mulling over this for a while. It’s a good question.
I’ve tried to answer it in two parts – Why does an ESP need / want a lot of IP addresses and How does an ESP acquire them from their regional internet registry.
Why does an ESP want / need lots of IP addresses?
One reason is “to be able to send a lot of email”. Any sane network setup is going to need at least one external IP address for each mailserver. A low end mailserver can send a few tens of thousands of emails an hour, in the best case, while a mid-range commercial email cannon can send ~100k an hour, and a high end (expensive) one maybe 10x that. At least in perfect conditions. In the real world, where recipients mailservers are overloaded, have long timeouts, defer connections and are just generally slow the number of SMTP connections you need to keep open simultaneously to get that sort of delivery rate goes up spectacularly – and that tends to be a limiting factor.
So, realistically, if a client wants to be able to send a message to most of their 100k recipients in a half hour window, they might to need to send from several addresses.
One mail engineer I talked to mentioned a customer with half a million recipients who was concerned about getting mail out to their recipients in a tight window. Switching from one source IP to two ended up delivering the same mail to the same recipients a third faster.
While modern spam filtering is sophisticated, and relies heavily on domain-keyed reputation and message content, the first level of filtering is IP-keyed.
Mail from an IP address that’s consistently sent reasonable quantities of messages that have been generally liked by the recipients is going to be able to deliver mail faster and more reliably than an IP address that doesn’t have a recent history of sending mail, or which has a history of sending unwanted or mediocre mail.
Put simply, consistently “good” senders get decent delivery, while “bad” or “mediocre” or “inconsistent” senders tend not to. Even if a sender sends wanted email, if they’re not sending it fairly regularly recipient ISPs will likely forget about their good behaviour and they’ll have problems when they try a large delivery after a long pause.
So there are two things that are needed to keep delivery healthy from an IP address. Mail needs to be sent at fairly consistent volumes, fairly regularly and it needs to be consistenly good email.
This leads to shared IPs (and shared pools) vs dedicated IPs.
If you have multiple customers sending from the same IP address then the reputation of those customers will be shared. Poor customers will be rewarded by better delivery rates than they’d get sending on their own, while good customers will be punished by having the quality of their delivery dragged down by the poor senders on the same address. That’s the opposite of what you want to happen if you’re encouraging customers to focus on behaving well.
But if you put customers who send very little email on their own dedicated IP addresses then they won’t send enough mail for that IP address to build or maintain a reputation with recipient ISPs. For those customers, sharing an IP address with others – of similar quality – will bring up the total volume sent from that address to a level that will benefit delivery.
So from the perspective of an ESP who wants to encourage customers to focus on what they’re doing and to behave well in all aspects of their email campaigns the ideal is to have any customer who is sending enough email to justify it on their own dedicated IP addresses, while having enough shared IP addresses to put smaller customers on. The larger customers’ delivery success will be driven primarily by their behaviour.
In the case of customers who send several decidedly different mail streams (transactional vs marketing, say) you may want to segregate those out to different IP addresses too, so that their marketing and ops groups have separate incentives to keep their marketing and transactional campaigns, respectively, clean.
So a couple of dedicated IP addresses per medium-to-large customer isn’t unreasonable operationally. More, if they send a lot of mail.
Customers also like dedicated IPs for several tangential reasons. They can use their own domain in reverse DNS and hostnames, which doesn’t have much operational effect but they like. It’s much easier to get data from feedback loops from a dedicated IP address, and third party reputation monitoring is more meaningful.
Stuff breaks. Mailservers crash, network connections go down, DNS infrastructure falls over, datacenters catch fire, backbone providers have contract disputes. And, sometimes, mail from an IP address will end up being undeliverable to a particular destination for no good reason.
So if your mail is valuable, you want a backup plan.
With most services you can engineer your system to be fairly robust against outages – keep backend data replicated between redundant sites, spin up new front end servers as needed or keep them idle at the backup site until you need them.
Because of the whole IP reputation thing, that’s nowhere near as simple to do if you’re sending email.
You could justify having a second set of IP addresses for each customer, duplicating their primary set. And, in order to keep those addresses “warm” you might want to spread deliveries across both sets normally, and still have the capacity to send their normal level of mail on just a single set if something breaks (or if you need to take something down for maintenance, or …).
I’ve very rarely seen this level of redundancy used in the wild, at least not by reputable ESPs. If a router goes down for six hours and it delays a customers send … that’s annoying, rather than mission critical.
I have seen less reputable ESPs (avoiding the “S” word here, but you know what sort of sender I mean) who’ll move a customer from IP address to IP address when they get blocked. That’s not what I mean by redundancy.
So, that’s all technical justifications for assigning IP addresses based on customers. What does a more real world policy look like?
I asked a couple of policy people at ESPs – ESPs I consider to be pretty smart, both in technical and business / policy ways – what they did.
“They get one.”
I like this policy. It’s simple to explain and to justify. Any customer that can benefit from a dedicated IP address gets a single IP address. This ESP doesn’t tend to have customers who send so much mail that they’ll overwhelm the capacity of a single IP address, so there’s no need to look at higher volumes than that.
“At 100k-200k/mo they get one. At 2M+/day they get one per 2M/day. If they’re segmenting marketing mail from transactional they can get additional ones as needed.”
Again, easy enough to explain and to justify. Customers who are too small to benefit from a dedicated IP address don’t get one. Once you get significantly over 2M/day delivery metrics start to get worse, deferrals tend to happen more… so 2M/day/IP is operationally justified.
What’s the downside, from a policy perspective? Until you actually run out of IPv4 addresses it’s easy for sales reps to keep giving them out to new customers. Customers that could send perfectly successfully from shared addresses – especially pooled addresses that were stratified by customer reputation and volume – still want a dedicated IP address when they’re talking to sales. And you can’t move a customer from a dedicated IP address to a shared pool without loud objections.
So the number of IP addresses an ESP is using hardly ever decreases, rather it keeps ratcheting up until you’re using many more than you actually need.
On the one hand using them profligately rather than facing the policy and engineering challenges now is going to make hitting the wall of “we can’t get any more IPv4 addresses” more painful. On the other hand it does mean you’ll have a hoard of them to reallocate to soften the blow when it does happen.
I’ll talk about the “how did the ESP get all these IPv4 addresses anyway?” aspect on Monday.
Permission is always a hot topic in email marketing. Permission is key! the experts tell us. Get permission to send email! the ISPs tell us.
Marketers have responded by setting up processes to “get” permission from recipients before adding them to mailing lists. They point to their privacy polices and signup forms and say “Look! the recipient gave us permission.”
In many cases, though, the permission isn’t given to the sender, permission is taken from the recipient.
Yes, permission is being TAKEN by the sender. At the point of address collection many senders set the default to be the recipient gets mail. These processes take any notion of giving permission out of the equation. The recipient doesn’t have to give permission, permission is assumed.
This isn’t real permission. No process that requires the user to take action to stop themselves from being opted in is real permission. A default state of yes takes the actual opt-in step away from the recipient.
Permission just isn’t about saying “well, we told the user if they gave us an email address we’d send them mail and they gave us an email address anyway.” Permission is about giving the recipients a choice in what they want to receive. All too often senders take permission from recipients instead of asking for permission to be given.
Since that post was originally written, some things have changed.
CASL has come into effect. CASL prevents marketers from taking permission as egregiously as what prompted this post. Under CASL, pre-checked opt-in boxes do not count as explicit permission. The law does have a category of implicit permission, which consists of an active consumer / vendor relationship. This implicit permission is limited in scope and senders have to stop mailing 2 years after the last activity.
The other change is in Gmail filters. Whatever they’re doing these days seems to really pick out mail that doesn’t have great permission. Business models that would work a few years ago are now struggling to get to the inbox at Gmail. Many of these are non-relationship emails – one off confirmations, tickets, receipts. There isn’t much of a relationship between the sender and the recipient, so the filters are biased against the mail.
Permission is still key, but these days I’m not sure even informed permission is enough.
Those of you who have seen some of my talks have seen this model of email delivery before. The concept is that there are a host of factors that contribute to the reputation of a particular email, but that at many ISPs the email reputation is only one factor in email delivery. Recipient preferences drive whether an email ends up in the bulk folder or the inbox.
The individual recipient preferences can be explicit or implicit. Users who add a sender to their address book, or block a sender, or create a specific filter for an email are stating an explicit preference. Additionally, ISPs monitor some user behavior to determine how wanted an email is. A recipient who moves an email from the bulk folder to the inbox is stating a preference. A person who hits “this-is-spam” is stating a preference. Other actions are also measured to give a user specific reputation for a mail.
Seed accounts aren’t like normal accounts. They don’t send mail ever. They only download it. They don’t ever dig anything out of the junk folder, they never hit this is spam. They are different than a user account – and ISPs can track this.
This tells us we have to take inbox monitoring tools with a grain of salt. I believe, though, they’re still valuable tools in the deliverability arsenal. The best use of these tools is monitoring for changes. If seed lists show less than 100% inbox, but response rates are good, then it’s unlikely the seed boxes are correctly reporting delivery to actual recipients. But if seed lists show 100% inbox and then change and go down, then that’s the time to start looking harder at the overall program.
The other time seed lists are useful is when troubleshooting delivery. It’s nice to be able to see if changes are making a difference in delivery. Again, the results aren’t 100% accurate but they are the best we have right now.
For a long time it was a known fact that certain ISPs recycled abandoned addresses into spamtraps. There were long discussions by senders about this process and how it happened. Then at a conference a few years ago representatives of ISPs got up and announced that they do not recycle addresses. This led to quite a bit of consternation about how deliverability folks were making things up and were untrustworthy and deceptive.
In the early 2000’s ISPs were throwing a lot of things at the wall to deal with mail streams that were 80 – 90% bulk. They tried many different things to try and tame volumes that were overwhelming infrastructure. ISPs did try recycled traps. I know, absolutely know, two did. I am very sure that others did, too, but don’t have specific memories of talking to specific people about it.
At that time, a lot of deliverability knowledge was shared through word of mouth. That turned into a bit of an oral history. The problem with oral history is that context and details get lost. We can use the story of the ISP that did/did not recycle traps as an example.
Deliverability folks talk about an ISP that recycles traps. They don’t mention how often it happens. Some folks make the assumption that this is an ongoing process. It’s not, but anyone who knows it’s not risks violating confidences if they correct it. Besides, if senders believe it’s an ongoing process maybe they’ll be better behaved. Eventually, the story becomes all ISPs recycle traps all the time. This is our “fact” that’s actually a myth.
Then an ISP employee goes to a conference an definitively states they don’t recycle traps. I believe he stated the truth as he knows it to be. That ISP moved on from recycled traps to other kinds of traps because there were better ways to monitor spam.
We were talking about this on one of the deliverability lists and I told another story.
[ISP] recycled addresses once – back when JD was there which must have been, oh, around 2005/6 or so. I heard this directly from JD. It wasn’t done again, but a whole bunch of people just assumed it was an ongoing thing. Since my knowledge was a private conversation between JD and me, I never felt comfortable sharing the information. Given the circumstances, I’ve decided it’s OK to start sharing that end of the story a little more freely.
No one set out to create a myth, it just happened. No one intended to mislead. But sometimes it happens.
I regularly tell clients to be transparent with their sends. With email, permission is better than forgiveness. A surprise change in mail frequency or type leads to complaints. Complaints lead to bulk foldering. Once mail is in the bulk folder, it’s hard to get out of there, particularly at some webmail providers.
The permission is better than forgiveness is hard for a lot of senders to understand. Much of marketing is about assuming the yes in the absence of a no. Sure, they’ll back off when there’s a no, in DMA terms it’s the “one bite at the apple rule.” Unfortunately for senders the one bite rule doesn’t work in the email space.
There are a couple reasons that permission is better than forgiveness in the email space. The biggest is that the ISPs own the mailbox and as the owners they make decisions about who gets access. They prioritize the wants and needs of their customers / users over the wants and needs of advertisers. It’s easy for users to give feedback; in many cases they just have to hit a button. But that’s another whole blog post.
Today I get an email from The Guardian. They’re modifying and expanding their newsletter program, so they sent subscribers an update about it.
This is a quick email to let you know we are making some changes to how we communicate our events to you.
Up until now we have sent information regarding events for members in your weekly newsletter. We will continue to send you information on events we think may be of interest, but will also send more in-depth coverage of our full Guardian Live programme such as topical panel debates, special guest interviews and Guardian Masterclasses.
In future these emails will come from Guardian Live rather than Guardian Members. If you would prefer not to receive these emails please opt-out here. Though you will no longer hear about our full events programme you will continue to receive the weekly members newsletter if you opt-out of Guardian Live.
If you chose to receive these emails and change your mind later on, you can opt-out at any time using the unsubscribe link at the bottom of every email.
For further information please visit our FAQs page.
This is the kind of thing that I regularly recommend to clients. I’m so pleased to have an example from an international publication.
Gmail’s approach to filtering; as described by one of the old timers. This person was dealing with network abuse back when I was still slinging DNA around as my job and just reading headers as a hobby.
Gmail uses a 10+ year old neural network that analyzes thousands of factors, related to email, IP, and web, integrated with all Google products, and with 99.9%+ accuracy for identifying certain types of messages, combined with an email-specific domain-based reputation system that combines IP reputation, content, read rates, reputation of other senders with similar content.
This excerpt was shared with a bunch of delivery experts and every one of them agreed. The Gmail filters are incredibly complex and they measure thousands of different things about email. Yes, sometimes you can remove a link or a URL and get mail to the inbox for a while. That doesn’t mean the block was against the URL, simply that changing the URL changed the score enough for the mail to go to the inbox.
This is part of what makes Gmail delivery issues so difficult to troubleshoot. There isn’t one thing, it’s all the things that contribute to where an email ends up. We, as senders and deliverability experts, don’t have access to the Gmail data. The poster goes on to say:
Trying to fix this using only inaccurate proxy data where there is no mediation pathway in a matter of weeks is complex. We consume data from a multitude of sources, compile and analyze the data, determine which of the hundreds of factors we can influence should be adjusted, come up with the easiest plan to address the most influential factors, and explain that to the customer using the clearest language possible to individuals who are not educated on the definition of a complaint.
We do our best, with limited data and try and tell you how to fix things.
One of the biggest challenges with Gmail delivery is I am convinced they look at your profile of recipients. They can map someone who is collecting addresses through third parties, or buying lists based on the specific Gmail accounts targeted by a mailing. Gmail has publicly stated and has on their website that they don’t think co-reg or purchased lists are opt-in. They have the technology and ability to track that. I think it’s one reason senders trying to use email for acquisition have such a challenge getting into the inbox and Gmail. I think it’s a feature, not a bug for them.
It's been a wild week here in the US. I have to admit, the current political climate is affecting my ability to blog about email. I've always said email is not life or death. And how can I focus on the minutia of deliverability when things are in such turmoil and uncertainty? There are many things I want to write about, including some resources for those of us who are struggling with the current administration and changes in the US. What we can do. What we must do. It just takes work and focus I don't have right now.
Reminder for folks, AOL is changing their FBL from address starting on Jan 17th.
The (in)famous firstname.lastname@example.org is going away to be replaced by fbl-no-reply @ postmaster.aol.com. These messages will be signed with the d= mx.postmaster.aol.com.
Time to update your scripts!No Comments