Yesterday there was a lot of talk about Verizon moving out of email and transitioning all their customers over to the AOL backend. The source was a page in the Verizon help center about transitioning an email address. There is no date on the page, so it’s unclear when this is going to happen or when it started.
I posted about Verizon beginning this transition back in May of 2016: Changes coming to Verizon email. The wording on the AOL page I link to is very similar to the wording on the page that was passed around yesterday.
Without a date it’s hard to really provide any advice, other than to maintain your list hygiene (you do have list hygiene programs, right?) and remove addresses that hard bounce. Quite honestly, I don’t think this will really have any effect on delivery. It doesn’t appear these changes are happening all at once, and Verizon customers have the option to keep their verizon.net address. They’re just going to have to access it differently.
For companies that use an email address as a primary key for logins or accounts, it’s probably a good time to contact customers with a verizon.net address and ask them to update their address. That’s a good idea most of the time, but when we know changes are happening at a domain level, it’s a requirement.
This morning, CSO and MacKeeper published joint articles on a massive data leak from a marketing company. This company, River City Media (RCM), failed to put a password on their online backups sometime. This leaked all of the company’s data out to the Internet at large. MacKeeper Security Researcher, Chris Vickery discovered the breach back in December and shared the information with Spamhaus and CSO online.
The group has spent months going through the data from this spammer. As of this morning, the existence of the breach and an overview of the extent of their operation were revealed by CSO and MacKeeper. Additionally, Spamhaus listed the network on the Register of Known Spamming Operations (ROKSO).
There are a couple interesting pieces of this story relevant to legitimate marketers.
The biggest issue is the number of brands who are paying spammers to send mail from them. The CSO article lists just some of the brands that were buying mail services from RCM:
[…] Nike, LifeLock, Liberty Mutual, Fidelity, MetLife, Victoria’s Secret, Kitchen Aide, Yankee Candle, Bath & Body Works, Gillette, Match.com, Dollar Shave Club, Dewalt, DirecTV, Covergirl, Clinique, Maybelline, Terminix, and AT&T.
This shouldn’t be a surprise to anyone who has been paying attention to the industry. We described this many years ago in a series of articles about mainstream spam. (Note: the organization in the article has cleaned up their act and no longer uses affiliates).
Addresses were collected through many ways, including the use of co-reg. Chris Vickery explains:
Well-informed individuals did not choose to sign up for bulk advertisements over a billion times. The most likely scenario is a combination of techniques. One is called co-registration. That’s when you click on the “Submit” or “I agree” box next to all the small text on a website. Without knowing it, you have potentially agreed your personal details can be shared with affiliates of the site.
You are never told who the affiliates are and groups like River City Media capitalize on that aspect. One line of the leaked chat logs explains it all very succinctly:
“The key is sincerity. Once you can fake that…”
Legitimate companies do buy co-reg data, still. The problem is that there’s no real permission associated with the address. In the absolute best case scenario, permission is taken by the co-reg provider rather than given by the recipient. All too many co-reg vendors go out of their way to hide the fact that they will sell the addresses in their privacy policies. This isn’t transparent. This isn’t real permission.
One argument I’ve heard over and over about laws, particularly CASL, is that it’s targeting the wrong companies. As the argument goes, the real problem with spam is spammers, not legitimate companies. But CASL and other laws target legitimate companies. I never really bought into that argument as it’s clear to me a lot of the money supporting spammers comes from the legitimate companies spending real marketing funds.
Legitimate companies are paying third parties to send spam on their behalf and are profiting. For a long time brands have pretended they’re not responsible for the mail. This recent breach shows that they are paying spammers to send mail on their behalf.
Looks like maybe the laws are targeting the right companies.
As always, I blogged about best practices with subscriptions, and shared a great example of subscription transparency that I received from The Guardian. I also wrote about what happens to the small pool of people who fail to complete a confirmed opt-in (or double opt-in) subscription process. While there are many reasons that someone might not complete that process, ultimately that person has not given permission to receive email, and marketers need to respect that. I revisited an older post on permission which is still entirely relevant.
On the topic of industry veterans, myths and truths, I looked at the “little bit right, little bit wrong” set of opinions in the world of email. It’s interesting to see the kinds of proclamations people make and how those line up against what we see in the world.
Steve took a turn as our guest columnist for “Ask Laura” this month with a terrific post on why ESPs need so many IP addresses. As always, we’d love to get more questions on all things email — please get in touch!
A few moments ago, I cancelled one of my email addresses. This is an address that has been mine since somewhere around 1993 or 4. It was old enough to vote. And now it’s no more.
I am not even sure why I kept it for so long. It was my dialup account back when I was in grad school in Delaware. When I moved to Madison to work at the university, I kept it as a shell account and email address. I gave it up as my primary email address about the time it was bought by a giant networking company. By then I had my own domain and a mail server living behind the futon in the living room. That was back when we started WttW, somewhere around 2002.
15 years the address has mostly laid dormant. I used it for a couple yahoo groups accounts, but just lists that I lurked on.
I did use it as research for some past clients, typically the ones using affiliate marketers. “Our affiliates only ever send opt in mail!” Yeah, no. See, look, your affiliate is spamming me. My favorite was when said customer put me on the phone with the affiliate.
Affiliate: If we have your email address you must have signed up with us.
Me: The last time I used that address to sign up for anything was the late ‘9os. Your company hasn’t been in business that long.
Affiliate: Well, we probably bought a company you did sign up with.
Me: Do you know who that was?
Me: So how can you have permission to mail me?
Affiliate: Well, you gave permission to someone at some point.
Me: But I didn’t give permission to you, ever.
Affiliate: We only send permission based email.
I’m going to miss the old email address, but it’s time to move on. But you can have my 20 year old same-lhs hotmail, yahoo, aol and gmail addresses when you pry them from my cold, dead fingers. Those aren’t as useful as spamtraps, though, because the filtering is better and the companies regularly stop accepting mail to to them if I don’t log in for a while.
Looking back through my archives it’s been about 4 years or so since I wrote about confirmed opt in. The last post was how COI wasn’t important, but making sure you were reaching the right person was important. Of course, I’ve also written about confirmed opt-in in general and how it was a tool somewhat akin to a sledgehammer. I’m inspired to write about it today because it’s been a topic of discussion on multiple mailing lists today and I’ve already written a bunch about it (cut-n-paste-n-edit blog post! win!).
Confirmed opt-in is the process where you send an email to a recipient and ask them to click on a link to confirm they want the mail. It’s also called double opt-in, although there are some folks who think that’s “spammer” terminology. It’s not, but that’s a story for another day. The question we were discussing was what to do with the addresses that don’t click. Can you email them? Should you email them? Is there still value in them?
We have to treat the addresses as a non-homogenous pool. There are a lot of reasons confirmation links don’t get clicked.
Some recipients aren’t going to click because they really don’t want the mail and the extra step is too much effort. If the mail had just shown up they might read it, might even engage with it, but don’t want to actually have to make an effort to engage with the confirmation.
Some recipients aren’t going to click because they’ve already gotten what they want, like access to a website or a white paper or free download.
Some recipients aren’t going to click because they aren’t your customer. Someone used their email address to sign up and they sincerely do not want the mail.
Some recipients aren’t going to click because they never saw the mail. It may have gone to bulk, they may have not recognized the subject line and just deleted it, it may have ended up dropped on the floor. Whatever happened, it wasn’t seen by the recipient.
Some recipients aren’t going to click because there is no recipient. Sure, the mail is accepted by the receiving mail server, but the user never logs in, or it’s a spamtrap.
There is some value in the pool, but statistically, some of that value is negative. Each company needs to do their own risk analysis and determine what best to do with these addresses.
Different subscription techniques are going to generate different subscriber pools. Those different pools are going to have different risk profiles. Some subscription processes will generate more of one type of subscriber than another. That means the risk of mailing users who didn’t click on the link is going to vary depending on the pool.
Lots of “no recipients” on the list means sending followups is high risk. Lots of non-customers on the list, ditto. But if the pool is lots of people who can’t be bothered or missed the email the first time? That’s probably OK to mail once or twice.
Overall, the entire goal here is to get a list of email addresses that are owned by people who want mail from you. There are two parts to that: identity and permission. The identity part is tying the email address to the person who is your customers / subscriber / lottery winner / potential future customer who wants to know what you sell. The permission part is discovering if they want mail from you.
Traditional COI combines the identity and permission piece into one step. Send the person a mail and ask for permission to mail them more email. That covers the identity and the permission – if the person clicks you have both. But there are other ways to prove identity and there are other ways to gauge permission.
We’re back at work after a trip to M3AAWG. This conference was a little different for me than previous ones. I spent a lot of time just talking with people – about email, about abuse, about the industry, about the ecosystem. Sometimes when you’re in a position like mine, you get focused way too much on the trees.
Of course, it’s the focusing on the trees that makes me good for my clients. I follow what’s going on closely, so they don’t have to. I pay attention so I can distill things into useable chunks for them to implement. Sometimes, though, I need to remember to look around and appreciate the forest. That’s what I got to do last week. I got to talk with so many great people. I got to hear what they think about email. The different perspectives are invaluable. They serve to deepen my understanding of delivery, email and where the industry is going.
One of the things that really came into focus for me is how critical protecting messaging infrastructure is. I haven’t spoken very much here about the election and the consequences and the changes and challenges we’re facing. That doesn’t mean I’m not worried about them or I don’t have some significant reservations about the new administration. It just means I don’t know how to articulate it or even if there is a solution.
The conference gave me hope. Because there are people at a lot of places who are in a place to protect users and protect privacy and protect individuals. Many of those folks were at the conference. The collaboration is still there. The concern for how we can stop or minimize bad behavior and what the implications are. Some of the most difficult conversations around policy involve the question who will this affect. In big systems, simple policies that seem like a no-brainer… aren’t. We’re seeing the effects of this with some of the realities the new administration and the Republican leaders of congress are realizing. Health care is hard, and complex. Banning an entire religion may not be a great idea. Governing is not like running a business.
Talking with smart people, especially with smart people who disagree with me, is one of the things that lets me see the forest. And I am so grateful for the time I spend with them.
One of the things that regularly happens at email conferences is a bunch of representatives from various ISPs and sometimes deliverability companies get up on stage and entertain questions from the audience about how to get email to the inbox. I’ve sat in many of these sessions – on both sides of the stage. The questions are completely predictable.
Almost invariably, someone asks if they can quote the ISP representative, because there is this belief that if you connect a statement with an employee name that will give the statement more weight. Except it doesn’t really. People who aren’t going to listen to the advice won’t listen to it even if there are names attached.
A lot of what I publish here is based on things the ISP reps have said. In some cases the reps actually review and comment on the post before I publish it. I don’t really believe attaching names to these posts will make them any more accurate. In fact, it will decrease the amount of information I can share and will increase the amount of time it takes to get posts out.
Last night I was joking with some folks that I should just make up names for attribution. Al did that many years ago, coining the pseudonym Barry for ISP reps. Even better, many of the ISP employees adopted Barry personas and used them to participate in different online spaces. Barry A. says X. Barry B. says Y. Barry C. says W. Barry D.
It doesn’t matter what names I attach.
I think I’m going to start adding this disclaimer to the appropriate blog posts:
Any resemblance to persons living or dead should be plainly apparent to them and those that know them. All events described herein actually happened, though on occasion the author has taken certain, very small, liberties with narrative.
Many years ago, back when huge levels of spam involved hundreds of thousands of emails, there was a group of people who spent a lot of time talking about what to do about abuse. One of the distinctions we made was abuse of the net as opposed to abuse on the net. We were looking at abuse of the network, that is activity that made the internet less useable. At the time abuse of the network was primarily spam; sure, there were worms and some malicious traffic, but we were focused on email abuse.
In the last 20 years, multiple industries have arisen around network abuse. I’m sitting at a conference with hundreds of people discussing how to address and mitigate abuse online. In the context of the early discussions, we’re mostly focused on abuse of the network, not abuse on the network.
But abuse on the network is an issue. It’s a growing issue, IMO. The internet has contributed to the rise and normalization of the alt-right. Social media is a medium used for abuse on the net. Incidents range from bullying of school kids to harassment of celebrities to sharing of child abuse material. All of these things are abuse on the net. They are an issue. They need to be addressed.
Today M3AAWG gave the 2017 Mary Litynski Award to Mick Moran from Interpol for his work in fighting child exploitation and abuse on the net. As I tweeted during the session, I have a phenomenal amount of respect for Mick and people like him who work tirelessly to protect children online. I don’t talk much about child abuse materials*, but I know the problem is there and it’s bad.
One of the discussions I’ve had with some folks lately is how we can better fight abuse on the net. Many of the tools we’ve built over the years are focused on volume – more complaints mean a more serious incident. But in the case of abuse on the net, or who is wrong. volume isn’t really an issue. It’s a hard problem to solve. It’s easy to create a system that lets the good guys get information, but it’s hard to create a system that also keeps the bad guys out and prevents gaming and is effective and values single complaints of problems.
Folks like Mick, and the abuse teams at ISPs all over the world, are integral to finding and rescuing abused and exploited children. Their work is so important, and most people have no idea they exist. On top of that, the work is emotionally difficult. Some of my friends work in that space, dealing with child abuse materials, and all of them have the untold story of the one that haunts them. They don’t talk about it, but you can see it in their eyes and faces.
We can do better. We should do better. We must do better.
*Note: Throughout this post I use the term “child abuse materials” to describe what is commonly called child pornography. This is because porn isn’t necessarily bad nor abusive and the term child porn minimizes the issue. It’s important to make it clear that children are abused, sometimes for years, in order to make this material.
That time of year when my friends and colleagues join the annual migration to San Francisco for 3 days and 4 nights of messaging, mobile, malware, and midnight meetings. We’re headed up to the conference later today. Do stop by and say hi!
I recently looked at a popular ESP’s IPv4 space and I was astounded. How does an ESP get an IP allocation of 20,480 IPs? ARIN guidelines do not allow “MX/Mailing” IPs to count towards a valid justification especially in the case when each and every IP is being used for this purpose. That’s 80 /24’s…and at a time when we are out of IPv4 space….Would love to see a blog post with your insight about this issue….
I’ve been mulling over this for a while. It’s a good question.
I’ve tried to answer it in two parts – Why does an ESP need / want a lot of IP addresses and How does an ESP acquire them from their regional internet registry.
Why does an ESP want / need lots of IP addresses?
One reason is “to be able to send a lot of email”. Any sane network setup is going to need at least one external IP address for each mailserver. A low end mailserver can send a few tens of thousands of emails an hour, in the best case, while a mid-range commercial email cannon can send ~100k an hour, and a high end (expensive) one maybe 10x that. At least in perfect conditions. In the real world, where recipients mailservers are overloaded, have long timeouts, defer connections and are just generally slow the number of SMTP connections you need to keep open simultaneously to get that sort of delivery rate goes up spectacularly – and that tends to be a limiting factor.
So, realistically, if a client wants to be able to send a message to most of their 100k recipients in a half hour window, they might to need to send from several addresses.
One mail engineer I talked to mentioned a customer with half a million recipients who was concerned about getting mail out to their recipients in a tight window. Switching from one source IP to two ended up delivering the same mail to the same recipients a third faster.
While modern spam filtering is sophisticated, and relies heavily on domain-keyed reputation and message content, the first level of filtering is IP-keyed.
Mail from an IP address that’s consistently sent reasonable quantities of messages that have been generally liked by the recipients is going to be able to deliver mail faster and more reliably than an IP address that doesn’t have a recent history of sending mail, or which has a history of sending unwanted or mediocre mail.
Put simply, consistently “good” senders get decent delivery, while “bad” or “mediocre” or “inconsistent” senders tend not to. Even if a sender sends wanted email, if they’re not sending it fairly regularly recipient ISPs will likely forget about their good behaviour and they’ll have problems when they try a large delivery after a long pause.
So there are two things that are needed to keep delivery healthy from an IP address. Mail needs to be sent at fairly consistent volumes, fairly regularly and it needs to be consistenly good email.
This leads to shared IPs (and shared pools) vs dedicated IPs.
If you have multiple customers sending from the same IP address then the reputation of those customers will be shared. Poor customers will be rewarded by better delivery rates than they’d get sending on their own, while good customers will be punished by having the quality of their delivery dragged down by the poor senders on the same address. That’s the opposite of what you want to happen if you’re encouraging customers to focus on behaving well.
But if you put customers who send very little email on their own dedicated IP addresses then they won’t send enough mail for that IP address to build or maintain a reputation with recipient ISPs. For those customers, sharing an IP address with others – of similar quality – will bring up the total volume sent from that address to a level that will benefit delivery.
So from the perspective of an ESP who wants to encourage customers to focus on what they’re doing and to behave well in all aspects of their email campaigns the ideal is to have any customer who is sending enough email to justify it on their own dedicated IP addresses, while having enough shared IP addresses to put smaller customers on. The larger customers’ delivery success will be driven primarily by their behaviour.
In the case of customers who send several decidedly different mail streams (transactional vs marketing, say) you may want to segregate those out to different IP addresses too, so that their marketing and ops groups have separate incentives to keep their marketing and transactional campaigns, respectively, clean.
So a couple of dedicated IP addresses per medium-to-large customer isn’t unreasonable operationally. More, if they send a lot of mail.
Customers also like dedicated IPs for several tangential reasons. They can use their own domain in reverse DNS and hostnames, which doesn’t have much operational effect but they like. It’s much easier to get data from feedback loops from a dedicated IP address, and third party reputation monitoring is more meaningful.
Stuff breaks. Mailservers crash, network connections go down, DNS infrastructure falls over, datacenters catch fire, backbone providers have contract disputes. And, sometimes, mail from an IP address will end up being undeliverable to a particular destination for no good reason.
So if your mail is valuable, you want a backup plan.
With most services you can engineer your system to be fairly robust against outages – keep backend data replicated between redundant sites, spin up new front end servers as needed or keep them idle at the backup site until you need them.
Because of the whole IP reputation thing, that’s nowhere near as simple to do if you’re sending email.
You could justify having a second set of IP addresses for each customer, duplicating their primary set. And, in order to keep those addresses “warm” you might want to spread deliveries across both sets normally, and still have the capacity to send their normal level of mail on just a single set if something breaks (or if you need to take something down for maintenance, or …).
I’ve very rarely seen this level of redundancy used in the wild, at least not by reputable ESPs. If a router goes down for six hours and it delays a customers send … that’s annoying, rather than mission critical.
I have seen less reputable ESPs (avoiding the “S” word here, but you know what sort of sender I mean) who’ll move a customer from IP address to IP address when they get blocked. That’s not what I mean by redundancy.
So, that’s all technical justifications for assigning IP addresses based on customers. What does a more real world policy look like?
I asked a couple of policy people at ESPs – ESPs I consider to be pretty smart, both in technical and business / policy ways – what they did.
“They get one.”
I like this policy. It’s simple to explain and to justify. Any customer that can benefit from a dedicated IP address gets a single IP address. This ESP doesn’t tend to have customers who send so much mail that they’ll overwhelm the capacity of a single IP address, so there’s no need to look at higher volumes than that.
“At 100k-200k/mo they get one. At 2M+/day they get one per 2M/day. If they’re segmenting marketing mail from transactional they can get additional ones as needed.”
Again, easy enough to explain and to justify. Customers who are too small to benefit from a dedicated IP address don’t get one. Once you get significantly over 2M/day delivery metrics start to get worse, deferrals tend to happen more… so 2M/day/IP is operationally justified.
What’s the downside, from a policy perspective? Until you actually run out of IPv4 addresses it’s easy for sales reps to keep giving them out to new customers. Customers that could send perfectly successfully from shared addresses – especially pooled addresses that were stratified by customer reputation and volume – still want a dedicated IP address when they’re talking to sales. And you can’t move a customer from a dedicated IP address to a shared pool without loud objections.
So the number of IP addresses an ESP is using hardly ever decreases, rather it keeps ratcheting up until you’re using many more than you actually need.
On the one hand using them profligately rather than facing the policy and engineering challenges now is going to make hitting the wall of “we can’t get any more IPv4 addresses” more painful. On the other hand it does mean you’ll have a hoard of them to reallocate to soften the blow when it does happen.
I’ll talk about the “how did the ESP get all these IPv4 addresses anyway?” aspect on Monday.
Today the US government unsealed an indictment against 2 Russian agents and 2 hackers for breaking into Yahoo's servers and stealing personal information. The information gathered during the hack was used to target government officials, security employees and private individuals.
Email is so central to our online identity. Compromise an email account and you can get access to social media, and other accounts. Email is the key to the kingdom.No Comments
It's been a wild week here in the US. I have to admit, the current political climate is affecting my ability to blog about email. I've always said email is not life or death. And how can I focus on the minutia of deliverability when things are in such turmoil and uncertainty? There are many things I want to write about, including some resources for those of us who are struggling with the current administration and changes in the US. What we can do. What we must do. It just takes work and focus I don't have right now.