Data Verification

Address verification doesn’t fix any real problem

Would you trust an address verification company that used twitter spam to advertise their product?

Catchall domains

Catchall domains accept any mail to any email address at that domain. They were quite common, particularly at smaller domains, a long time ago. For various reasons, most of them having to do with spammers, they’re less common now.

Most folks think catchall domains are only used for spamtraps. As a consequence, many of the address verification tools will filter out, or recommend filtering out, any address that goes to a catchall domain. They test this by trying to send emails to random addresses like sldqwhhxbe+ym7ajymw23gm0@clientspecific.domain.example.
But not all catchall domains are used for spamtraps. Every client here at WttW gets a domain assigned to them and those domains are catchalls. Emails to those domains go into a database for analysis. Clients (and I!) can create any LHS on the fly to test signups, look at mail flows. Having a catchall means we don’t have to actually configure any address so I can test multiple signups and encode the data about the signup in the to: address.
This works most of the time, at least until verification services mark those addresses as bad and they don’t get imported into the client’s processes. We have some workarounds, and can still get mail despite the services making assumptions.

Trust the list broker

Over the years I’ve worked with companies who admit to me that they’ve purchased data at one point or another. Let’s face it, as bad a practice as it is, people and companies still think they can succeed in email marketing with purchased lists.
As part of the cleanup process, I start to ask questions about the list. Who did you buy it from? How were the addressees collected? Are these addresses shared with others? What did the seller tell you about the list.

Clients are rarely able to tell me about where the addresses are collected or if they’re shared.
It’s amazing to me how many companies choose to outsource the creation of such a valuable asset. They don’t know anything about it, but it’s a huge asset and so important they won’t let go when it doesn’t work.
Some of it is the sunk cost fallacy. But I think in some cases my clients don’t really believe the person who sold them the list wasn’t truthful. They really believe there is value in the list, if they can only unlock it.
Companies selling lists don’t really have any incentive to spend time or money making sure they have permission or that the lists are good. That’s just expense to them and returns no value. The value is in the number of addresses they can sell, not in the number of responsive addresses.
How many companies buy a list and immediately take it to a list cleansing service? Why should they? Shouldn’t the company SELLING the list make sure they’re selling deliverable addresses? Shouldn’t the seller spend the money for verification?
The very fact that so many companies believe they need to clean a purchased list speaks to the horrible quality of purchased lists. And, yet, companies are addicted to the idea of purchasing lists. They trust that the addresses are collected in a permission based manner. They believe when sellers tell them the addresses are good and valid – even when they see that 10 or 20 or 30% of the list is cleaned off by the list services.
List sellers won’t do the cleaning because they know they’re not providing the product. It’s a con and it’s a swindle and yet marketers still think they’re getting something of value from list sellers. And they still discover purchased lists are horrible in terms of deliverability and performance.

Organizational security and doxxing

The security risks of organizational doxxing.
These are risks every email marketer needs to understand. As collectors of data they are a major target for hackers and other bad people. Even worse, many marketers don’t collect valid data and risk implicating the wrong people if their data is ever stolen. I have repeatedly talked about incidents where people get mail not intended for them. I’ve talked about this before, in a number of posts talking about misdirected email. Consumerist, as well, has documented many incidents of companies mailing the wrong person with PII. Many of these stories end with the company not allowing the recipient to remove the address on the account because the user can’t prove they own the account.
I generally focus on the benefits to the company to verify addresses. There are definite deliverability advantages to making sure email address belongs to the account owner. But there’s also the PR benefits of not revealing PII attached to the wrong email address. With Ashley Madison nearly every article mentioned that the email address was never confirmed. But how many other companies don’t verify email addresses and risk losing personally damaging data belonging to non customers.
Data verification is so important. So very, very important. We’ve gone beyond the point where any big sender should just believe that the addresses users give them are accurate. They need to do it for their own business reasons and they need to do it to prevent incorrect PII from being leaked and shared.

It's not about the spamtraps

I’ve talked about spamtraps in the past but they keep coming up in so many different discussions I have with people about delivery that I feel the need to write another blog post about them.
Spamtraps are …
… addresses that did not or could not sign up to receive mail from a sender.
… often mistakenly entered into signup forms (typos or people who don’t know their email addresses).
… often found on older lists.
… sometimes scraped off websites and sold by list brokers.
… sometimes caused by terrible bounce management.
… only a symptom …

Linking identities to email addresses

As I predicted yesterday, a bunch of sites have popped up where you can input email addresses and find out if the address was part of the Ashley Madison hack. My spam trap address isn’t on it, which makes me wonder if unsubscribe data was kept elsewhere or if they just never bothered to save the requests.
One of the things I’m seeing in most articles about the hack is reassurance that Ashley Madison doesn’t verify addresses, so the accounts may not belong to the email address in question. We can’t say that the email address owner is the cheater, because Ashley Madison didn’t care who owned the email address.
The warnings have been published in security blogs.

Yes, Virginia, there is list churn

Yesterday I talked about how data collection, management, and maintenance play a crucial role in deliverability. I mentioned, briefly, the idea that bad data can accumulate on a list that isn’t well managed. Today I’d like to dig into that a little more and talk about the non-permanence of email addresses.
A common statistic used to describe list churn is that 30% of addresses become invalid in a year. This was research done by Return Path back in the early 2000’s. The actual research report is hard to find, but I found a couple articles and press releases discussing the info.

Data is the key to deliverability

Last week I had the pleasure of speaking to the Sendgrid Customer Advisory Board about email and deliverability. As usually happens when I give talks, I learned a bunch of new things that I’m now integrating into my mental model of email.
One thing that bubbled up to take over a lot of my thought processes is how important data collection and data maintenance is to deliverability. In fact, I’m reaching the conclusion that the vast majority of deliverability problems stem from data issues. How data is collected, how data is managed, how data is maintained all impact how well email is delivered.
Collecting Data
There are many pathways used to collect data for email: online purchases, in-store purchases, signups on websites, registration cards, trade shows, fishbowl drops, purchases, co-reg… the list goes on and on. In today’s world there is a big push to make data collection as frictionless as possible. Making collection processes frictionless (or low friction) often means limiting data checking and correction. In email this can result in mail going to people who never signed up. Filters are actually really good at identifying mail streams going to the wrong people.
The end result of poor data collection processes is poor delivery.
There are lots of way to collect data that incorporates some level of data checking and verifying the customer’s identity. There are ways to do this without adding any friction, even. About 8 years ago I was working with a major retailer that was dealing with a SBL listing due to bad addresses in their store signup program. What they ended up implementing was tagged coupons emailed to the user. When the user went to the store to redeem the coupons, the email address was confirmed as associated with the account. We took what the customers were doing anyway, and turned it into a way to do closed loop confirmation of their email address.
Managing Data
Data management is a major challenge for lots of senders. Data gets pulled out of the database of record and then put into silos for different marketing efforts. If the data flow isn’t managed well, the different streams can have different bounce or activity data. In a worst case scenario, bad addressees like spamtraps, can be reactivated and lead to blocking.
This isn’t theoretical. Last year I worked with a major political group that was dealing with a SBL issue directly related to poor data management. Multiple databases were used to store data and there was no central database. Because of this, unsubscribed and inactivated addresses were reactivated. This included a set of data that was inactivated to deal with a previous SBL listing. Eventually, spamtraps were mailed again and they were blocked. Working with the client data team, we clarified and improved the data flow so that inactive addresses could not get accidentally or unknowingly reactivated.
Maintaining Data
A dozen years ago few companies needed to think about any data maintenance processes other than “it bounces and we remove it.” Most mailbox accounts were tied into dialup or broadband accounts. Accounts lasted until the user stopped paying and then mail started bouncing. Additionally, mailbox accounts often had small limits on how much data they could hold. My first ISP account was limited to 10MB, and that included anything I published on my website. I would archive mail monthly to keep mail from bouncing due to a full mailbox.
But that’s not how email works today. Many people have migrated to free webmail providers for email. This means they can create (and abandon) addresses at any time. Free webmail providers have their own rules for bouncing mail, but generally accounts last for months or even years after the user has stopped logging into them. With the advent of multi gigabyte storage limits, accounts almost never fill up.
These days, companies need to address what they’re going to do with data if there’s no interaction with the recipient in a certain time period. Otherwise, bad data just keeps accumulating and lowering deliverability.
Deliverability is all about the data. Good data collection and good data management and good data maintenance results in good email delivery. Doing the wrong thing with data leads to delivery problems.

Email verification services

Just yesterday a group of delivery folks were discussing email verification services over IRC. We were talking about the pros and cons, when we’d suggest using them, when we wouldn’t, which ones we’ve worked with and what our experiences have been. I’ve been contemplating writing up some of my thoughts about verification services but it’s a post I wanted to spend some time on to really address the good parts and the bad parts of verification services.
Today, Spamhaus beat me to the punch and posted a long article on how they view email verification services. (I know that some Spamhaus folks are part of that IRC channel, but I don’t think anyone was around for the discussion we had yesterday.)
It’s well worth a read for anyone who wants some insight into how email verification is viewed by Spamhaus. Their viewpoints are pretty consistent with what I’ve heard from various ISP representatives as well.
In terms of my own thoughts on verification services, I think it’s important to remember that the bulk of the verification services only verify that an address is deliverable. The services do not verify that the address belongs to the person who input it into a form. The services do not verify that an address matches a purchased profile. The services do not verify that the recipient wants email from the senders.
Some of the services claim they remove spamtraps, but their knowledge of spamtraps is limited. Yes, stick around this industry long enough and you’ll identify different spamtraps, and even spamtrap domains. I could probably rattle off a few dozen traps if pressed, but that’s not going to be enough to protect any sender from significant problems.
Some services can be used for real time verification, and that is a place where I think verification can be useful. But I also know there are a number of creative ways to do verification that also check things like permission and data validity.
From an ESP perspective, verification services remove bounces. This means that ESPs have less data to apply to compliance decisions. Bounce rate, particularly for new lists, tells the ESP a lot about the health of the mailing list. Without that, they are mostly relying on complaint data to determine if a customer is following the AUP.
Spamhaus talks about what practices verification services should adopt in order to be above board. They mention actions like clearly identifying their IPs and domains, not switching IPs to avoid blocks and not using dozens or hundreds of IPs. I fully support these recommendations.
Email verification services do provide some benefit to some senders. I can’t help feeling, though, that their main benefit is simply lowering bounce rates and not actually improving the quality of their customers’ signup processes.

Misdirected email

While this does seem to be more common with gmail addresses, it’s not solely limited to gmail. I’ve written about this frequently.

Growing your list carefully

Karl Murray wrote a great set of recommendations for growing an email marketing list. I really can’t think of anything I would have said differently. Touching customers and getting contact information from them is great, but there are situations where this gets bad addresses. Too many bad addresses can impact delivery.
So how do you grow your list without falling into a delivery trap? The specific recommendations, as always, depend on your specific situation. But knowing how bad addresses get onto your list will allow you to implement mitigation strategies that actually work.

Sending mail to the wrong person, part eleventy

Another person has written another blog post talking about their experiences with an email address a lot of people add to mailing lists without actually owning the email address. In this case the address isn’t a person’s name, but is rather just what happens when you type across rows on they keyboard.
These are similar suggestions to those I (and others) have made in the past. It all boils down to allow people who never signed up for your list, even if someone gave you their email address, to tell you ‘This isn’t me.” A simple link in the mail, and a process to stop all mail to that address (and confirm it is true if someone tries to give it to you again), will stop a lot of unwanted and unasked for email.

Verifying addresses after POS collection

Collecting email addresses at point of sale is a challenge. Some stores collect the addresses electronically, where the clerk or the customer types addresses directly into the register. Smaller stores, however, typically collect addresses on a sheet of paper at the cash register. Eventually someone takes the list and types it into whatever contact management system the store maintains.
There are all sorts of errors that can happen when someone types in an address, but those errors are only compounded when the addresses are written on a sheet of paper for later transcription. Not all of us have perfect, copperplate handwriting and many of us have barely legible scribbles. In one case I had a sender read the tag in my email address wrong causing all their mail to me to bounce.
One person found an interesting solution to the problem of illegible addresses: using Facebook’s lookup to clarify illegible addresses.

Email verification – what are we verifying

One of the ongoing discussions in the email space is the one about address verification. Multiple companies have sprung up to do “real time” email address verification. They ensure that addresses collected at the point of sale are valid.
But what does valid mean? In most of these contexts, valid means that the addresses don’t bounce and aren’t spam traps. And that is one part of validating email addresses.
That isn’t the only part, though. In my opinion, an even more important thing to validate is that the email address belongs to the person giving it to you. The Consumerist has had an ongoing series of articles discussing people getting mis-directed email from various companies.
Today the culprit is AT&T, who are sending a lot of personal information to an email address of someone totally unconnected to that account. There are a lot of big problems with this, and it’s not just in the realm of email delivery.
The biggest problem, as I see it, is that AT&T is exposing personally identifiable information (PII) to third parties. What’s even worse, though, is that AT&T has no process in place for the recipient to correct the issue. Even when notified of the problem, support can’t do anything to fix the problem.

Can you verify email addresses in real time?

In a recent discussion about spamtraps and address lists and data collection a participant commented, “[E]very site should be utilizing a real-time email address hygiene and correction service on the front end.” He went on to explain that real time hygiene prevents undeliverable addresses and spamtraps and all sorts of list problems. I was skeptical to say the least.
Yes, there are APIs that can be queried at some of the larger ISPs to identify if an account name is taken, but this doesn’t mean that there is an associated email address. Yes, senders can do a real time SMTP transaction, but ISPs are quick to block SMTP transactions that quit before DATA.
I decided to check out one service to see how accurate it was. I’m somewhat lucky in that I created a username at Yahoo Groups over a dozen years ago but never activated the associated email address. This means that the account is shown as taken and no one else can register that address at Yahoo. But the address doesn’t accept any mail.

Opt-in Reconfirmation in the Wild

What’s an opt-in reconfirmation email? Also called, as fellow blogger Al
Iverson mentioned lately, a re-engagement email, or a permission pass email.
Al links to DJ Waldow’s write up on Shop.org’s recent re-engagement
strategy, and today I see that Janine Popick, CEO of VerticalResponse,
talking about Coach’s turn at culling their list through this process. What’s interesting here is that, according to Janine, Coach didn’t target this reconfirmation email only at recipients who never open or click. She says she does both, regularly, and received this email message anyway. Another friend of mine, who is also a Coach subscriber, reports to me that she receives regular emails from them (most recently as just about
ten days ago), but that she did not receive this reconfirmation email message.

Garbage in… garbage out

Ken Magill (hereafter known as Mr. Stupid Poopypants) has a follow up article today on his article from last week about the Obama campaign’s mailing practices. While poking Dylan a bit, his message is that marketers really need to look harder at double opt-in.