Recent Posts

Using Google to taunt coworkers

Happy Friday, all. This has been a rough week for so many people, I thought we needed a little humor.
From Tim Norton (@norton_tim) on Twitter.

IP reputation and email delivery

IP reputation is a measure of how much wanted mail a particular IP address sends. This wanted mail is measured as a portion of the total email sent from that IP. Initially IP reputation was really the be all and end all of reputation, there was no real good way to authenticate a domain or a from address. Many ISPs built complex IP reputation models to evaluate mail based on the IP that sent the mail.
These IP reputation models were the best we had, but there were a lot of ways for spammers to game the system. Some spammers would create lots of accounts at ISPs and use them to open and interact with mail. Other spammers would trickle their mail out over hundreds or thousands of IPs in the hopes of diluting the badness enough to get to the inbox. Through it all they kept trying to get mail out through reputable ESPs, either by posing as legitimate customers or compromising servers.
These things worked for a while, but the ISPs started looking harder at the recipient pool in order to figure out if the interactions were real or not. They started looking at the total amount of identical mail coming from multiple IP addresses. The ISPs couldn’t rely on IP reputation so they started to dig down and get into content based filtering.
As the ISPs got better at identifying content and filtering on factors other than source IP, the importance of the IP address on inbox delivery changed. No longer was it good enough to have a high reputation IP sending mail.
These days your IP reputation dictates how fast you can send mail to a particular ISP. But a high reputation IP isn’t sufficient to get all the mail in the inbox. It’s really content that drives the inbox / bulk folder decisions these days.

Generally IPs that the ISP has not seen email traffic from before start out with a slight negative reputation. This is because most new IPs are actually infected machines. The negative reputation translates to rate limiting. The rate limiting minimizes people getting spam while the ISP works out if this is a real sender or a spammer.
Some ISPs put mail in the inbox and bulk foldering during the whitelisting process. In this case what they’re doing is seeing if your recipients care enough about your mail to look for it in the bulk folder. If they do, and they mark the mail as “not spam” then this feeds back to the sender reputation and the IP reputation.
If you’re seeing a lot of bulk foldering of mail, it’s unlikely there’s anything IP reputation based to do. Instead of worrying about IP reputation, focus instead on the content of the mail and see what you may need to do to improve the reputation of the domains and URLs (or landing pages) in the emails.

Thoughts on Gmail and the inbox

Over the last few months more and more marketers are finding their primary delivery challenge is the Gmail inbox. I’ve been thinking about why Gmail might be such a challenge for marketers. Certainly I have gotten a lot of calls from people struggling to figure out how to get into the Gmail inbox. I’ve also seen aggressive domain based filtering from Gmail, where any mention of a particular domain results in mail going to the bulk folder.
It’s one of those things that’s a challenge, because in most of these cases there isn’t one cause for bulk foldering. Instead there’s a whole host of things that are individually very small but taken together convince Gmail that the mail doesn’t need to be in the inbox.
A pattern that I’m starting to see is that Gmail is taking a more holistic look at all the mail from a sender. If the mail is connected to an organization, all that mail is measured as part of their delivery decision making. This is hurting some ESPs and bulk senders. I’ve had multiple ESPs contact me in the last 6 months looking for help because all their customer emails are going to bulk folder.
Gmail’s filtering is extremely aggressive. From my perspective it always has been. I did get an invite for a Gmail account way back in the day. I moved a couple mailing lists over to that account to test it with some volume and discussion lists. I gave up not long after because no matter what I did I couldn’t get gmail to put all the mail from that list into the tag I had set up for it. Inevitably some mail from some certain people would end up in my spam folder.
Gmail has gotten better, now they will let you override their filters but give you a big warning that the message would have been delivered to spam otherwise.

What are mailers to do? Right now I don’t have a good answer. Sending mail people want is still good advice for individual senders. But I am not sure what can be done about this ESP wide filtering that I’m starting to see. It’s possible Gmail is monitoring all the mail from a particular sender or ESP and applying a “source network” score. Networks letting customers send mail Gmail doesn’t like (such as affiliate mail or payday mail, things they mentioned specifically at M³AAWG) are having all their customers affected.
I suspect this means that ESPs seeing problems across their customer base are going to have to work harder to police their customers and remove problematic mail streams completely. Hopefully, ESPs that can get on the Gmail FBL can identify the problem customers faster before those customers tank mail for all their senders.

Spam is not a moral judgement

Mention an email is spam to some senders and watch them dance around trying to explain all the ways they aren’t spammers. At some point, calling an email spam seems to have gone from a statement of fact into some sort of moral judgement on the sender. But calling an email spam is not a moral judgement. It’s just a statement of what a particular recipient thinks of an email.
There are lots of reasons mail can be blocked and not all those reasons are spam related. Sometimes it’s a policy based rejection. Mailbox providers publishing a DMARC record with a reject policy caused a lot of mail to bounce, but none of that was because that user (or that mailing list) was sending spam. Most cable companies prohibit customers from running mail servers on their cable connection and mail from those companies is widely rejected, but that doesn’t mean the mail is spam.
Sometimes a block is because some of the mail is being sent to people who didn’t ask for it or are complaining about it. This doesn’t make the sender a bad person. It doesn’t make the sending company bad. It just means that there is some issue with a part of the marketing program that need to be addressed.
The biggest problem I see is some senders get so invested in convincing receivers, delivery experts and filtering companies that they’re not spammers, that they miss actually fixing the problem. They are so worried that someone might think they’re spammers, they don’t actually listen to what’s being said by the blocking organization, or by their ISP or by their ESP.
Calling email spam isn’t a moral judgement. But, if too many people call a particular email spam, it’s going to be challenging to get that mail to the inbox. Instead of arguing with those people, and the filters that listen to them, a better use of time and energy is fixing the reasons people aren’t liking your email.

More from Gmail

Campaign Monitor has an interview with Gmail looking at how to get mail to the Gmail inbox. It’s a great article and I think everyone should go read it.
One of the most important things it talks about is how complex filters are.

April: The month in email

April was a big month of changes in the email world, and here at Word to the Wise as we launched our new site, blog and logo.
DMARC
The big story this month has been DMARC, which started with a policy change Yahoo made on April 4 updating their DMARC policy from “report” to “reject”. We began our coverage with a brief DMARC primer to explain the basics around these policy statements and why senders are moving in this direction. We shared some example bounces due to Yahoo’s p=reject, and talked about how to fix discussion lists to work with the new Yahoo policy. We gathered some pointers to other articles worth reading on the Yahoo DMARC situation, and suggested some options for dealing with DMARC for mail intermediaries. Yahoo issued a statement about this on April 11th, explaining that it had been highly effective in reducing spoofed email. We also noted a great writeup on the situation from Christine at ReturnPath. On April 22nd, AOL also announced a DMARC p=reject record. We talked a bit about who might be next (Gmail?) and discussed how Comcast chose to implement DMARC policies, using p=reject not for user email, but only for the domains they use to communicate directly with customers. We expect to see more discussion and policy changes over the next few weeks, so stay tuned.
Spamtraps
We wrote three posts in our continuing discussion about spamtraps. The first was in response to a webinar from the DMA and EEC, where we talked about how different kinds of traps are used in different ways, and, again, how spamtraps are just a symptom of a larger problem. Following that, we wrote more about some ongoing debate on traps as we continued to point out that each trap represents a lost opportunity for marketers to connect with customers, which is really where we hope email program managers will focus. And finally, we tried to put some myths about typo traps to rest. As I mentioned in that last post, I feel like I’m repeating myself over and over again, but I want to make sure that people get good information about how these tools are used and misused.
Security
We started the month by saying “Security has to become a bigger priority for companies” and indeed, the internet continued to see security breaches in April, including the very serious Heartbleed vulnerability in SSL. In the email world, AOL experienced a compromise, which contributed to some of the DMARC policy changes we discussed above. In a followup post, we talked about how these breaches appear to be escalating. Again, we expect to hear more about this in the next weeks and months.
Best Practices
Ending on a positive note, we had a few posts about best practices and some email basics. We started with a pointer to Al Iverson’s post on masking whois info and why not to do it. Steve wrote up a comprehensive post with everything you ever wanted to know about the From header and RFC5322. I talked about how companies ignore opt-outs, and why they shouldn’t. I shared a really good example of a third-party email message, and also talked about message volume. And finally, we talked about how and why we warm up IP addresses.
Let us know if there’s anything you’d like to hear more about in May!

The true facts of spam traps and typo traps

I’m seeing an increase in the number of articles stating wildly wrong things about spam traps. Some have started claiming that typo traps are new. Or that typo traps are newly used by Spamhaus. These claims make for great copy, I guess. Wild claims about how the evil anti-commerce self-appointed internet police are actively trying to trap marketers get clicks. These claims also reinforce the martyr complex some senders have and gives them something to commiserate about over drinks at the next email conference.
I strongly recommend ignoring any article that claims Spamhaus started using typo traps in December 2012. In fact, you can immediately dismiss absolutely everything they have to say. They are wrong and have proven they can’t be bothered to do any fact checking.
I can’t figure out why so many people repeat the same false statements over and over and over again. They’re wrong, and no amount of explaining the truth seems to make any difference. I went looking for evidence.
First, I asked on Facebook. A bunch of my contacts on Facebook have have been running spam traps for a long time. Multiple people commented that they, personally, have been using typos to track spam since the late ’90s. These typos were on both the right hand side of the @ sign (the domain side) but also on the left hand side of the @ sign (the username).
Then, I looked through my archives of one of the anti-spam mailing lists and I see a Spamhaus volunteer mentioning that he had already been using typo traps in 2007. I asked him about this and he pointed out these are some of his older traps and had been around for many years before that mention.
Of course, we’ve written about typo domains used by an anti-spam group to catch spam.
The truth is, typo traps are not new and they’re not a new set of traps for Spamhaus. I’ve talked about traps over and over again. But I’m seeing more and more articles pop up that make verifiably wrong statements about spam traps. Here are a few facts about spam traps.

DMARC and organizations

Comcast recently published a statement on DMARC over on their postmaster page. The short version is that Comcast is publishing a DMARC record, but has no current intentions to publish a p=reject policy for Comcast user email. Comcast will be publishing a p=reject for some of their domains that they use exclusively to communicate with customers, like billing notices and security notices.
Comcast does point out that Yahoo! and AOL’s usage of p=reject is “not common usage.”
This is something a lot of people have been arguing loudly about on various mail operations lists and network lists. DMARC is about organizational identity. In fact, I was contacted about my DMARC primer and told that I didn’t mention that it’s not about domains, it’s about organizations.
The way I read the DMARC spec, it is all about organizational identity. The underlying theme being that the domain name is linked to a particular organization and everyone using email at that domain has some official relationship with that organization. I’ve always read the spec mentally replacing organization with corporate brand. This was for brands and organizations that strictly control how their domains are used, who can use those domains and how the mail is sent with those domains.
I never expected any mailbox provider or commercial ISP to publish a p=reject message as it would just break way too much of the way customers use email. And it did break a lot of legitimate and end user uses of email. Many organizations have had to scramble to update mailing list software to avoid bouncing users off the lists. Some of these upgrades have broken mailbox filters, forcing endusers to change how they manage their mailboxes.
Even organizations see challenges with a p=reject message and can have legitimate mail blocked. At M³AAWG 30 in San Francisco I was talking with some folks who have been actively deploying DMARC for organizations. From my point of view anyone who wants to publish a DMARC p=reject should spend at least 6 months monitoring DMARC failures to identify legitimate sources of email. The person I was talking to said he recommends a minimum of 12 months.
This is just an example of how difficult it is to capture all the legitimate sources of emails from a domain and effectively authenticate that mail. For a mailbox provider, I think it’s nearly impossible to capture all the legitimate uses of email and authenticate them.
It remains to be seen if the other mailbox providers imitate Yahoo! and AOL or if they push back against the use of DMARC reject policies at mailbox providers. Whatever the outcome, this is a significant shift in how email is used. And we’re all going to have to deal with the fallout of that.

AOL admits to security breach

According to Reuters AOL has admitted there was a breach of their network security that compromised 2% of their accounts. Users are being told to reset their passwords, and security questions.
AOL started investigating the attack after users started reporting an uptick in spam from aol.com addresses. This spam was using @aol.com addresses to send mail to addresses in that user’s address book.
According to the AOL mail team, they are still investigating the attack, but they do not believe financial information was compromised. Their statement reads in part:

Is gmail next?

I’m hearing hints that there are some malware or phishing links being sent out to gmail address books, “from” those gmail addresses. If that is what’s happening then it’s much the same thing as has been happening at Yahoo for a while, and AOL more recently, and that triggered their deployment of DMARC p=reject records.
It’s going to be interesting to see what happens over the next few days.
I’ve not seen any analysis of how the compromises happened at Yahoo and AOL – do they share a server-side (XSS?) security flaw, or is this a client-side compromise that affects many end users, and is just being targeted at freemail providers one at a time?
Does anyone have any technical details that go any deeper than #AOLHacked and #gmailhacked?