Deliverability and IP addresses

laura
Industry
May 29, 2015

Almost 2 years ago I wrote a blog post titled The Death of IP Based Reputation. These days I’m even more sure that IP based reputation is well and truly dead for legitimate senders.
There are a lot of reasons for this continued change.

Improved computing power

I touched on the increase in computing power in my 2013 post. The power and the complexity of filters in even greater now than then. Filters can sort through all sorts of variables faster than ever. They’re now fast enough to keep up with the high volume of email received at most incoming servers.

Planning for IPv6

In a world with 340 trillion trillion trillion IP addresses IP based reputation isn’t going to work very well. We’re not yet at the point where a lot of email is going out over IPv6, but many people are working on filtering already. When a spammer can use an IP per email, IPs are just not useable for blocking email. The smart folks who develop and maintain spam filters are planning ahead and developing technologies that don’t rely on the IP address. These technologies are being developed on current IPv4 infrastructure, and we’re seeing the effects.

Consumer demand

Years ago I was sitting in a meeting talking with a lot of very smart people about filtering. During the discussion a representative from Yahoo! mentioned that it was hard to make global decisions about email when some people really wanted it and some people really didn’t want it. Consumers want the email they want and they don’t want email they don’t want. This demand has helped drive filters away from the all or nothing approach.

Better anti-spam programs

Spam and other types of malicious emails are a global problem. Criminals are using email to gain access to individual computers and using that access to launch bigger compromises. Many of the major compromises over the past few years started with email, including Target, The Oak Ridge National Laboratory, ICANN and even the RSA. Over the past few years filtering companies and organizations have worked closely with law enforcement across the world. These groups have identified and removed a number of criminal gangs from the internet, and even society. In addition to the legal work being done, many legitimate ISPs and network providers police and disconnect spammers and other email abusers. Overall, the effect is to restrict truly bad senders and criminals away from legitimate IPs. We now have bad IP neighborhoods and good IP neighborhoods. Senders in good IP neighborhoods see less IP blocking than senders in bad IP neighborhoods.

Better communication and partnerships

Filtering companies, ESPs, ISPs and large senders are working together on the spam problem. M³AAWG is part of that by simply giving diverse groups a place to talk, interact and develop relationships. These relationships were tentative at first, but continue to develop. M³AAWG is not the sole reason for the increase in cooperation. A number of individuals, on both the sending and the filtering side, took a risk and reached across the divide to work together. These relationships have changed how filters look at senders and how senders look at filters. This leads to less abuse and less need for IP based filtering.

Domain based authentication

Technologies like DKIM and SPF and DMARC have given filters and ISPs the ability to trust more data than the connecting IP address. These frameworks provide other data points that can be trusted. Trusted data can be used for reputation and that reputation can be used to filter email.
IP reputation was always a big, big club. It could affect lots of email from different senders (think shared IPs at an ESP or an ISPs outgoing servers). But most IPs don’t send all good email or all bad email. Most IPs send mixed email, and IP based reputation is really bad at dealing with that.
Newer filtering technologies mean that IP reputation isn’t as important for deliverability as it used to be. IP reputation is important for the SMTP transaction, and it will always be. There are too many “blackhat” IPs for blocking to go away completely. But once a receiver has accepted an email the IP reputation has done its part for delivery. The receiver knows that the email is coming from an IP with a minimum reputation. All the other reputation factors (domains, links, content, images) influence where that email ends up.
We don’t need the IP club any more. This is good news for good senders. Deliverability now depends more on that specific email and that specific recipient than the IP the message was sent from. This means senders really can focus more on meeting the needs of their recipients and less time worried about the health of their sending IP.

Why do we "warmup" IP addresses

IP address warmup is a big issue for anyone moving to a new IP address for sending.
I’m constantly being asked how to warm up an IP. My answer is always the same. There’s no right way to warm up an IP nor is there a specific formula that everyone should follow.
What warming up is about is introducing mail traffic to receiving spam filters in a way that lets the filter know this is a legitimate email stream. This means sending small but regular amounts of mail that recipients interact with. As the filters adjust to the amount of mail from that IP, more mail can be sent over that IP. Increase the mail volume over the next few weeks until the desired volume is reached.
There are a couple things to remember about warming up.

Delivery challenges increasing

Return Path published their most recent Global Deliverability report this morning. (Get the Report) This shows that inbox placement of mail has decreased 6% in the second half of 2011. This decrease is the largest decrease Return Path has seen in their years of doing this report.
To be honest, I’m not surprised at the decrease. Filters are getting more sophisticated. This means they’re not relying on simply IP reputation for inbox delivery any longer. IP reputation gets mail through the SMTP transaction, but after that mail is subject to content filters. Those content filters are getting a lot better at sorting out “wanted” from “unwanted” mail.
I’m also hearing a lot of anecdotal reports that bulk folder placements at a couple large ISPs increased in the first quarter of 2012. This is after the RP study was finished, and tells me increased bulk folder placement is more likely to be a trend and not a blip.
One of the other interesting things from the RP study is that the differences are not across all mail streams, but are concentrated in certain streams and they vary across different regions.

IP reputation and email delivery

IP reputation is a measure of how much wanted mail a particular IP address sends. This wanted mail is measured as a portion of the total email sent from that IP. Initially IP reputation was really the be all and end all of reputation, there was no real good way to authenticate a domain or a from address. Many ISPs built complex IP reputation models to evaluate mail based on the IP that sent the mail.
These IP reputation models were the best we had, but there were a lot of ways for spammers to game the system. Some spammers would create lots of accounts at ISPs and use them to open and interact with mail. Other spammers would trickle their mail out over hundreds or thousands of IPs in the hopes of diluting the badness enough to get to the inbox. Through it all they kept trying to get mail out through reputable ESPs, either by posing as legitimate customers or compromising servers.
These things worked for a while, but the ISPs started looking harder at the recipient pool in order to figure out if the interactions were real or not. They started looking at the total amount of identical mail coming from multiple IP addresses. The ISPs couldn’t rely on IP reputation so they started to dig down and get into content based filtering.
As the ISPs got better at identifying content and filtering on factors other than source IP, the importance of the IP address on inbox delivery changed. No longer was it good enough to have a high reputation IP sending mail.
These days your IP reputation dictates how fast you can send mail to a particular ISP. But a high reputation IP isn’t sufficient to get all the mail in the inbox. It’s really content that drives the inbox / bulk folder decisions these days.

Generally IPs that the ISP has not seen email traffic from before start out with a slight negative reputation. This is because most new IPs are actually infected machines. The negative reputation translates to rate limiting. The rate limiting minimizes people getting spam while the ISP works out if this is a real sender or a spammer.
Some ISPs put mail in the inbox and bulk foldering during the whitelisting process. In this case what they’re doing is seeing if your recipients care enough about your mail to look for it in the bulk folder. If they do, and they mark the mail as “not spam” then this feeds back to the sender reputation and the IP reputation.
If you’re seeing a lot of bulk foldering of mail, it’s unlikely there’s anything IP reputation based to do. Instead of worrying about IP reputation, focus instead on the content of the mail and see what you may need to do to improve the reputation of the domains and URLs (or landing pages) in the emails.