Reputation is in the eye of the beholder

A few years ago reputation was generally recognised as one thing. If a sending reputation or IP reputation was good in one place it was likely good in other places. Different entities mostly reputation using the same set of signals albeit slightly tweaked to meet their own needs. More recently there is a divergence in how reputation is measured, meaning delivery can be vastly different across entities.

In simpler times reputation was based on the number of complaints an email generated and the number of non-existent addresses a sender hit with some spamtrap and blocklist data thrown in for good measure. These were pretty easy for each recipient domain to measure and, in fact, most senders could measure them as well. These measurements mostly reflected how well the sender collected and managed data. We could assume that data management was the same no matter what the recipient ISP was. Thus, we could assume reputation was mostly the same across domains.

As spammers adapted to reputation, filters had to evolve to keep mailboxes clean for their users. In most cases, the evolution was based on the data the recipient domains had access to. For webmail providers that manage the mail client as well as the mail server, they had a lot of data related to how users interacted with mail. Thus, engagement emerged as an important metric for those providers. There are a lot of providers, including cable companies, third party filters and blocklists, that don’t have any access to engagement data. They had to find other things to use.

The different data sources drove significant divergence among filters and that changes how we have to cope with them.

For example, engagement filters are really only relevant at the consumer webmail providers. There are two reasons for this. First, they have the data to see how folks interact with mail. Second, they’re the ones who care. The webmail providers want to delivery mail their users want. Thus, they’re going to measure what mail their users want. Other providers can’t get the data, as they don’t control the mail client. In some cases the user’s needs are not as important as the customer needs, like when the provider manages inbound mail for a business.

Cloud MX providers, that accept mail filter it and either deliver directly to the users inbox or to another MTA, often can’t or won’t look at engagement data. But they do have access to traffic data across thousands of different domains. They can use that traffic data as part of their reputation calculations.

There are non-email sources of data as well. For instance, many of the commercial email filters are maintained by companies that also track all types of security events online. Many of these companies include this data in their email reputation metrics as well.

All of this means that we can’t treat reputation as a monolith. It’s no longer the case that the inputs are the same and the algorithms are tuned a little differently. Instead, we’re in a situation where filter maintainers have access to very different types of data and thus the reputations are really specific to the provider measuring the reputation.

Related Posts

All filters are not equal

Many questions about delivery problems often assume that there is one standard email filter and the rules are the same across all of them. Unfortunately, this isn’t really the case.

Read More

What's up with microsoft?

A c/p from an email I sent to a mailing list.
I think we’re seeing a new normal, or are still on the pathway to a new normal. Here’s my theory.
1) Hotmail made a lot of underlying code changes, learning from 2 decades of spam filtering. They had a chance to write a new codebase and they took it.
2) The changes had some interesting effects that they couldn’t test for and didn’t expect. They spent a month or two shaking out the effects and learning how to really use the new code.
3) They spent a month or two monitoring. Just watching. How are their users reacting? How are senders reacting? How are the systems handling everything?
3a) They also snagged test data along the way and started learning how their new code base worked and what it can do.
4) As they learned more about the code base they realized they can do different and much more sophisticated filtering.
5) The differences mean that some mail that was previously OK and making it to the inbox isn’t any longer.
5a) From Microsoft’s perspective, this is a feature not a bug. Some mail that was making it to the inbox previously isn’t mail MS thinks users want in their inbox. So they’re filtering it to bulk. I’ll also step out on a limb and say that most of the recipients aren’t noticing or caring about the missing mail, so MS sees no reason to make changes to the filters.
6) Expect at least another few rounds of tweak and monitor before things settle into something that changes more gradually.
Overall, I think delivery at Microsoft really is more difficult and given some of the statements coming out of MS (and some of the pointed silence) I don’t think they’re unhappy with this.

Read More