How accurate are reports?

One of the big topics of discussion in various deliverability circles is the problems many places are seeing with delivery to Microsoft properties. One of the challenges is that Microsoft seems to be happy with how their filters are working, while senders are seeing vastly different data. I started thinking about reporting, how we generate reports and how do we know the reports are correct.

Everyone I know has bodged a SQL query at some point or another. I shared one of my scripts with Steve just recently and he pointed out that I left out a % so that one line wasn’t going to match. When I’m creating scripts, I check and compare them with manual queries and making sure the the right number of records are updated. But, apparently I missed this one query. What it does mean is all my reporting will be wrong. Now, in this case, it’s not a huge deal. The domain in question belonged to a free email provider acquired back in 2016 and who may or may not actually provide email services any longer.

Microsoft has its own history of problematic reporting in their SNDS product. They provide clear “red, yellow, green” coding of mail. According to their documentation red is definitely spam, green is definitely not spam and yellow is the 80% in the middle. Makes total sense and sounds awesome. The problem is that the colors seem to have no correlation to how mail is delivered. I’ve had clients with solid red and great inbox delivery and solid green and all their mail goes to spam. The reporting doesn’t match the behaviour. In fact, my go to answer for SNDS color questions is “the colors are a lie.”

Thing is, the colors have been a lie for as long as I’ve been using SNDS. I’ve told MS folks the colors are a lie. I’ve filed reports about the colors not accurately reflecting delivery. Most of the time the MS employees simply agree with me.

All of which led me to the place that maybe some of the problem with Microsoft is that some of their internal reporting is wrong. That what they’re seeing isn’t accurately reflecting what’s happening with delivery. Maybe someone bodged a SQL query but there’s no incentive to go back and check all the queries. When you’re monitoring delivery and filtering, there has to be reporting, there’s just too much information to go through it by hand.

The part about Microsoft is all rampant speculation on my part. But there is a lesson here for anyone working in “big data” – and these days email marketing and deliverability is big data. Regularly run your reports against a known data set, make sure it’s reporting what you think it is and that it’s giving you accurate information. Inaccurate reports are unactionable but unless you check you’d never know if your reporting was broken.

 

Related Posts

SNDS issues and new Gmail

A bunch of folks reported problems with Microsoft’s SNDS page earlier today. This afternoon, our friendly Microsoft rep told the mailop mailing list that it should be fixed. If you see problems again, you can report it to mailop or your ESP and the message will get shared to the folks who can fix it.
The other big thing that happened today was Gmail rolled out their new inbox layout.
It’s… nice. I’ll be honest, I am not a big gmail user and have never been a huge fan. I got my first account way-back-during-the-beta. I used it to handle some of my mailing list mail. I could never work out how to get it to stop breaking threads by deciding to put some mail into the junk folder. I just gave up and went back to my shell with procmail (now sieve) scripts. I still have a couple lists routed to my gmail account, and the filtering is much improved – I can at least tell it to never bulk folder certain email.
The feature I’m really interested in is the confidential, expiring email. I’m interested in how that’s going to work with non-Gmail accounts. Within Gmail makes perfect sense, but I don’t think Gmail can control mail once it’s off their system.

My best guess is that Gmail will end up sending some type of secure link to recipients using non-Gmail mail servers. The message itself will stay inside Google and recipients will only be able to view mail through the web. That’s how the vast majority of secure mail systems work.
If anyone has the secure message already, feel free to send me a secure message. I’ll report back as to how it works.

Read More

SNDS News

A number of people have mentioned over the last week or so that they’re seeing a lot of outages, failures and general ickiness with SNDS. I contacted Microsoft and asked about it. SNDS has been undergoing some upgrades and improvements and the outages were not intended to be end user visible. They’re going to keep a closer eye on things, while they finish the upgrades.
The good news in all of this is that SNDS is being upgraded and maintained. SNDS is still a functioning part of the Microsoft infrastructure, and this is good news for anyone who uses it as a data source.

Read More

What kind of mail do filters target?

All to often we think of filters as a linear scale. There’s blocking on one end, and there’s an inbox on the other. Every email falls somewhere on that line.
Makes sense, right? Bad mail is blocked, good mail goes to the inbox. The bulk folder exists for mail that’s not bad enough to block, but isn’t good enough to go to the inbox.
Once we get to that model, we can think of filters as just different tolerances for what is bad and good. Using the same model, we can see aggressive filters block more mail and send more mail to bulk, while letting less into the inbox. There are also permissive filters that block very little mail and send most mail to the inbox.
That’s a somewhat useful model, but it doesn’t really capture the full complexity of filters. There isn’t just good mail and bad mail. Mail isn’t simply solicited or unsolicited. Filters take into account any number of factors before deciding what to do with mail.

Read More