Meaningless metrics

I’ve been having some conversations with fellow delivery folks about metrics and delivery and bad practices. Sometimes, a sender will have what appear to be good metrics, but really aren’t getting them through any good practices. They’re managing to avoid the clear indicators of bad practices (complaints, SBL listings, blocks, etc), but only because the metrics aren’t good.
This made me laugh when a friend posted a link to a Business Insider article about how many website metrics aren’t useful indicators of the business value of a website.  Then I found the original blog post referenced in the article: Bullshit Metrics. It’s a great post, you should go read it.
I’d say the concluding paragraph has as much relevance to email marketing as to web marketing.

Despite the internet’s evolution, bullshit metrics perpetuate a constant cycle of poor understanding. Let’s strive to understand how our businesses are doing and to pick better metrics–the harsher, the better. Let’s stop fooling ourselves with numbers that don’t represent reality. And let’s push the industry forward as a whole because collectively we’ll all benefit.

The sooner we can get away from opens as a useful email metric, the better the email industry is going to be.
 

Related Posts

Standardizing email metrics

Slogging towards e-mail metrics standardization a report by Direct Mag on the efforts of the Email Experience Council to standardize definitions related to email marketing.

Read More

Reporting email disposition

Most regular readers know I think open and click through rates are actually proxy measurements. That is they measure things that correlate with reading and interacting with an email and can be used to estimate how much an email is wanted by the recipients.
The holy grail is, of course, having ISPs report back exact metrics on what a user did with an email. Did the user read it? Did it stay open on their screen a long time? Did the user just mark it read or throw it away? What happened to the message. Marketers would love this information.
It’s unlikely the ISPs will ever provide this information to marketers. Take away all the technical challenges, and there are some significant ones there are still social challenges to making this data available. Current user contracts protect the privacy of the user, local laws prohibit sharing this data. And, there is the vocal group of privacy advocates that will protest and raise a big stink.
I’m not sure why email is gets the special treatment of expecting the channel owners to provide detailed disposition data. In no other direct marketing venue is that information collected or provided. TV stations can’t tell advertisers whether or not someone watched a commercial, fast forwarded through it or got up to grab a beer from the fridge. The post office can’t tell direct mail marketers whether or not a recipient read the mail or just dumped it in the big recycling bin the post office provides for unwanted messages. Billboard owners can’t tell advertisers how many people read the billboard.
Since we can’t get exact read rates from ISPs, what do we do? We look at proxy numbers.
Read rate directly measures who opened the message. Open rate is a proxy. It’s who displayed images in the message.
Read rate can be measured only by people who have access to the user’s inbox. The ISPs can measure read rate because they have full access to the mailbox, but this requires the user to access the mailbox through webmail or IMAP. Some third party mailbox addons can measure it, but this requires the cooperation of the mailbox owner. If the mailbox owner doesn’t install the reporting tool, then the 3rd party doesn’t have access to the data. Only groups with access to the end users mailbox can measure this rate.
Open rate can be measured by people who have access to the server images are hosted. Senders and ESPs and 3rd parties can measure it if they provide unique image IDs or tracking pixels in their emails. Open tracking does require the cooperation of the recipient – they have to have images on. No images on, no open tracking. Ironically, ISPs cannot measure open rate, because they have no access to the image hosting servers.
Click rate can be measured by people who have access to the server that hosts the website. The same people who can measure opens can measure clicks. Some ISPs can measure clicks, Hotmail used to pass every URL through a proxy they hosted and they could count clicks this way. AOL controls the client so they could measure number of clicks on a link. I’ve heard trustworthy folks claim that ISPs are measuring clicks and that they’re not measuring clicks (any of the Barry’s want to comment?).
Without controlling the inbox, though, senders have to rely on proxy measurements to judge the effectiveness of any particular campaign. But at least email marketers have proxies to use for measurement.

Read More

Delivery Metrics

Last week ReturnPath published a study that shows 20% of permission based email fails to be delivered to the inbox. For this study, ReturnPath looked at the mail sent by their mailbox monitor customers and counted the number of deliveries to the inbox, the number of deliveries to the bulk folder and the number of emails that were not delivered.
At US ISPs 21% of the permission based emails sent to the ReturnPath probe network did not make it to the inbox. 3% of the emails sent went to the bulk folder and 17% did not make it to the mailbox at all.  MSN/Hotmail and Gmail were the worst ISPs to get mail to. They each failed to deliver more than 20% of the mail that was sent to them. At Canadian ISPs, even less of the mail made it to the inbox, primarily because primus.ca is such a large portion of the Canadian market and they use Postini as a filter. Postini is a quite aggressive filter and takes no feedback from senders.
ReturnPath’s take home message on the survey is that one set of metrics is not enough to effectively evaluate a marketing program. Senders need to know more about their mailings than they can discover from just the bounce rate or the revenue rate or response rate or open rate.
There are a lot of reasons an email doesn’t get to the recipient’s inbox or bulk folder. Mail can be hard blocked at the MTA, and rejected by the ISP outright. Mail can be soft blocked at the MTA and the ISP can slow down sending. Sometimes this is enough to cause the sending MTA to stop attempting to deliver the mail, thus causing mail to not show up. Both of these types of blocks are usually visible when looking at the bounce rate.
Some ISPs accept mail but then fail to deliver it to the recipient. Everything on the sender end says the ISP accepted it for delivery but the ISP just drops it on the floor. This is the type of block that a mailbox monitoring program is best able to identify.
Despite all the discussions of numbers, many marketers are still not measuring the variables in their email campaigns. Ken Magill wrote today about a study released by eROI that indicates more than a third of marketers are not doing any testing on their mailings.
Now, both of these studies are done in an attempt to sell products, however, the numbers discussed should be making smart senders think about what they are measuring in regards to their email campaign, how they are measuring those factors and what the measurements mean.

Read More