Ask Laura: What should we be measuring?

Dear Laura,
We are trying to evaluate the success of our email programs, and I don’t have a good sense of what metrics we should be monitoring. We have a lot of data, but I don’t have a good sense of what matters and what doesn’t. Can you advise us what we should look at and why?
Thanks,
Metrics Are Hard

Dear Metrically-Challenged,
You’re not going to like this answer, but here goes.
It depends.
If you’re sending newsletters and general brand mail, you’ll want to track clicks and opens to look at how engaged your recipients are with your content. This will help you evaluate the success of individual messages and campaigns, as well as your larger program efforts. You can also use this information to further segment and market to your most engaged (or least engaged) recipients.
If you’re sending marketing mail, you need to look at revenue as well. You need to understand how email engagement translates to purchases, both by campaign and over the customer lifetime.
And for any kind of mail you send, you need to keep an eye on bounces, complaints and unsubscribes. These can be valuable early indicators of both technical issues and marketing success.
The biggest question is: what data do you have access to? When we talk to clients, we often find that they have SO MUCH DATA, but they have no idea how to analyze it and make sense of what they’re seeing. As you point out, there are a lot of numbers to look at. Whether you’re sending mail directly or working with an email service provider, you likely have more dashboards and reports than you know what to do with. You need to figure out what you have and what matters most to you.
On the deliverability front, you can look at your logs to see if there are any ISPs temp failing mail. This will tell you if there’s some reputation issues. Y! and AOL both have specific codes for “come back later” and they’re helpful to ID if there’s something problematic with your reputation.
“Unknown users” is also a valuable metric. If you’re using a data hygiene service, you’ll want to monitor how many addresses they’re removing. If it’s more than 1 – 5%, then you need to look at your address collection process.
Opens and clicks are reasonable metrics to measure. Marketers also look at click-to-open-rate (CTOR), but that’s not something I use for deliverability — It’s more about how many people are interacting with your mail.
Mailbox monitoring tools are less useful than they were, but can still provide interesting information.
Another useful thing to consider is to identify what filters your mail goes through. We do this by taking the MX for every domain on a mailing list, and then identifying the number of email addresses behind each MX.
Overall, you want to make sure you’re looking at the same metrics over time so you can be aware of significant changes in delivery and marketing effectiveness. Depending on your mail types and volumes, there are numbers you’ll want to look at daily, others weekly or monthly, and still others only as needed. Ultimately, there’s no one-size-fits-all answer to what metrics matter to businesses — it’s up to you to determine what matters most to you.

Confused about delivery in general? Trying to keep up on changing policies and terminology? Need some Email 101 basics? This is the place to ask. We can’t answer specific questions about your server configuration or look at your message structure for the column (please get in touch if you’d like our help with more technical or forensic investigations!), but we’d love to answer your questions about how email works, trends in the industry, or the joys and challenges of cohabiting with felines.