In this part of my series on Campaign Stats and Measurements I will be examining open rates, how they are used, where they fail and how the can be effectively used.
There has been an lot written about open rates recently, but there are two posts that stand out to me. One was the EEC’s post on renaming open rate to render rate and Mark Brownlow’s excellent post on what open rate does and does not measure. I’ve also weighed in on the subject.
Overall, I find open rates to be a very frustrating metric. Some senders, particularly those relatively new to email marketing, are so sure they know what open rate is and what it means, that they don’t take any time to actually understand the number. While the name “open rate” seems self explanatory, it’s actually not. Open rate is actually not a measure of how many recipients open an email. However, there are times where open rate is a useful metric for measuring a marketing program over time.
What is an open?
If asked, most people will tell you that open rate is the number of emails that were opened by the recipients. The problem is that this isn’t actually true. An open is counted when a tagged image in an email is rendered by the recipient’s email client. Not all mail clients render images by default, but the emails are still available for the recipient to read. If a user clicks on a link in an email that has not had an image rendered, some ESPs count that as an open as well as a click. In other cases, visiting a link in an email with no image rendered is just a click, no open is recorded.
What is the open rate?
Open rate is generally the percentage of email opens divided by some number representing the number of emails sent. Many senders use the number of emails sent minus the number of bounced emails, others use just the number of emails sent without factoring in the number of emails bounced.
Open rate is a secondary metric. While it does not measure the success, or failure, of a campaign directly, it can be used as a indicator for campaigns. Many people use open rate as a metric because it’s easy to measure. Direct metrics, such as clicks or average purchase or total purchase, may take days or even weeks to collect and analyze. Open rates can be calculated quickly and easily.
What the open rate isn’t
Open rate is not a measure of how many people opened a mail. It is not a measure of how many people read a mail. It really only records that an image in a particular email is loaded and, sometimes, that a link was clicked on. Open rates can be wildly different depending on how the sender measures opens and how the sender measures sends.
What senders use open rates for
To compare their open rates with industry averages
As I talked about above, this use of open rates is problematic at best. You cannot compare numbers, even when they have the same name, if the numbers were arrived at using different calculations. Open rate is not open rate and unless you know the underlying algorithm used you cannot compare two open rates. This is a poor use of open rate.
As a metric for advertising rates
Since a sender can manipulate the open rate by using different calculation methods, this is a good metric for the advertiser to use. It is not so great for the purchaser though, who is at the mercy of the sender’s metrics. There are contractual ways a purchaser can protect herself from an unscrupulous marketer, but only if she understands how open rate can be manipulated and takes steps to define what open rate is in use.
To judge the success of campaigns over time
A single open data point doesn’t mean very much, however, using consistently measured open rates a sender can measure trends. Open trends over time are one area that open rates can help senders judge the success, or failure, of a marketing campaign.
As one metric in A/B testing
Comparing open rates in A/B testing gives some indication of which campaigns recipients may be more interested in. As with trends over time, the lone measurement isn’t useful, but as a comparative metric, it may provide senders with insight into a particular mailing.
To judge the engagement of recipients
Over the long term, recipients who do not interact with a mailing become dead weight on the list. Too many non responders can hurt a sender’s reputation at an ISP. List hygiene, in the form of removing people who never open or click on an email, is an important part of reputation management.
As metrics for email campaigns go, open rate is limited in what it measures about an email campaign. However, as a quick way to measure trending or do head to head comparisons it is a useful metric.
Measuring open rate
M
RSS - Posts
1. What you define as “Open” is not what many ESP define as Open. Many counts also “clicked but not rendered” emails.
2. EEC is not proposing to RENAME OpenRate to RenderRate. They are 2 different things and RenderRate is a NEW metric. IMHO both can survive, RenderRate simply leave less space for personalization of the metric (as you see we, experts, already disagree on the definition of OpenRate).
3. As you explained at the top of the article, when I ask to senders what OpenRate is they tell me the definition you wrote under “What the open rate isn’t” 😉 So I’d add a “Should” to “What senders use open rates for”.
And now my question: don’t you think that RenderRate will slightly reduce the misunderstanding of the metric?
Laura,
Thanks for putting this together, and for not just broadly disparaging/disregarding open rate. It’s an imperfect metric, but not a useless one. This is the most thought-out critique of its uses and shortcomings I’ve seen.
Stefano,
I believe Laura touches on inferred opens (“clicked but not rendered”) in the 4th paragraph (“What is an open?”).
Hi, Stefano
1) I briefly addressed that some clicks with no renders were counted. “If a user clicks on a link in an email that has not had an image rendered, some ESPs count that as an open as well as a click. In other cases, visiting a link in an email with no image rendered is just a click, no open is recorded.”
2) I didn’t really want to get into the RenderRate discussion in this post as it would distract from the point which was to clear up some of the misconceptions about Open Rate. However, there are some systemic problems with the RenderRate. I think the name indicates one thing: number of emails that had an image rendered. Unfortunately, the calculation is something completely different: number of emails that had an image rendered plus the number of emails with no image rendered but did have a click. That’s why I said rename, because the instance you include an email that did not render an image, you’re actually measuring something other than the number of images rendered. I also think that the measurement doesn’t measure anything relevant about performance. RenderRate is also a secondary measurement. I prefer to measure things that directly improve the client’s bottom line.
3) At least a couple of the examples I used are things that people do, but shouldn’t, use Open Rate for. As I said in a private email to someone, I believe he second best thing to do with open rate is standardize the calculation and rename. The best is, of course, take it out back and put it out of our misery.
Don’t you think that RenderRate will slightly reduce the misunderstanding of the metric?
Not really. The EEC is including non-rendered emails in the calculation which is not going to decrease any confusion. They’re also attempting to change an existing lexicon in an entire industry which is difficult at best. I don’t see RenderRate replacing Open Rate
@Laura
1) Right, sorry! When I replied I already forgot what i read 🙁
2) As far as I understood it the Render Rate DOES NOT include the clicks, but only the image rendering. That’s the whole point of it existence. It’s not a new name for the OpenRate, it is something different that measure a different thing.
I just downloaded the “The Email Render Rate – 01-09.pdf” and the definition only include “images”, never talk about clicks. clicks are included in the Actions metrics.
If they changed this, and RenderRate is simply a new name for OpenRate then I agree that we don’t need it, but I hope it’s not.
Hi, Stefano,
I remember reading the click / render definition in the EEC pdf. However, looking back it seems I remember what I read on http://blog.emailexperience.org/2009/04/the_value_of_the_render_rate.html
That link also define the render rate not including clicks:
——
Render Rate is calculated using *only* the first method, while Action Rate is calculated by combining the unique results of both methods.
——
The 2 methods are defined as
——
1) A unique tracking image inserted into the email was loaded: confirming that the images in the email were rendered.
2) A link in the email was clicked: since obviously a link cannot be clicked if the email was not opened, this method allows us to track some of the people missed by the first method.
——
[…] Part 1: Campaign Stats and Measurements Part 2: Measuring Open Rate […]