The data are what they are

I’ve had a lot less opportunity to blog at the recent M3AAWG conference than I expected. Some of it because of the great content and conversations. Another piece has to do with lack of time and focus to edit and refine a longer post prompted by the conference. The final issue is the confidential nature of what we talk about.
With that being said, I can talk about a discussion I had with different folks over the looking at A/B testing blog post from Mailchimp. The whole post is worth a quick read, but the short version is when you’re doing A/B testing, design the test so you’re testing the relevant outcomes. If you are looking for the best whatever to get engagement, then your outcome should be engagement. If you’re looking for the best thing to improve revenue, then test for revenue.
Of course, this makes perfect sense. If you do a test, the test should measure the outcome you want. Using a test that looks at engagement and hoping that translates to revenue is no better than just picking one option at random.
That particular blog post garnered a round of discussion in another forum where folks disagreed with the data. To listen to the posters, the data had to be wrong because it doesn’t conform to “common wisdom.” The fact that data doesn’t conform to common wisdom doesn’t make that data wrong. The data is the data. It may not answer the question the researcher thought they were asking. It may not conform to common wisdom. But barring fraud or massive collection error, the data are always that. I give Mailchimp the benefit of the doubt when it comes to how they collect data as I know they have a number of data scientists on staff. I’ve also talked with various employees about digging into their data.
At the same time the online discussion of the Mailchimp data was happening, there was a similar discussion happening at the conference. A group of researchers got together to ask a question. They did their literature review, they stated their hypothesis, they designed the tests, they ran the tests. Unfortunately, despite this all being done well, the data showed that their test condition had no effect. The data were negative. They asked the question a different way, still negative. They asked a third way and still saw no difference between the controls and the test.
They presented this data at the conference. Well, this data went against common wisdom, too, and many of the session participants challenged the data. Not because it was collected badly, it wasn’t, but because they wanted it to say something else. It was the conference session equivalent of data dredging or p-hacking.

 
Overall, the data collected in any test from a simple marketing A/B testing through to a phase III clinical trial, is the answer to the question you asked. But just having the data doesn’t always make the next step clear. Sometimes the question you asked isn’t what you tested. This doesn’t mean you can retroactively find signal in the noise.
Mailchimp’s research shows that A/B testing for open rates doesn’t have any affect on revenue. If your final goal is to know which copy or subject line makes more revenue, then you need to test for revenue. No amount of arguing is going to change that data.
 
 

Related Posts

Incentivizing incites fraud

There are few address acquisition processes that make me cringe as badly as incentivized point of sale collection. Companies have tried many different ways to incentivize address collection at the point of sale. Some offer the benefit to the shopper, like offering discounts if they supply an email address. Some offer the benefits to the employee. Some offer punishments to the employee if they don’t collect addresses from a certain percentage of customers.
All of these types of incentive programs are problematic for email collection.
listshoppingcart
On the shopper side, if they want mail from a retailer, they’ll give an address simply because they want that mail.  In fact, asking for an address without offering any incentive is way more likely to get their real address. If they don’t want mail but there is a financial incentive, they’re likely to give a made up address. Sometimes it will be deliverable, but belong to another person. Sometimes it will be undeliverable. And sometimes it will be a spamtrap. One of my delivery colleagues occasionally shares addresses she’s found in customer lists over on her FB page. It’s mostly fun stuff like “dont@wantyourmail.com” and “notonyour@life.com” and many addresses consisting of NSFW type words.
On the employee side there can also be abuses. Retailers have tried to tie employee evaluations, raises and promotions to the number of email addresses collected. Other retailers will actively demote or fire employees who don’t collect a certain number of addresses. In either case, the progression is the same. Employees know that most customers don’t want the mail, and they feel bad asking. But they’re expected to ask, so they do. But they don’t push, so they don’t get enough addresses. Eventually, to protect their jobs, they start putting in addresses they make up.
Either way, incentivizing point of sale collection of information leads to fraud. In a case I read about in the NY Times, it can lead to fraud much more serious than a little spam. In fact, Wells Fargo employees committed bank fraud because of the incentives related to selling additional banking products at the teller.

Read More

Do you know where your signups are?

Here at Word to the Wise we sign up for a lot of email from our customers. There are multiple reasons we do this.

Read More

Edison acquires part of Return Path

Today Matt Blumberg announced that Edison Software acquired Return Path’s Consumer Insight division, current customers and some Return Path staff.
Congrats to everyone involved.

Read More