Research

The data are what they are

I’ve had a lot less opportunity to blog at the recent M3AAWG conference than I expected. Some of it because of the great content and conversations. Another piece has to do with lack of time and focus to edit and refine a longer post prompted by the conference. The final issue is the confidential nature of what we talk about.
With that being said, I can talk about a discussion I had with different folks over the looking at A/B testing blog post from Mailchimp. The whole post is worth a quick read, but the short version is when you’re doing A/B testing, design the test so you’re testing the relevant outcomes. If you are looking for the best whatever to get engagement, then your outcome should be engagement. If you’re looking for the best thing to improve revenue, then test for revenue.
Of course, this makes perfect sense. If you do a test, the test should measure the outcome you want. Using a test that looks at engagement and hoping that translates to revenue is no better than just picking one option at random.
That particular blog post garnered a round of discussion in another forum where folks disagreed with the data. To listen to the posters, the data had to be wrong because it doesn’t conform to “common wisdom.” The fact that data doesn’t conform to common wisdom doesn’t make that data wrong. The data is the data. It may not answer the question the researcher thought they were asking. It may not conform to common wisdom. But barring fraud or massive collection error, the data are always that. I give Mailchimp the benefit of the doubt when it comes to how they collect data as I know they have a number of data scientists on staff. I’ve also talked with various employees about digging into their data.
At the same time the online discussion of the Mailchimp data was happening, there was a similar discussion happening at the conference. A group of researchers got together to ask a question. They did their literature review, they stated their hypothesis, they designed the tests, they ran the tests. Unfortunately, despite this all being done well, the data showed that their test condition had no effect. The data were negative. They asked the question a different way, still negative. They asked a third way and still saw no difference between the controls and the test.
They presented this data at the conference. Well, this data went against common wisdom, too, and many of the session participants challenged the data. Not because it was collected badly, it wasn’t, but because they wanted it to say something else. It was the conference session equivalent of data dredging or p-hacking.

Overall, the data collected in any test from a simple marketing A/B testing through to a phase III clinical trial, is the answer to the question you asked. But just having the data doesn’t always make the next step clear. Sometimes the question you asked isn’t what you tested. This doesn’t mean you can retroactively find signal in the noise.
Mailchimp’s research shows that A/B testing for open rates doesn’t have any affect on revenue. If your final goal is to know which copy or subject line makes more revenue, then you need to test for revenue. No amount of arguing is going to change that data.

GDPR and Whois data

For folks who aren’t following the discussion about whois records and GDPR compliance there’s a decent summary at vice.com: What Is Going to Happen With Whois?

Information sharing and the Internet

Many years ago I was working at the UW-Madison. Madison is a great town, I loved it a lot. One of the good bits was this local satire paper called The Onion. This paper would show up around campus on Wednesdays. Our lab, like many university employees and students, looked forward to Wednesday and the new humor The Onion would bring to us.
At the same time, I was internet friends with an employee of JPL. I’d met him, like I met many of my online acquaintances, through a pet related mailing list.
One Wednesday, The Onion published an article Mir Scientists Study Effects of Weightlessness on Mortal Terror. As this was the time when the Internet consisted of people banging rocks together, there was not an online link to Onion articles. But I was sure my friend at JPL, and all his friends, would appreciate the joke. That night I stayed late at the lab and typed the article into an email (with full credit to the Onion) and mailed it off to him.
As expected, the article garnered quite a few chuckles and was passed around to various folks inside JPL. What wasn’t expected was another friend, from totally different circles, sending me a copy of that same article 3 days later. Yes, in 1997 it took three days for information to be shared full circle on the Internet.
Information sharing is a whole lot quicker now, with things coming full circle in mere seconds. But that doesn’t make the information any more reliable and true. Take a recent article in ZDNet Research: Spammers actively harvesting emails from Twitter in real-time.
ZDNet links to a study published by Websense, claiming that email addresses on Twitter were available for harvesting.
That’s all well and good, but all ZDNet and Websense are saying is that email addresses are available for harvesting. I’ve not seen any evidence, yet, that spammers are harvesting and sending to them. This doesn’t, of course, mean they’re not, but it would be nice to see the spam email received at an address only shared on twitter.
Well, I have unique addresses and an un-spamfiltered domain. I went ahead and seeded a tagged address onto twitter. We’ll see if it gets harvested and spammers start sending to it. I’ll be sure to keep you updated.

Character encoding

This morning, someone asked an interesting question.

Last time I worked with the actual HTML design of emails (a long time ago), <head> was not really needed. Is this still true for the most part? Any reason why you still want to include <head> + meta, title tags in emails nowadays?
Read More