“Overall, a 30% increase in positive Tweets is four times more effective in driving sales than a 30% increase in existing above-the-line advertising.”
That is an impressive statistic, quoted directly from a Twitter infographic and attributed to a study by Deloitte. With the inclusion of a big name audit and consulting firm doing a study of 100 different game releases, it has all of the marks of credibility.
Unfortunately, it isn’t true.
“Question everything you hear, see or read.
Everyone has a bias.
Everyone applies their own spin.
Everyone has something to sell.” (source)
Twitter’s claim highlights why questioning is so important. Twitter, of course, does have a bias here. If marketers believe Twitter is key to their success, marketers will focus more of their attention on Twitter. As a marketer, you can’t afford to take Twitter’s word for it.
Here are some of the issues I see with the study from Twitter and Deloitte and its conclusions:
1. Confusing Cause and Correlation
Like so much of the research in social media today, Twitter and Deloitte are jumping to the conclusion that correlation means causation. It doesn’t. The kicker here is Twitter didn’t even ask Deloitte to study if Tweets impacted sales, they only asked if tweets were statistically predictive of sales.
This is the equivalent of a study to determine the relationship between home value and the presence of high speed internet connections. If they are positively correlated, it doesn’t mean a faster internet connection is the reason a home is more valuable!
2. Volume Barely Matters
Deloitte used a traditional advertising stock model for the value of Tweets. While it seems like a reasonable approach (and the model approach would actually have indicated if it was egregiously wrong), the result is a major red flag:
Tweet volume almost didn’t matter.
In other words, actually seeing positive or negative tweets isn’t nearly as important as their existence. How can tweets you don’t see impact what you buy? They can’t. (They can predict it though, we will get to that in a moment).
The minimal impact of volume on sales flies in the face of traditional theory on the impact of communications. That doesn’t mean its incorrect per se, but it should have been a trigger for Twitter and Deloitte to step back and confirm there are no other issues in the structure of the study.
3. Ignoring Key Variables
What would happen if you studied the factors that predict household ownership of luxury cars but you excluded household income, net worth and all other financial metrics from the study? The result would point to things like location, profession and age as indicators because they are correlated with those key financial metrics.
This study did something similar. We know that word of mouth has a powerful influence on purchase behavior. We also know that positive media coverage and reviews have a positive impact on purchase behavior. However, the study only considered sales data (sales volume, pricing, etc), advertising spend and activity on Twitter.
Where is the impact of a great versus average versus disappointing game in the study? Where is the impact of media coverage and editorial reviews? In short, they are excluded.
Connecting the Dots
What does the research actually find? Here are the key points to connect:
- Tweet sentiment is far more important than the actual volume of that sentiment on Twitter.
- The experience playing a game, or our expectations for a game, are reflected in Tweet sentiment.
- Game quality and media coverage are not directly included in the research.
What impact does a high quality game and positive media coverage have? For one, it increases the number of positive Tweets about a game, from people sharing positive publication reviews, sharing their own experience or sharing their expectation of the game.
Why is the impact of volume so small? Because the balance of positive versus negative Twitter sentiment, not reach, is acting as a proxy in the research for positive media coverage, game quality and positive word of mouth beyond Twitter. Positive Tweets are not actually the cause of the increase, they are simply correlated.
Let Me Fix That For You
Twitter: “A 30% increase in positive Tweets is four times more effective in driving sales than a 30% increase in existing above-the-line advertising.”
Fixed: “A significant improvement in the quality of your product improves word of mouth and increases sales.”
Unfortunately for Twitter, the fixed version does not conclude that Tweets are the reason for the sales increase. It also doesn’t say Tweets do not drive sales. Based on my review of the full research report, I simply do not believe it supports any conclusion about the direct impact of Tweets on sales.
Credit to Tom Webster, I’ve enjoyed his let me fix that for you posts in his reviews of research and statistics recently.
At what point is “spin” of research results move from acceptable to misleading to outright fraudulent? I’d love to hear your thoughts in the comments below or on Twitter (@wittlake).
Photo Credit: Lie to Me by Rosy on Flickr