Last week, media outlets were abuzz with news that Amazon was claiming to have passed Apple as the third largest video service site, coming in behind Netflix and YouTube.
Here’s The Verge, reporting on the news:
Amazon’s been trying to turn Instant Video into a major player in the streaming space, and it looks like its dedication is starting to pay off: Amazon says that Prime video streams have nearly tripled year over year, and it cites video-delivery firm Qwilt to say that Instant Video is now the third largest video site overall, behind Netflix and YouTube.
That’s a fairly representative blurb about Qwilt’s numbers and Amazon’s willingness to cite them in a press release.
A quick look at Qwilt’s announcement raised some red flags:
- There was no source data listed.
- The accompanying blog post included pro-Amazon hyperbole.
- The included infographic provided no useful comparative data but does include a picture of an Amazon-branded rocket ship next to a skyrocketing arrow.
Here’s some sample language (emphasis mine):
If Amazon says they will boil the ocean, better run to the beach and hop in fast before the water is scalding…
The growth of Amazon continues to amaze and confound Wall Street. CEOs across the globe marvel at Amazon”s insatiable appetite for new markets, new products and new revenue…
Amazon”s traffic volumes, as measured by Qwilt in March of 2014, increased by 94% over the previous 12 months. In some US operator networks, between March 2013 and March 2014, Amazon”s streaming video traffic increase was nearly 300%…
Of course, on seeing these developments, we smile knowingly and approvingly…
Despite the red flags and obvious questions regarding bias, Qwilt’s claims were widely reported as a major win for Amazon in the streaming space.
I immediately questioned the findings (and the subsequent rush to report) on Twitter:
I’m fucking shocked that people (ahem, @verge) are reporting on this @qwilt claim without questioning the data 04/08/14
The fact that @amazon cited an @qwilt report with unofficial numbers rather than providing official numbers is a HUGE red flag. 04/08/14
@Mark1Fisher Where is the underlying data on this claim? 04/08/14
Qwilt’s Mark Fisher eventually responded (both on Twitter and in the comments on his blog post) with a promise to reveal the underlying data, which was eventually tacked on as an update to the original post.
As I expected, the story gets less interesting for Amazon as soon as you see the data, which I’ve summarized in a chart:
Yes — based on Qwilt’s data — Amazon did in fact pass Apple to take the third spot behind Netflix and YouTube but the more reasonable takeaway to report is that there’s Netflix and YouTube — and then there’s everyone else. (Granted, it was impossible to know that, let alone report that, without demanding the underlying data, which no one bothered to do.)
Indeed, Qwilt’s data reveals that Amazon’s movement isn’t the most interesting shift from 2013 to 2014, which just makes Fisher’s hyperbole all the more obnoxious:
- Netflix increased its percentage from 52.5% to 57.5% in one year. (This means there’s a 54.5% gap in 2014 between #1 Netflix and #3 Amazon that isn’t discoverable in Qwilt’s initial post.)
- YouTube dropped from 28.2% to 16.9% in one year. (This 11.3% drop is impossible to glean by reading Qwilt’s initial post.)
- Even though Amazon passed Apple to reach #3, Twitch (what the hell is Twitch?) also passed Apple to climb to #5, pushing Apple back to #6. (Twitch wasn’t even ranked in Qwilt’s 2013 top 10 and isn’t mentioned at all in the original blog post. Indeed, Qwilt included a top 5 graphic — sans underlying data — that includes Apple while omitting Twitch.)
- HBO Go did not rank in Qwilt’s 2013 top 10, but has moved to 0.5% in 2014. (This increase is not evident in Qwilt’s initial post.)
- Apple actually gained 0.2%, even as it lost two spots on Qwilt’s list.
Keep in mind that Apple doesn’t actually provide a one-to-one competitor to Netflix or Amazon Instant Video, both of which are subscription models that provide all-you-can watch access to a curated library of video content. Apple (currently) only provides rental and purchase options for individual titles from a curated library.
It’s also worth noting that Qwilt offers absolutely no data about how people are watching any of this content: Netflix streams are attributed to Netflix but Qwilt does not break down those streams by device.
This means that it’s entirely possible that more Amazon Instant Video content is streamed from Apple’s iDevices than is streamed from Amazon’s Kindle devices.
It’s also not clear how any of this data relates to revenue or profit: Does Netflix make more than Amazon who makes more than Apple, or does Apple (due to a rental/purchase model) bring in more money than Amazon despite a drop to #6 in 2014?
My guess is that Qwilt is another company in a long line of companies that knows that headlines that lead with Apple draw more attention than headlines that lead with virtually any other company. If it takes a little data manipulation (a snip here and the failure to mention Twitch there) in order to get the emphasis tuned juuuuuuust right for maximum page views, well, why not?
Especially if no media outlet is going to question the results?