If you watched the CNN's Iowa caucus coverage Tuesday night (or caught our highlight reel of the same), you probably were treated to another extension of the network's long and strange fascination with Twitter. Way back in the day, the Most Trusted Name used to devote whole segments to reading Twitter to America. Tuesday night, it built an entire "social media wall" to demonstrate that Americans were tweeting stuff about politics, to the withering snark of Anderson Cooper.
People be tweetin', yo! And so, this now becomes a serious thing that political analysts have decided they need to grapple with as an essential part of explaining What It All Really Means. It's a great way for cable news types to "connect with ordinary people" without ever actually having to meet any. In the past year, we've seen Meet The Press project Tweetdeck on an entire wall of its studio, so David Gregory can gawk at it and exclaim, "Shiny!" and "Bouncy!" and "Wow!" and talk about what topics are "trending."
And more recently, the Washington Post unleashed its "@mentionmachine." Alex Pareene notes that "until it adjusts its algorithm, Ron Paul will 'win' every day, because he's got a psycho Internet cult." (The "algorithm," by the way, seems to be "adding up raw Twitter mentions." So, everyone with Twitter should open their accounts and send 100 tweets that read "Jon Huntsman," right now, because it will really confuse the Washington Post and lead to a few stories about how Huntsman is "surging in New Hampshire."
The open question that everyone seems to want to know is whether Twitter can help predict electoral outcomes. Why the media needs a new way of predicting electoral outcomes is anybody's guess. This is a pastime the media has happily pursued in a consequence-free environment since the time "political prognosticating" involved chucking turkey entrails around a room and reading the splatter patterns.
But the folks over at Mashable decided to take up the matter after the Iowa caucuses concluded to see if anything could be said about Twitter as a tool for forecasting:
In the closest Iowa caucus the nation has ever witnessed, former Massachusetts Gov. Mitt Romney edged out former Pennsylvania Sen. Rick Santorum by only 8 votes. That’s not 8 percent of votes, it’s just 8 actual votes. No pundit could have predicted such a neck-and-neck finish.
Actually, it seems to me that lots of pundits could have predicted this finish, and if you, say, took all of the panelists on the Sunday morning chat show panels as a barometer, you would have probably concluded that Romney and Santorum were neck and neck. (Matthew Dowd predicted a "three-way tie at the top" and that he "didn't know" who'd get the most votes, and it seems to me this is how the evening played out.)
Nevertheless, Mashable partnered with Globalpoint to track "positive sentiment" on Twitter and compare it to the results of the latest NBC News-Marist national poll. Well, their findings were as follows:
Information from Twitter matched up with pre-Iowa polling data from NBC/Marist, with one glaring difference: On Twitter, Rick Santorum was on fire.
No, Twitter could not have predicted that Iowa would been won by a minivan's worth of caucus-goers. But it did a much better job of anticipating Santorum’s excellent performance than the national polls accomplished.
Well, it's good that we all agree that Twitter "could not have predicted" anything. The only thing I'd point out is that you don't need to go to all this trouble, tracking Twitter mentions, to determine whether Santorum's performance in the Iowa caucuses was not going to be reflected with tremendous accuracy by a national poll. You could have just used "political science" or "history" or "common sense" to tell you this. Why the NBC-Marist poll was grabbed at random as the basis for comparison is beyond me -- if you want to predict the Iowa caucus, why use a national poll? We all understand that national polls don't have any bearing on the results of state contests, right?
When I showed the details of this Mashable study to our own polling expert, Mark Blumenthal, his response was -- and I'm paraphrasing here -- "LOL," basically.
(Also, you really can't impugn the accuracy of "the national polls" -- plural -- when you only actually used one national poll as your basis for comparison.)
[UPDATE: You can't really impugn the accuracy of "the national polls" when you don't use any as your basis for comparison either, and Mary Azzoli of The Marist Institute for Public Opinion assures me that the poll that was being used was conducted in Iowa, it was not a national poll, and it was conducted on December 27th and 28th. Knowing that, I've no idea why Mashable though any of this said something about Twitter's ability to predict outcomes better than "the national polls."]
But I hate having to throw cold water on Mashable's findings. So I'll let Larry Huynh, partner at Trilogy Interactive, do it for me. In a press release obtained by the Huffington Post, Huynh is pretty definitive about how Twitter fared as a prognostication tool in Iowa:
With all the excitement around the Iowa caucuses in New Media Land, you could be forgiven for thinking the biggest contest of the night was seeing who could most convincingly predict the results on Twitter and Facebook. As Mashable asked, "Did Twitter Predict the Iowa Caucus Better Than Pundits?"
After looking at several models, the answer is, unfortunately, no.
Trilogy examined the findings of four social media studies, from the aforementioned GlobalPoint, as well as surveys from Ensomo (which looked at "social media mentions, likes, and retweets of the GOP candidates, from Dec. 23-30"), SocialBakers (which examined "Facebook's 'people talking about' metric in the week leading up to Jan. 2, and the total number of Facebook fans"), and Sociagility (which used a proprietary technology to score social media mentions in mid-December).
[UPDATE, 9:19pm: Niall Cook, co-founder of Sociagility, writes in to point out that Trilogy erred by lumping them in here. Says Cook: "The purpose of our study was to see how the candidates were performing as at 21 December 2011 using our proprietary measure of social media performance normally reserved for brands. In doing so, we found a statistically significant positive correlation between performance and voting intention as measured by Public Policy Polling data around the same time. It would have been foolish to use this as a basis to predict the final result, and we didn't."]
The results, according to Huynh?
None of these metrics came even close to a significant correlation to the final caucus results, with one exception: Globalpoint, with a suspiciously strong correlation of 0.99 -- almost perfect, and well ahead of the traditional gold standard, the Des Moines Register's poll, which came in at a 0.86 correlation with the final results.
We contacted Huynh for details on GlobalPoint's "suspiciously strong correlation."
"GlobalPoint, by their own admission, they didn't have the full dataset," Huynh said. "And if you had the full dataset, it would have told a different story."
Missing from the story in this instance were two full days of Ron Paul tweets, withheld by Twitter in this case because of their substantially large volume.
"As great as Twitter is," said Huynh, "if you only look at the raw positive mentions included in their dataset, it just falls apart when you don't account for the Ron Paul data. If that data is included, then GlobalPoint does just as poor as any of the others."
Obviously, social media will have a big role in political campaigns and political analysis going forward. And Huynh says it's entirely appropriate to be enthusiastic about this, but it's still "important that you use these tools in the right way." In this case, the absent data, combined with the vagaries of the timing in which this particular snapshot of enthusiasm and the misguided approach to comparing a state caucus result to a national political poll, combined for a result steeped in fallacies.
Let me put it like this: at various points in the first quarter of last night's Orange Bowl, it's all but certain that positive mentions of Clemson University spiked on Twitter as the Tigers ended the fourth quarter with a 17-14 lead over West Virginia. Clemson went on to get outscored, 56 to 16, in the remaining three quarters.
But they were doing so well on Twitter there, for a while!
[Would you like to follow me on Twitter? Because why not?]