The Wall Street Journal's Carl Bialik weighed in today on the Strategic Vision, LLC controversy in both his print column and a separate blog item. Collectively, like Shaila Dewan's New York Times story on Sunday, they provide a decent, concise overview of the story for those that have not been following it. Unfortunately, there is little news for those of us that have been following the story closely. According to Bialik, Strategic Vision CEO David Johnson "didn't respond to Wall Street Journal requests for comment."
Bialik did seek comment from various mathematicians regarding the claims of statistical irregularities in Strategic Vision's results made by Nate Silver (here, here and here) and others at FiveThirtyEight.com:
This week, Mr. Silver brought in a physicist and commenter on his blog to calculate the probability, which shrank to 5,000 to 1 against, when removing what he said was an unproven assumption that each digit should appear equally often. Several mathematicians said the shift in odds doesn't diminish Mr. Silver's finding that the Strategic Vision numbers were unlikely to arise by a quirk of fate.
He added additional details in the blog post:
Mathematicians said the Silver analysis -- finding that certain digits showed up far more often than others in Strategic Vision polls -- was troubling but want to see more evidence. Jordan Ellenberg, a University of Wisconsin, Madison, mathematician, blogged that the case isn't as persuasive as investigations into possible fraud in the Iranian election. "It's not so substantial that I would have gone public with it, if it were me," Ellenberg said, but he does think it merits further investigation.
"To strengthen the argument that Strategic Vision's (or any other polling group's) numbers seem unusual, the next step would be to assess the observed variation across a number of similar polling organizations and see where various groups fall," said Lance Waller, a biostatistician at Emory University.
One bit of news that Bialik passes along is that some of Strategic Vision's clients are now pressing the company for more verification of their data:
Two think tanks that are clients of Strategic Vision also are seeking more details on the firm's methods in light of Silver's analysis. The Goldwater Institute, which calls itself a free-market think tank, and the Oklahoma Council of Public Affairs hired Strategic Vision to test high-school students' civic knowledge in Arizona and Oklahoma, respectively.
After Silver questioned the Oklahoma results as being too bleak, both think tanks sought verification from Strategic Vision. "Although I find it very unlikely that Strategic Vision manufactured this data, I have asked for receipts from the marketing firm from which they purchased the contact data just to make certain," Matthew Ladner, vice president of research for Goldwater Institute, said.
Brandon Dutcher, vice president for policy for the Oklahoma group, isn't making up his mind just yet. "I have requested voluminous survey data from them, as well as answers to some methodological questions -- all of which I expect they can and will provide so that they can go about defending their firm and I can go about defending this survey," Dutcher said. "If not, however, then of course I would want my money back and wouldn't hire them again.
See both the article and blog post for all the details.
Bialik observes that while the controversy has gotten "bogged down in threats of litigation and arcane calculations," the controversy "has shed light on an inconvenient truth about widely reported political polls: Verifying their numbers is nearly impossible." That conclusion is hard to argue with and a big reason why better disclosure -- the issue behind the AAPOR reprimand that helped bring these issues to a head -- is so important. Only greater transparency will prevent these sorts of controversies in the future.