THE BLOG

Research 2000, DailyKos and Transparency

06/29/2010 04:17 pm ET | Updated May 25, 2011

Today the polling world was rocked by claims that polls by Research2000 for DailyKos are substantially flawed:

We do not know exactly how the weekly R2K results were created, but we are confident they could not accurately describe random polls

.

Coming on the heels of recent arguments by Nate Silver at FiveThirtyEight.com that Strategic Vision faked their polling over several years, this is a new blow to the credibility of public polling.

Mark Blumenthal here at Pollster.com has lead a concerted effort over the last two years to increase the degree of disclosure expected from polling firms, an effort that paid off in new disclosure requirements from the American Association for Public Opinion Research (AAPOR) this spring. Some three dozen firms immediately signed on to the new disclosure requirements, but there are many firms that produce widely cited polls that have not yet agreed to disclose as much as required.

I've only had time for a single quick read of the Research2000 analysis by Mark Grebner, Michael Weissman, and Jonathan Weissman. It seems to be done seriously and it raises important doubts about Research2000's practices. But with my academic's hat on, I'd like to see it receive serious review by professional statisticians with polling experience. Academic journals typically reject 80-90% of articles submitted to them because on close inspection by experts, flaws are found in the theory or the analysis. These are serious charges, and they deserve to be vetted by professionals qualified to do such an evaluation. If flaws in the analysis are discovered, they can be fixed and the conclusions corrected. If the analysis is found to be sound, then the evidence is even more compelling and worrisome for its implications for the polling industry.

There is one element of disclosure that has not been pushed, but which could significantly and easily reduce the chance of "pollsters" making up their data. Every media firm, including DailyKos, should write into their contracts the requirement that the raw data and complete questionnaire be deposited within two months with the Roper Center Polling Archive at the University of Connecticut. Two months is long enough that there is little remaining news value, but rapid enough that meaningful vetting and analysis is possible. By forcing this disclosure, by contract, the sponsors of polling would gain credibility for their polls while insisting that their pollsters live up to the standards of disclosure by AAPOR as well as making the raw data available for subsequent scrutiny.

Most major media polls already deposit their raw data with the Roper Center (including, Gallup, ABC/Post, CBS/NYT, NBC/WSJ, Pew, Time, Newsweek), though not necessarily as quickly as two months. Their example should encourage others to also deposit their data.

But most importantly, it is in the interest of the sponsors of polling to protect their reputation by requiring full disclosure and deposit of the data. Such practice would enhance the value of their polls, not diminish it.

More:

Pollster