Daily Kos founder Markos Moulitsas today rocked the polling world by posting an analysis that he says shows "quite convincingly" that the national surveys conducted for his website by Research 2000 since early 2009 were "largely bunk." Just three weeks ago, Moulitsas fired Research 2000 on the basis of low accuracy scores tabulated by Nate Silver. Today, on the basis of the work of "statistics wizards" Mark Grebner, Michael Weissman, and Jonathan Weissman, Moulitsas announced that Daily Kos "will be filing suit" against its former pollster "within the next day or two."
The core of his extraordinary explanation is worth reading in full:
We contracted with Research 2000 to conduct polling and to provide us with the results of their surveys. Based on the report of the statisticians, it's clear that we did not get what we paid for. We were defrauded by Research 2000, and while we don't know if some or all of the data was fabricated or manipulated beyond recognition, we know we can't trust it. Meanwhile, Research 2000 has refused to offer any explanation. Early in this process, I asked for and they offered to provide us with their raw data for independent analysis -- which could potentially exculpate them. That was two weeks ago, and despite repeated promises to provide us that data, Research 2000 ultimately refused to do so. At one point, they claimed they couldn't deliver them because their computers were down and they had to work out of a Kinkos office. Research 2000 was delivered a copy of the report early Monday morning, and though they quickly responded and promised a full response, once again the authors of the report heard nothing more.
While the investigation didn't look at all of Research 2000 polling conducted for us, fact is I no longer have any confidence in any of it, and neither should anyone else. I ask that all poll tracking sites remove any Research 2000 polls commissioned by us from their databases. I hereby renounce any post we've written based exclusively on Research 2000 polling.
Separately, Charles Franklin will soon post some thoughts on the evidence presented by Grebner, Weissman and Weissman, but for the moment let's consider the issues of disclosure raised.
First, there is some further history. Two years ago, when the American Association for Public Opinion Research (AAPOR) launched an investigation into the polling miscues during the 2008 presidential primary elections, it asked 21 firms to provide response rate information. Research 2000 could not calculate specific response rates for the calls they made. The estimates they provided were "about 1 complete of every eight attempts" in New Hampshire and "about" 1 in 9 in Wisconsin. They were also unable to provide "a full set of dispositions" (a tally of how many calls resulted in completed interviews, no answers, refusals to be interviewed, and so on).
Their incomplete answer did not result in a formal censure, since Research 2000 claimed they were sharing whatever information they had. Another half dozen or so pollsters also claimed they failed to keep an accounting necessary to enable such calculations. But whether a failure of disclosure or quality control, the inability to provide such a basic metric speaks to the importance of demanding greater disclosure as a matter of routine. That's why AAPOR's Disclosure Initiative is so important. If you have not yet read my column from last month on this subject yet, I hope you will. It is highly relevant to this story (h/t to Nirml for making the same point on Twitter).
Second, the most damning information for the layperson about Research 2000 provided in today's announcement -- and is certainly most troubling to me -- is their apparent reluctance to share raw data with their own client. AAPOR's Disclosure Initiative will not mandate the release of raw, respondent level data, but it is worth considering that some of the most respected media pollsters already make their raw data available to scholars by routinely depositing it the Roper Center archives. The Pew Research Center makes their raw data available to the general public through its own web site.
As this story broke, NBC's Chuck Todd noted the "stain" that irresponsible pollsters are leaving on the community of credible pollsters. Many are asking what the polling profession can do. Whatever one might conclude about Research 2000, there are two clear answers: Full support for and participation in AAPOR's Transparency Initiative and a greater willingness to deposit raw data to the Roper Archives.