We interrupt the previous post still in progress to bring you a feature Pollster readers will definitely want to read in full. Carl Bialik, the "Numbers Guy" from Wall Street Journal Interactive did some comparisons of the performance of five pollsters that were particularly active in statewide elections: Rasmussen Reports, SurveyUSA, Mason Dixon and Zogby International (twice, once for its telephone surveys and once for Internet panel surveys).
The most important lesson in Bialik's piece is his appropriate reluctance to "crown a winner." As he puts it, "the science of evaluating polls remains very much a work in progress." That's one reason why we have not rushed to do our own evaluation of how the polls did in 2006. Bialik provides a concise but remarkably accessible review of the history of efforts to measure polling error (including a quote from Professor Franklin) and a clear explanation of his own calculations.
Again, the column -- which is free to all -- is worth reading in full, but I have to share what is for us, the "money graph:"
There were some interesting trends: Phone polls tended to be better than online surveys, and companies that used recorded voices rather than live humans in their surveys were standouts. Nearly everyone had some big misses, though, such as predicting that races would be too close to call when in fact they were won by healthy margins. Also, I found that being loyal to a particular polling outfit may not be wise. Taking an average of the five most recent polls for a given state, regardless of the author -- a measure compiled by Pollster.com -- yielded a higher accuracy rate than most individual pollsters.
Thanks Carl. We needed that today. Now do keep in mind the one obvious limitation of Bialik's approach. He only looked at polls by four organizations, including just one online pollster (Zogby) and just two that used live interviewers (Mason Dixon and Zogby). There were obviously many more "conventional pollsters," although few conducted anywhere near as many surveys as the four he looked at.
Another worthy excerpt involves Bialik's conclusions about the Zogby Interactive online surveys, especially since nearly all of those surveys were conducted by Zogby on behalf of the Wall Street Journal Interactive -- Bialik's employer.
But the performance of Zogby Interactive, the unit that conducts surveys online, demonstrates the dubious value of judging polls only by whether they pick winners correctly. As Zogby noted in a press release, its online polls identified 18 of 19 Senate winners correctly. But its predictions missed by an average of 8.6 percentage points in those polls -- at least twice the average miss of four other polling operations I examined. Zogby predicted a nine-point win for Democrat Herb Kohl in Wisconsin; he won by 37 points. Democrat Maria Cantwell was expected to win by four points in Washington; she won by 17.
Again...go read it all.