THE BLOG

Crowley: Survey Says

10/20/2008 05:02 pm ET | Updated May 25, 2011

The New Republic's Michael Crowley has produced a nearly 4,000 word must-read on the new and changing world of polls, pollsters and polling websites in the 2008 campaign. For Pollster.com regulars, and anyone intrigued by the business of polling these days, it is truly worth reading in full but here are three paragraphs that capture some of the important conflict in the world of political survey research:

Many pollsters insist that the way to restore public confidence in their profession is to re-impose methodological standards, which treat polling as a social science rather than an amateur's pursuit. But the problem is that, despite the meticulous nature of Gallup and Pew--and the quicker, less orthodox approach of Leve and some other newcomers to the field-the establishment firms don't necessarily have all the answers. Jay Leve['s firm SurveyUSA], as it turns out, actually has a very strong track record. In March, Nate Silver ranked SurveyUSA as the most accurate of 18 major national firms, ahead of more venerable outfits like Gallup and CNN's pollster, Opinion Research.

Leve's winning track record may be the result of the technology he uses; automated polling allows him to tap into a larger pool of voters at a faster rate than live human polling--offering an instant snapshot rather than results blurred by time, as with a slow camera shutter. But Leve's technique is subject to some of the same criticisms as traditional polling. For example, some phone surveys are missing young voters with cell phones but not land lines. Overall, response rates have been declining for years. And, this season, race and gender have added tricky new variables. In short, Leve's success is hard to explain-- affirming a postmodern sense that methodology no longer ensures accuracy more than instinct and dumb luck.

The irony is that this perception, one that critics use to deride polling, is now widespread even among pollsters themselves. Last January, after all the big pollsters failed to predict Clinton's stunning New Hampshire primary victory, aapor asked them to hand over their raw data for evaluation. Leve complied immediately but says others dragged their feet. "That report has never come out," he says with a shake of his head. Mathiowetz, the former aapor president, says the report is coming soon. But, she says, a wider lack of transparency is a sign of changing times. It wasn't just the Chicago Tribune that blew the 1948 Truman-Dewey presidential election call--it was the entire polling establishment. Afterward, a panel of public opinion professionals studied what went wrong. Almost every major polling firm cooperated and submitted data about their research. The thought of this lost era clearly moves Mathiowetz: "What a lovely ..." she says, trailing off. "It just kind of brings tears to my eyes."

Crowley's piece is thorough, smart and covers the most important ground with respect to polls and how they are produced in 2008. I know it's a blogger cliche, but I mean it: go read it all.