THE BLOG

A Question I Won't Answer

11/04/2009 11:35 am ET | Updated May 25, 2011

"So who was the most accurate pollster yesterday?"

If I had $100 for every time I've been asked that question by a reporter on the Wednesday morning after an election, I could retire early. And after five years of blogging on this beat, it's a question I'm determined to refuse to answer today.

Why?

First, all the votes are not yet counted (including 7% of the precincts in NY-23), and the counts that are available do not yet include the absentee and provisional ballots that will be added later and are not reflected in those percentage-of-precincts-reporting statistics you see on all the media vote counts morning. Take a look at this snap judgement from November 5, 2008. It declared a "big winner" among prognosticators on the assumption that Barack Obama won by 6.1 percentage points (52.4 to 46.3), but when all the ballots were counted the margin was 7.2 (52.9% to 45.7%). So that particular snap judgement picked the wrong "big winner."

Second, the whole notion of crowing a "big winner" based on a handful of polls in a handful of states is foolish. The final polls yesterday had random sampling error of at least +/- 3 percentage points. If a poll produces a forecast outside its margin of error, that's important. But if several polls capture the actual result within their standard error, chance alone is as likely as anything else to determine which one "nails it" and which miss by a point or two.

Third, there are sometimes other problems with making too much of "hitting a bullet with a bullet" on the final poll, when the polls leading up to it provide different results.

Yes, there are several good stories about what went right and what went wrong with yesterday's polling, including some important lessons about the value of automated polling. Some pollsters certainly did better yesterday than others. And I'm hoping to have something written and posted on that subject later today, provided that I don't get bogged down by the calls and emails from reporters wanting me to tell them, "who was the most accurate pollster yesterday?

More:

Pollster