On Polls and Astrophysics

06/09/2008 11:37 am ET | Updated May 25, 2011
  • Mark Blumenthal Mark Blumenthal is the Head of Election Polling at SurveyMonkey.

Last Friday's New York Times included an op-ed column by Neil deGrasse Tyson, an author and astrophysicist who used current polling state-by-state polls to conclude that "if the general election were held today, Barack Obama would lose to John McCain, while Mr. McCain would lose to Mrs. Clinton."

His assertion depended on "a new method of analysis on the statistics of polls" developed by two astrophysicists, J. Richard Gott III and Wes Colley, "that has been accepted for publication in the journal Mathematical and Computer Modeling."

Here is the gist of their method:

[I]n swing states, the median result of all the polls conducted in the weeks prior to an election is an especially effective predictor of which candidate will win that election — even in states where the polls consistently fall within the margin of error.

So the idea is this: Ignore sampling error and award states to the candidates if they are "ahead" (even by a single percentage point) in a greater number of polls. That method may work out roughly the same as an average in a state with a large number of polls. Tyson does not say how many polls the Gott-Colley method used in each state in 2004, but in that year the Kerry-Bush results remained reasonably stable over time (so counting up all polls over a 2-3 month period may have worked as well as watching the final week). Moreover, you could have skipped the "rocket science" in 2004 and just examined the final RealClearPolitics averages. The candidate leading on their final averages won every state except Wisconsin.

However, apply that counting method to the "past six weeks of polls," as Tyson did, and you run into a problem: Very few polls, so the counting method starts to break down. Thus, for the crucial 20 electoral votes available in the state of Ohio, Tyson tells us:

In Ohio, for example, Mr. McCain beats Mr. Obama two polls to one. But Mrs. Clinton beats Mr. McCain two polls to nothing. So Ohio, which Mr. Kerry did not win in 2004, would go into Mrs. Clinton’s column, giving her an additional 20 electoral votes.

Shazam! That's not a "two to one" margin, that's three polls, total: One (from SurveyUSA) showing Obama leading by 9, one from Quinnipiac University showing McCain leading by 4, and the tie-breaker from Rasmussen Reports showing McCain with a single percentage point advantage. Never mind that if you average the three (as RealClearPolitics does), they give Obama with a one-point advantage. Or if you draw a regression line through all the available polls -- as we do -- you get a the same one-point Obama edge. Altogether, these data suggest a very close race in Ohio, as of mid-May, not a clear leader. But never mind all that. Tyson applies the Gott-Colley procedure and gives the state to McCain.

The bigger problem with the Tyson column is is the phrase, "if the general election were held today." It isn't. The notion that polls are just a "snapshot" of opinion may be a hackneyed cliche, but it is nonetheless true. The current batch of horse-race polls tell us about voter preferences over the last six weeks. The next six weeks may be different.

Andrew Gelman reproduced a chart from a paper he co-authored 15 years ago to remind us all that polls conducted five or six months before an presidential election are often poor predictors of the final outcome (h/t Monkey Cage):

06-09_Gellman chart.jpg