I was on vacation last week, but nearly interrupted it when I saw the press release from D.C. public relations firm Clarus, touting the results of its new survey. "PALIN SUPPORT FOR GOP NOMINATION SINKS," the headline blared, followed by this lead paragraph:
A new nationwide survey of Republican voters finds that support for former Alaska Gov. Sarah Palin to win the GOP's 2012 presidential nomination has fallen by one-third since March, sliding from 18 points to 12 points. Palin is now running in fourth place for the nomination behind former Massachusetts Gov. Mitt Romney, former Arkansas Gov. Mike Huckabee, and former House Speaker Newt Gingrich.
The release has several lessons to teach us about how to best interpret horse race polling. First, the headline struck me as overly dramatic, especially when I checked the methodology. The survey, conducted from July 26-27, interviewed just 374 "registered voters nationwide who self-identified as Republicans or as Independents who lean Republican," yielding a reported margin of sampling error of +/- 5%. The March survey interviewed 415 Republicans or Republican leaners, so the margin of error would have been roughly the same.
It's not hard to do the math on that. Eighteen percentage points minus five (or 13%) is less than 12 percentage points plus five (17%). So I assumed, at first glance, that the much heralded drop in Palin's support was not statistically significant.
Problem is, the margin of error is a little more complicated than my quick arithmetic. While the references at the bottom of news articles and press releases rarely explain it, the margin of error gets smaller as a given result gets closer to zero or one hundred percent (explained in more detail here). In this case the sampling error probably shrinks just enough to make 18% and 12% "significantly" different had the two questions asked in March and July been identical (and I say "probably" because without knowing how severely Clarus weighted their samples, I can't calculate the precise margins of error).
And that brings me to the second lesson: The margin of error tells us nothing about what happens when the pollster changes the question, which Clarus did here in two important ways. First, in March Clarus asked Republicans to choose among seven potential candidates: Sarah Palin, Mitt Romney, Mike Huckabee, Newt Gingrich, Jeb Bush, John Thune, and Mitch Daniels. Last month, they presented nine choices. They dropped Bush (who received 8%) and added Lamar Alexander, Haley Barbour and Tim Pawlenty (who received a combined 8%). So was the apparent change in Palin's support about a decline in her support, or were Barbour, Pawlenty and Alexander collectively more attractive to some potential Palin supporters than Jeb Bush?
Equally important, Clarus changed the root question. In March, they asked Republicans which candidate they "would now most likely favor." On the more recent survey, they asked which they would favor "if you had to vote today." Is it a coincidence that the undecided percentage grew by five points (from 10% to 15%) when respondents were pressed how they would have to vote "today?" I think not.
We might also consider the results of other polls. CNN, which has asked an identical Republican preference question three times this year, shows Palin with exactly the same support a week ago (18%) as in March (18%). A new PPP survey released just this afternoon finds essentially the same result.
Also, a dozen or so national pollsters have been asking national samples of adults or registered voters to rate Palin (favorably or unfavorably), and our chart shows no consistent pattern in their measurements over the course of 2010 (click on the individual black and red dots on the interactive chart below to see the trends of individual pollsters).
And finally, there is a lesson about the value of this sort of horse race question, especially when asked at this stage of the contest. What they tell us about the Republican nomination race shaping up for 2012 is that none of the potential candidates -- not Palin, Romney, Gingrich nor Huckabee -- are the sort dominant front runner likely to begin with a huge advantage based on early name recognition or support. The same was true at this point four years ago when polls showed Rudy Guiliani as the "front runner" in early trial heat questions. Those early "leads" turned out to be meaningless as the real races in the early primary states got underway.
Over the weekend, Kevin Madden, the former press secretary to Mitt Romney during the 2008 campaign, tweeted that "2012 horserace polls are like pre-season football: Fun to watch for a few minutes until you realize they don't matter." That's about right.