There has been a considerable buzz over the last two days about the surveys released yesterday by SurveyUSA that test both McCain-Obama and McCain-Clinton trial-heat questions in all 50-states. Putting aside the concerns some have about SurveyUSA's automated methodology and the other usual caveats about horse race polling at this stage in the campaign, I tend to agree with the critique from Matt Yglesias (via Sullivan):
Each of these polls has a sample size of 600, so the margin of error will come into play. What's more, there are 100 separate polls being aggregate here, so the odds are that several of these are just bad samples.
True on both counts. SurveyUSA colors in states on their maps even if a candidate leads by a point or two, margins that are not close to achieving statistical significance. However, since SurveyUSA says they did 600 interviews in each state, we can take their analysis a step further, applying statistical sampling error to the candidates' margins in each state.
Professor Franklin and I have done just that, classifying each state based on the statistical significance of the candidate's lead. We call a state "strong" for the candidate if they lead by a margin that is statistically significant at a 95% level of confidence, the level typically used to calculate the "margin of error" attached to most surveys. We label as "lean" any state where a candidate leads by more than one standard deviation, which amounts to a 68% confidence level. We label all other states as toss-ups.
Note also that these significance tests assume "simple random sampling," which produces a smaller error margin than we would get if we could take into account that SurveyUSA, like virtually all pollsters, weights its data. We would need access to the raw data and weights in order to do truly correct significance testing.
The tables and maps appear below, followed by some discussion. First, here are the results and a map showing an Obama vs. McCain match-up (you can click on any of the images for a larger size version):
And here are the results and a map showing an Clinton vs. McCain match-up:
If you would prefer, you can also download the spreadsheet that we used to create the tables.
Now that you have all of the data before you, let's consider the merits of the project and a few caveats about the data. First, this sort of project -- which involved 30,000 interviews completed in 50 states over a three-day period (February 26-28) -- would not have been feasible with live interviewers.
On the other hand, the automated methodology is controversial with traditional survey researchers. I wrote about the arguments for and against IVR (interactive voice response) surveys Public Opinion Quarterly, and I have blogged often on the subject often, both here at Pollster and on its forerunner MysteryPollster. Readers are obviously welcome to share their opinions about the IVR methodology in the comments.
The other caveats noted by SurveyUSA are worth repeating: They surveyed all self-reported registered voters, and did not attempt to screen for "likely voters" (although many national pollsters do the same at this stage, feeling that we are too early in the process to attempt to predict what voters will actually cast ballots). McCain would likely do slightly better in both match-ups under a "likely voter" screen. Also, we are obviously still eight months from the election. Much can and will change in terms of voter perceptions and preferences.
Let us also keep in mind the limitations of random sampling error. It tells us only about the variability that comes from calling a sample of households rather than dialing every working phone number in every state. As with any survey, it tells us nothing about the potential for error based on the wording of the questions, the selection of respondents within the household and the voters missed because they lack land-line phones or do not participate in the survey. Be careful about using the misnomer "statistical tie" to describe states in the toss-up category. One candidate would likely show a "significant" lead if we could increase the sample size -- we just lack the statistical power to know which candidate that would be.
Finally, keep in mind that since we are looking at 100 tests (2 each in 50 states), these results probably misclassify five states by chance alone (as opposed to the way we would classify them if SurveyUSA had called every working telephone in the 50 states).
With all the caveats out of the way, what does all this data tell us? Consider this summary of the electoral vote totals**:
These data are less useful in forecasting the ultimate result than they are in gauging the relative strength of both Clinton and Obama as of last week (February 26 to February 28). Those dates are important, since both the Gallup Daily and Rasmussen Reports automated tracking have shown Clinton gaining ground on Obama nationally over the last week.
Nonetheless, as of last week, Hillary Clinton led in states that add up to a slightly greater electoral vote total counting the leaners (250 for Clinton vs. 244 for Obama. Still Obama appeared to put more states into play (138 pure toss-up for an Obama-McCain race vs. a Clinton-McCain race). So Obama's initial electoral vote advantage is greater.
The most interesting aspect of these surveys is the states that explain those differences. Let's consider first the states where Obama does better than Clinton:
- Obama moves three states from lean McCain to strong Obama: Colorado, Iowa and Oregon
On the other hand, Clinton does better than Obama in a smaller number of states:
- Clinton moves one state from strong McCain to strong Clinton: Arkansas
Here is another table that makes it easier to see these comparisons (again, click on the image to see a full size version):
So, Pollster readers, what do you think?
**And yes, after putting these tables together I see that SurveyUSA split the Nebraska electoral votes based on on the vote totals, something I did not do.
Update: Nick Beaudrot (via Yglesias) creates thematic maps based on the same data keyed to the size of the candidate margin.