It's hard to blog about polling and not comment on last night's surprise outcome in the Democratic primary for Governor in Alabama, although I followed the Alabama contests about as closely as this humorist/blogger watched Sex and the City 2 before reviewing it (which is to say, hardly at all). I won't follow his lead and declare myself "somewhat of an expert" on Alabama politics simply because "I know how to type into a blog," so I'll limit myself to the difficulties of polling in primary elections.
The gist of the Alabama surprise is that the one recent public poll on the race -- a survey of 400 likely Democratic primary voters conducted two weeks ago by Research 2000 for Daily Kos -- had Rep. Artur Davis leading primary opponent Ron Sparks by eight percentage points (41% to 33% with 8% for "other" and 18%11% undecided). Other polls conducted earlier, including an internal Davis campaign poll in early May and a PPP survey in March had Davis leading by somewhat larger margins.
But last night, Sparks crushed Davis by a 24-point margin (61% to 37%). Nate Silver quickly tweeted that the polling error on Sparks-Davis margin amounted to the "5th largest error in my database of 4500 polls."
So what might have gone wrong?
First, let's remember that pre-election polling is generally less accurate in primary elections. According to Silver, the top 10 errors in his database all involve primaries (which, he says, are about one fifth of his database). And, as he wrote last week, the average error in primaries (7 points on the margin) is roughly double that of general elections for president (3-4 points).
Much of that difference has to do with greater indecision among voters as compared to general elections, where a majority of voters are anchored to candidates by their party affiliation. Primary campaigns are more likely to communicate to a narrower electorate via channels -- direct mail, endorsements -- that reach voters very late in the campaign and thus add to the polling challenge.
So second, keep in mind that Research 2000 fielded their Alabama poll a full two weeks before election day, a very long time in a primary campaign. If the polling on Virginia's 2009 Democratic primary for governor had stopped two weeks out, the polling errors would have been as big or bigger than in Alabama. Subsequent polling in Virginia still understated Creigh Deeds' ultimate margin of victory, but did show him gaining rapidly, so few were surprised by the outcome. As such, PPP's Tom Jensen is right to argue that more polling in Alabama would likely have lessened last night's surprise.
Third, consider the challenge of polling and "modeling" a primary electorate like the Alabama Democrats. The 324,767 votes counted as of this writing represent slightly less than 10% of the state's eligible adults. Research 2000 used a random-digit-dial sample, which means they had to first obtain a representative sample of adults -- no easy feat when at least a quarter lack landline phones -- and then use screen questions and weighting to approximate the likely electorate.
How representative was their final sample of 400 likely Democratic primary voters? While DailyKos shares a lot of data, when it comes to this sample of likely primary voters, they tell us only that the Democrats were 54% female and 25% African American. They could also tell us about the regional composition of likely Democratic primary voters and the percentage of all adults that they deemed as "likely," but they don't.
Finally, consider Silver's finding that the highest errors in primary elections in his database have occurred in Democratic primaries in the South, particularly in 2008:
Much of this is driven by 2008, when the polls tended to lowball Barack Obama's performance in Southern states. In those primaries, the polls missed by an average of 9.4 points. Outside of those Clinton-Obama contests, the sample size is fairly small: about 35 polls covering roughly 10 Democratic contests. Still, the error on those non-2008 races is above-average: 7.6 points, higher than what its been for Democrats in other regions.
In South Carolina in 2008, we saw that one reason for the lowballing was that Obama's margin was especially understated among African-American voters. In Alabama this year, the error may have been the opposite, that polls either overstated the African-American turnout or Davis' share of the African-American vote. The Birmingham News reported last night:
While hard numbers were not yet available late Tuesday, long time observers in Davis' camp said Sparks' victory appears to have been achieved, at least in part, because of low voter turnout among blacks who, unlike two years ago when they showed up in big numbers to vote for Barack Obama, showed no such enthusiasm for Davis on Tuesday.
Either way, something seems to go wrong more often in Democratic primaries in the South with large African American electorates. It may be about the accuracy of the way polls measure vote preferences in these contests or among African American voters, or it may be about how well (or poorly) samples represent the true electorate, but either way, something is off more often.
The best advice I can offer to political junkies is that with primaries, especially when polling is sparse, expect the unexpected.
Update: First, as reader jmartin4s points out in a comment below, Artur Davis had a net negative favorable rating (28% favorable, 34% unfavorable) among likely Democratic primary voters in their survey in March. Davis' favorable rating among likely Democrats in the May Research2000 poll was very different -- 55% favorable, 22% unfavorable. It's very difficult to explain that discrepancy away on the basis of timing, a late trend or question wording; a difference in the representation of the likely electorate is a more likely explanation.
Second, both polls were more consistent in showing Davis with a lead, but only about half the vote, among African-American primary voters (48% to 23% on the PPP poll in March; 54% to 20% on the Research 2000 poll two weeks ago). There was no exit poll, but Tom Schaller has a post up estimating that "Sparks won something approaching half the African-American vote," based on looking at vote returns in counties with heavily African American populations. So for what it's worth, both polls had Davis' ultimate percentage of the black vote about right but vastly understated support for Sparks.