10/27/2010 06:49 am ET Updated May 25, 2011

Is An 'Attentiveness Gap' Blowing Up Pollster's Likely Voter Models?

WASHINGTON - On Monday the Gallup Organization released its latest results for what political junkies call the "generic" ballot -- the question that asks voters whether they will choose the Democratic or Republican candidate for Congress in their district. As has been their practice for the last four weeks, Gallup reported three sets of results. Among all registered voters Republicans led by 4 percentage points (48% to 44%). Among those likely to vote in a "high turnout scenario" the Republican lead by a 9-point margin (52% to 43%). Under a "low turnout scenario" it expands to 14 points (41% to 55%).

Four weeks ago, when Gallup first released results using its classic likely voter model, the numbers produced under both scenarios looked far more pessimistic for the Democrats than what other polls were showing. The change has been modest, but over the last four weeks, the Republican margins as measured by Gallup have been slowly moving closer to the other polls - from 18 to 14 points under a low turnout scenario, from 13 points to 9 points under the higher turnout scenario. The narrower margin now falls within a few percentage points of recent surveys by the Pew Research Center, Rasmussen Reports, AP-GfK, the GWU/Politco/Battleground poll, YouGov/Polimetrix and our overall trend estimate for the generic ballot.

The narrowing of the leader's margin in October is consistent with a previous history that Gallup acknowledges: In 2006, 2002 and 1994, Gallup reported leads for one party in early October that faded significantly by election day.

What is this late fade about? And can it tell us something about a similar pattern we are seeing at the statewide level, where Democratic candidates appear to be closing gaps in states like Colorado and Pennsylvania? I have a theory that this year's clear gap in enthusiasm, which no doubt tells us that Republicans will enjoy a turnout advantage next week, may have blown up some pollsters' likely voter models, especially in the late summer and fall, producing Republican margins that were far greater in some polls than other. The rest of this post explores data that supports the theory.

But let's be clear: This is not a theory that should bring much cheer to Democrats. Even if the true Republican lead on the generic ballot question is just 5 or 6 percentage points, Republicans should win a majority of the House and come very close to a Senate majority. This is also not a theory about a late Democratic "surge." If anything it suggests that some of the apparent "narrowing" is really an artifact of polling leads among likely voters that were artificially high a few weeks or months ago,

Questions Used to Model Turnout

A few weeks ago, I wrote an article that explained the many different ways that pollsters attempt to "model" or otherwise select likely voters. The underlying issue, as evident in the table below (based on data collected and tabulated by Michael McDonald) is that the turnout in mid-term elections is typically less than half of eligible adults:


The challenge for pollsters, of course, is to try to select or "model" this likely electorate from the much larger population of adults. While their approaches vary widely, most depend to some degree on questions about past voting, intent to vote this year and interest in the upcoming election to screen or model the electorate. I want to take a closer look at how some of these measures behave over time, using survey data collected by the Pew Research Center.

The Pew Research surveys are especially helpful for two reasons. First, they make their archive of past reports and, more importantly, of respondent-level data, available for download. Second, the national surveys they conduct in September and October of even numbered years include most of the same questions used by Gallup for its classic likely voter model.

Let's start with their question about past voting: "How often would you say you vote, always, nearly always, part of the time, seldom or never?" The chart below shows the percentage of registered voters that says they "always" vote in the surveys conducted in the fall of mid-term elections since 1994:


The "always" response typically varies between 50% and 60% of the electorate, with no discernible pattern over time.

Now consider the vote likelihood question:

I'd like you to rate your chance of voting in November on a scale of 10 to 1. If TEN
represents a person who definitely will vote and ONE represents a person who definitely will NOT vote, where on this scale of 10 to 1 would you place yourself?

The chart below shows the percentage that answers with a 10 on the 1-10 scale, indicating they will "definitely" vote:


This measure also shows very little trend, although if you look closely, you can see that the "definite" percentage typically increases very slightly between early September and late October. In the last four off-year elections, the "definite" percentage increased by a few percentages points over the fall campaign.

Finally, consider the pattern in the question about attention paid to the "How much thought have you given to the coming November election: Quite a lot or only a little?"


This picture looks much different than the others. Note two things: First, voters tend to report paying more attention to the campaign as it progresses. The "a lot" percentage typically grows by double-digit margins during the fall of a mid-term campaign. Second, the overall level of attention paid throughout the campaign has been significantly higher for the last two contests -- this year and 2006 -- than in the three previous mid-term elections (a topic discussed in the most recent Pew Center report).

Recall, however, that turnout in 2006 was only slightly higher than in 2002 and 1998, not quite the same order of magnitude of difference as the levels of self-reported attention paid to the campaigns.

Nevertheless, it's this last question that I believe explains some of the discrepancies we have seen among likely voter polls this fall. To understand why, we need to look at how these three questions interact with partisanship and voter preferences.


The table below illustrates how Democrats and Republicans differed in 2002 in the way they reported their past turnout, their intent to vote and how closely they were following the campaign. Each line plots the result of subtracting the Republican percentage from the Democratic percentage:


Two things are evident in the chart. First, the likely-to-vote and past voting questions show a modest "gap" favoring the Republicans. Most of the data points are below zero, as is the typical pattern. Mid-term voters tend to be older, better educated and white -- all constituencies that are typically better for Republicans -- but the differences are usually modest. Second, there is no clear trend up or down in these gaps over the course of the fall campaign.

Next consider how vote preference differed in 2002 among the two subgroups that are more likely to vote: Those who said they would definitely vote (10 on the 1-10 scale) to vote and those who said they were paying "a lot of attention" to the campaign.


In 2002, the differences between all registered voters, "definite" voters and attentive voters were relatively small. Voters who said they paid a lot of attention to the campaign were not much different than all registered voters. The "definites" were typically a few points more likely to vote Republican, as were the "likely voters" selected using the full Pew Research likely voter model (which is similar to Gallup's).

My assumption is that 2002 was a more typical election, one in which the likely voter questions helped "nudge" the sample in slightly more Republican direction.


Now consider the same data for 2006, a year in which pollsters found an "enthusiasm gap" favoring Democrats that persisted throughout the campaign:


The likely-to-vote and past voting questions show the same pattern as in 2002: little or no trend, with gaps a few points below zero favoring Republicans. The result for "paying a lot of attention" however, looks very different: It shows large gap favoring the Democrats that emerges in early October but then disappears on the final survey.

What was happening, as demonstrated by the table below, is that Democrats started paying attention the campaign sooner, but Republicans caught up in the end. It's not surprising that attention to the campaign increased as the campaign progressed -- that's the usual pattern. What was unusual was the way Democrats got interested sooner than Republicans.


Not surprisingly, voters who paid more attention to the campaign gave Democrats a much bigger lead on the generic ballot, especially in early and mid-October, than voters who said they would "definitely" vote:


Also as we would expect, the Democrats' lead among those paying a lot of attention faded as the gap between Democrats and Republicans on that question closed.

This Year

Now let's look at what the Pew Research surveys have been showing so far this year (with special thanks to the Pew Center's Scott Keeter for sharing the following data). First, let's consider the gaps between Democrats and Republicans on the three turnout measures:


All three lines look different this year, although the gap in attention paid to the campaign shows essentially the reverse of the pattern from 2006. A enormous attentiveness gap has emerged favoring the Republicans, far bigger than in 2006. Meanwhile, all three lines have diverged farther from zero -- indicating bigger gaps between Republicans and Democrats -- than in the past two mid-term elections. While the gaps favoring Republicans have closed on the most recent survey, the questions about past voting and intent to vote still show slightly bigger gaps favoring Republicans than they did in 2006. These bigger gaps are presumably an indicator of a coming turnout. They tell us that the 2010 electorate will be more Republican in composition than in 2006.

Not surprisingly, the big gap in attentiveness translates into similarly huge differences between voters who are paying "a lot of attention" to the campaign compared to the other low turnout subgroups. The gaps were bigger in August and September, but even on the most recent survey, Republicans lead by 15 points among those who are paying "a lot of attention" to the campaign (53% to 38%) but by 9 points (49% to 40%) among the definites.


I believe this last table is a road map to some of the discrepancies we have seen among some "likely voter" polls and turnout scenarios this year. If we could see everything about how pollsters model likely voters and all of their data, my educated guess is that the more these models rely on self-reported attention paid to the campaign, the more they tend to produce outsized Republican leads. My sense is that polls that have depended more on other means to model the likely electorate, including the internal campaign polls that also make use of previous vote history gleaned from voter files, have produced results that have been more consistent over time.

Now in fairness, this is just a theory, and attention paid to the campaign is just one of seven questions that Gallup uses to build its likely voter index. I should also stress that the attentiveness gaps in 2006 and 2010 have not "blown up" the version of the likely voter model applied by the Pew Research Center.

Still it's worth noting that in the 2006 Pew Research data, the fade in the attentiveness gap between Democrats and Republicans coincided with a narrowing of the Democratic lead among just the "definite" voters (from +15 in early October to +9 in late October to +4 on the final survey). So perhaps in years with large enthusiasm gaps, the likely voter measurements that seem to work well in selecting the likely electorate a few days before the election tend to overrepresent the most "fired up" voters a few weeks or months earlier.

Bob Erikson, Costas Panagopoulos and Christopher Wlezien made a similar case in the pages of Public Opinion Quarterly six years ago. They looked at the wide swings in support among likely voters during the 2000 presidential campaign and concluded that pollsters were mistaking "shifts in the excitement level of the two candidates' core supporters for real lasting changes in preferences."

We will know in a few days whether the narrowing that was evident in 2006 in the Gallup, Pew Research and other surveys repeats this year. It may not. And again, even if it does, Democrats are in for a drubbing that is likely to equal or exceed their loses in 1994. But at some point, pollsters need to address why their results sometimes diverge widely a few weeks or months before the election, only to converge at the end.

Note: The chart showing the gap in turnout measures from 2010 has been updated to correct an error in the original version o this article.

Follow Mark Blumenthal and HuffPost Pollster on Twitter

Subscribe to the Politics email.
How will Trump’s administration impact you?