Our update to the Slate Election Scorecard yesterday tries to put the results for the generic congressional vote from the USA Today/Gallup survey into some perspective. It also reintroduces the controversy over likely voter models in general with specfic attention to the Gallup likely voter model. More on that below. Given the obviously high interest in this particular survey, as reflected in the sometimes heated debate in the comments section yesterday, I want to first share some of my own reactions.
First, remember it's just one poll. One of the inherent weaknesses in political polls is that they come with a lot of built in variation. Some comes from interviewing a sample rather than the whole population. Some comes from other methodological differences across surveys. As such, it is always better to look at more surveys than few. We like to average results across polls, despite some theoretical shortcomings, for just that reason.
In hindsight, the single discordant poll my be just a random statistical outlier, but not always. Sometimes it can be the proverbial "canary in the coal mine" that warns us of some new and emerging trend. So we pay attention to polls like yesterday's Gallup Poll, even if we typically recommend caution in interpreting them.
Second, let's put aside the likely voter conundrum and focus on the larger sample of adults and compare the Bush job approval rating among all adults to trends on other surveys. I have updated the table from Monday's post below, and I averaged the two Gallup polls conducted in September to try to make the data as comparable across pollsters as possible.
The pattern is now strong and obvious: While the precise level of approval shows the usual variation across pollsters, eight of the nine pollsters show some small increase in the Bush job rating between August and September. That is a highly improbable result by chance alone, analogous to flipping a coin and having it come up heads eight of nine times (roughly 2% according to my favorite binomial calculator).
Third, consider one issue that everyone overlooked except one very alert MP reader: On previous Gallup polls, the Bush job rating came first on the questionnaire, or at least before questions about congressional vote preference. This is the first pre-election poll in which Gallup switched the order, asking the congressional ballot question first and then the Bush job rating.
This practice is not unusual. Media pollsters frequently juggle the order of questions with events, especially those that conduct surveys on a wide variety of topics year-round. They will generally try to position the most important (or newsworthy) questions first to reduce the chance of bias. The problem is that in moving questions around, they sometimes introduce some unforeseen new bias that unintentionally skews a time series trend.
We have no way to know whether that happened on the latest Gallup poll (absent a controlled experiment**), but it is certainly possible the change in question increased the Bush approval rating by a few points.
Fourth, as many comments on yesterday's post have noted, the 48% to 48% tie in the generic Congressional ballot question was based on the sometimes controversial Gallup likely voter model. Our Slate update yesterday provided a quick and dirty summary:
Ideally, pollsters and pundits prefer to watch likely voters because, well, they're more likely to vote than those who are simply registered. But identifying the likely electorate is much more difficult when an election is still months away, because respondents are less able to honestly assess whether they're really going to vote. (Getting a large enough sample of likely voters also costs more money, so media pollsters usually wait until closer to the election.)
The problem is that once pollsters start screening for likely voters, their methodologies vary widely. This produces the scattershot results we've seen recently. An AP-IPSOS poll conducted last week showed likely voters preferring the Democrats by a 14-point margin (53 percent to 39 percent). Other surveys conducted over the last two weeks by Zogby, Harris, and Fox show results that were more encouraging for Republicans but not the even split that Gallup shows. Further, the Gallup poll's likely voter model has been criticized for producing volatile results, especially when used a month or more before the election.
I have written extensively about the way pollsters choose likely voters. For those without time to read it all, the key point is that while evidence shows the Gallup likely voter model typically provides a better estimate of the vote on the final poll than looking at all registed voters, it can produce a lot of volatility before October. As Mickey Kaus put it yesterday, the model may tell us more about:
'who would vote if the election were held today' as opposed to what we really want to know, which is 'whom would the people who are going to vote on November 7 vote for if the election were held today.'
Finally, consider that we are obsessing over measures with very limited utility to predict election outcomes -- the generic congressional ballot and presidential approval rating. In the races for Senate, on the other hand, we have direct measures of the actual contests and far more surveys to compare. Our Slate Senate Scoreboard has logged 36 new state-level Senate polls released in September alone. And despite the modest increase in the Bush job rating, as of today, these continue to indicate momentum toward the Democrats. Of course, the Scorecard also shows 49 seats currently held or at least leaning Republican, with 46 held or leaning Democrat.
With seven weeks remaining until the election, the data above should leave no one feeling too confident about the outcome. All of these trends can and probably will change. We'll be watching, so stay tuned...