The latest Gallup (5/7-10/09) poll has party identification tied at 32-32 and caused an immediate howl of "outlier!" in the comments at Pollster.com. In this case, the howl is justified. Compared to all recent Gallup polls (so we compare apples to apples) this latest stands out quite a bit from the rest.
The chart above shows the trends since January 2001 for Gallup polls for Dems, Reps and the Rep minus Dem margin. This latest poll is circled for easy reference. The Dem percentage is a bit below trend, while the Rep percentage is considerably higher than trend. The difference of the two is therefore quite high- a difference of zero compared to a trend estimate of -9.3.
The Dem value is just inside the 95% confidence interval while the Rep value is outside the CI, as is the difference. (The Dem and Rep estimates are, of course, correlated, though not perfectly so since Independent is an available option as well.)
Is there anything that might explain this outlier? One intriguing possibility is that this Gallup poll is their annual "Values and Beliefs" survey, which spends most of the interview asking questions about many of the classically partisan moral issues including abortion, gay rights, stem cell research, the death penalty and other topics. (Gallup has not yet released the full questionnaire so my list here is based on the content of past surveys in this series and may be wrong about some details.)
We know how powerfully question order can sometimes affect subsequent responses. Since party id is usually asked late in the survey it is plausible that a lengthy set of questions about these social issues would affect independents who might have recently called themselves Republican. Priming these issues might be expected to provoke more pro-Republican considerations, which in turn could shift the partisan balance. That would be interesting as a finding because of the supposed durability of party id and would invert the outlier finding because the outlier would be the result of the content of the survey, rather than a mere random fluctuation in the makeup of the sample.
Happily we can test this using the five previous Values and Beliefs surveys, which are done annually in early May. (No party ID data have been released for the 2001-2003 surveys, so this analysis is based on 2004-2009 only.) If the question order effect is real, we should see the pid distribution shift in a pro-Republican direction each year, since the content is similar and should provoke similar imbalance in considerations among independents who might lean Rep if they thought about these issues. The chart below shows the residuals for the pid series, this time circling all the Values and Beliefs polls.
The data are quite clear: There is no evidence at all that the Beliefs and Values surveys are systematically skewed in party id. Only the new 2009 edition of this series is an outlier. It looks like idiosyncratic sampling fluctuation remains a more plausible explanation.
This "null finding" is worth appreciating for a moment. I had high hopes for the hypothesis, which fits well with research on both priming and question order effects. And yet we find no difference. This too is consistent with the literature: question order effects have proved irritatingly difficult to provoke on purpose. We find large order effects sometimes, yet when we try a different treatment which is seemingly just as likely to have an impact, we get nothing. This seeming randomness in when we can and cannot provoke order effects is a cautionary tale about being too quick with post-hoc explanations for outliers based on question order. These effects may be real, but require substantial evidence before we accept them. Theory alone is a poor guide to empirical reality in this case.
It is easier to be confident about the outlier status of this poll than to account for why it is so clearly out of line with previous Gallup results. At least we can address the outlier status empirically and with some statistical confidence. They "why" of that status must remain the always true maxim: "Outliers Happen."