Gallup's Frank Newport Responds To Criticism
share this story
Gallup Poll Editor-in-Chief Frank Newport posted an online response this morning to the critical analysis of the Gallup likely voter model by Emory University Professor Alan Abramowitz that appeared on the Huffington Post earlier this week. Newport argues that Abramowitz misinterpreted Gallup's data and simply "did not like" the "double-digit Republican lead among likely voters" in the Gallup's recent poll.
For the past two weeks, Gallup has released results showing a huge advantage for Republicans among likely voters on the generic ballot for Congress, the question that asks voters whether they will support the Democrat or the Republican candidate in their own congressional district. They have released results for two scenarios: When they assume a turnout level of 55% of adults nationwide, Gallup's most recent release (based on interviews conducted between September 30 and October 10) gave Republicans a 12-point lead (53% to 41%). When they assume a turnout of 40%, which is consistent with typical off-year voting, they show an even wider, 17-point Republican lead (56% to 39%).
Abramowitz considers the results "wildly implausible" in comparison to findings from previous exit polls. In particular, he focused on results that Gallup shared with him among the non-white voters that show the Republican candidate leading by a 10-point margin (52% to 42%). Drawing on exit polls, he infers that this subgroup is likely two-thirds Latino and concludes that, as such, the apparent shift to Republicans is the "most implausible result" in the Gallup Poll.
Some Democrats have seized on the Abramowitz analysis, including Simon Rosenberg of the New Democratic Network (NDN), who concluded that the poll "is so statistically flawed that Gallup should revise the model and its results or take it down from its website immediately."
In response, Newport addresses Abramowitz' subgroup analysis:
Having the in-depth Gallup tabs in hand, Alan took issue with various voting patterns among subgroups, mainly saying that they were "too Republican". But of course, in a situation in which the Republicans have a historically high lead on the generic ballot, Republicans will mathematically have a historically high lead in many of the subgroups within the overall pools of voters. That point should, for most observers, go without saying.
One of the specifics of Alan's micro-analysis of estimated votes among smaller subgroups focused on "nonblack, nonwhites." That's actually a group not represented in the cross-tabular data we typically use and was not in the data provided Alan, but one Alan apparently attempted to identify by performing his own calculations. (Typically a scholar would contact us or inquire about aspects of the data they are unsure of, but I don't believe we heard from Alan on this one.) In this particular case, we would have told Alan that nonwhites in our usual procedures is a broad, mixed group of respondents, including blacks, Hispanics, Asians, other races, and a significant number of respondents who chose not to identify their race. Alan attempted to make guesses or assumptions about the composition of this group, and made an assumption as a result that Hispanics in the likely voter sample must be too Republican in voting orientation.
In fact, like most pollsters, we typically are cautious and do not report data for subgroups when there is low sample size involved. Hispanics are one of these. Certainly our analyses of broad, aggregated datasets has shown that Hispanic registered voters as a national group skew Democratic, as we have pointed out many times. But Gallup has also shown that Hispanics' support for President Obama tumbled into the low 50s earlier this year and has only recently recovered some. Further, among likely voters within all subgroups, those most likely to vote this year are disproportionately Republican in their orientation compared to the subgroup as a whole.
So the gist of Newport's response, which is worth reading in full, is essentially that Abramowitz was wrong to assume that most of the "non-white" respondents are Latino, and that more generally, their results will prove to be plausible if they are correct. I was curious, however, why Newport's post never spelled out just how many respondents were Latino. So I asked via email, and he sent the following response:
The overall percentage of Hispanics in the Sep 30-Oct 10 likely voter sample (using the low turnout model) is 4% and it is 8% in the national [adult] sample. Although, as noted, we do not report exact numbers for smaller subgroups, the vote choices of the Hispanics in the likely voter sample skew substantially Democratic. It is important to note that the percent of Hispanics in our final likely voter sample was 4% in 2002 and 5% in 2006, and both of the generic ballot estimates from these years were remarkably accurate compared to the overall national vote, even with one being a Republican year and the other a Democratic year. So this year's sampling appears to be on track with previous, accurate likely voter modeling. As I noted in my discussion, and as I think you understand, micro analysis of the composition of samples is often not productive, particularly when comparisons are made to other polls, particularly from past elections (such as exit polls), and misses the larger point that samples are accurate taken as a whole.
So Hispanics represented 4% of the weighted value of their likely voter sample rather than the roughly 85 that Abramowitz assumed. If the unweighted sample size is roughly the same as the weighted value, that 4% translates into roughly 75 interviews out of roughly 1900, a subgroup that would have a margin of error of at least +/- 12%. Newport is not sharing the actual result. Is it implausible? Well maybe it is, Newport implies, but so what -- it's a small sample size with a huge margin of error that has little impact on the overall numbers.
While I agree that a "micro" focus on small subgroup inconsistencies is often inconclusive, simply brushing aside criticism as griping from partisans about results they "dislike" misses the larger issue. The results of Gallup's traditional likely voter model, using their traditional assumptions about turnout, have been very different as compared to other pollsters this year. Given Gallup's outsized influence on the campaign narrative, we need to understand all we can about why their results look the way they do. Critical analysis from all corners is warranted, whether we dislike the results or not.
Loading comments… 
First Posted: 10/22/10 07:16 PM ET Updated: 05/25/11 07:05 PM ET