Huffpost Politics
Mark Blumenthal Headshot
Ariel Edwards-Levy Headshot

Gallup Poll Reveals 4 Reasons It Got The 2012 Election Wrong

Posted: Updated:
Print

WASHINGTON -- Gallup, the polling firm that took a major hit last year for finding better results for GOP presidential nominee Mitt Romney than other pollsters found, offered a detailed mea culpa on Tuesday, citing a confluence of small issues that tipped its results rightward.

Over the final weeks of the 2012 campaign, Gallup's daily tracking poll showed President Barack Obama consistently trailing Romney, including a final survey that gave Romney a 1-point edge. Instead, Obama won by nearly 4 percentage points. While Gallup was far from the only polling firm to call the election wrong, its visibility and reputation, as well as the size of its error, made it one of the most notable.

On Tuesday, at a press briefing and in a 17-page report, Gallup named four factors that lessened the accuracy of its polling. As expected, no one problem alone accounted for its difficulties in 2012.

"None of these factors are large, in and of themselves," said Gallup Editor-in-Chief Frank Newport, "in the sense that they are not moving the numbers 10 points, 5 points." However, each of the four nudged Gallup's numbers slightly in Romney's favor, making them collectively "significant enough," according to Newport, "that we think they made a significant difference in our overall estimate of who was going to win the presidential election last fall."

Here’s a brief summary of those issues, several of which The Huffington Post addressed earlier in more detail:

Misidentification of Likely Voters. Like most other media pollsters, Gallup attempts to determine which respondents are likely to actually vote in November, using a procedure developed in the 1950s that involves a battery of questions, such as how often they've voted in the past and how much attention they're paying to the election. Last year, this likely voter model moved Gallup's estimate of the margin separating Obama and Romney 4 points in Romney's direction.

"That, in and of itself," Newport said, "was at least 1 point more towards Romney than the average of other polls that were using some time of likely voter model." Thus, "on that sense alone," he concluded, "they moved things too far in moving the sample towards Mitt Romney."

While Gallup is continuing to investigate its likely voter procedures with a series of experiments in the fall 2013 state elections, it pointed out one big issue in 2012: that many Obama voters said they hadn't given much thought to the election, removing them from the likely voter pool even though they intended to cast a ballot. Gallup is researching whether it needs to majorly overhaul or even replace the way it identifies likely voters.

Under-Representation of Regions. Gallup also weights its data by a variety of factors, including broad geographical regions such as the Midwest and the South. But each region contains several time zones. Due to differing response rates, Gallup didn't interview enough people in certain time zones within some regions, effectively undersampling states that vote more Democratic.

Faulty Representation of Race and Ethnicity. As HuffPost first reported in June 2012, Gallup in recent years has used an unusual method to ask about race that distorted the racial composition of its samples when the data were weighted. Unlike most other pollsters, who ask respondents to select from a list of racial and ethnic categories, Gallup asked respondents whether they identified with each racial and ethnic group one by one. This led to a disproportionate number of people who said they were multiracial, and that in turn distorted the weighting procedure, effectively giving too much weight to some white voters.

Earlier this year, Gallup eliminated the yes/no racial questions and made some additional technical changes to its weighting procedures. As a result, Newport said, "we [now] do a better job moving our sample to the parameters we have as our gold standard ... for overall race representation."

Nonstandard Sampling Method. Before 2011, Gallup had selected phone numbers using random digit dialing, or RDD, which calls randomly generated numbers. This is the procedure that most national media polls have used for decades. As reported by HuffPost earlier this year, Gallup made a significant change in 2011, when it dropped the RDD methodology for its landline sample, using instead numbers randomly selected from those listed in residential telephone directories.

Gallup was alone among national pollsters in making this move, which promised to cut costs by eliminating unproductive calling to business lines and nonworking numbers. But the change came with a downside: Not everyone who has a landline has a listed number. Although Gallup's initial research indicated that cell phone calls would cover the difference, they didn't: The listed sample turned out to be older and more heavily Republican than the RDD sample.

Gallup is now in the process of switching back to RDD sampling for both landline and mobile phones, Newport said.

"Although in theory the use of the landline listed sample made sense ... it probably was a contributing factor to a skew of the sample that we felt was inappropriate, and therefore, we're making that change," he said.

In addition, Gallup also identified a number of factors it said didn't influence its results. Among them: the tracking poll format, the use of the name "Gallup," the race and gender of interviewers, the handling of third-party candidates, the order in which candidates' names were read, and the way undecided voters were nudged into picking a candidate.

But the firm said it's continuing to research these and other areas, using two 2013 elections -- the New Jersey and Virginia gubernatorial races -- as a sort of polling test lab. The results, which won’t be released to media, are intended to shed light on some of the questions plaguing Gallup individually and the polling industry as a whole.

"We think this is a perfect environment in the real world to see what's going on," Newport said. "It's very difficult to do retrospectively without an actual election to use."

Gallup's research later this year will include experimental surveys that attempt to validate the actual turnout of sampled voters in order to assess the accuracy of the likely voter selection process. These surveys will be designed to test a theory on which HuffPost previously reported, that many nonlikely voters were, in effect, screening themselves out -- by declining to participate in the survey.

"Does the fact that [respondents] have gone through the gantlet to the point where they are in our survey," Newport asked, "could that make a difference in terms of saying that they're already more likely to vote than the people who don't enter the sample?"

The ongoing research is being led by Michael Traugott, a renowned political scientist and survey methodologist at the University of Michigan, and several other academic experts in survey methods and statistics. Traugott will organize a fall seminar involving students in the Joint Program in Survey Methodology at Michigan and the University of Maryland, who will participate in the experiments' design. Newport confirmed that Gallup will make the "raw" respondent-level data available to scholars generally through the Roper Center Public Opinion Archives.

Also on HuffPost:

Close
100 Years Of Election Night Losers
of
Share
Tweet
Advertisement
Share this
close
Current Slide

Suggest a correction

Around the Web

Gallup identifies flaws in 2012 election polls

Gallup's 2012 Mea Culpa: What To Expect

Gallup Identifies Factors That Led to Its GOP Skew in November