WASHINGTON -- Conflicting survey results are often a fact of political life, and as we reported earlier this week, such inconsistency is apparent in some recent surveys of Egypt. But underlying this latest polling kerfuffle is a simple idea, one that anyone who reports on survey data needs to understand: If we want to measure public opinion in a country, a survey designed to represent everyone in that country is far more appropriate than a survey designed to represent less than a quarter of its population.
On Monday, The New York Times' Nate Silver published results of a BBC World Service survey conducted in late 2009 by GlobeScan and the Program on International Policy Attitudes (PIPA) at the University of Maryland, showing that "Egyptian sentiment toward the United States has improved dramatically" since 2007. I subsequently linked to a second survey, conducted by the Pew Research Center's Global Attitudes Project, that shows the Egyptian public with overwhelmingly negative attitudes toward the U.S. that have grown somewhat more negative since 2006. I also noted that the BBC survey sampled only urban areas of the country, just 22% according to the BBC report.
Silver later posted an update in which he conceded it "worries me a bit" that "the BBC poll's numbers have shifted so significantly" especially in comparison to the Pew survey, though he reasserts his belief that the BBC results are "slightly more plausible." He also cautions against the temptation to to "categorize one of these surveys as being 'right' and the other as being 'wrong."
But the point is not whether any particular survey was right or wrong, good or bad. The issue is that one survey was designed to represent all of Egypt and the other was not.
Three years ago, then-ABC News Polling Director Gary Langer took issue with the same BBC global survey for some of the same reasons:
The poll's disclosure notes that nearly half the samples - 16 of the 34 - were urban-only, with noncoverage of the full adult national population ranging from 26 percent to 95 percent, averaging 66 percent (leaving aside India, where the coverage is unclear). That's the share of the national population in each country that had no chance of being selected for participation in the survey - a high to extremely high level of noncoverage.
Does it matter? Unknown, without full national data to compare. But as I always say, imagine an urban-only poll in the United States: President Kerry would love the results. The point is that we wouldn't accept urban-only coverage in a U.S. survey - which raises the question of why we'd regard it as acceptable in other countries.
In this case, we do have full national data available and it tells a very different story than the low-coverage BBC poll. A survey covering 98% of the Egyptian population conducted in April and May 2010 by the Pew Research Center's Global Attitudes Project finds just 17% of Egyptians reporting a favorable opinion of the United States, while 82% have an unfavorable opinion.
A third survey conducted by Gallup last year reports just 19% of Egyptians expressing approval of "the job performance of the leadership of the United States" (49% disapprove). As Gallup explains here, their sample covers "the entire country, including rural areas" in "every country" they survey.
In contrast, the most recent BBC survey of a slice of "urban" Egypt finds 45% who say the United States has a "mainly positive influence in the world," while just 29% say the U.S. has a mainly negative influence.
Silver dismisses concerns about poor coverage as "something of a red herring in this particular case, since, as Mr. Blumenthal noted, the Pew survey showed no appreciable urban-rural split." He refers to data from Pew Research that we published on Monday showing slightly more negative views of the United States in urban areas of Egypt (16% favorable, 83% unfavorable) than in rural areas (19% favorable, 81% unfavorable) on their most recent survey.
The problem with this line of argument is that the surveys define and sample "urban" Egypt differently, so the lack of a large urban/rural difference in the Pew Global does not rule out sample coverage as a problem.
The Pew Global Attitudes Project bases their urban/rural classification on geography, according to Associate Director Richard Wike. They design their sample to randomly select respondents in urban and rural areas in proportion to their share of the overall population. As a result, the Pew Research survey classifies 43% of the Egyptian population as urban, a level that Wike says is consistent with population data from the Egypt's Central Agency for Public Mobilization Statistics.
The BBC/PIPA survey characterizes its Egyptian sample as "urban." Their report adds the following information:
In Egypt the survey was conducted in Alexandria, Cairo, Giza, and Shoubra al-Khaima, representing 22% of the total national adult population.
They provide no further details, but regardless of the definition or measurement applied, simple arithmetic tells us that the BBC/PIPA sample reached a far narrower portion of "urban" Egypt (22%) than the Pew Research urban subgroup (43%).
Yet another Egyptian survey conducted by PIPA in April and May 2009 adds one more wrinkle. It asked an identically worded question about the influence of the United States and used the same research partner in Egypt (Attitude Market Research) yet appears to sample a larger slice of "urban" Egypt. The PIPA report describes the "face-to-face" sample for the April/May survey as follows:
In Egypt, the survey was executed in the urban areas of Cairo, Alexandria, Giza, and Subra. These four urbanized areas represent 75 percent of Egypt's urban population, which is 42 percent of the national population.
While that verbiage may be confusing, PIPA Director Steven Kull clarifies that the April/May survey sampled 31% of the national population (or 75% of 42%). Mohamed Al Gendy of Attitude Market Research also confirms that Subra and Shoubra al-Khaima both refer to the same geographic area in greater Cairo, but neither Kull nor Al Gendy could explain the apparent discrepancy between the 21% to 22% coverage of the BBC surveys and the reported 31% coverage of the 2009 Attitude Market Research survey. Via email, Al Gendy promises to check his files on the 2009 survey "early next week when we can go back to work," understandably citing "what is going on in Egypt" as a barrier.
Whatever the explanation, the April/May 2009 PIPA survey appears to have sampled a larger slice of Egypt's urban population than PIPA's surveys for the BBC. Also, note that PIPA's own report describes "Egypt's urban population" as "42 percent of the national population" (they did so again here), a number that matches estimate produced by the Pew Global survey.
Moreover, the same PIPA survey produced a very different result for the same question asked on the BBC surveys about whether the United States has a positive or negative influence in the world. As the table below shows, the larger-slice PIPA survey conducted in April and found just 15% of urban Egyptians rating the influence of the U.S. as positive. Yet smaller-slice surveys of the urban population conducted by the same survey partner using an identical question found dramatically more positive readings just four months before (40%) and seven months after (45%).
PIPA's Kull argues that real events explain much of the variation. He notes, as illustrated below, that the Gallup survey in early 2009 appears to confirm a brief improvement in Egyptian attitudes toward the U.S. registered in the BBC/PIPA survey in late December 2008 following Barack Obama's election the previous month. And both the Gallup and Pew Research surveys show improvement in perceptions of the U.S. following Barack Obama's speech in Cairo in early June 2009 (although the speech fell in the middle of the field period of the Pew Research survey).
But neither the Gallup nor Pew Research surveys confirm the apparently dramatic decline in the PIPA trend line between late 2008 and April/May 2009. These are very large shifts -- far more than can be explained by random sampling error.
So we have something of a puzzle. Perhaps urban-is-urban-is-urban, in which case the Egyptian urban public exhibits an unusually high degree of volatility in its views of the United States. Perhaps other aspects of the methodology or survey administration can explain the conflicting results. Or perhaps the degree of urban coverage matters. We may never know the answer, but the differences in coverage are too large to simply dismiss as a "red herring."
Those differences also take us back to the fundamental question that I put to PIPA's Steven Kull. Whatever the explanation for the conflicting numbers, why rely on a survey that covers just 21% of Egypt's population?
"We would love to have national samples when we can get them," he says, "but there are a lot of constraints we have to deal with." He also points out that PIPA explicitly identifies the Egyptian samples discussed here as "urban" and has never asserted that its urban surveys are "something you can extrapolate to a national sample." Nonetheless, he argues that results from urban samples usually mirror those from national surveys. "Analyzing national data from numerous developing countries including Egypt we have found that urban and rural samples vary only slightly after controlling for [don't know responses]."
Kull also cautions against making too much of BBC/PIPA's Egyptian results. "We have never made the assertion that attitudes among the Egyptian people toward the U.S. have fundamentally changed," he says. "Before doing that, I would want to see polling data be more stable over time and come from multiple sources. We have made no such assertions."
All this may leave ordinary readers confused, but we need not get lost in the pollster puzzle. When it comes to Egypt, we do have readily available national samplings of public opinion. And given that high quality in-person, random probability surveys are available that cover close to 100% of Egyptians, why bother with a survey that covers just 22%?