Experts On Afghanistan Doubt Survey On Foreign Occupation: Results Are Impossible
A new survey of the Afghan people is being touted as evidence that hearts and minds may, in fact, be warming to the U.S.'s military presence, which is heading into its ninth year.
MSNBC's Rachel Maddow, generally a critic of the continuation of the war, heralded the survey as hopeful news on Tuesday night.
But can it be taken seriously?
In a word, no, say people who have worked extensively on the ground in Afghanistan.
HuffPost interviewed Prakhar Sharma, head of research at the Centre for Conflict and Peace Studies (CAPS) in Kabul, who has done a large amount of public-opinion research work in Afghanistan, where he is based; Matthew Hoh, a foreign service officer who resigned last September in protest of the administration's Afghan policy; Anand Gopal, a Wall Street Journal reporter who has traveled widely in Afghanistan; and Christian Parenti, a reporter with The Nation who travels frequently to Afghanistan and was the field producer of the Afghanistan-based documentary The Fixer
Four of the five say that reliable survey results in Afghanistan are impossible for several obvious reasons, and some not so obvious. The obvious ones first: The Taliban controls large swaths of the country and the war has made much of the country unsafe to travel through. The Taliban doesn't do surveys, so anybody approached by somebody with a clipboard knows that the person either represents foreign troops, the central government or a private company associated with one or both.
Then there are the not-so-immediately obvious reasons: Afghanistan is a highly patriarchal society, meaning that getting a woman's true opinion is extremely hard. Sharma said that his research teams have never been able to get even close to the 50-50 male/female split that the ABC survey claims.
Getting a man's honest opinion is no simple task, either, he said, because the responses are calculated to protect and benefit the respondent's family and village. "The Afghans know it when they see sudden changes in development assistance, changes in government officers, police tashkils/numbers [the Tashkils are the national police], more/less operations immediately after the polls. It is difficult to pretend to them that the polls do not matter. Their responses are therefore calculated," he said.
Those with experience in Afghanistan were skeptical that the surveyors actually went where they said they did. "If you look at it, the polling was conducted in built-up areas, in urban areas where we have our bases and where the Afghan government has a presence, primarily off the major highways," said Hoh. "So through the South and West of the country, primarily it was done right along Highway 1 where the government has control and where we have control. Off those areas, we don't have control."
Feld said that his company didn't target safe areas. "The villages are actually randomly selected. There's no convenience selection. If it's on the top of a mountain, that's where we'd go," he said. "Our guys would walk into a village and we ask questions."
Hoh said he simply doesn't buy it, both because the areas are impenetrable and because the answers make no sense. "I just don't really believe that, because what I saw in both the East and the South of the country...where all the fighting's really going on, this doesn't jive with, it doesn't agree with what you get when you go out and meet with local villagers. When you go out to these valleys and meet with folks, it doesn't square that they see a central government as a force for good," said Hoh. "I just don't think it's possible to get accurate polling in a war zone like that, particularly one that's been at war for 30 years, where the government has been oppressive and they can't trust it, whether it's been the communists, the Taliban or the Karzai government."
Kabul, Parenti noted, has been trying to put down its rural surroundings for decades, and for just as long, the rural areas have fought back. Many urban Afghans, he said, may indeed support the escalation to the extent that it increases Kabul's power over its surroundings. In the countryside, though, the reverse would be true. Finding that they all of a sudden support the central government doesn't ring realistic to him, Parenti said, and he doesn't believe that the researchers got to the relevant areas. "Kabul headed East or South and even to the West a bit, it's very dangerous," he said. "They wouldn't even go into an area that they didn't have control over. Huge parts would be no-go for people associated with the government." (Disclosure note: I've done foreign reporting from Bolivia and Iraq with Parenti.)
Karl Feld, research manager for Virginia-based D3 Systems, whose Afghan subsidiary was hired by ABC News, the BBC and ARD to conduct the survey, said that his interviewers were not seen as associated with the government because they are Afghans. "Interviewing teams all live in the area they work in. They're not viewed as outsiders," he said.
Sharma said it's nowhere near that simple. "Yes, the interview teams are usually supposed to be recruited from districts where they belong, to minimize the suspicion, but that does not mean that you could do surveys in Khas Uruzgan or Zurmat or Panjwai or get a perfect gender balance. As far as I can tell, we were not able to survey certain provinces at all for months in 2008 because of insecurity. The locals (they were from the same district) refused to be seen as holding questionnaires in their hands and talking to people to 'elicit' their responses. It was way too risky. Things have not improved in the South. They have, in fact deteriorated in most of the South and the Southeast during the last one year," he said.
Gopal and the others said that Afghan respondents try to figure out what the interviewer wants. "They almost always tell the surveyor what he wants to hear. Moreover, people generally understand that the Taliban do not do surveys, so any surveyor is seen as representing the government. In many parts of Afghanistan, where they almost never see people from outside their district, people coming and asking them political questions gives the impression that they are representatives of the government or foreign forces (and they often are)," wrote Gopal. "I've seen this first hand when I accompanied surveyors in the field a couple of years ago."
The interviews were conducted December 11 to 23, and reached 1,534 Afghan adults, the survey claims. Sharma said he has worked directly with the D3 subsidiary and found them to be the best qualified survey contractor among an unimpressive field. But even with that outfit they found "data falsified for insecure provinces (90 respondents in Ghazni had identical responses to all governance related questions, for instance)," Sharma wrote in an e-mail.
The survey claims to use a "Kish-Grid" approach to select individuals for questioning. Sharma is skeptical. "I have personally spoken with regional coordinators who would manage field teams and they used to laugh at 'Kish-Grid' approach because it is simply not feasible in Afghanistan," said Sharma.
The survey takers said that they ran into only marginal security issues: "Of the 101 districts initially drawn in the sample, 11 were inaccessible for security reasons and were randomly replaced with other districts in the same province; a 12th was inaccessible because a road washed out, and likewise was replaced. At the settlement level, 21 of the 194 sampling points were replaced for various accessibility reasons, a customary number of settlement-level replacements."
Sharma and others who've traveled throughout Afghanistan said that it is simply implausible that researchers could get to 90 percent of the areas they sought. Feld insists that they did.
What's more likely, the critics said, is that the interviewers spoke to people they already knew.
Gopal explains why that would be. "The way the surveys work is by recruiting, say, 34 people for the 34 provinces," he writes. "Each of these people are then tasked with finding participants for the survey in their province. In rural Afghanistan, with geographical, logistical and security concerns, these people can't very well go door to door. Moreover, they can't randomly select phone numbers here because there are no area codes like in the States (so that you can ensure an even distribution geographic distribution) and only major urban areas have good network coverage. Therefore the surveyors usually find participants by polling their friends and family. This means that you don't have a random sample, and the results of the survey depend entirely on the political outlook of [the] person in charge. Since the surveyors are often educated people who live in urban areas and have ties to the government (in most provincial urban centers, almost every educated person--and there's not many--have family members working for the government, because that's the only job available to them.), there's a heavy pro-government and pro-coalition bias in the surveys."
Loading comments… 
First Posted: 03/18/10 06:12 AM ET Updated: 05/25/11 04:10 PM ET