04/22/2010 01:38 pm ET Updated May 25, 2011

Not a Survey That Will Not Confuse

A message to pollsters from their elementary school grammar teachers: don't use double negatives! In their survey released today, the Quinnipiac University poll ran afoul of this principle of survey questionnaire design, and it should seem obvious why the offending questions could be confusing to a survey respondent.

Take a look at the first of these questions:

40. Do you approve or disapprove of - not using nuclear weapons against countries that do not have them?

Jean Converse and Stanley Presser, in their paper on survey questionnaire design, Survey Questions: Handcrafting the Standardized Questionnaire*, warn that "Double negatives... are to be much avoided; they can introduce needless confusion and they can creep in unobserved."

Converse and Presser give an example of a question with a double negative:

Please tell me whether you agree or disagree with the following statement about teachers in the public schools: Teachers should not be required to supervise students in the halls, the lunchroom, and school parking lots.

And go on to explain why this sows confusion:

...the Disagree side gets tangled, for it means "I do not think that teachers should not be required to supervise children outside of their classooms" -- that is, teachers should be required.

Generally speaking, survey questions ought to be asked in as simple and direct a manner as possible -- for one thing, most respondents have been called with no advanced warning and likely have more important things on their minds than the 10-20 minutes of questions asked in most surveys. They certainly should not be expected to concentrate on just how many "nots" were in a given question, much less on their exact placement and meaning to the question.

Quinnipiac piled on to their hapless respondents with the following, which likely made several of the interviewees' heads spontaneously explode:

41. Do you approve or disapprove of - not using nuclear weapons against countries that do not have them, even if they launched a biological or chemical weapons attack against the United States?

The double negative was confusing enough without the additional complexity of the caveat. It's likely that many of the respondents simply didn't know what they were being asked here. The likelihood that a respondent missed at least one important word is very high, making it difficult to put much if any stock in the results reported by Quinnipiac on these particular questions.

As an added note: while we don't usually like piling on pollsters, John Sides noted yesterday that Quinnipiac's headline for their release on domestic issues inaccurately portrayed their recent results as showing an Obama "bounce" that had since "gone flat." His piece is worth clicking through to read.

*Converse, J. M., & Presser, S. (1986). Survey questions: Handcrafting the standardized questionnaire. Newbury Park, CA: Sage Publications.