The polling head-scratcher of the day: In a survey conducted March 12-15 (among 1,019 adults, margin of sampling error +/-3%), CNN/ORC obtained these results, producing the headline "Obama's Taken on Too Much":
Since he became president, do you think Barack Obama has tried to handle more issues than he should have, or hasn't he done that?
55% - More issues than he should have
43% - Has not done that
2% - No opinion
But just yesterday, the Pew Research Center released a survey conducted March 9-12 (among 1,308 adults, margin of error +/- 3%) with this question, which supported the sub-heading "Obama Not Seen as Overextended" within their report:
So far, do you think Barack Obama is [read and randomize response options, with option 3 always last]
35% - Trying to address too many issues at once
4% - Focusing on too few issues [OR]
56% - Doing about right
5% - Don’t know/Refused
Huh?
Let's stipulate that the following is speculation, but the usual culprit in these sorts of discrepancies is that pollsters are asking about something that a lot of respondents have not really considered before. Some call these "non-attitudes." As a general rule, it is hard to understate how often ordinary Americans are oblivious to the controversies that seem oh-so important inside the beltway, on cable news or the blogosphere. I would wager that the is-Obama-overextended meme is probably one of them (same goes for the word "meme").
One big clue is that although both questions ostensibly ask about the same idea, they use different wording. In particular, the Pew question is far more explicit about labeling one option as approving of Obama's performance ("Obama is...doing about right"), while the CNN equivalent is a little more vague ("Obama has tried to handle more issues than he should have"). Perhaps some respondents interpret the words "should have" differently, agreeing that Obama is having to handle more issues than he "should have" were these ordinary times. Or perhaps some are not hearing the words "than he should have" at all, and instead interpret the question as asking whether Obama is trying to handle many issues or few issues.
Notice also that the CNN question offers only two options, while the Pew question offers three. Most important, the "doing about right" response is the middle option on the Pew question, and survey methodologists know that when respondents are confused or uncertain, they often flock to the middle choice.
In the course work I did a dozen years ago through the University of Maryland's Joint Program in Survey Methodology (JPSM), one of the most valuable experiences was a class project that conducted "cognitive pre-testing" of survey questions . Each student had to find 10 subjects -- at least half without a college education -- willing to "think out loud" while they answered 20 minutes worth of survey questions. We did the interviews ourselves, and then compared notes on problems we discovered (for more info, see also Methods for Testing and Evaluating Survey Questionnaires).
Even though I had written survey questions for many years, the experience was an eye-opener. It was stunning to see how respondents failed to hear key portions of a question, ignoring introductory text altogether or hearing something or answering in a way completely different from what the authors had intended. Some stock questions formats I had used for years failed miserably. Unfortunately in-depth cognitive pre-testing is too costly and time consuming to be used on most political and media surveys. A true cognitive pretest of the CNN or Pew questionnaires would take weeks and might end up costing more than the surveys themselves.
But back to the questions about Obama doing too much. While I cannot prove that question wording and structure is the culprit here, it is worth restating a favorite rule of thumb: When small differences in question wording, structure or order produce wildly different results, the thing we are trying to measure is probably not a fully formed, pre-existing opinion for a significant number of respondents.
PS: Thanks for the Washington Post's Dan Froomkin for suggesting this topic.
Follow Mark Blumenthal on Twitter: www.twitter.com/MysteryPollster
Loading comments… 
