Earlier today, PPP's Tom Jensen teased an odd result from a just completed South Carolina survey that will be released tomorrow. The survey asked the following question about South Carolina's very conservative Republican Senator:
Do you think that Jim DeMint is too liberal, too conservative, or about right? If too liberal, press 1. If too conservative, press 2. If about right, press 3.
Jensen says he was surprised at some partial results showing that 10% of voters saying DeMint is "too liberal." At first he assumed that most were Republicans whose ideology is even further right than DeMint's. But then he looked at the crosstabs:
Breaking down the final numbers this morning though, 51% of respondents who said he was too liberal were Democrats and only 34% were Republicans. I'm guessing those folks were either saying he was too liberal for their own amusement or answering the poll strategically, hoping to put some data point out there that might make DeMint go out even further on a limb and perhaps make himself too extreme even for South Carolina's very conservative electorate.
Jensen concludes that this result is "a good reminder that people can game polls if they want to." I'll wait to see the full results, but I'm dubious.
Pollsters, as David Hill wrote a few weeks ago, are often confronted by the "seemingly disorganized and apparently contradictory views" expressed by survey respondents. Hill cautions, however, that the apparent illogic often results from our own shortcomings as analysts. "Ask enough questions or poke around in [respondents'] heads long enough," he writes, and "there's sense in their ostensible nonsense."
With that advice in mind, let's ask ourselves a few questions about PPP's odd result:
- How many South Carolina voters either don't know DeMint at all or recognize his name but know little else about him? Without an option would uncertain respondents offer when pressed about his ideology?
- How many were not listening carefully and heard the question differently than intended?
- How many were simply confused and unsure of how to answer?
Tom Jensen confirms via email that the DeMint ideology question offered no "unsure" option. All respondents were forced to choose from the responses listed above, even those who might not recognize DeMint or know him well enough to rate him.
The issue of respondent confusion highlights one of the potential shortcomings of automated surveys: With a live interviewer, the respondent always has the option to ask to hear the question again. If a respondent offers only silence or expresses confusion, a good interviewer knows to repeat some or all of the question in order to help prod the respondent to answer. Also, if the respondent is adament that they "don't know," most pollsters allow interviewers to accept that as an answer. In the absence of a pre-programmed "repeat-the-question" option, a PPP respondent unsure of how to answer the DeMint ideology question had only two choices: Enter a number between 1 and 3 -- perhaps even a random number -- or hang up.
Of course, even surveys conducted by live interviewers produce strange results, especially when they push respondents to answer difficult questions. The theory of "satisficing," described in the abstract of 1991 journal article, by Stanford Professor Jon Krosnick, helps to explain the odd responses that sometimes result:
[W]hen optimally answering a survey question would require substantial cognitive effort, some repondents simply provide a satisfactory answer instead. This behaviour, called satisficing, can take the form of either (1) incomplete or biased information retrieval and/or information integration, or (2) no information retrieval or integration at all. Satisficing may lead respondents to employ a variety of response strategies, including choosing the first response alternative that seems to constitute a reasonable answer, agreeing with an assertion made by a question, endorsing the status quo instead of endorsing social change, failing to differentiate among a set of diverse objects in ratings, saying don't know instead of reporting an opinion, and randomly choosing among the response alternatives offered.
Gaming the poll? Nah. More likely just confusion about the question.
Update (12/8): PPP has released the full results of their South Carolina survey. Although more than a quarter (27%) of the sampled voters are "not sure" about whether they approve or disapprove of Senator Jim DeMint's performance, all (100%) express an opinion about whether DeMint is too liberal (12%), too conservative (32%) or about right (56%). The first and most logical thing to check about those who think DeMint is too liberal: How many are Democrats or independents who cannot rate DeMint's job performance?