Last week our colleague David Moore highlighted some contradictory results on the issue of the proposed economic stimulus program to illustrate, as he put it, "how easy it is for pollsters to manipulate public opinion into something different from what it really is." He pointed out that a question asked by Gallup produced just 11% with no opinion, while a similar question posed by the NBC/Wall Street Journal poll produced 30% in that category when it prompted respondents to say when they "do not have an opinion either way."
David is nothing if not provocative, and he comes to his opinions from a long career in the field of survey research. For those reasons we are glad he has become a regular contributor. I also agree with him that pollsters too rarely report probes of overall awareness of specific proposals, or conversely, the reluctance of respondents to offer an opinion when asked questions about complex public policy issues. More often than not, issue questions on media polls measure the way the public reacts to the information presented rather than a pre-existing opinion on the proposal being discussed. That habit, as Moore argues, tends to exaggerate our sense of the public's engagement in matters of public policy. But I part company with David when he implies a nefarious intent, arguing that pollsters "are generally not interested in realistic measures of public opinion," and choose instead to deliberately "create" or "manipulate" public opinion.
By way of explanation, consider all of the questions asked over the last few weeks that attempt to gauge support for the proposed economic stimulus legislation. I have ranked the table below on the percentage expressing support (all are from January, follow the links for details on specific dates, sample sizes and reported margins of error).
When you compare the results this way, a few obvious patterns emerge:
1) When asked whether they favor or oppose an "economic stimulus" plan that would cost $800 billion or so (give or take a hundred million), Americans generally express support in the mid-50-percent range.
2) When pollsters also provide an explicit "do not have an opinion" option, as NBC/Wall Street Journal and (presumably) Rasmussen do, support falls to the mid-40 percent range and (on the NBC/WSJ poll at least) opposition also falls proportionately.
3) When the questions provide more information on how the $800 billion (or so) will be spent, usually specifying a combination of tax cuts and transportation, education and energy projects," support grows to mid-60 low-70 percent range.
4) Only one question -- again from NBC/Wall Street Journal -- poses explicit arguments for and against the proposal, and it produces a slightly higher level of support (57%) than the first category of questions that mention only the overall price tag and omits a specific prompt for "no opinion."
5) Every question shows net support for the proposal.
David Moore argues, in effect, that only the second category of question provides a "realistic" measure of public opinion. There we disagree. Yes, it is important to understand that many Americans lack a specific opinion the "economic stimulus" legislation per se, something stressed by too few pollsters. Still, that finding is just one part of "public opinion" on this issue. Reactions to new information are also important, as are the underlying values driving responses to all of the questions reproduced above.
Think about it this way. Suppose you had the chance (and the time and sheer stamina) to sit down and ask a random sample of Americans, one at a time, to share their opinions about what the government in Washington should be doing to "stimulate" the economy. A substantial number, though a minority, would have strong awareness of the details of the plans being debated and would offer highly informed, strongly held opinions. A smaller number, I'm guessing, would offer little more than a blank stare, having no idea what plans are being debated and would express no real opinion about what the government should do even when provided some vague details on the options being considered.
The vast majority, however, would fall somewhere in between. The would offer some opinion about what the government should do, even if they know little about the "economic stimulus" proposal now being debated in Washington. We know (from questions on the recent ABC/Washington Post or the Pew Research Center surveys) that at least three out of four Americans want their government to do something to strengthen the economy. Some are ready to trust (or distrust) whatever the new Obama administration is proposing, on the basis of their trust (or distrust) of our new president alone. Others will have general attitudes about government spending, taxes or specific priorities they want addressed or about other players in the debate.
And the important point is that all of this defines "public opinion" on the issue of economic stimulus. Poll analysts and journalists get in trouble when we try to boil all of public opinion on any legislative proposal down to a a single question, when we try to pick the most "correct" question when their wording differs (see the comments from my old friend Professor M here and here) or when we forget that survey questions sometimes tap both pre-existing opinions and reactions to new information.
PS: To be clear, the polls linked to above asked many questions on the economy and economic stimulus, not just the one or two from each I included above. The Polling Report also has a helpful compilation.