Wisconsin Standoff: Rasmussen's Leading Questions
WASHINGTON -- Amid the ongoing protests and legislative standoff in Wisconsin, a new automated Rasmussen Reports survey purports to show more Americans backing Republican Gov. Scott Walker than the public employee unions. Additional national survey data is not yet available on this issue, but there are good reasons to be skeptical of the Rasmussen poll, which in this case appears to lead respondents to a desired result.
Any pollster attempting to measure reactions to a public policy debate faces a big challenge in the large number of Americans who will know little or nothing about an issue. Some will have well-formed attitudes to share, others will be less familiar and will form opinions on the spot in response to the text of the questions presented.
In this case, Rasmussen begins by asking respondents how closely they have been following the Wisconsin controversy and reports that 37 percent say they are following the story very closely and a total of 67 percent at least somewhat closely. Presumably the remaining third say they are following the story not closely or not at all (based on how Rasmussen has asked about news stories in the past). How those not following the story closely react to the language of the questions that follow is critical.
Here is what Rasmussen asked, in order, on their automated telephone survey of 1,000 likely voters nationwide conducted this past Friday and Saturday:
1* How closely have you followed news reports about the Wisconsin governor's effort to limit collective bargaining rights for most state employees?
2* Does the average public employee in your state earn more than the average private sector worker in your state, less than the average private sector worker in your state or do they earn about the same amount?
3* Should teachers, firemen and policemen be allowed to go on strike?
4* In the dispute between the governor and the union workers, do you agree more with the governor or the union for teachers and other state employees?
5* Would you favor or oppose reducing your state government payroll 1% a year for 10 years, either by reducing the number of state employees or by cutting the pay of state workers?
The question of greatest interest is the fourth, which finds 48 percent siding with Walker, 38 percent with the public employee unions and 14 percent undecided. But consider the context set by the questions that come first.
Rasmussen does begin by describing the issue as involving "the Wisconsin governor's effort to limit collective bargaining rights for most state employees." But that verbiage assumes that respondents are familiar with the term "collective bargaining rights." The Rasmussen survey then asks two questions that prime messages frequently offered by Walker's supporters: The supposed earning disparity between public and private sector employees and the specter of teachers and fire and police personnel going out on strike.
The issue is not that they asked about salary disparities or striking by teachers or police or firefighters, but that they asked those questions before the more general probe of whether respondents side with Walker or the unions. The more typical approach would involve asking a more general version of question one ("how closely have you been following the dispute between the Governor of Wisconsin and the public employee unions in Wisconsin?") and then go immediately to something like question four.
Without a side-by-side experiment, we cannot know for certain that the context set by the order of Rasmussen's questions helped push some respondents, especially those who know little about the issue, toward support of Gov. Walker's position. But consult one of the most widely used textbooks on survey methodology (p. 232) and the guidance is clear: "When asking general and specific questions about a topic, ask the general questions first." Rasmussen ignored that standard practice in this case.
Compare the information provided by the Rasmussen question with another robo-survey conducted in Wisconsin last week by WeAskAmerica, a for-profit subsidiary of the Illinois Manufacturers' Association (an organization that has endorsed Republican candidates for state office).
WeAskAmerica used their automated methodology to ask the following question of 2,397 registered voters they were able to reach an interview using and automated telephone methodology this past Thursday:
As you may know, Gov. Scott Walker has proposed a plan to limit the pay of government workers and teachers, increase their share of the cost of benefits, and strip some public-employ unions of much of their power. We'd like to know if APPROVE or DISAPPROVE of Gov. Walker's plan.
They found 43 percent of the Wisconsin voters interviewed approve of the plan, 52 percent disapprove and 5 percent are uncertain.
WeAskAmerica informs respondents that Walker's plan would "limit the pay of government workers and teachers" and "increase their share of the cost of benefits." The Rasmussen questionnaire says nothing about Walker's proposed pay cuts and benefit changes.
WeAskAmerica tells respondents in plain language that the Walker plan would "strip some public employee unions of much of their power," while Rasmussen uses more technocratic terms, though earlier in the survey.
For this sort of issue, no single question is perfect. The odds are good that further national surveys will produce a range of responses that will vary depending on what aspects of the controversy the pollsters choose to describe. Rasmussen's habit of reporting only the views of "likely voters," whose demographics and political leanings are never disclosed, may also make these results differ from others to follow that will mostly sample all adults.
We will eventually have more probes of public opinion on this issue to consider, but for now, Rasmussen's results raise more questions than they answer.
P.S. The British sitcom "Yes, Prime Minister" offers a more concise, satirical description of this style of survey design:
Loading comments… 
First Posted: 02/21/11 06:24 PM ET Updated: 05/25/11 07:35 PM ET