04/04/2007 04:04 pm ET Updated May 25, 2011

Order Effects in The Fox News Poll?

A series of unusual questions near the end of a Fox
News/Opinion Dynamics survey
released last week generated criticism from the left side of the blogosphere
(see Jonathan Singer on MyDD,
, Carpetbagger
, Keith Olbermann via Crooks
and Liars
). As Carpetbagger's Steve Benen puts it, "FNC [Fox News Channel]
is practically offering professors of quantitative analysis case studies in
what not to do in a poll." He has a point. The wording and ordering of some of the
question in the Fox News survey are a lesson in things pollsters should avoid.

Let's start near the end of the survey with a question about
the recent congressional vote on the Iraq War (the Fox poll was fielded March
27-28 among 900 registered voters nationwide):

40. Last week the U.S.
House voted to remove U.S.
troops from Iraq
by no later than September 2008 -- would you describe this as a correct and
good decision or a dangerous and bad decision?

44% Correct and good

45% Dangerous and bad

11% (Don't know)

Now compare these results to findings from four other recent
surveys (each sampled all adults rather than registered voters - text and
results via Polling Report):

Newsweek Poll, March 28-29, 2007, n=1,004 adults - Do you support or
oppose the legislation passed this week by the U.S. Senate calling for the
withdrawal of U.S. troops from Iraq by March 2008?

57% Support

36% Oppose

7% Unsure

CBS News Poll, March
26-27, 2006, n=831 adults - Do you think the United States should or should not
set a timetable for the withdrawal of U.S. troops from Iraq that would have
MOST troops out by September 2008?

59% Should

37% Should not

4% Unsure

USA Today/Gallup,
March 23-25, 2007, n=1,007 adults - Would you favor or oppose Congress taking
each of the following actions in regards to the war in Iraq? How about setting a
time-table for withdrawing all U.S.
troops from Iraq
no later than the fall of 2008?

60% Favor

38% Oppose

2% Unsure
Pew Research Center, March 22-25, 2007, n=1,245 adults - And thinking about a specific proposal: The Congress is now debating future funding for the war in Iraq. Would you like to see your congressional representative vote FOR or AGAINST a bill that calls for a withdrawal of troops from Iraq to be completed by August of 2008? .

59% Vote For

33% Vote Against

8% Unsure

So we have four slightly different questions from four
different survey organizations asked during roughly the same period that
provide very similar results. Support for the congressional vote to withdraw
troops from Iraq
in 2008 varies between 57% and 60%. But the Fox News poll shows only 45%
support. Why the difference?

One contributing factor may be the oddly double-barreled answer
categories on the Fox question that imply "danger" in the congressional vote. Put
another way, the Fox item asks two questions: Was the vote "good" or "bad," and
was the vote "correct" or "dangerous?" Good pollsters avoid double barreled
questions, but even then, the wording of this question is odd. "Dangerous" is
not exactly the converse of "correct." Some Americans may see the vote is both good
and dangerous.** How should they
answer the question?

But the bigger potential problem has to do with question order.
Consider the question that Fox asked just before
the item on the Congressional vote:

39. Who do you trust
more to decide when U.S.
troops should leave Iraq -- U.S.
military commanders or Members of Congress? (ROTATE)

69% Commanders

18% Congress

7% (Both)

3% (Neither)

3% (Don't know)

40. Last week the U.S.
House voted to remove U.S.
troops from Iraq
by no later than September 2008 -- would you describe this as a correct and
good decision or a dangerous and bad decision?

44% Correct and good

45% Dangerous and bad

11% (Don't know)

While I can only speculate, I suspect that the order of
these two questions primed respondents with the notion that the House vote was
at odds with the recommendations of military commanders (although the initial
item certainly said nothing about the views of those commanders). The only way
to know for certain would be to conduct a controlled experiment that divides
respondents into two random samples, asking Q39 first for half the respondents
and Q40 first for the other half. And even that sort of experiment would tell
us nothing about the potential impact of the 38 questions that came first.

Academic methodologists have conducted many such experiments
over the years showing that the order of questions can sometimes affect
results, often in very subtle and unexpected ways. In fact, the academic evidence
on these "order effects: tends to involve far more subtle examples of priming
than the one theoretically at work above (for examples, see Chapter 2 of the
seminal text by Howard Schuman and Stanley Presser, Questions
and Answers in Attitude Surveys

I have sympathy for the task media pollsters often face in
ordering questions on long surveys covering a wide range of topics that change
from month to month. They seem to struggle to resolve goals that often come
into conflict: On the one hand, they want to preserve the order of questions
asked previously. On the other hand, inserting new items can risk an order
effect in which the older items asked earlier bias the new questions inserted
later. So they often face no-win choices.

In this case, however, the intent of the author of these
questions seems clear. Check the question (Q36) that nearly the
liberal bloggers cited as a self-evident example of partisan bias:

36. After the 2004
presidential election, the president of the left-wing political
action committee made the following comment about the Democratic Party, "In the
last year, grassroots contributors like us gave more than $300 million to the
Kerry campaign and the DNC, and proved that the Party doesn't need corporate
cash to be competitive. Now it's our Party: we bought it, we own it and we're
going to take it back." Do you think the Democratic Party should allow a
grassroots organization like to take it over or should it resist
this type of takeover?

16% Yes, allow a
grassroots organization to run the party

61% No, don't
allow a grassroots organization to run the party

24% Don't know

Could this clearly partisan "message test" have had an order
effect of its own on the subsequent questions on the congressional vote on Iraq? Your
guess is as good as mine, but either way, the items at the end of this Fox News
survey are not exactly "by the book."

**See for example, the last two questions on this same Fox
poll. On Q41, 59% agree that the "killing or capturing of Usama bin Laden"
would be a "huge" or "major" accomplishment. But on Q42, 47% go on to say that
bin Laden's capture would also "encourage additional terrorist attacks," while
only 24% say it would "discourage" further terrorism. Many Americans apparently
consider that outcome both desirable and