This morning, we posted results from a new survey of Arkansas conducted by the firm Research 2000 on behalf of Democracy for America. The survey was conducted on Tuesday night among 500 "Democratic voters" who said the had voted in the Democratic primary that day. It shows Bill Halter with a not-statistically-significant two point lead over Senator Blanche Lincoln (48% to 46%).
While the DFA release is less than transparent about several aspects of their methodology (including how voters were sampled and whether and how the results were weighted), they do disclose the two most important things you need to know about this particular poll: The order in which questions were asked and the fact that Democracy for America is a political action committee that has formally endorsed Bill Halter.
Why does question order matter? Take a look at the questions they asked in the order they presented in the DFA Release (emphasis added):
QUESTION: Who did you vote for in the Democratic Primary election today, Blanche Lincoln, Bill Halter, or D.C. Morrison?
QUESTION: What do you think is the most important problem facing this country today?
QUESTION: Generally speaking do you think Democrats in Washington, DC are fighting hard enough against corporate special interests like Wall Street and Insurance Companies?
QUESTION: Generally speaking, do you think Democrats in Washington, DC are fighting hard enough to challenge the Republican policies of the Bush years, aren't fighting hard enough to change those policies, or are fighting about right?
QUESTION: Is the issue of national health care reform very important, somewhat important, or not important when deciding how or if to vote?
QUESTION: Do you favor or oppose the health care reform bill passed by Congress this year?
QUESTION: If oppose, do you think it goes too far or doesn't go far enough?
QUESTION: Do you favor or oppose the national government offering everyone the opportunity to voluntarily buy into a system like Medicare that would compete with private health insurance plans?
QUESTION: Is the issue of the economy very important, somewhat important, or not important when deciding how or if to vote?
QUESTION: Who do you blame more for our nation's current economic problems: The government or Wall Street?
QUESTION: What would do more to improve our nation's economic conditions: Decreasing government spending OR tightening government regulation of Wall Street and corporate executives?
I'm going to read you several pairs of statements, please tell us which comes closest to your views:
QUESTION: Democrats' economic policy is more focused on helping Wall Street than helping Main Street, OR Democrats' economic policy is more focused on helping Main Street than helping Wall Street
QUESTION: Blanche Lincoln is more on my side than on the side of the lobbyists and special interests, OR Blanche Lincoln is more on the side of the lobbyists and special interests than on the side of people like me.
QUESTION: Bill Halter is more on my side than on the side of the lobbyists and special interests, OR Bill Halter is more on the side of the lobbyists and special interests than on the side of people like me.
QUESTION: If the Democratic primary for Senate results in a run-off election between Blanche Lincoln and Bill Halter on June 8, do you plan to vote?
QUESTION: If yes, do you plan to vote for Blanche Lincoln or Bill Halter?
I probably oversimplify when I say that pollsters fall into roughly three schools of thought in terms of when they ask the vote preference question in a pre-election survey interview. Most media pollsters typically ask about vote preference as the very first question, or at least only after any questions to screen for likely voters, on the theory that asking any other questions might bias vote preference. Most campaign pollsters typically ask a few very neutral, general items before the vote (such as candidate favorable ratings or the so-called "right direction-wrong track" question), on the theory that they only serve to get voters thinking about the candidates (something they will inevitably do before voting) but do not prime specific issues or themes. And then there's the third category: Pollsters who either fell asleep in questionnaire design class or (dialing back the snark) are willing or eager to risk creating bias in the vote preference question.
Now I exaggerate a little, and I am certainly going to pick on Research 2000 today, but I have seen a disturbing number of independent media polls ask vote preference questions deep into surveys following questions with the potential to prime preferences.
The question order used in the DFA Arkansas survey, however, is especially problematic. Consider that if you visit Bill Halter's web site, you will learn that he "supported a bill that would have allowed the public to buy into a system that would have also provided more competition and choice." And more important, if you have been watching any commercial television in Arkansas the last month or so, you have seen advertisements from the Halter campaign claiming that Halter believes that "Washington is broken" and "cares more about special interests than you," that Senator Lincoln has "gone Washington," that she voted "aye" on "bailing out Wall Street" and has taken "millions in campaign cash from Wall Street."
In other words, the questions that immediately precede vote preference get voters thinking at considerable length about precisely the themes emphasized by the Halter campaign in its advertising. And we know that Halter has a perceived advantage on these themes, at least among the 500 respondents to this survey, because of the results of the two questions that immediately precede the vote preference question: By a margin of 46% to 42%, the voters sampled say Lincoln is "more on the side of the lobbyists and special interests than on the side of people like me." They say Halter is on the side of "people like me" by a margin of 35% to 14% (with 51% unsure).
Academics have a word for what this sort of question order can do to vote choices. it's called "priming." In 1987, political scientists Shanto Iyengar and Donald Kinder published the book, News That Matters, that details a set of experiments that proved that "by priming certain aspects of national life while ignoring others," news programs could "[set] the terms by which political judgements are rendered and political choices made" (p.4). The same principles extend to survey questions.
I asked Del Ali, the president of Research 2000, to confirm and explain the order of the questions on this survey. His response:
It was asked in the order that DFA had asked us too. I think you have a good point, however, let me a few points that I think one has to consider:
The poll was pretty close to the actual primary vote, therefore, one could argue that primary voters who are committed to their candidate and have something at stake by the fact they showed up to vote are not going to jump ship on any exit polling questions that may be deemed as negative towards one candidate or the other. In fact, I felt the questions were fair for all the candidates in that we asked the same question about each of them when it came to lobbyists, Wall street, etc... I don't think there is anything that we asked that is close to Lincoln beats her husband and Halter is the next Father Flannagan
I know the criticism is that we should have asked the run-off question right after we asked whom did they voted for on Tuesday, and that is fair. However, I do not feel we moved any of the voters preference and am confident that this race is a true toss up as of today.
And to be fair, it is certainly possible that vote preferences in Arkansas are now so firmly rooted that the order of questions on this survey had no priming effect. More straightforward surveys by other organizations may yield similar results. We will see. For now, however, we have good reason to be skeptical about the results of this survey. If it turns out to be something of an outlier, you will certainly know why.