Over the weekend, our friend Chuck Todd shared this bit of poll skepticism via Twitter:
Why I'm always skeptical of Rasmussen robo-polls: it's results like this: NO WAY this guy Bob Krause has name I.D. of 63% in Iowa SEN.
I know little about Bob Krause except that he has not held public office since serving in the Iowa state representative in the 1970s and running unsuccessfully for Iowa Treasurer in 1978. As such, handicappers assume he begins his campaign with little true name recognition statewide. So how could Rasmussen show 63% of Iowa likely voters able to rate him?
Well, they start by asking a favorable rating question that offers no neutral category and that lacks an explicit prompt to say that the name is unknown or unfamiliar. Here is the text:
I'm going to read you a short list of people in the News. For each, please let me know if you have a very favorable, somewhat favorable, somewhat unfavorable, or very unfavorable impression.
According to Scott Rasmussen, their script then reads each name, with a prompts that "If 'very favorable, press 1; if somewhat favorable, press 2" and so on. It concludes with, "If you are not sure, press 5."
Next, they ask favorable ratings after asking the vote preference question, Rasmussen's standard practice. Doing so tells respondents two things about Krause they probably didn't know: He is a Democratic candidate for Senate and the presumed opponent of Republican Senator Charles Grassley in 2010.
Now combine all of that with the operating theory that survey methodologists use to explain how inattentive respondents answer pollsters' questions: They "satisfice," which is shorthand to say that they work just hard enough to provide a satisfactory answer. So suppose you're responding to the poll, you really not sure who Bob Krause is, but the pollster (a) does tell you that "never heard of him" is an acceptable answer and (b) has already told you that Krause is the Democratic candidate for Senate. Under those circumstances, you might say "somewhat" favorable or unfavorable (reasoning, "well, I've never heard of him, but he's a Democrat, so I'll say I'm just somewhat [favorable or unfavorable]"). That answer is especially likely if you're in a hurry and not in a mood to wait for the automated system to additional answers after those listed in the introduction of the question.
Look at the the Rasmussen results, and you see a pattern consistent with that theory: Only 13% express a "strongly favorable" (5%) or "strongly unfavorable" (8%) opinion of Krause, and I'd wager that most of those were strong partisans reacting mostly to Krause's Democratic affiliation. Most of the rest with an opinion said it was either "somewhat favorable" (28%) or "somewhat unfavorable" (22%). And of course, more than a third (37%) did hang on long enough to choose "not sure."
Now, is it possible that aside from the way respondents answer questions, automated polls attract a different kind of respondent that is better informed about politics? Yes, that's possible, but not nearly proven by this example. Most of what makes this result seem so improbable to Chuck Todd and the other political pros that he talks to can be explained by the combination of question text and order. I share Chuck's skepticism with Rasmussen's favorable ratings, but necessarily of their underlying sample or their vote preference results.
P.S. A related issue: We discovered over the last few days that what we believed to be a weekly update of Barack Obama's favorable rating on RasmussenReports.com had been changed at some point to a weekly job approval rating, even though the labels still read "favorable" and "unfavorable" rather than "approve" and "disapprove" up until about a week ago. As such, we have removed the non-favorable ratings from our national Barack Obama favorable rating chart.
How will Donald Trump’s first 100 days impact YOU? Subscribe, choose the community that you most identify with or want to learn more about and we’ll send you the news that matters most once a week throughout Trump’s first 100 days in office. Learn more