It's not surprising there are so many polling organizations these days. The 24 hour news cycle demands fresh and exciting information to share with the public. Polls provide much of this drug. Yet, consumers of poll data should hold a healthy skepticism about the popular interpretations of poll numbers presented on television, in papers and online blogs, and on the radio.
When it comes to following proper ethical and methodological guidelines, polling organizations are not all created equal.
Unfortunately, during the election season, you are likely to find high quality survey work replaced by expediency and bias. This happens regardless of the partisan or ideological leanings of the pollsters.
One of the main sources of error in interpreting poll results is poor question wording. In surveys questions are supposed to be accurate measures of one's attitudes, opinions, beliefs, and behaviors; just like the scales in bathrooms accurately measure one's weight. They should at least be well written and easily understood by respondents, and they should not be biased toward a particular viewpoint.
As a professor in Delaware, I've been following the polls targeting the now infamous U.S. Senate election and other national issues. Delaware does not have regular polling, but when it does happen, it's important to do it right.
Take for example the most recent Fox News poll conducted by Pulse Opinion Research on September 18th in the state of Delaware. The interactive voice response (IVR) survey -- where a computerized voice asks the questions -- contacted 1,000 likely voters in the state (margin of error + 3%).
See the Poll's Results here (PDF)
There are [at least] three questionable "questions" on the brief poll that caught my attention. The first asked respondents about their preferences regarding government.
"All in all, would you rather have bigger government that provides more services or smaller government that provides fewer services?"
Approximately 35% said, "bigger government" providing more services", 53% said "smaller government" with fewer services, and 12% were unsure.
First, it's unclear what services respondents would prefer. Perhaps Delaware residents would prefer only military protection, social security, and Medicare services; likely the three largest increasing budgetary expenditures in the federal budget. Second, it's unclear what is meant by "bigger" or "smaller" government. The question could be referring to spending, the number of agencies, red tape and bureaucracy, or the "out-reach" of government into the private sphere. In general, the question is ambiguous and should be rewritten to be more specific.
I'm not a fan of the spirit of the question, but If Pulse wants to stay consistent with this line of wording, I'd suggest, "would you prefer a [federal] government that provides fewer services or one that provides more services."
Another question on the survey is "double-barreled." It asks the following:
"Do you agree or disagree with the following statement: the federal government has gotten totally out of control and threatens our basic liberties unless we clear house and commit to drastic change."
What? Let's count how many assertions are in this one statement: 1) the government has gotten out of control, 2) the government threatens our basic liberties, 3) unless we clean house, and 4) commit to drastic change. These are really four separate questions that can be asked of respondents. To put them as a single assertion to agree or disagree with is problematic. I won't even address the loaded language -- "control and threaten" -- in the question.
And, it's not clear what aspects of government are out of control; is it the budget, the over-reach of government, not listening to the people, the wars, or what? I'd suggest simply dropping this item and returning to the drawing board.
A third question presents inaccurate and illogical information, and is poorly worded. It asks respondents:
"Thinking about the health care law that was passed earlier this year, would you favor repealing the new law to keep it from going into effect, or would you oppose repealing the new law?"
First, many parts of the health care legislation have already gone into effect, so it seems impossible to "keep [the law] from going into effect." Thus, I take the item to suggest that there are specific parts of the law that should be repealed before they go into effect and that everything that's already in effect is okay; however, if I (or respondents) have to make assumptions about the question's intent, then it's a bad question.
Second, there are actually two parts to the first part of the statement, one about repealing the new laws and another about keeping them from going into effect. This is the double barrel problem all over again. Finally, the wording of the question is imbalanced. The statement places "favor" with "repealing the new laws to keep it from going into effect" and trivially tosses in the "oppose" option at the end. Why not simply ask, "do you favor or oppose repealing the new health care law?" Even this wording is too general, but it's much better than the status quo.
Questions about the 2009 health care legislation are notoriously bad because they don't speak to any specifics. What aspect of the legislation should be repealed? Is it the part about pre-existing conditions, the individual mandate, or the part about adult children being able to stay on their parent's health insurance? The public should be skeptical of health care generalities because people may disagree with the narrative of politicized health care legislation, but not with the specifics of it.
Finally, the poll asks respondents about the state of the economy in Delaware.
"Generally speaking how would you rate the condition of Delaware's economy... excellent, good, not so good, or poor?"
This seems like a clear cut question, but the topline response categories presented by Pulse raise some addition concerns. The topline report shows 1% saying "excellent," 31% saying "good," 50% saying "no so good," 16% saying "fair," and 2% saying "not sure." Where the heck did "fair" and "not sure" come from? They are not a part of the question. And, why use two closely related categories like "not so good" and "poor?" What's the difference? If I'm asking, respondents are likely asking also.
Pulse, as well as other polling organizations, and ideologues should not interpret these critiques as indictments of their pollsters; rather this type of criticism speaks to the very future of public opinion polling as an industry. Public opinion researchers and survey methodologists are finding it hard enough to do high quality work in a changing social world of cell phones and high geographic mobility.
The errors mentioned above are not difficult to resolve; but if the same wording comes out from the same polling organizations again, it would speak volumes about the agenda of the pollster or their client. The bottom line is that survey and polling researchers need to stay focused on good measurement, lest the polling industry become as politicized as many in the public currently view the media.
Follow David C. Wilson on Twitter: www.twitter.com/dcwilsonphd
Just post an article like this every week, taking apart a "recent" poll.
I hope we keep seing articles on how these polls are used to shape and manipulate opinion, not report opinion.
I hang up.
I will not answer questions that aren't really questions.
The questions are insane, and as David Wilson has pointed are full of "push type" assumptions.
I don't believe the polls, none of them, and I won't participate in them, none of them.
I took a course in correct polling as a Political Science minor, and believe me, many, many of the current polling practices are NOT going by the rules of good poll writing.
Thanks again,
DK
I understand it is not part of the topic of discussion, but have you ever run across a "poll" whose questions were so egregiously biased that you could not just classify it as erroneous, but had to take a more forceful stand against it?
And this thought occurred to me as well: if we were to assume that the Fox News Polls were all intentionally biased, and therefore their surface results tainted, could we not still glean valuable information from them, from the manner in and direction toward which they obfuscate?
Normal people don't have a response - so they don't poll.
Ignore the surveys. They've become another tool in the Rovesque handbook.
The only poll that will count is NOV 2nd.
Get out and VOTE.
I feel sorry for the large number of people who only get their "news" from FOX. They are being conditioned and lied to by people without morals or concience.
This news brought to you tonight by the Koch brothers.
I understand your point and it is almost valid. You can take a sample of my blood to tell if I am anemic because all of the blood in my body has been filtered and adjusted by my body to maintain homeostasis. But the thoughts of the person living at 2121 Indiana is not the same as the thoughts of the person living at 2123 Indiana. People are individuals and no sampling ever tells the truth, it can't.
Statistically speaking, no poll takes into effect outliers. "A frequent cause of outliers is a mixture of two distributions, which may be two distinct sub-populations, or may indicate 'correct trial' versus 'measurement error'; this is modeled by a mixture model." That means it can only be accurate for a certain demographic and the people who pay for polls pay to get information from a certain demographic. That invalidates the poll if you are claiming it represents the entire population. I can not call your synopsis wrong, I just have no reason to believe it. As I said, I have never been polled, I don't know anyone that has. Therefore I have no real reason to believe a poll ever happened. I can't see it, I can't verify it and not knowing the original questions, I cannot even verify the results. Statistics are designed to get to a predetermined point, they are by their very nature slanted. Polls are statistical values and are not valid for anything except entertainment. There are only about 6 billion people on this earth but statistics will tell you that you can have a 1 in 500 billion chance of being the father of a certain baby. That is an invalid result set.