Survey response rates have hit all-time lows, yet telephone pollsters argue their samples can still be representative and accurate. Online pollsters, whose methods don't include starting by selecting respondents at random, argue their methods produce reliable results.
A panel of experts convened at survey conference in Baltimore on Tuesday tackled these issues and more in a wide ranging discussion of the future of election polling that included an exchange at the heart of the debate over what makes a poll "scientific."
It was inspired by a NYT op-ed column by Rutgers political scientist Cliff Zukin wrote in June that detailed the "near crisis" now facing election polls, which has made "high-quality research much more expensive." That trend, Zukin argued, has "opened the door for less scientifically based, less well-tested techniques."
In the op-ed, Zukin chronicled the fall of survey response rates from the 80 percent levels considered acceptable when he "first started doing telephone surveys in New Jersey in the late 1970s," to the 9 percent response rate reported by the Pew Research Center in their most recent study of the phenomenon in 2012.
Yet "strangely, for some reason that no one really understands," Zukin wrote, "well-done probability samples seem to have retained their representative character despite the meager response rate."
In Baltimore, he expanded on that issue and posed three questions:
One, why does probability sampling continue to work at 10 percent?...If you’d said to any of us 50 years ago that your response rate was going to be 10 percent…I don’t know how you’d survive. I’d still don’t see why, and I’d like to know the answer to that.
Zukin's second and third questions deal with what pollsters call "non-probability sampling" -- the method used to conduct most internet surveys -- which doesn't begin with a random sample but rather uses large pools of respondents who have volunteered to complete online surveys.
Second...why should the non-probability samples work, theoretically?....
And third, and this is the one where empiricism needs theory, [it] goes very simply, under what conditions will a non-probability sample behave more and less like a probability sample? Truth is I often surprised they’re as close as they are.
In response, Andrew Gelman, a professor of political science and statistics at Columbia, gets to the heart of the debate of the future of polling:
The surveys we work with are not probability samples. We do not know the probability of people being included in the sample. We don’t know it. To the answer to why does probability sample work is I don’t know because I’ve never seen a probability sample.
I think what Cliff means is, why do surveys work that have probability sampling followed by non-response, which cannot be modeled probabilistically. And…those surveys work to the extent that you believe that selection bias is small. What we do in surveys and analysis is we correct for known differences between the sampled population....
The only part [of what Cliff said] I disagree with is his distinction between probability sampling and non-probability sampling, because all of our samples are non-probability. What it’s really distinguishing is the very first stage of the sampling. Are you picking phone numbers out of a list, or are you having people opt in. That’s the issue. I’d rather have people out of a list than opt-in, but it’s not so central. And from the perspective of statistical theory, it’s not really important at all, because once we have 90 percent non-response rates we’re out of the probability sampling range.
ONLINE SURVEYS CAPTURE MOST BUT NOT ALL - Pew Research: "Drawn both by lower data collection costs and substantial advantages in how questionnaires can be designed and administered, survey research is rapidly moving to the Web. Adoption of the Web for data collection is occurring in all sectors of the industry from the U.S. Census Bureau, which now allows respondents to take the American Community Survey on the Web, to political polling, where the cost and difficulty of targeting phone calls geographically is prompting more organizations to rely on online polls, to the marketing research community, which has moved virtually all consumer surveys to the Web. One key factor that has made this shift possible is the fact that the vast majority of Americans now use the internet. Pew Research Center telephone surveys have documented the rise in internet adoption, which has grown from 14% of the U.S. adult population in 1996 to 89% today. But 89% is not 100%, and surveys that include only those who use the internet (and are willing to take surveys online) run the risk of producing biased results….One area of generally positive news for Web surveys is that estimates of political preferences and engagement are relatively unaffected by internet coverage bias, even among subgroups of the population with relatively lower levels of internet use. On a 10-item index of ideological consistency, there was almost no difference between the Web and total samples across age groups. Similarly, the pattern of reported voter registration by educational attainment was very similar in the online and total samples." [Pew Research]
REPUBLICAN VOTERS ARE DEEPLY UNHAPPY WITH THEIR PARTY ESTABLISHMENT - Dana Blanton: "Most Republicans feel betrayed by their party -- and show their displeasure by supporting outsiders over establishment candidates in the GOP presidential race….The appeal of outsiders comes from significant dissatisfaction with the party establishment: 62 percent of Republican primary voters feel 'betrayed' by politicians in their party, and another 66 percent say the recent Republican majorities in Washington have failed to do all they could to block or reverse President Obama’s agenda. For comparison, 40 percent of Democratic primary voters feel betrayed by their party." [Fox]
TRUMP'S LEAD SHOWS SIGNS OF STALLING - David Lauter: "There's almost nothing Donald Trump likes to talk about more than his steady rise in polls. He may have to find a new topic. A series of polls conducted since last week's Republican debate at the Reagan Library continue to show Trump leading the pack. But they also show a plateau, or perhaps a small decline. The evidence for at least a plateau is strong. Polls by Fox News, Bloomberg, and CNN, all released in the last several days, have shown Trump stuck in place with about one-quarter of the GOP vote. A fourth poll, by Quinnipiac University, showed a small decline, from 28% in August to 25% now." [LA Times]
HOW MUCH WILL THE PARTY DECIDE IN THIS YEAR'S GOP PRIMARY? - Andrew Prokop: "The idea, put forth in a 2008 book of that name by political scientists Marty Cohen, David Karol, Hans Noel, and John Zaller, is that modern presidential primaries aren't driven by the whims of voters. Instead, the most important players are party insiders, who use a variety of tools to try and deliver the nomination to the candidate they prefer — and usually get their way….So far, the GOP establishment appears paralyzed and uncertain, with Republican elected officials issuing endorsements at their slowest pace in decades. As of mid-September, the party hadn't decided to throw its weight behind anyone. But is this because elites just haven't made up their minds yet — or because voters are refusing to let them?...It turns out that in most of the cases in which the party does decide, there was one candidate who clearly had a higher standing than the others from the start. That, The Party Decides co-author and University of Maryland professor David Karol told me, is his main 'skepticism' about his own theory. 'In the cases where the party has coordinated on a candidate,' he says, 'it's almost always been somebody fairly obvious.'" [Vox]
THIS WEEK'S POLLS
-Nearly half of America sees the federal government as an imminent threat. [Gallup]
-Americans like the Pope better than they like the Catholic Church. [WashPost]
-Americans feel favorable towards the Pope, but doubtful that he'll be able to influence U.S. policy. [YouGov]
-Republicans aren't feeling too hot about the Holy See's climate change stance. [Bloomberg]
-Catholic Democrats, Catholic Republicans, and Catholic Latinos all really like the Pontifex. [Roll Call]
-Fifty-eight percent of Americans say race relations are worse today than one year ago. [PBS]
-The debate improved Americans' opinions of Carly Fiorina, Marco Rubio and Ben Carson the most. [HuffPost]
-George W. Bush's legacy is anything but toxic to Republican voters. [HuffPost]
-Republicans say that today's GOPers are less conservative than Ronald Reagan. [HuffPost]
-A quarter of Democrats would vote for Joe Biden if he jumped into the Presidential race. [Bloomberg]
-Political outsiders may be popular right now, but old-timers Jeb Bush and Hillary Clinton seem the most presidential to Americans. [Bloomberg]
-Quinnipiac finds Donald Trump's poll numbers steady as Carly Fiorina, Ben Carson and Jeb Bush getting a boost. [Quinnipiac]
-Another poll shows Bernie Sanders making headway against Hillary Clinton in New Hampshire. [NYT, University of New Hampshire]
-Hillary Clinton seems to be in a good position in Iowa against Bernie Sanders. [PPP]
-Twenty years later, black and white Americans seem to finally agree that O.J. Simpson is guilty. [WashPost]
HUFFPOLLSTER VIA EMAIL! - You can receive this weekly update every Friday morning via email! Just click here. Enter your email address, and click "sign up." That's all there is to it (and you can unsubscribe anytime).
THIS WEEK'S 'OUTLIERS' - Links to the best of news at the intersection of polling, politics and political data:
-Sasha Issenberg weighs in on campaign journalists' and political scientists' clashing views of Donald Trump. [Bloomberg]
-A bill in California could register millions of new voters, but it's not clear how much turnout will be affected. [HuffPost]
-A poll that finds Hillary Clinton's lead both growing and shrinking demonstrates the futility of overanalyzing early horserace numbers. [HuffPost]
-Most people don't want to abolish the penny. [Harris]