Does it matter that many polls -- including the vast majority that we are currently watching at the state and congressional district level -- do not call Americans who use only a cell phone and thus lack landline telephone service? Yes it does. It creates a growing bias that appears to benefit Republican candidates. That's the message of a new analysis released this afternoon by the Pew Research Center.
Since 2006, a rapidly increasing percentage of American households lacks landline phone service. The most recent government estimates find that one in four American households is reachable by cell phone only. Pollsters have been reluctant to sample and call Americans on their cell phones, partly because it costs more and partly because federal law requires hand dialing any call placed to a cell phone, which makes such calls less efficient and puts cell phone polling off limits to automated survey methodologies.
For the last four years, the Pew Research Center has conducted public opinion surveys involving separate, parallel samples of both landline and mobile phones. Their design allows for a comparison between combined samples of landline and cell interviews and samples based only on landline calls.
Before the 2008 election, they found that calling only landline phones introduced a "small but real" bias in favor of John McCain, an average bias of 2.3 percentage points on the margin on nine national surveys conducted between June and October of that year.
This year, according to today's report, the Pew Center finds that sampling only landline phones creates an even bigger bias -- "differences of four to six points on the margin" - in favor of the Republicans. The most recent survey in the study, conducted in late August and early September, also involved comparisons based on a subgroup of "likely voters" chosen using a traditional seven question turnout scale (similar to the classic Gallup likely voter model):
The combined landline and cell estimate produced a seven-point Republican advantage: 50% supported the GOP candidate for Congress in their district while 43% backed the Democratic candidate. The Republican lead would have been 12 points if only the landline sample had been interviewed, a significant difference from the combined sample of five points in the margin.
The impact such a bias may have on this year's pre-election polls depends in part on the polls involved. At the national level, many organizations now routinely sample and call both landline and mobile phones. These include, in addition to the Pew Center, ABC News/Washington Post, AP/GfK, CBS News/New York Times, Gallup (both their daily tracking and the surveys in partnership with USA Today), Kaiser Family Foundation, McClatchy/Marist University, NBC News/Wall Street Journal and Newsweek.
At the statewide level, however, more expensive cell phone interviewing is far more rare. Except for a single experiment conducted by SurveyUSA this summer (involving live interview calls to cell phones) we have not seen any cell phone sampling or calling by the pollsters that use an automated, recorded voice methodology. The organizations we know of that are currently calling samples of both cell and mobile phones include California's Field Poll and Public Policy Institute of California, the University of Cincinnati Ohio poll, and the Marist Poll's statewide surveys. While the Quinnipiac University announced plans to begin calling cell phone samples earlier this year, polling director Doug Schwartz tells the Huffington Post that they "decided to suspend cell phone calling, which is much less efficient than calling landlines," until after the election.
It is also worth noting that while the Pew Center found a modest bias for landline-only samples in their survey experiments in 2008, their post-election analysis found that other national surveys that called on both cell and landline phones were no more accurate than national surveys that called only landline. Moreover, at the statewide level where virtually all polls were landline only, the errors were "still relatively small" and favored Republicans more often than Democrats.
Related: see the archive of reporting on cell phones and surveys from Pollster.com.