Few in the field of survey research have examined the problem of cell phones and surveys as closely over the last four years as Scott Keeter and his colleagues at the Pew Research Center. They conducted one of the first large-scale political surveys by cell phone two years ago and have been leaders in testing and developing new methods to conduct political surveys by combining samples of landline telephones and cell phone.
Today they released a new must-read report summarizing findings from "three major election surveys [conducted] with both cell phone and landline samples since the conclusion of the primaries." The verdict? "Pew's surveys this year suggest at least the possibility of a small bias in landline surveys."
The key details (emphasis added):
In each of the surveys, there were only small, and not statistically significant, differences between presidential horserace estimates based on the combined interviews and estimates based on the landline surveys only. Yet a virtually identical pattern is seen across all three surveys: In each case, including cell phone interviews resulted in slightly more support for Obama and slightly less for McCain, a consistent difference of two-to-three points in the margin.
Pollsters have long understood that the cell phone only population -- those who have cell phone but no landline telephone service -- tend to be younger, and that the growth of that population has made it more difficult to reach 18-29-year olds. However, the conventional wisdom among pollsters has held that weighting by age could mostly alleviate any potential bias, as they did they did in 2004.
The new Pew report shows why weighting by age may not have the same effect now:
Traditional landline surveys are typically weighted to compensate for age and other demographic differences, but the process depends on the assumption that the people reached over landlines are similar politically to their cell-only counterparts. These surveys suggest that this assumption is increasingly questionable, particularly among younger people. [...]
In the pooled [August-September] data, cell-only young people are considerably less likely than young people reached by landline to identify with or lean to the Republican Party, and even less likely to say they support John McCain. Among landline respondents under age 30, there is an 18-point gap in party identification - 54% identify or lean Democratic while 36% are Republican. Among the cell-only respondents under age 30, there is a 34-point gap - 62% are Democrats, 28% Republican. The difference among registered voters on the horserace is similar: 39% of registered voters under 30 reached by landline favor McCain, compared with just 27% of cell-only respondents. Obama is backed by 52% of landline respondents under 30, compared with 62% of the cell-only.
The roughly two-to-three point difference in the margin favoring Obama is, as it happens, very close to the effect Nate Silver obtained over the weekend by comparing results from pollsters that have been interviewing by cell phone (including Pew) with a control group that has not.
Finally, the Pew Report has a bit of a bonus with implications for the ongoing debate over how to model "likely voters" this year:
While 18-29-year-olds reached by cell phone tend to have less experience voting than their landline counterparts, they are just as interested in the 2008 campaign, and express just as much intention to vote this year.
For all the details, go read the whole thing.