04/21/2009 03:35 pm ET Updated May 25, 2011

Pew's Practical Issues in Cell Phone Polling

I attended a presentation last week at the Pew Research Center (sponsored by the DC AAPOR chapter) on some of the practical issues they have encountered in their innovative work on cell phone polling. I'm still catching up from a few hectic days that have followed but want to pass along a few interesting details they shared.

Most of what was new in the session will be of more interest to pollsters than to political junkies wondering about how pollsters are dealing with the growing number of Americans without landline phone service. Fortunately, for those of you in the latter category, the PRC shared most of their more general data obtained from calling cell phones during the 2008 campaign in a report released this past December (see their summary, full report pdf and our review).

Here are a few highlights that seemed especially noteworthy or new in the presentation by Pew's survey research director Scott Keeter, associate director Michael Dimock and research associate Leah Christian:

  • Pew has now conducted 18 surveys (14 in 2008 and and 4 in 2009) featured a "dual frame" sample of both landline and mobile phones. In those surveys, they have interviewed approximately 9,400 adults via cell phone. Keeter explained that these dual frame samples are now "standard policy" for Pew's political surveys.
  • They have found cell phone users just as willing to answer their phones and cooperate as landline users. Their response rate for cell phones (23%) is virtually the same as that for landlines (24%)
  • Pew's calling center is finding it "cheaper and easier" to interview by cell phone. Just a few months ago, Keeter reported that it cost Pew "two to two and half times" as much for cell phone interviews as landline. Now the cell phone costs are "closer to two times as expensive" as landline interviews. Keeter attributed the improvement to their call centers "getting more familiar with the tricks of doing successful cell phone interviewing." Note that Pew interviews all cell phone users, and weights down respondents less likely to be "cell phone only." If they had screened for just the cell phone only users, the differential would be closer to four times the cost of landline interviews.
  • Pew continues to offer a $10 per interview incentive to cell phone respondents although, according to Keeter, other pollsters such as Gallup do not pay incentives to the cell phone respondents.
  • The biggest continuing methodological challenge to pollsters is determining how to combine and weight their landline and cell phone samples, partly because of what some are now calling the "cell phone mostly" problem. This issues involves those who have both mobile and landline phones but make "all" or "almost all" of their calls on the mobile phone (see pp. 6-8 from the December report, and my previous discussion of the issue for more details).

In this presentation, Dimock presented results showing that the cell phone mostly respondents were less likely to say they could be reached "right now" on a landline phone (52%) than those who use their cell phones for only some or few calls (63%). "This is really a spectrum," Dimock explained, without "a clear cut line between the group of people who absolutely can't be reached on a landline and another group who absolutely can't," but rather "a probability across the spectrum of people." As such, Pew's approach is to interview everyone reached via cell phone and weight on the inverse of their probability of selection.

  • Christian presented data on the problems identifying the actual geography of respondents based on their telephone number. Before the widespread adoption of cell phones, telephone area codes could identify the state and time zone of each number with great accuracy, as landline telephone numbers are closely associated with geography. Cell phone numbers, on the other hand, are assigned based on the geography where you first purchased your cell phone. The relatively new ability to "transport" cell phone numbers from one carrier is creating a growing mismatch between phone numbers and geography.

Pew has been able to confirm and quantify that trend by asking respondents to provide their zip codes. That additional data shows that a 5% mismatch for their cell phone samples on region (presumably census region), a 9% mismatch on state and a 39% mismatch on county. This discrepancy has two practical implications: First, obviously, when interviewing via cell phone, pollsters cannot treat phone numbers as an accurate measure of geography, especially at the county level. Second, they have to be careful about scheduling call time based on area code. A few years ago, pollsters could safely dial west coast area codes until 9:00 p.m. Pacific time. Now it is all to easy to ring someone much later at night, so pollsters have had to modify their procedures.

A hat tip and thanks to Susanna Fox of the Pew Internet Project and our friend Alex Lundry, for their helpful notes posted notes during the session on Twitter