My NationalJournal.com column, on those wildly variant automated polls in North Carolina from Public Policy Polling (PPP), is now online.
A few additional pieces of the story: First, I get a lot of email asking about the firms like PPP. Who are they? Who pays for the polls? PPP was founded by a North Carolina businessman named Dean Debnam, and its clients are mostly Democratic candidates holding or seeking local office in North Carolina. According to Tom Jensen, PPP's communications director, Debnam founded the company to help provide "low cost, high quality polling" to candidates for local offices who "could never afford a $12,000 poll."
PPP is a good example of a growing trend that the automated (or interactive voice response - IVR) technology makes possible. It is easier than ever for organizations with little prior experience in survey research to make calls, ask questions, tabulate the results and disseminate them via the Internet. Where my pollster colleagues disagree -- often vehemently -- is whether the new firms like PPP are delivering the "high quality" polling they promise. For example, one campaign pollster friend I talked to this week said he had a "hearty laugh" about the change in the sample selection methodology I describe in the column because, "I doubt seriously that they had one in the first place."
I should say that Jensen has been very responsive on behalf of PPP and as transparent about their methods as any pollster we have dealt with. On the other hand, PPP made no reference to their changed sample selection in their most recent releases (here and here, though they did note the change in a separate blog entry). They also neglected to extract the relevant vote history data from the sample that would have allowed a simple tabulation of the results from the latest survey using the older, narrower universe of past primary voters. Those are the kinds of mistakes that fuel skepticism among experienced pollsters.
Professor Franklin and I share an attitude about these sorts of surveys that sometimes puts us at odds with many of our colleagues in survey research. We believe we ought to judge all surveys by their performance rather than simply dismissing them by their methodology alone. Skepticism is certainly appropriate for newcomers using relatively unproven methods, but we will continue to track and follow the results from companies like PPP in order to evaluate their ultimate success or failure in achieving their stated goals.