Over the last few weeks, I had the chance to attend two different events that covered similar ground: a day-long conference on survey quality organized by the Harvard University Program on Survey Research and a private (but on-the-record) meeting of a number of media pollsters last week hosted by Stan Greenberg's Democracy Corps. Thanks to those events, as well as the recent AAPOR report on the problems of the New Hampshire and other primary polls in 2008, my notebook is full of new ideas and material for future posts. Throw in the upcoming AAPOR conference that is now just not quite two weeks away, and my inbox is overflowing. The sheer volume of has been a little overwhelming.
So today I want to try to review a few themes that have been constant across these various events.
What is Quality? One obvious theme is the way the survey research profession is struggling for consensus on the meaning of quality in an environment where new technologies are reshaping everything. New technologies are limiting our ability to reach respondents as we used to, but they are also producing new methods. The growth of cell-phone only households, the reality of response rates below thirty percent, the increased use of listed samples of registered voters and the proliferation of automated telephone polls and online surveys drawn from volunteer panels have many polling practitioners asking each other hard questions.
It is true, as former CBS polling director Kathy Frankovic points out, that the questions raised by technical advances are not new. Another "proliferation" of newer polls using new technology that "makes polling easier" occurred in the 1970s, she reports, when telephones replaced in-person interviewing, and again in the 1980s and 1990s, when personal computers replaced mainframes making data analysis cheaper and easer. Still, that knowledge comes as little help.
One of the striking things about the Harvard conference was the occasional disconnect between the presentations of the most respected academic survey scientists, often concentrating on advanced theoretical constructs of the measurement of survey error, and the questions that often followed from pollster and researchers in the audience: What practical things should we be doing now? What are the "best practices" we should follow? What characteristics should journalists and journal editors look for in a quality survey?
Twenty years ago, the speakers at this sort of conference would have been much closer to consensus on a set of best practices for telephone surveys on politics. Now such questions are much more difficult to answer. As one speaker at the Harvard conference put it, we are at "this odd moment in history where there are conceptual tensions in the field."
Standards and the "Margin of Error" - For Gary Langer, the ABC News director of surveys, "the solution" to the question of what makes for good data is firmly rooted in the standards they require to put polls on the air at ABC News. Here is an excerpt from the description of those standards on the ABC News web site:
[I]n all or nearly all cases we require a probability sample, with high levels of coverage of a credible sampling frame. Self-selected or so-called "convenience" samples, including internet, e-mail, "blast fax," call-in, street intercept, and non-probability mail-in samples do not meet our standards for validity and reliability, and we recommend against reporting them.
We do accept some probability-sample surveys that do not meet our own methodological standards – in terms of within-household respondent selection, for example – but may recommend cautious use of such data, with qualifying language. We recommend against reporting others, such as pre-recorded autodialed surveys, even when a random-digit dialed telephone sample is employed.
At the Harvard conference, Langer spoke out in particular against surveys based on opt-in Internet panels that claim a margin of error for their results, and followed up with a column in the same vein last week. "A probability sample is a hallmark of good data," he wrote, "a sign that it lives within the framework of inferential statistics." Opt-in internet surveys that claim a margin of error, on the other hand, amount to "samples taken outside that framework [that] try to nose their way out of the yard and into the house. They don't belong there."
[Interests disclosed: The owner of Pollster.com is YouGov/Polimetrix, a company that conducts opt-in internet surveys and was among those singled out by Langer for criticism at the Harvard conference. That said, I have issued similar criticisms of the reporting of a margin of error for internet surveys].
Stanford Professor Jon Krosnick, another speaker at the Harvard Conference, took a slightly different tack. He presented a compilation of efforts to measure and quantify survey accuracy using seven different methods across a wide variety of different surveys. Krosnick explained that his intent was to provide "a supplement to the margin of error... our only statement of accuracy" in most surveys as a way to account for the various "other" forms of variation not accounted for by random sampling error.
What makes Krosnick's approach different is his willingness to use measurements of accuracy to evaluate new forms of surveys, including opt-in panels and automated telephone surveys.
Assessing accuracy is useful for comparing various methods and to identify more accurate ones. So if you want to know, well, does representative sampling help you or not? Is representative sampling going to lead to more accurate data?
Krosnick examined seven methods to assess accuracy, although only a few tried to compare results from traditional telephone methods to those based on opt-in Internet panel. On some methods he found bigger errors for opt-in surveys, on others he found much closer results. Still, this was as much a demonstration of a method of comparison than an attempt at a comprehensive review.
Nevertheless, these approaches of Langer and Krosnick may be on something of a collision course. Langer's philosophy, shared by many others in the field, has been consistent: "a good sample is determined not by what comes out of a survey but what goes into it." This philosophy warns against efforts to measure individual poll accuracy since it may be no guarantee of quality. Krosnick's approach -- in theory, at least-- is to try to quantify total error for all surveys, including those gathered using opt-in internet panels. So what happens if non-probability sampling can produce data that is effectively as accurate as surveys that reach out to random samples? At that point will we need to reassess the standards?
Disclosure. A constant theme across all of the meetings and presentations is the need for more and better disclosure for those who put survey data into the public domain. At the Harvard Conference, Phil Meyer, a long-time journalist, survey researcher (author of the highly regarded text book, Precision Journalism) and now professor emeritus in journalism at the University of North Carolina, gave a talk on "the rise and fall of truth in polling." He shared the frustration expressed by so many others regarding the lack of cooperation by many pollsters with AAPOR's recent investigation of New Hampshire and other 2008 primary polls.
What I found intriguing, however, was his hope that the "marketplace" might react by creating a "reputation-based hierarchy," that would reward pollsters that disclose and punish those that withhold. "The real accountability system," he said, "is the market -- we need an efficient market, a public that cares." Meyer also noted that the mass media have tried to create that market with mixed results. Now he looks to specialized media (blogs like this one) to become a forum where "specialists in polling might become the arbiters of standards," a development that would amount to journalism healing itself. He closed with the image of the lighthouse used in the logo of the Scripps-Howard news chain along with their corporate motto: "Give light and the people will find their own way."
The resistance that many pollsters showed to repeated requests from AAPOR's investigation on the New Hampshire polling problem demonstrates the limits of formal enforcement mechanisms of professional organizations like AAPOR. As Frankovic put it, the discussion around standards and disclosure rarely extends beyond the "professional polling community."
I believe that sites like Pollster, RealClearPolitics, and FiveThirtyEight have a real opportunity to broaden the dialogue and help create the reputation based hierarchy that Meyer talked about. But that is a much bigger topic. I will have more to say about it in the coming months. Stay tuned.
(A hat-tip and thanks to Mike Mokrzycki for his excellent Twittered notes on the Harvard conference that helped fill in a few blank spots in my own).