Today's Guest Pollster article comes from David W. Moore, a senior fellow with the Carsey Institute at the University of New Hampshire. He is a former vice president and senior editor with the Gallup Poll, where he worked for 13 years, and is the founder and former director of the UNH Survey Center. He manages the blogsite, Skeptical Pollster.com.
This month, the American Association for Public Opinion Research (AAPOR) gave its most coveted honor, the AAPOR Lifetime Achievement Award, to Kathleen A. Frankovic, director the CBS News polls and a former AAPOR president. In her acceptance speech, she referred to her presidential address of a decade and a half ago, when she leveled several incisive criticisms at the media polls - criticisms that deserve to be re-examined today.
Since joining CBS News over thirty years ago, Frankovic has amassed an impressive set of accomplishments, including being president of both AAPOR and its sister organization, the World Association for Public Opinion Research (WAPOR); a member of the Market Research Council; a trustee both of The Roper Center for Public Opinion Research and, separately, the National Council on Public Polls; and a former chair of the Research Industry Coalition. She is also the author of numerous published articles and book chapters on public opinion. If anyone could be considered a pillar of the polling industry establishment, she would be it.
Yet, her writings on polls have not been effusive encomiums to the presumed benefits they bring to society. While always attributing much importance to the role of media polls in American politics, she has also expressed concerns about them, nowhere more evident than in her 1993 AAPOR presidential address, "Noise and Clamor: The Unintended Consequences of Success."
Her theme is reflected in the title, as she raised questions about the increased frequency of polls and the lack of thought that goes into many poll questions - "Immediate response is more important than what the response is or what it really means. In other words, we may no longer have to think." She also worried about the decreased value and import of polls - "It's so easy to conduct polls now that it may actually cheapen the value of each one we do. Instead of meaning, we may just be getting noise - noise and clamor."
She noted that with the advent of scientific polls, we now have a "continuous ballot box," the dream of early democratic idealists. But is that good? Not always, apparently. In the several months prior to her presidential address, polls taken at two-week intervals had showed President Bill Clinton's approval rating bouncing all over the map, "from 58 percent to 53 percent to 59 percent to 53 percent to 57 percent to 49 percent to 57 percent to 45 percent." She lamented, "This is information, but how informative is it? It's almost like what Truman Capote once remarked about Jack Kerouac's novel, On The Road: 'That isn't writing-it's typing.' Continuing ballot boxes shouldn't bounce around so much." Indeed.
Today, the uninformative information provided by polls is even more acute, obvious to anyone who has followed the pollsters' fascination with the 2008 national Democratic electorate. It isn't the results at two-week intervals, but contemporaneous results that bounce all over the place these days. One has to look only at pollster.com from April 30 to May 4 to find five polls with three different results: Gallup by itself, reporting a dead heat (Obama up by two points); Gallup with USA Today and, separately, AP/Ipsos each reporting Clinton leading by 7 points; while CBS/NYT and Diageo/Hotline each reporting double digit leads (12 and 11 points respectively) for Obama. And this isn't the only time Gallup has contradicted itself this campaign season, or that different polling organizations have come up with contrasting results when interviewing in the same time period. (See comments by ABC's Gary Langer, Dec. 12, 2007 and Feb. 26, 2008; and Mark Blumenthal's "Dueling Gallups.")
Despite her criticisms, Frankovic proposed no remedies, nor special panels to investigate the problems, perhaps in recognition that they might entail a fundamental change in the way that polls are currently conducted. In the wake of the miscalls in the New Hampshire Democratic Primary, AAPOR's president, Nancy Mathieowetz, did in fact establish a special panel "to examine what occurred, provide a timely report of our findings, and promote future research on pre-election primary polls." No such panel has been established to examine all the subsequent conflicting polls, though the New Hampshire panel might want to consider broadening its scope. The "continuing ballot boxes" are not just bouncing around, they're running into each other going in opposite directions.