Later this week, I'll be in Chicago for the annual conference of the American Association for Public Opinion Research (AAPOR). One of the more newsworthy aspects of this year's conference is the "Transparency Initiative" of AAPOR's current president, Peter Miller.
Until this week, the initiative has been mostly an idea, born out of AAPOR's recently higher profile in publicizing "the failure of survey organizations to be open about their research methods," as Miller put it last fall. Regular readers may recall AAPOR's work to investigate the polling failures in New Hampshire and elsewhere during the 2008 presidential primaries, and its recent public censure of two non-members for failing to disclose basic facts about their methodologies: Dr. Gilbert Burnham, regarding research he published on civilian deaths in Iraq, and Strategic Vision, LLC, regarding pre-election polling data they released in 2008.
Last fall, Miller concluded that what AAPOR's efforts to date have been inadequate. "Despite decades of work," he wrote "transparency in public opinion and survey research remains an elusive goal." The investigation of the polls in New Hampshire in 2008 was a focal point:
AAPOR's Ad Hoc Committee that studied pre-primary polls in the winter and spring of 2008 intended to release its report in time for our annual meeting in May of that year. The members of the committee hoped their findings would inform polling practice in the general election. Instead, the committee issued its report in April 2009, about a year late, because many organizations that published pre-primary poll results took so long providing methodological information. In the end, the committee had to publish its findings based on partial data.
It is obvious that if an AAPOR committee cannot efficiently gather methodological information for a report commissioned in the aftermath of a significant polling failure (in New Hampshire in 2008), then transparency is not the guiding norm that it should be in our profession.
So Miller proposed that AAPOR follow a different course. Rather than focus solely on violations of the AAPOR ethical code, Miller proposed to create positive incentives, to "give AAPOR's stamp of approval to survey organizations for timely and complete methodological disclosure." Toward that end, he also proposed to create an AAPOR administered archive -- a "system for collecting and storing disclosed information in one place" -- and to "provide education and assistance" to survey organizations that pledge to routinely deposit information about their surveys to that archive.
This morning, Miller gave a hint of what is coming later this week. He sent an email message to the entire AAPOR membership, urging members "in a position to decide whether your survey organization can participate...to join the initiative." Miller writes that he plans "to publicize the names of organizations that have agreed to help during my Presidential Address" on Friday. He adds:
This is a big commitment for AAPOR, maybe the biggest thing that the Association has ever tried to do. At the same time, it appears that such a program is essential at this time when the status and credibility of our profession is under unprecedented threat. It can move AAPOR from an occasional, largely ineffective re-actor in the realm of survey standards to a proactive positive force for the profession. And it can coalesce polling and survey organizations around a common goal of openness and integrity.
Regular readers will know that Miller's initiative jives neatly with my own ongoing interest in improving disclosure of polling methodology (discussed most completely here). As such, it should come as no surprise that I strongly support this initiative. Among other things, I will be a participant along with Miller in the opening plenary session on Thursday night. I look forward to reporting more details about Miller's transparency initiative, and about the rest of the AAPOR conference. This week especially, you will want to stay tuned...
[Interests disclosed: I served on AAPOR's Executive Council from 2006 to 2008]