As the final act of a yearlong investigation of the polling mishaps leading up to the New Hampshire and other primary elections last year, the American Association for Public Opinion Research (AAPOR) today criticized the firm Strategic Vision LLC for refusing to disclose "essential facts" about surveys it conducted prior to the 2008 New Hampshire and Wisconsin primaries. An AAPOR press release described Strategic Vision's nondisclosure as "inconsistent with the association's Code of Professional Ethics and Practices and contrary to basic principles of scientific research."
When AAPOR's special ad hoc committee released its report last April, over a year after the New Hampshire miscues it was created to investigate, it had still not received "minimal disclosure" from 3 of 21 firms that had received requests a year earlier (though Strategic Vision's evasions were notworthy and consistent with my own experience). Today's AAPOR release completes the story:
For more than one year, AAPOR was unable to obtain the following basic information about Strategic Vision LLC's polling in New Hampshire and Wisconsin: who sponsored the survey; who conducted it; a description of the underlying sampling frame; an accounting of how "likely voters" were identified and selected; response rates; and a description of any weighting or estimating procedures used. AAPOR considers the release of this information for public polls to be a minimum requirement for professional behavior among those who conduct public opinion research.
Following Strategic Vision LLC's failure to respond to AAPOR's inquiries, a complaint was filed alleging a violation of the association's Code of Professional Ethics and Practices. The investigation process included two notices of non-compliance to Mr. David Johnson, CEO of Strategic Vision LLC, who explicitly refused to provide the requested information. Later, after receiving notification of the association's initial findings of a violation, Mr. Johnson offered partial but incomplete information. AAPOR never received any information about response rates, weighting, or estimating procedures. The AAPOR Executive Council now concludes that the repeated noncompliance by Strategic Vision LLC was a violation of the AAPOR Code.
AAPOR's action comes with no penalty, since no one associated with Strategic Vision is an AAPOR member.
The release also notes that this action "concludes AAPOR's official evaluation" of the 2008 primary polling mishaps while also noting that Strategic Vision was the "only polling firm" that failed to meet its minimal disclosure standards in response to their committee's requests. That action implies that other firms have provided additional information since the release of the committee's report in April. As of this hour, however, the reports available on the Roper Center web page appear to include no new information on the South Carolina surveys by Clemson University and Ebony/Jet/Lester & Associates beyond what they publicly released in 2008 (both firms had been singled out in the April report, along with Strategic Vision, for failing to respond to committee requests).
So there you have it. Twenty months after announcing its intention to request data related to New Hampshire primary poll, six months after reporting that only 7 of 21 firms had gone beyond the minimal disclosure that AAPOR mandates for public release "in any report of research results," AAPOR today "raises objections" about the response of one firm.
To put it simply: The process of "on demand" disclosure backed by the sort of punitive sanction issued today is not working. As I wrote in August, there may be a better way.
[Interests disclosed: I'm an active AAPOR member and served on the AAPOR's Executive Council from 2006 to 2008].
Update: ABC's Gary Langer has more, including news that "AAPOR also said an updated and final version of its report on the pre-primary polls is now available." An AAPOR spokesperson tells me that Gary is in error and that the report has not been recently updated.