The retraction of a major gay marriage study highlights polling's "secrecy problem," as well as the relevance of a newly launched transparency initiative. This is HuffPollster for Friday, May 29, 2015.
JOURNAL RETRACTS DISCREDITED GAY MARRIAGE STUDY - Natalie Jackson and Lila Shapiro: "Science magazine officially retracted a major study on same-sex marriage and public opinion on Thursday without the consent of the lead author, UCLA graduate student Michael J. LaCour. The events leading to the retraction became public on May 19, when the article’s second author, Columbia University political science professor Donald Green, issued a request for the retraction based on evidence that the study data were at least in part falsified....Thursday's statement from Science Editor-in-Chief Marcia McNutt cites two reasons for retracting the study. First, claims that survey respondents had been given cash payments to participate and refer family and friends to participate in the survey were inaccurate. Second, the funding statement was false. LaCour had misrepresented that funding for the study came from the Williams Institute, the Ford Foundation, and the Evelyn and Walter Haas Jr. Fund. McNutt's statement notes that both of these falsifications have been confirmed by LaCour’s attorney, although it also points out that LaCour does not agree with the retraction." [HuffPost, Science retraction statement]
Cohn: A 'secrecy problem' for polling... - Nate Cohn: "[E]ven before the LaCour case, it was becoming obvious that a different group of public opinion researchers — public pollsters — adhere to much lower levels of transparency than academic social science does. Much of the polling world remains shielded from the kind of scrutiny that is necessary to identify and deter questionable practices.Consider how Mr. LaCour’s study fell apart. First, a team of political scientists conducted a new study to try to replicate his findings. When they could not do so, they scrutinized the original data set of individual respondents, called microdata, and found pervasive irregularities. In the polling world, no survey firm releases its microdata in a timely manner. When pollsters release it at all — usually months after publication, to an archive that requires a paid subscription for access — they seldom provide the detailed methodological explanations necessary to replicate the survey results." [NYT]
...but AAPOR's Transparency Initiative 'falls short' - More from Cohn: "This month, at the annual conference of the American Association of Public Opinion Research, which represents pollsters, herding was a popular topic. People discussed it in public panels and in smaller conversations. And the association recently began a transparency initiative intended to address some of the concerns. But few polling organizations have signed up so far, and the transparency initiative’s disclosure standards generally fall short of what is needed to allow others researchers to identify or deter dubious practices. The initiative would not require polling firms to release either microdata or the methodological details necessary to replicate the results. It’s hard to say what’s more telling: that the transparency standards wouldn’t be enough to preclude bad practices, or that so few pollsters seem willing to adhere to even those requirements." The Huffington Post is a charter member of AAPOR's Transparency Initiative.
WHY THE TRANSPARENCY INITIATIVE MATTERS - The release of raw data from individual survey respondents would be the most powerful protection against outright fraud, but Cohn understates the value of the methodological details mandated for disclosure by the AAPOR Transparency Initiative (T.I.) and the value these had or could have had even in uncovering fraud in the LaCour study.
First, the two reasons cited by Science editor McNutt for retracting the article -- untrue statements about survey sponsorship and incentive payments -- both involve methodological details mandated for disclosure by AAPOR's Initiative.
Second, neither the original LaCour-Green article in Science, nor the voluminous "supplemental" materials published by Science online, disclosed several key details also mandated by the TI: (1) the name of the organization that conducted the survey, (2) how the respondents were initially selected or (3) how they were recruited to the online panel (including the mode of interview used for the initial contact).
Uncertainty about the survey organization purportedly used by LaCour turned out to be a pivotal issue for David Broockman and Joshua Kalla, the two political scientists whose investigation ultimately led to the study's retraction. Before they scrutinized the raw data of the original study, Broockman and Kalla attempted "to contact the survey firm we believed had performed the original study and ask to speak to the staffer at the firm who we believed helped perform [it]...The survey firm claimed they had no familiarity with the project and that they had never had an employee with the name of the staffer we were asking for. The firm also denied having the capabilities to perform many aspects of the recruitment procedures described in [the] LaCour and Green [article]." With their suspicions raised, the two researchers decided to dig deeper.
How did Broockman and Kalla learn the name of that organization? In their paper, they say only that "LaCour and Green were both responsive to requests for advice about design details when queried." Contacted by HuffPollster for clarification, Broockman explained that they were told the name of the survey company by LaCour.
Had Science required publication of these details, per AAPOR's standards, others may have raised hard questions about LaCour's data earlier in the process. If nothing else, requiring publication of these more easily verified details in the future would provide an additional layer of deterrent to fraud.
Cohn isn't buying it. "I'm increasingly of the view that it's microdata releases or bust in polling transparency. AAPOR TI falls short," Cohn writes on Twitter. Yes, tougher standards would make for more complete disclosure, but if AAPOR's Initiative goes bust, then what?
"In general, the AAPOR Transparency Initiative is just a huge step forward in generating far more transparency in publicly generated polls than there has ever been in the past," AAPOR President Mollyann Brodie tells HuffPollster.
Why not mandate release of individual data? "Reflecting our fundamental goals of transparency and replicability, AAPOR strongly encourages the release of microdata and data sets, and we have recommended protocols to follow it. In fact that’s part of our revised Code of Professional Ethics and Practices." At the same time, "real legitimate and practical issues are at play that make it difficult for any voluntary organization, including AAPOR, to mandate the immediate release of such data, as some have called for."
AAPOR's initiative stands at a crossroads. Forty-five organizations have joined as charter members since its launch last year, but so far, most are academic and non-profit survey centers. The charter members to date include only two news media companies, The Washington Post and The Huffington Post. If the initiative is to succeed, the news organizations that initially expressed support for the Initiative, including Cohn's New York Times, will need to participate in earnest.
More on LaCour and evaluating survey data:
-Jesse Singal reports that LaCour made up the largest funding source listed on his CV and promises a new "long piece" coming later on Friday. [NYMag, @jessesingal]
-LaCour may have also faked awards listed on his CV. [Mediate]
-An Emory University political scientist questions anomalies in another LaCour paper. [Emory]
-Steve Koczela has background on other efforts to detect data fabrication. [MassInc]
-Natalie Jackson provide six questions to determine if you should trust the data you're seeing. [HuffPost]
MORE OF THIS WEEK'S NATIONAL POLLS
-Most Republicans are totally fine with capping their debates at 10 candidates. [HuffPost]
-"Undecided" continues to lead in the GOP primary race. [HuffPost, Quinnipiac]
-Americans see the benefits of free trade agreements but are concerned they lead to lower wages and job losses. [Pew]
-Gallup's economic confidence index was down in May from highs measured earlier in the year. [Gallup]
-Americans have taken a moral shift to the left, becoming more accepting of gay relationships, children out of wedlock and pre-marital sex. [Gallup]
-For the first time in seven years, more Americans are pro-choice than pro-life. [Gallup]
-Most Americans support drone strikes as a method to deal with extremist groups. [Pew Research]
HUFFPOLLSTER VIA EMAIL! - You can receive this weekly update every Friday morning via email! Just click here, enter your email address, and click "sign up." That's all there is to it (and you can unsubscribe anytime).
THIS WEEK'S 'OUTLIERS' - Links to the best of news at the intersection of polling, politics and political data:
-Harry Enten plots two charts to show why George Pataki's candidacy is virtually hopeless. [538]
-Ipsos predicts a decline in voter turnout in 2016 compared to 2012. [Ipsos]
-Steve Singiser summarizes some of the perils of grading pollster accuracy. [DailyKos]
-No, says Andrew Gelman, public opinion is not driven by "unreasoning bias and emotion." [WashPost]
-Amy Walter says Hillary Clinton should be more concerned with the rising perception that she's untrustworthy than with her overall favorability rating. [Cook Political]
-Stu Rothenberg and Lee Miringoff argue against using poll averages to determine debate participation. [Roll Call, Marist]
-The number of white Christians drops, while the number of religiously unaffiliated climbs, in nine battleground states. [PRRI]
-Karthick Ramakrishnan and Sono Shah plot two maps to show how changing "one person, one vote" would affect congressional reapportionment. [WashPost]
-Republican pollster Margaret Kenski dies. [Arizona Daily Star]
Before You Go
