They're back. As reported yesterday by Politico, Strategic Vision, LLC posted results** from what they claim is a survey of Georgia. As per our previous entries on this subject (here and here), we will no longer publish their results as "poll updates" or in our poll charts. Yesterday's release does virtually nothing to answer of questions raised by well over 200 purported surveys released by Strategic Vision since 2004. It also falls well short of the minimal standards of disclosure that got the company into trouble in the first place.
For the uninitiated, the saga began with a rare censure by the American Association for Public Opinion Research (AAPOR) last fall resulting from Strategic Vision's failure to comply with requests for information about their response rates and weighting procedures -- information that 21 other organizations provided upon request in connection with AAPOR's investigation of the primary election polls of 2008.
Following AAPOR's action, blogger Nate Silver raised the possibility of fraud and subsequently found a pattern in the trailing digits of the percentages reported in Strategic Vision polls suggesting a "possibility of fraud." Michael Weissman, a retired professor of Physics at the University of Illinois and frequent commenter on Silver's site, did some additional number crunching (a Fourier analysis) and concluded that the odds were 1 in 5,000 that the pattern in Strategic Vision could have been produced by chance alone.
The issues raised by Silver and Weissman were highly technical and difficult for mathematical mortals to evaluate, but even more troubling was Strategic Vision's strange pattern of half-truths and evasion. Commenters on FiveThirtyEight discovered that the four offices listed on the Strategic Vision web site were UPS store mailboxes. In the wake of the initial stories, Strategic Vision CEO David Johnson announced to at least five news organizations that he would soon take legal action against AAPOR and Silver. He promised to release additional subgroup tabulations of the contested data. None of this ever happened.
"We intend to vindicate ourselves," Johnson told Politico a few days after the AAPOR Censure. If his surveys were real, if they had been conducted by live interviewers at actual call centers, Johnson should have a wealth of evidence at his disposal to silence his critics. The public polls released by Strategic Vision since 2004 (archived here and here by Harry Enten) add up to more than 200,000 interviews. That many interviews would leave a lot of witnesses: Call center managers, supervisors, probably hundreds of interviewers, any one of which could come forward to vouch for the process that produced the numbers. And there should be electronic records of the actual survey data somewhere -- at least for the most recent projects. Why hasn't Strategic Vision taken any steps to present some of this evidence and vindicate themselves?
Yesterday, Johnson tried to Strategic Vision stopped releasing public polling in September and went dark in terms of public polling until this week. Here's what he told Politico:
[Johnson said] that the lull in his firm's work had represented a deliberate choice to take some time off in the light of the allegations and let the scrutiny subside. He also said a family illness prevented him from polling the Georgia gubernatorial race earlier in the year.
"Some of the stuff was getting to me. I felt it was best to take some time off," Johnson said. "You know the old adage - lawyers should never defend themselves. I should never try to be my own PR person."
He also told Atlanta Journal Constitution columnist Jim Galloway that his libel suit threats "was me speaking in anger because I was really outraged at the time."
Yesterday's release includes two new twists. For this first time, Strategic Vision sent Galloway and other reporters a set of tables in a compressed file showing results tabulated by gender, age, race and income, and all of their percentages are computed to one decimal point. Is it more likely that these results are based on some sort of interview data, for what that's worth.
I've got a counter-intuitive guess: maybe SV-LLC will start doing real polling. It's not hard, it doesn't cost much money, and it would serve more than one purpose.
One thing is that it would restore their status as serious participants in conservative politics. A second benefit is that it would undermine the case against them, at least in the public's mind.
Maybe they'll never release another result, but if they do, I'd guess it would be genuine.
Doesn't affect the utter bogusness of everything they've done to date, of course.
He's right that new surveys, even if "genuine," do nothing to resolve the serious questions raised about Strategic Vision's previous work.
But let's ponder the meaning of "genuine" as we consider what Strategic Vision's latest release does not tell us: They say nothing about the mode of the survey (whether it used live interviewers or some automated method), the sample frame (whether telephone numbers were selected from some sort of list or via a random digit method), the weighting procedure (whether results were weighted and the variables used to weight them), and they do not identify of who conducted the survey (the call center or field-work provider, if these used one). These basic facts are part of the minimal disclosure requirements of both AAPOR and the National Council on Public Polls (NCPP).
NCPP also requires that its members describe the "size and description of the subsample, if the survey report relies primarily on less than the total sample." It is not clear whether NCPP's mandate applies to cross-tabulations, but it is very clear that Strategic Vision tables provide no information about the size of each demographic subgroup.
Both organizations also mandate that releases tell us, "who paid for the poll?" Strategic Vision's release says nothing about how this poll was paid for and, as an alert Pollster reader informs me, fails to disclose a significant conflict of interest: A search of Georgia campaign finance records shows that Strategic Vision was paid $3,500 to conduct a poll in 2009 for Ralph Hudgens, a candidate in the Republican primary contest for Insurance Commissioner tested in the new survey.
So again, for all of these reasons, we will no longer publish results by Strategic Vision, LLC on Pollster.com. But that raises a much bigger problem: Strategic Vision is not the only polling organization that has fallen far short of the minimal disclosure requirements of organizations like AAPOR and NCPP, and their results do appear on Pollster.com. That shortcoming is something I want to discuss at greater length this week. Stay tuned.
**For what it's worth: That link and the rest of the Strategic Vision, LLC web site remains inaccessible to computers in our offices and to our colleagues at the National Journal Group and Atlantic Media.