The Strategic Vision story is getting far more interesting. In the wake of a public reprimand from the American Association for Public Opinion Research (AAPOR) for failing to disclose "essential facts" about his company's methods, and after more than a year of doing his best to avoid public comment, Strategic Vision CEO David Johnson now has much to say and is threatening legal action. Meanwhile, over at FiveThirtyEight, Nate Silver says he has found evidence that "suggests, perhaps strongly, the possibility of fraud" in Strategic Vision's numbers.
On Wednesday, he provided this response to a call from Jim Galloway at the Atlanta Journal Constitution:
Strategic Vision CEO David Johnson said his firm had wanted to appeal the judgment, and said a Sept. 17 hearing had been scheduled - and then canceled by the AAPOR. "We've asked for a copy of the complaint that was filed against us, and who filed it," Johnson said. "How can you respond to something when you don't know who filed the complaint."
Moreover, he added, "We're not a member of their organization. I don't know anything about them."
Johnson also gave Galloway an email sent by AAPOR to Johnson this past June acknowledging receipt of "some of the information requested regarding polls in New Hampshire and Wisconsin" (emphasis added).
Yesterday, James Verrinder of the website Research reports that Johnson now "vows legal action" against AAPOR and some of its members:
Johnson said he "disagreed completely" with the charge levied at his firm by AAPOR and vowed to take legal action against the association. He said the firm had supplied AAPOR with all the information it had requested on 19 June this year, and had electronic proof of what was sent.
Johnson believes a competitor is behind the original complaint to AAPOR, and wants to see the source of the action against his firm. "I find it unusual," he said, "that an organisation that says they are all about transparency won't supply us with details of the complaint. What they were asking for were trade secrets."
He said: "We will be taking legal action. We have spoken with our attorneys and have gotten them the documentation and should know exactly the venue and specific charges that we will be filing against AAPOR specifically and individual members of AAPOR personally."
Johnson alleges that the AAPOR's acted "maliciously" in issuing its ruling. "I think it was timed to coincide with the results of a poll we had out yesterday [on the gubernatorial elections in Georgia]," he said.
Both accounts also include responses from AAPOR President Peter Miller. Miller told Galloway that "AAPOR had sent Johnson notices four times asking him to confirm his attendance at that hearing last week, and finally ended up canceling because of the lack of any response." Miller told Verrinder that it was "completely wrong" that a competitor had filed the complaint and reiterated that Johnson's 2009 response, received after the release of the AAPOR report in April, did not include all of the requested information. In a previous article, also published today, Verrinder had reported that the reply still did not provide requested information about response rates and weighting or estimating procedures.
Now separately, Nate Silver claims to have found evidence of a non-random pattern in the trailing digits of the percentages reported by Strategic Vision in their public polling since 2005, and the implications of that assertion are pretty explosive. Last night, he raised some suggestive questions that mirror some of the unsubstantiated gossip and prodding I've received via email for years from a Democratic activist or two in Georgia. But this morning, as Silver puts it himself, he's making a much more concrete allegation (emphasis his):
Certain statistical properties of the results reported by Strategic Vision, LLC suggest, perhaps strongly, the possibility of fraud, although they certainly do not prove it and further investigation will be required.
In other words, Nate is suggesting that Strategic Vision has been making up its numbers. The analysis he reports this morning is
based on Benford's Law, the same principle similar to the principles behind much the analysis of Iran election fraud that we reported this summer [see the clarification below]. The idea is the last digit of numbers with two or more digits should have a uniform distribution. A '1' should occur as often as a '2,' a '3' etc. According to Nate, the pattern for Strategic Vision is far from uniform:
[T]his data is not random. It's not close to random. It's not close to close. Which brings up the other possibility: Strategic Vision is cooking the books. And whoever is doing so is doing a pretty sloppy job. They'd seem to have a strong, unconscious preference for numbers ending in '7', for instance, as opposed to those ending in '6'. They tend to go with round numbers that end in '5' or '0' slightly too often. And they much prefer numbers with high trailing digits like 49 and 38 to those with low ones like 51 and 42.
I haven't really seen anyone approach polling data like this before, and I certainly haven't done so myself. So, we cannot rule out the possibility that there is some mathematical rationale for this that I haven't thought of. But it looks really, really bad. There is a substantial possibility -- far from a certainty -- that much of Strategic Vision's polling over the past several years has been forged.
I recognize the gravity of this claim. I've accused pollsters -- deservedly I think in most cases -- of all and sundry types of incompetence and bias. But that is all garden-variety stuff, as compared against the possibility that a prominent polling firm is making up numbers whole cloth.
I would emphasize, however, that at this stage, all of this represents circumstantial evidence. We are discussing a possibility. If we're keeping score, it's a possibility that I would never have thought to look into if Strategic Vision had been more professional about their disclosure standards. And if we're being frank, it's a possibility that might actually be a probability. But it's only that. A possibility. An hypothesis -- as yet unproven.
Predictably enough, my email box is filling with the same question: What do you think of this? My first reaction is similar to "Mark" (not me) and some other commenters on FiveThirtyEight: The analysis is intriguing, but I would find it far more convincing if he ran comparable statistics for some of the other prolific pollsters in the same contests since 2005 (Rasmussen, SurveyUSA, Quinnipiac, ARG, Zogby, Mason-Dixon, etc). If the Strategic Vision pattern is really different from all the rest, then it would reduce the possibility that the pattern Nate found "is a function of polling in general" (as commenter Matt puts it).
Also, while I stipulate that I am no expert in Benford's Law, my sense from reviewing the analysis of the Iranian election is that its assumptions can get extremely complex. As such, we need to very cautious about jumping to conclusions based on the pattern that Silver is reporting. That said, I will have more about Strategic Vision, AAPOR and the theme of transparency in my National Journal column on Monday.
Update and Clarification: To prove I'm no expert, I initially described Silver's analysis incorrectly. Mark Lindeman is right in the comment below when he says that Nate Silver is expecting a uniform distribution, not a Benford distribution.
I also exchanged email with Walter Mebane, the University of Michigan professor whose has done much work in this area, most recently on fraud in Iran. He reviewed Silver's post and urges caution, saying that some of the comments there (such as those from Mark, Allen and Zach) "cover the kinds of further questions" he would want to ask. Like Zach, Mebane says that with two-digit numbers, we should not expect a uniform distribution of the last digit, especially if it is based on
percentages that have been rounded in a biased manner. Echoing commenter Mark, he says that a "comparison with other polling houses would probably be the most informative and quickest thing to do."
Update 2: For those wondering whether Strategic Vision, LLC has any real clients, a colleague passes along some indisputable proof that they do. The Friedman Foundation for Educational Choice has used Strategic Vision to conduct a series of statewide surveys since 2007 (click on any link showing a state's "Opinion on K-12 Education and School Choice"). I count 14 surveys in all since 2007, the most recent released just last wewek.
Ironically, these reports show that the Friedman Foundation demonstrates a prominent commitment to "methods and transparency." Check Page 2 of the most recent report for Nebraska: "We are committed to sound research and to provide quality information in a transparent and efficient manner." A methodology section found within includes the sort of information -- including response rates and call disposition reports -- that Strategic Vision continues to resist releasing for their political surveys.