Jeff the Productivity Sapper points me to this insulting open letter to Nate Silver written by pollster John Zogby. I'll go through bits of Zogby's note line by line. (Conflict of interest warning: I have collaborated with Nate and I blog on his site). Zogby writes:
Here is some advice from someone [Zogby] who has been where you [Silver] are today.
Sorry, John. (I can call you that, right? Since you're calling Nate "Nate"?). Yes, you were once the hot pollster. But, no, you were never where Nate is today. Don't kid yourself. Zogby writes:
You [Nate] are hot right now -- using an aggregate of other people's work, you got 49 of 50 states right in 2008.
Yes, Nate used other people's work. That's what's called "making use of available data." Or, to use a more technical term employed in statistics, it's called "not being an idiot." Only in the wacky world of polling are you supposed to draw inferences about the U.S.A. using only a single survey organization. I do agree that it wasn't particularly impressive to get 49 states out of 50. What made Nate's name was not his routine election-night forecast but his exceptional insight during the primary election season, followed up by strong and timely analyses during the following months. If you think that's so easy, fine: You do it. Until then, how about some division of labor, where analysts such as Nate make use of polling data and pollsters such as John respect that their polls will be used in all sorts of interesting ways once their data go out the door. Zogby writes:
Hey, I have been right within a few tenths of a percent - but you are a probabilities guy and even a 95% confidence level and a margin of sampling error are not enough for some.
A basic understanding of sampling and nonsampling error will tell you that being "right within a few tenths of a percent" is luck, not skill. If you're within a percent or two, great. A few tenths of a percent on a poll with a 3% margin of error . . . sure, that'll happen sometime, but perhaps it's a good idea for you as a pollster to explain to people how randomness works. Zogby writes:
You [Silver] take other people's polls, compare records for predictions, add in some purely arbitrary (and not transparent) weights, then make your own projections and rankings.
There's that "other people's polls" thing again. What's your problem? Would it make you happier if Nate only used his own polls? Even if Nate conducted his own polls -- heck, maybe he's doing that right now, I have no idea -- he'd be a complete an utter idiot to make forecasts from them and not use others' polls as well. As a forecaster, you're not gonna go to heaven because you never used anybody else's data. Zogby writes:
We pollsters are data-based problem-solvers. We work with clients to solve problems, plan the future, project trends, and test effective messages and models. This involves lots of people skills, a passion to get it right and do right by people who trust us. We are so much more than where we stand on election day. Your [Silver's] ratings come with and generate a lot of vitriol. How does that make our world a better place?
Hey, John. The good news is you're getting paid for those services! Now take the perspective of those of us who analyze "other people's polls" (as you put it). You wouldn't want us to treat all polls equally, would you? To mix a high quality poll such as yours with some crappy robopoll or some discredited partisan hack job? Once we're using data, it's our duty to evaluate the quality of our data. Sure, Nate could rate the pollsters and keep his rating a secret, but I think it actually will "make our world a better place" (as you put it) for Nate to be completely open about his procedures and release his poll ratings publicly, where they can be shared, challenged, and improved upon. Earlier in your note you criticized Nate for using weights that are "not transparent." Releasing poll ratings is a way to increase transparency, no? Zogby writes:
You [Nate] are a statistician - a very good one - but you are not a pollster. You should conduct some polls and learn that the rest of us good pollsters survey people, not statistics.
Haven't you ever heard about the division of labor? Actually, I agree that Nate would learn a lot by being involved in a survey operation (and, as noted above, maybe he's already doing so). In the meantime, I have no particular interest in discounting his work because of his lack of experience in this area. Any more than I'd discount Bill James's 1982 Baseball Abstract because of his lack of direct experience in major league baseball. An outsider's perspective can be useful. Zogby writes:
The numbers tell the story; preconceived ideologies and fuzzy-math statistical models do not.
Huh? Say again?
P.S. As I think is clear from the above, I think John went just a bit over the top in his criticism of Nate. In all seriousness, though, I think pollsters are extremely important in forming our understandings about politics, and I agree that there are all sorts of skills that good pollsters have. There's no way I could ever conduct a good poll myself, and I rely in so much of my research on the expertise of pollsters in the field. There's no reason for John to be so defensive: Nate Silver, Andy Gelman, Larry Bartels and all the rest of us rely crucially on the efforts of public opinion professionals from Gallup to Zogby to gather the information we use in our analyses.
This post originally appeared at Statistical Modeling, Causal Inference, and Social Science.