The American Association of Public Opinion Research is a justly well-respected organization whose publications and conferences are an important means of communication between academics and practitioners. The organization's position at the intersection of academia and business makes it unique in how it can help foster constructive discussion and innovation in its field. That is why the organization's dismissive official reaction to the use of YouGov polling by the New York Times is so disturbing. According to AAPOR:
Recent actions involving the removal of meaningful standards of publication for polling data by at least one leading media outlet and the publication of polling stories using opt-in Internet survey data have raised concerns among many in the field that we are witnessing some of the potential dangers of rushing to embrace new approaches without an adequate understanding of the limits of these nascent methodologies.
We work with YouGov on multiple projects, so we hardly claim to be disinterested observers, but our experiences in this area make us realize how ridiculous the AAPOR claims are. Many in the polling community, including ourselves, have the exact opposite concern: that, with some notable exceptions such as The New York Times, leading organizations are maintaining rigid faith in technology and theories or "standards" determined in the 1930s. We worry that this traditionalism is holding back our understanding of public opinion, not only endangering our ability to innovate, but putting the industry and research at risk of being unprepared for the end of landline phones and other changes to existing "standards."
Nate Cohn of the New York Times methodically laid out the distinctions, and then costs and benefits, of using YouGov polling to supplement traditional probability polling to forecast the 2014 mid-term election:
In a probability sample, every voter should have an equal chance of being randomly selected, making the sample representative. ... Instead, YouGov attempts to build a large, diverse panel and then match its panelists to demographically similar respondents from the American Community Survey, an extremely rigorous probability survey conducted by the Census Bureau.
The costs of YouGov's approach are clear: there is still a higher portion of the population on phones than the Internet and there needs to be a model of the electorate. The benefits of the YouGov method are clear as well: it is faster, less costly, and it is a panel rather than the traditional cross-section.
Cohn continues by outlining the less obvious benefit of YouGov's approach as compared to traditional probability polling methodology: rapidly falling response rates on the phone are raising the time, costs, and potential accuracy of traditional methods. The idea that using a phone can provide equal probability of reaching any voter has a lot of assumptions: (1) everyone is reachable by phone, (2) people are all equally likely to answer the phone, (3) people are all equally likely to participate in a survey if asked, and (4) the pollster can define the voting population from the general population. In practice, the probability pollster needs to make massive and changing assumptions about the method of reaching people, as the breakdown of landline only, cell phone/landline, and cell phone only households switches. There is no known ground truth to how people can be reached and the quantity of people at each phone. As response rates fall below 10% probability polls need to make more and more decisions about how to adjust for systematic differences between respondents and the general population. And, pollsters continue to make decisions about their models for the likelihood that a respondent will turn out to vote.
In short, probability pollsters need to make many assumptions about selection in their polls, just as YouGov does! An important difference is that while YouGov examines their selection issues aggressively and publically, probability pollsters sometimes ignore the growing lists of selection issues they face. While academics and practitioners alike have studied the issue, traditional probability polling still reports a margin of error that is based on the assumption of 100% response rates for a random and representative sample of the population. AAPOR writes of non-probability polling, "In general, these methods have little grounding in theory and the results can vary widely based on the particular method used." In fact, the theory used by YouGov and in other non-probability polling contexts is well-founded and publically disclosed, based on the general principles of adjusting for known differences between sample and population.
Yet, oddly, AAPOR's letter barely mentions methodology, but instead focuses on transparency; they accuse the New York Times of obfuscating the methodology. That is odd because Doug Rivers of YouGov is a prolific writer who has detailed the methodology at length and subjected the methodology and results to public transparency that rivals the best practices of major polling companies. YouGov polls fare well when scrutinized along with the major traditional probability polling companies. Doug's (and other's) academic papers are published in the top peer review journals. If anything, people on the cutting edge of research are not hiding anything, on the contrary, we are fighting hard to overcome entrenched methods by being even more diligent and transparent.
Again, we do not pretend to be outsiders here. With collaborators at Columbia University, Microsoft Research, and elsewhere, we have done research on multilevel regression and poststratification and survey nonresponse. In this note we are not trying to prove that our methods are better than other approaches to public opinion polling; rather, our point is that, from the research perspective, there is not so much difference between our approach and traditional probability sampling methods. In either case, there is a data collection stage and a data analysis stage. In data collection, it is important to try to get as wide coverage as possible with minimal selection bias in the responses. In analysis, it is important to adjust for as many factors as possible. Traditional polling puts more of a burden on data collection, but with nonresponse rates as high as they are now, responsible pollsters need to perform a large amount of adjustment. In short, we are all dealing with the same problems, if with slightly different emphases, and we find any sharp division between probability sampling and model-based inference to be naïve in the context of 21st century polling.
We are concerned that the statement of AAPOR, while undoubtedly well-intentioned, could distract people from recent progress in the field of public opinion research. Neither we nor the New York Times are advocating replacing probability polling -- just supplementing polling where it is more cost effective or provides increased coverage. There is no reason to maintain a status quo of only expensive, slow, and increasingly theoretical shaky probabilistic polling. Regardless of the debate on transparency, non-probabilistic polling is going to become a major part of public opinion research at some point in the near future as research and technology makes non-probability polling cheaper and faster (the Internet is a very effective way to reach people) and more accurate (computers can quickly do advanced statistics). And shifting technology makes probability polling more expensive and slower and less accurate (who is going to have a landline or even a cell phone in 2024?). It is time for AAPOR to embrace the future or risk watching innovation pass it by.
This post was also published on The Washington Post's The Monkey Cage.