04/16/2013 06:04 pm ET | Updated Jun 16, 2013

Avoiding a Repeat of 2012 GOP Polling Collapse With Adaptive Sampling

The origin of the problem that plagued Republican pollsters in 2012 (and to a lesser extent Democratic pollsters in 2010 and Republicans in 2008) is that we're really good at what has historically been the big question we've been asked to answer: How are people going to vote?

But the "dirty secret" in political polling is that neither Republican nor Democratic pollsters historically have worked much at measuring who will vote. In fact, it is standard practice among campaign pollsters and many media pollsters to impose a model of what the electorate will look like on a sample via either weighting or stratification.

As a result, pollsters have nearly always had problems when turnout has been unpredictable -- in run-offs, special elections, and even in cases like the 2008 Democratic primary in New Hampshire. But those failings have been largely ignored because the big-ticket polls, those in federal general elections, have been mostly accurate.

The world is what it is, until it isn't.

General election polling has historically been accurate because the rates at which different groups in the electorate turn out have been predictable across similar elections. Mid-terms looked a lot like other mid-terms and presidential years looked a lot like presidential years.

This all changed over the last three election cycles.

In 2008, 61.6 percent of eligible voters turned out and voted, a participation rate not seen since 1968. The electorate that voted for Barack Obama included significantly more minorities than previous presidential electorates, more so than simple demographic change in the population would explain.

Many pollsters were slightly off in their models of the electorate and wrong in their results. But the errors were relatively small and, in part because of the magnitude of Obama's victory, not particularly embarrassing.

The 2010 elections saw a different set of mistakes as the electorate rebounded to an older, whiter and more working class electorate than in previous mid-terms. This wasn't simply reversal of the "outlier" election in 2008; it was a departure from the previously predictable composition of mid-term elections.

Which brings us to 2012. The rumblings under the mountain of electoral stability that had been mostly ignored after 2008 and 2010 were followed by an eruption in 2012 and left a number of pollsters -- especially Republicans -- looking bad. While the authoritative Census Bureau/Bureau of Labor Statistics Current Population Survey numbers have still not been released, examination of exit polls and other post-election estimates yield a startling picture:

  • Hundreds of thousands of white, working class voters who had voted in 2008 simply stayed home;
  • Minority turnout rates (and not just raw numbers) surpassed even 2008 levels; and
  • Young, unmarried women participated at rates surpassing previous rates.

These changes meant that pollsters, almost universally, got it wrong. While Republican pollsters, who had been relying on a voter model that split the difference between 2004 and 2008, were the most embarrassingly wrong, many Democratic and non-partisan pollsters under-estimated the magnitude of Obama's victory as well.

It's not a one-time problem; it's not a one-time fix.

Most of the post-election analysis of Republican polling failures has focused on assessing how the incorrect assumptions Republicans made about the electorate diverged from reality, and offering simple solutions to correcting these "models" of the electorate.

But arbitrarily adding more young people or minorities to polling samples treats 2012 as an isolated case. This "business as usual" solution is no solution at all, but instead risk repeating the same failures again and again in this new era of unstable electorates.

A new question, a new solution

By now, it should be evident that the previous practice of imposing an electoral model on survey data will almost certainly run a high risk of resulting in biased results. In the future, good pollsters are going to have to measure two equally important questions:

  • Who is going to vote?
  • How are they going to vote?

The days of assuming the answer to the first question based on past evidence have ended. We must now measure it to avoid biased results.

At Wilson Perkins Allen, we have begun answering this question using a method called Adaptive Sampling. This tool, originally developed for the kind of run-off election with which pollsters have historically struggled, offers a proven path toward measuring, rather than assuming, voter participation.

The AAPC awarded our firm a "Pollie" for our use of Adaptive Sampling in the Ted Cruz for Senate (R-TX) primary run-off. While traditional a priori techniques lead other pollsters astray and caused them to predict a substantial victory for David Dewhurst in the run-off, Adaptive Sampling allowed us to actually measure who would vote in the Cruz run-off with enough accuracy to predict a 15-point Cruz victory (Cruz won by 14.5 percent).

WPA's Adaptive Sampling uses participation questions already validated in polling and political science literature to measure respondents' likelihood of participating in the election.

Most pollsters use similar questions to screen respondents in or out of a poll, but then return to their assumptions about the electorate to structure their sample. Instead, we use the rate at which voters screen in and out of our polls and compare these to the anticipated participation rates for that group of voters based on data collected by the BLS and Census Bureau. These prior assumptions can then be updated using a formal mathematical process based in Bayesian statistics.

The result of this process is a picture of the electorate that changes based on our actual measurements of intention to vote, but one that also changes in a controlled and smooth way that uses all available information and avoids over-correction or rapid and random changes in the model.

The evidence of the last three cycles shows that pollsters need to measure, rather than assume, who will vote in general elections. Our Adaptive Stratification is one way to do this, and one that has already proven its worth and accuracy.