In politics, polls are everything. Media outlets use polls and surveys to guide their coverage decisions. Party officials use polls to decide which candidates to back in primaries. Donors look at polls to help them decide whether it's worth purchasing a particular candidate. Fox News and CNN are using polls to decide which candidates to include in the first GOP presidential debates.
But there are big problems with how much we rely on polls: They have almost no ability to predict the results of elections that are more than a year away. And it's getting worse: the increasing use of cell phones and the decline of people willing to answer surveys has put the election polling industry in a state of "near crisis," as one prominent expert recently put it. The problems are myriad. Here are a few.
Who doesn't get surveyed
Since the advent of cell phones, the number of people who have or answer a landline has been steadily declining. Most reputable pollsters call cell phones, but even with those included, Pew Research reported in 2012, about 9 percent of the people pollsters try to reach actually respond.
A low response rate isn't necessarily a problem if the people who respond are generally similar to people who don't, which Pew finds is generally true in their very high-quality samples. But think about it: there is now a particular type of person who's going to pick up the home phone and talk to a pollster. That person is typically older. Old people are different from younger people. That's a huge problem if the pollster doesn't call cell phones, as they'll have a hard time finding enough younger people.
Internet surveys are becoming more common, with the obvious problem that they can’t survey people who aren’t online. While 84 percent of Americans are online, far fewer will sign up with an online survey company to complete polls. Some survey companies, particularly YouGov, with whom The Huffington Post conducts polling, work very hard to make their surveys match the population as closely as possible. Some, however, do not.
There are a few simple things you can look for in pollsters' descriptions of their methods to get clues about the quality of the poll. If people were reached via telephone, the pollster should have called cell phones. If people were reached online, the information should include details on how the pollster made their results accurate, since Internet polls are usually not based on a random selection process. Regardless of how the data were collected, if the poll is of “likely voters” the pollsters should explain how they identify likely voters.
Who gets surveyed
Early primary polls usually look for registered Republicans or registered Democrats, though the people who show up on primary day will be a tiny sliver of that population and may have different characteristics. In 2008, the combined Democratic and Republican turnout in all primary and caucus contests nationwide was roughly a quarter of the adult population. It fell to just over 10 percent in 2012 thanks to a non-competitive Democratic contest, according to voter turnout statistics maintained by University of Florida Professor Michael McDonald.
National polls of registered partisans sample a pool of potential primary voters that is two to three times larger than the group of Americans who will actually show up on primary day. This practice occurs as much by necessity as by design, since there wouldn't be enough likely voters to analyze in a sample if pollsters narrowed it down to the proportion of people who usually vote in primaries. Also, models that aim to predict which poll respondents are likely to vote are problematic enough even in general elections. People who say they plan to vote may not actually vote for a variety of reasons and those who don’t think they will vote might end up with a reason to vote on Election Day. Trying to anticipate actual primary voting would require accounting for widely different turnout by states, including significantly higher turnout in primary than in caucus states.
Error in polls
In election polls, pollsters talk to a few hundred or a couple thousand people out of millions of potential voters. This introduces “error” in the poll, which means you would probably get a slightly different answer if you talked to everyone rather than a smaller group. Candidates within about 2-3 percentage points of each other in any one poll are in a very close race and neither may be leading. But only if many polls, done at similar times, consistently show a candidate leading by 2-3 percentage points is it likely to be a real lead.
After collecting the responses, pollsters make adjustments to the data to make sure the sample of people they polled resembles the demographic makeup of the population that they're trying to measure, whether it's Republican primary voters or all Americans. These procedures vary by pollster. For election polls, often pollsters will try to identify those most likely to vote and report their responses, and they use different methods of identifying likely voters. At any point in any of these processes, individual pollsters can make unique decisions, and those decisions will result in polls that are slightly (or very) different from one another.
Nobody's paying attention yet
The reality is that most people won’t pay close attention to the election until the primaries next year or even right before ballots are cast next November. Almost half will never pay close attention.
Since 1987, Pew Research has asked this survey question: “As I read a list of some stories covered by news organizations this past week, please tell me if you happened to follow each news story very closely, fairly closely, not too closely, or not at all closely. News about candidates for the [relevant year's] presidential election. ...”
This chart, originally published in early May, shows the share of Americans who said they were following news about presidential candidates "very closely" in election cycles from 1988 through 2016. Since then, Pew Research has put out an estimate of what proportion of Americans are paying close attention to the 2016 race. That number? Sixteen percent. That means a whole lot of the people getting polled aren't following election news.
When HuffPost surveyed people who care a great deal about politics and the upcoming presidential election, the results were quite different from polls that weren't so selective.
Candidates in the news get poll bumps that fade
Surveys' inclusion of people who aren't paying attention or might not end up voting may help explain some of the volatility we see in early polling. One pattern is fairly predictable: Candidates who briefly dominate campaign news tend to get small and usually temporary bumps in their support. In a Republican contest with so many candidates, this can make for some dramatic looking changes, but most of the "surges" so far have been small and have subsequently receded.
Consider the charts produced by Charles Franklin, the co-founder of Pollster.com, whose trend lines are very similar to those plotted by HuffPost Pollster charts. These show that Jeb Bush, Scott Walker, Marco Rubio, Ted Cruz and now Donald Trump have all received bumps, many temporary, from news surrounding campaign kickoffs or announcements of exploratory committees.
The same patterns of candidates rising and falling in the polls played out in 2012.
Early frontrunners often fade
All theory aside, in practice early polls often have folks leading who then go on to nowhere. Ask Presidents Cain, Gingrich, Dean, Lieberman and so on.
What we can get from polls
Finally, please remember that polls conducted months before an election are not predicting what will happen in the election. Polls provide valuable information -- in the form of snapshots of what the public thinks at the time the poll is conducted. As we get closer to the primaries and general election, the polls will provide better indicators of the frontrunners -- in 2012, many pollsters were remarkably close to the actual outcome -- but even then polls are not forecasts. We know that in the general election polls become predictive a few months before the election, but for primaries, outcomes are a lot less predictable. The 2016 GOP primary exacerbates the unpredictability by setting 17 candidates against each other, making it difficult to name “frontrunners.” Polls this early should not be interpreted as an indicator of who will win.