A recent Pollster.com blog post in this space referenced the real-world challenges that pollsters face in collecting accurate data under unusual circumstances -- recalling how Pew Research Center faced significant hurdles collecting data in the aftermath of Hurricane Sandy. Besides the fact that Pew's own call center was in New Jersey, Sandy had an obvious impact on response rates for all pollsters conducting research in the affected areas in the days, weeks and months after the storm hit.
In the case of Sandy, pollsters adjusted their methods accordingly. Some did not call into the affected areas (although that was not a feasible option for many in the waning days of a presidential election). Others, like Pew, made one-time adjustments because they understood that their data might be flawed or unrepresentative. At the firm where I work, Global Strategy Group (GSG), I recall debating how long we needed to wait after Sandy to conduct polls in Long Island, New Jersey, and Connecticut. Was it when 80 percent of the power was back in affected areas? 90 percent? 100 percent?
While natural disasters are an obvious case, pollsters often have to balance the need for timely data (and the needs of their clients) with the desire for accurate polling in an era of waning response rates. I can think of numerous cases when GSG has either delayed calling or adjusted our calling procedure to avoid certain individuals or areas. Sometimes this is due to concern over response rates or whether certain groups of individuals might be less willing to participate -- leading to an unrepresentative sample. Some examples include:
- Religious Holidays: Christmas and Easter are obvious ones to avoid, but Yom Kippur and Rosh Hashanah are also problematic in heavy Jewish areas. This is always tricky for campaign pollsters as the Jewish holidays occur in the fall during prime election polling season and only impact a subset of the electorate.
- Halloween: October 31st is one of the oddest nights of the year to try and conduct a poll and we have avoided it at all costs due to low response rates -- especially from parents. I recall once doing a nightly rolling tracking program late in campaign season and throwing out the data because it looked so different from previous and subsequent nights.
- Sporting events: Super Bowl Sunday is to be avoided, but other "big games" specific to certain locales can be a problem as well. Sports with high local viewership, like college and professional football, can be a real cause for concern.
Other times we have delayed fielding a poll due to external events that might cause shifts in public opinion. Political campaigns generally avoid calling during presidential debates and party conventions in case one side gets a "bump" that changes the electoral environment. But when it comes to other news events -- both the expected and unexpected -- determining when to poll in the news cycle can be tricky.
Unlike media pollsters who often poll in the middle of a crisis to show a spike in public opinion, internal campaign pollsters usually want to reflect a "normal" state of affairs. Outside of peak election season, internal campaign polls are used to inform long-term strategic decisions. But deciding when it is "normal" to poll is often a delicate balancing act. For example:
- How long after the Boston bombings was it OK to poll on topics even tangentially related to terrorism?
- Is it even worthwhile for campaigns to conduct polling about their 2014 races right now in the middle of the government shutdown? Or is this a moment in time that will not be reflective of the world after the shutdown is resolved?
There are also other external day-to-day events that may influence public opinion. If it rains for five days straight, does that make respondents more pessimistic about whether things are headed in the right direction or off on the wrong track? What if your local sports team goes on a terrible losing streak or loses a big game in heart-breaking fashion?
We at GSG often talk about the overlap between the world of politics and sports (both industries care an unhealthy amount about winning and losing) and decided to tackle that latter question -- conducting a small-scale experiment about whether the outcome of sporting events can impact public opinion across a variety of topics.
We conducted a short automated survey in weeks 1 and 2 of the NFL season on Sunday evening in the immediate aftermath of the early slate of football games. We asked the same questions in media markets where the local team won and the local team lost. We then compared the results of the "winners" and the "losers" controlling for demographics and attitudinal factors that might explain differences between the two groups.
The results were intriguing and, in some cases, we saw some differences. For example, respondents in "winning" markets were more likely to agree that "Americans can get ahead with hard work," to say the economy would improve in the next 12 months and to give positive ratings to NFL referees. We generally found modest effects -- the biggest ones were about 10 points among the subset of respondents who could accurately report whether their local team won or lost. Just as importantly, we found no effect on many questions -- crucially on major political measures like direction of the country and the president's job approval.
So what does this mean for pollsters? Does this mean we should avoid calling on Sunday evenings during the NFL season? No. First, we only found modest effects among a subset of respondents with knowledge about the games' outcome. Second, we conducted our poll only on Sunday evenings because of the nature of the experiment. But it's generally not a good practice to call exclusively on a specific day and time. The effects of winning and losing may not be long-lasting -- by spreading calling across nights, one can likely reduce or nearly eliminate the impact of short-term effects.
Finally, it is worth remembering that polls are snapshots in time. They reflect attitudes at the time they are conducted. A pollster cannot control for everything every time they go in the field. External events impact opinion in unusual ways on nearly every poll -- that's likely not the exception, but the rule. If we always waited to poll when everything was absolutely "normal", we might be waiting a long time.
All that said, if you are running for office right now in Pittsburgh, Jacksonville, New York, or Tampa Bay -- all cities with NFL teams that haven't won a game this season -- maybe you do need to be concerned that the electorate is a tad sour on life at the moment.