In the last week, the Quinnipiac Poll has Hillary Clinton up 10 points. The next day, the USC/LA Times poll has Hillary Clinton tied. That's a drop of 10 points in one day, so the race must be tightening, right?
I'll come back to those polls in a second... but first a related story. It's morning and time for me to weigh in... literally. I climb on the scale in my home bathroom, and for the sake of argument, let's say I weigh in at 205 pounds. I quickly realize that maybe the pizza this weekend wasn't such a great idea, so I go to the gym and work out hard... cardio and weight training. When I get done, I jump on the scale at the gym, and am thrilled to see I now weigh 199 pounds. Clearly, all my hard work paid off. I get dressed and head to the airport--I'm flying from Washington, DC to San Francisco for a business trip. When I check in, I find a scale in the hotel room, and when I weigh myself, I'm now at 203 pounds.
What happened? Did I eat too many peanuts on the plane? Which is the right measure of the change in my weight? Did I lose six pounds today, or did I gain four pounds? Or maybe I only lost two pounds today? I am the same person being weighed each time.
Here's the answer: each scale was calibrated differently. My home scale leaned heavier, my gym scale leaned lighter, and the hotel scale seemed to be in the middle. So by comparing results across the three scales, I could find three different results, even though they were measuring ostensibly exactly the same thing.
So, what can the tale of my three scales teach us about polling? Just as some scales lean heavier or lighter, some polls lean more Democratic or more Republican. For example, the USC/LA Times Poll this year has consistently showing Donald Trump's support levels higher than in any other national poll. Similarly, the Quinnipiac Poll has shown Hillary Clinton with a double digit lead, more than most other national polls. Two polls measuring exactly the same Presidential race but with very different outcomes.
What should you do as a smart consumer of data?
1. Look for trends across the same polls over time. If the Quinnipiac Poll has Clinton up 10 points one week, but only up 8 points the next week, that tells you her support may be declining. But if you are trying to compare results across different polls--for example, comparing the Quinnipiac poll vs. the USC/LA Times poll--you might get a misleading view of the trends because you can be observing differences in the underlying polls themselves. Each poll has a different list of questions, a different way of gathering the sample of people who are being questioned, and a different way of analyzing the results. These underlying poll-specific characteristics are important.
2. Be wary of frequent polls that might not really be detecting anything statistically meaningful. Some daily tracking polls offer new results every day--often very different than the day before. Can the race really swing widely within a single day, barring some major event? Perhaps. But, just as it isn't realistic to think my weight went down by six pounds (or increased by four pounds) during the course of the day, it's not realistic to think that daily tracking polls or wide swings across polls are capturing something meaningful in isolation. Often, these day-to-day changes can simply represent the imprecision of a poll, or a change in who responded on a given day. Look at the broader picture--if you see a trend across polls, then it's time to start to pay attention.
3. Not all polls are created equal. Just as a scale can be off with its measurements, a poll can be as well. The fact that a result is different than all the other polls doesn't imply it is necessarily wrong or bad in and of itself. But, it is worth more scrutiny to find out the how the poll is conducted, how accurate it has been in past elections, and if there are any particular differences in the methodologies that could be driving the results.
There's certainly no shortage of polls to analyze. But whether you're stepping on the scale in the morning or looking at an election poll, it's important to understand what the numbers really mean.