President Donald Trump’s approval ratings have bounced around pretty wildly during his first four weeks in office. There has been a 36-point swing across this period of time ― his net approval has ranged from a disapproval rating 18 points higher than his approval rating to an approval rating 18 points higher than his disapproval rating.
The HuffPost Pollster chart illustrates the instability:
Several articles have been written about these widely ranging numbers, and the HuffPost Pollster team has been considering the issue. The answer seems to be less about instability in what Americans think of Trump than it is about polling methods. The variation in Trump’s job approval ratings has been driven by a few factors: how and when the poll was conducted, who was polled and whether it was conducted by a specific pollster.
To look at what’s affecting these numbers, I took all of the polls on the new president’s approval rating collected in the HuffPost Pollster database. Seventy-one publicly released polls that meet our disclosure criteria have been conducted between Jan. 20 and Feb. 17 (excluding the results by party, which you’ll see on the downloadable spreadsheet).
The HuffPost Pollster database includes a few details about how the polls were conducted in addition to recording the numbers, sample size and dates. For one thing, it considers the poll’s “mode” ― if respondents were contacted and interviewed via live interviewer telephone, internet, or a combination of automated phone and internet. The database also considers the polling “population” ― meaning whether the respondents were American adults, registered voters or likely voters.
Those differences account for the vast majority of the differences in Trump’s net approval rating (percent approve minus percent disapprove). A model that includes the poll’s mode, population, sample size, length of time in the field (number of days), undecided proportion, week of the Trump presidency (numbered 1-4), and with variables for Rasmussen’s and Gallup’s daily trackers explains 86 percent of the variance in the approval ratings across polls.
Not all of those things are statistically significant predictors of variance, though. The chart below shows the regression model effects with 95 percent confidence intervals. If the vertical blue bar crosses the black horizontal zero line, the effect is not statistically significant ― meaning we can’t say with 95 percent certainty that it affects Trump’s ratings. If the blue vertical bar is completely above or below the zero line, then we can say with 95 percent certainty that the aspect of the poll affects Trump’s ratings.
The most important factors are population, mode, the timing of the poll ― week of the presidency and how many days the poll was collecting data ― and whether it was conducted by Rasmussen. The only factor that is even remotely attributable to Trump’s actual job performance is the week of the presidency. The rest are about the surveys’ methods.
The model also included the sample size of the poll and the proportion of undecided responses in the poll, but both had effects near zero, so were eliminated from the chart.
A brief technical note: Daily tracking polls from Rasmussen and Gallup only appear on the chart when all of the data are new (e.g., every three days for a three-day rolling average), but this analysis includes all of their daily reports. (R code is available here. Data available here.)
Rasmussen polls are giving Trump the largest bump in his net approval, even accounting for all of the other factors in the model. That can’t be attributed to their using a different mode or population than other polls, since both factors are also included in the model. We don’t have enough information to say what is causing it, but no other pollster has a significant effect on net approval, much less an effect that large.
Polls of registered voters give Trump about a 5 to 6 percentage point boost over those that report results for all American adults. That makes sense: All Americans tend to be more liberal than registered voters. Internet polls give Trump about an 8 percentage point jump in net approval over polls conducted by telephone. And for each additional day the poll is in the field, Trump loses a little more than a percentage point in his net approval rating.
The one variable that probably is related to Trump’s actual performance as president is the timing of the poll. As Trump’s presidency has progressed over the past month, his approval has dipped. He’s lost nearly 3 percentage points in net approval each week across the first four weeks.
Since the Rasmussen effect was so large, I re-ran the model without any of those polls. Nothing changed ― no other pollster stood out in Rasmussen’s absence, and the same variables were statistically significant at very near the same values. That model explained 78 percent of the variance in Trump’s net approval ratings.
All of these effects make a great case for looking at polling aggregates to assess Trump’s approval ratings. There’s no way of knowing which set of numbers most accurately measures what Americans think of the job Trump is doing in office. Unlike election polls, which can be compared to election results, there won’t be a national vote on approving or disapproving of Trump for comparison.
The discrepancy between individual poll results is precisely why polling aggregates are useful to those without the in-depth polling expertise to discern why numbers vary and which might be right. And HuffPost Pollster’s charts allow customization and filtering for those who do have the expertise and want to look at differences by population, mode or pollster.
So for Trump’s approval ratings, keep calm and look at the polling aggregates. And maybe consider Rasmussen a Trump-friendly outlier.