A Surprise In Obama's Poll Numbers

[Updated, see below for additional information from Mark Blumenthal of Pollster.com.]

It really is a bit early to focus on President Obama's approval ratings in the polls, I know. But, rather than looking at the overall picture of how he's doing, I have been noticing something interesting which I don't believe others have picked up on -- Obama's numbers dramatically improve depending on the sample used by the pollsters. When "likely voters" (LV) are polled, the numbers they give are different from when either "registered voters" (RV) or "all adults" (A) are polled. Obama's LV approval rating is about five points lower than the RV/A numbers. The difference is more pronounced in the disapproval ratings, where LV numbers are fully ten points higher than RV/A numbers.

Is this significant? I have to admit, I don't know enough about polling to come to any conclusions I can be confident of. Perhaps I should send this column to Nate Silver over at FiveThirtyEight.com to see what he thinks. Like many people, I did a lot of poll watching over the course of the presidential election, and even wrote a series of articles with graphs to track what I thought was going on. But I am strictly an amateur at this stuff, I fully admit.

Which is why I truly don't know the explanation of this phenomenon. But I know enough to identify it, at the very least.

You can see what I'm talking about most clearly by looking at Obama's tracking graph over at Pollster.com. The graph is interactive, so you can roll your mouse over each dot in the chart to see which poll it came from. Spend fifteen or twenty seconds with this graph, and you'll see the same effect I'm talking about. There is an average line for both approval and disapproval. Check the dots (data points from individual polls) both above and below the "approval" line. Almost all the points above are listed as either "n=A" or "n=RV," and almost all the points below the line are listed as "n=LV."

Now look at the "disapproval" line and dots. The effect is easier to see here. Of all the dots above the line, only three are not LV -- two A and one RV. Of the dots under the line, only three are LV, the rest are A or RV. And the difference between their answers has stayed at a pretty steady ten points since the post-election polling started.

Meaning we really need a better graph to see what is going on. Due to my rather limited experience creating charts (which begins and ends with a few things I know how to do in Excel), I turned once again to my favorite charting blogger, Sam Minter of abulsme.com. Sam helped with the charts I used in my "Electoral Math" series tracking the Electoral College numbers throughout last year, and graciously put together the chart I needed to clearly show this effect:

[Click on the charts to see a larger image.]

This shows quite obviously that the dark red and dark green lines (which track RV/A) show a clear divergence from the light green and pink lines (which track LV). This divergence, as I said, is consistently (roughly) five points for the approval rating, and a bigger gap of ten points on the disapproval rating.

Minter then decided to put together another chart, to see if this was just a normal effect of polling. He gathered numbers for the most recent few months worth of polling on President Bush (from 7/08 to 11/08), to see if the effect is a normal thing or not. Here is his result (you'll notice the "approval" is now on the bottom, and the "disapproval" is now on the top, due to Bush's generally low poll numbers at this point).

[Click on the charts to see a larger image.]

As you can see, Bush's average lines do not really diverge much, and actually cross here and there. There is not much difference (certainly not as high as ten percent, or even five percent) between the lines.

So what does this all mean? Well, this is the point where we jump from analyzing data to wild speculation. You have been warned. We are now entering the realm of opinion, not fact.

Looking at Pollster.com's breakdown of Democratic approval ratings, Republican approval ratings, and Independent approval ratings doesn't shed much light on what is going on either. The polling samples for these sub-groups are a lot smaller (meaning a lot higher error margins), and there isn't enough data to really show any more than general trends among the three groups. It certainly looks like the disconnect has nothing to do with partisanship, but again, the samples are too small and there don't appear to be enough of them to draw any solid conclusions on this issue.

One thing to consider when looking at all the charts and data is how pollsters pigeonhole each group. If they're polling all adults, then all they have to ask is: "Are you old enough to vote?" or maybe: "Are you eligible to vote?" Likewise, asking: "Are you a registered voter?" is enough to figure that one out as well. But what, exactly, is a "likely voter"? Some polling organizations just ask you: "How likely is it that you'll vote in the next election?" with a range of vague answers such as "very likely" or "not so likely." And then some pollsters ask "Did you vote in the last election?" or even "Did you vote in the last two elections?" -- you'll notice that the second one is a tougher bar to hit, since it means you'd have had to have voted in at least one "off-year" (non-presidential) election.

But however you define "likely voters," they are always a subset of the universe of "registered voters" (which, in itself is a subset of "all adults"). And although different polling organizations have different ways of determining the likelihood of a poll respondent actually voting, the numbers are pretty consistent. No matter which pollster is asking which questions, there still seems to be a pretty solid gap (five points on approval, ten points on disapproval) between all the LV polls and all the RV/A polls.

It's gotten to the point where I can just hear the poll numbers and know what their sample was. If the approval is in the high 50s and the disapproval is around 40, the poll used likely voters. If the approval is in the low 60s (or above) and the disapproval is around 30, it is either registered voters or adults. That's as of this writing, of course -- Obama's poll numbers will change over time.

So, having discounted partisanship and polling methodology, what is causing this difference? Again, causality is always very hard (if not impossible) to prove beyond a reasonable doubt, even for statisticians, so please remember that I'm only rationally guessing here.

Sam Minter, when he sent me the charts, pointed something out which I hadn't taken into consideration -- that there is also a difference in the number of people who don't answer either "approve" or "disapprove" when asked the question:

My gut feel here is that these differences are actually somewhat tied to how "undecideds" or "non-responders" are taken into account. If you take my "average" lines for approve and disapprove and sum them at each point, for All/Registered (RV/A) you will see that the sum varies from 79.2% to 92.4% with an average of 87.5%. Meanwhile, for likely voter the sum varies from 88.6% to 99.0% with an average of 95.1%.

This means that the RV/A group has a much higher undecided rate... which could either be a real effect or just that the "likely voter" polls push harder for an answer. It actually makes sense, though, that people who are less likely to vote are more undecided, so let's assume it is a real effect.

What would more undecided people result in? Hmmm... well, at first you'd think that it would lower BOTH the approve and disapprove numbers. But what we are seeing here is lower disapprove, but HIGHER approve. So what does this get you? Maybe there is a real effect here... something along the lines of the less-likely voters being more willing to give the benefit of the doubt and give an "Approve" rating, and less likely to take a leap and say "Disapprove."

He also pointed out later that the Bush numbers were taken from the very end of his term, and by that time most people had figured out their opinion of him, negative or positive, so there were less undecideds in general. Which makes a lot of sense. Perhaps this is some sort of "honeymoon effect" for Obama, while the public figures out how it feels about him. I would need data from the beginning of a few other presidential terms to check this, but did not have time to research it to this level of depth.

And maybe the pollsters are creating the phenomenon to some extent. If, as Minter suggests, they are pushing likely voters harder to answer either "approve" or "disapprove," and are more willing to accept "undecided" from a non-likely voter, then that would indeed skew the data a bit.

I have my own hypothesis, though, and I leave it for a real statistician to look into further to prove me right or wrong. Bear with me here for some math to set it up. One thing we don't know from the chart data is the relative size of the "likely voter" group when compared to either of the other two. Remember, the LV proportion of the population will always be smaller than the RV or A population. Meaning that in a poll of RV or A, some percentage of the respondents would have been LV (as well as RV or A). Some would not -- registered voters or adults who didn't qualify for "likely voter" status. Call the subsets LV and non-LV. But what is the ratio between the two? This is an important question to be able to draw any solid conclusions.

For example, say they were even. Posit (for the sake of argument) that half of people polled in a RV or A poll are LV, and half are non-LV. We already know what a poll of LV looks like -- 40 percent disapproval. So, to get to the final RV/A disapproval number of 30 percent, all the non-LV respondents would have to have answered at a 20 percent rate (because, since the samples are half-and-half, they'd have to be averaged to come up with 30 percent).

Still following this? Without the math terms, what it means is that the non-likely voters approve of Obama at a higher rate and disapprove at a lower rate than the graph even shows. To know exactly what the numbers are for the non-LV approval and disapproval, we would need to know the exact ratio between them and the LV sample. But no matter what this ratio is, the non-LV people, in order to move the final percentages, approve of Obama a lot more strongly than the likely voter segment of the public. Or, to put it even more simply, non-likely voters love Obama a lot more than these polls can show. Meaning the difference between the two groups is even more pronounced than you see on the charts.

You can discount this by saying that non-likely voters are also much more politically apathetic, and hence draw the conclusion that they're not paying as much attention to politics now. Or, as Minter did, you can say that since they are less informed and politically apathetic, they are much more likely to give Obama the benefit of the doubt at this point -- meaning the effect should decrease steadily over time, as people do make up their minds.

But I think there's a demographic trend here which has not been identified. I think there are a lot of people who voted for Barack Obama who either hadn't voted before, or hadn't voted in a long time. Both young voters and older previously-disillusioned voters. Obama drew more people into the voting process, in other words. This was confirmed by endless polling during the campaign and isn't even really debatable now -- more people were more interested in voting in 2008 than in the recent past.

What I think is that the people in the "non-likely voter" group are strong supporters of President Obama. And that they are still supporting him in large numbers now. But because they've only voted in one recent election, or because they answer some flavor of "I don't know" when asked if they plan on voting in the 2010 midterm congressional elections, that they are dropping off the radar somewhat by being pegged as "non-likely voters" by the professional pollsters.

As I said, I don't know whether I'm right or wrong on this. But I think that there is a segment of the American public which is big enough to steadily sway the approval polling by plus five points, and the disapproval numbers by minus ten points. I don't know the size of this segment, either in real numbers or in comparison to the "likely voter" demographic. But the smaller in size this group is (when compared to "likely voters"), by definition the stronger they have to be supporting Obama (in order to have this effect on the total polling numbers). I think this segment represents the first-time voters that Obama drew to the voting booth last November. And I further think that (because this effect is flying below the radar of the media and possibly even the pollsters themselves) it represents an unnoticed source of political capital for President Obama.

But then, I could be wrong. Maybe it's time to write Nate Silver that email....

 

[Update: Mark Blumenthal of Pollster.com wrote me a nice note with a few links I'd like to share with everyone. He wrote an op-ed article in the New York Times last February (in the midst of the primary campaign, and when the New Hampshire results were still fresh) in which he complains of the problem:

Despite 22 years of experience as a Democratic pollster, I can only speculate about what might be going wrong.

Why? Because so many pollsters fail to disclose basic facts about their methods. Very few, for instance, describe how they determine likely voters. Did they select voters based on their self-reported history of voting, their knowledge of voting procedures, their professed intent to vote or interest in the campaign? Did they use actual voting history gleaned from official lists of registered voters?

Fewer still report the percentage of eligible adults that their samples of likely voters are supposed to represent. This is a crucial statistic, given the relatively low percentage of eligible adults who participate in party primaries. (In California, for example, turnout surged in 2008 but still amounted to about 30 percent of the state's eligible adults.)

And, more recently, Blumenthal writes of the problem of determining likely voters, and how Rasmussen's numbers are largely to blame for the effect (towards the end of the article):

...the fact that Rasmussen screens for "likely voters" and perhaps other, harder to detect differences stemming from their use of an automated methodology -- yield lower Obama approval scores than other pollsters. Obama's average approval percentage has been roughly 3 points lower on Rasmussen than on other polls, and his disapproval score has been roughly 14 points higher. You can easily see the difference in the disapproval scores in our chart below. There are two bands of red disapproval dots -- click on any of the higher numbers and you will see that virtually all are Rasmussen polls.

Thanks to Mark for sharing these, which help to explain the LV gap a little better.]

 

Chris Weigant blogs at: ChrisWeigant.com

 

Comments

[MJJM]

WHAT????!!­!!?????!!!­!???

[MadJeffersonian]

“Oh, people can come up with statistics to prove anything, Kent. 14% of people know that.”

- Homer J Simpson.

[army193]

What good is polls when the voting machines are in question. Wait if election seem to be a close one there is no reason to dispute the machinery.

[luvangelHussein330]

I'm not nerdy enough to read through this past the first paragraph. But one thing I'm sure about is that all of these poll and grades for a President who just made 8wks or so is absolutley pathetic. Did we have this with either Bush or clinton...­nope. A peep wasn't said until more realisticaly two years in. So how about we collectively take a chill pill and let President Obama and his team do thier job

[research]

Obama's numbers are falling because

He has not stopped Swaps and Shorting.

He is letting the Bankster win, and the entire world lose bad.

[foxfan]

His poll numbers were too high to begin with. Even before he took office his approval numbers were around 80%. Once in office and actually making decisions I expected his numbers to drop. Short of waving a magic wand to fix everybody's problems in a day, there is no way he could have maintained that high an approval rating.

[MacQ]

His poll numbers are average for a president in office 8 weeks. That is troubling, given the adoration that he rode in on.

And, his numbers are falling.
I fully expect the race issue to surface soon. During his campaign, he introduced it every time he was slipping in the polls (did THEY mention I'm black?). No one did, and no one had, ever. Not his opponent.
Watch for it!

[Principaldad]

"His poll numbers are average for a president in office 8 weeks. That is troubling, given the adoration that he rode in on."

Actually, that is absolutely amazing given the current state of the economy.an­d rocketing unemployment. Also, his numbers are currently ABOVE average for recent presidents. In fact, you have to go back to Carter to get higher overall numbers. The highly partisan trend in recent years has caused initial presidential approval ratings to be much lower. Obama is still bucking that trend. This in spite of a massive, unprecedented, and often dishonest media blitz by Republicans aimed solely at undercutting Obama's credibility.

What makes Obama's overall averages even more impressive is that they include polls like Rasmussen which didn't exist years ago. Rasmussen consistently and intentionally slants to the conservative side. The only reason that real pollsters look at Rasmussen is that they are consistent and conduct polls on a regular basis. Analysts also take into account the conservative bias when they look at Rasmussen. However, ther Rasmussen numbers do bring down the average.

[LiquidJak]

Everyone, except those who remain bitter, acknowledge Obama did everything possible to keep race out of the equation. Please give an example to prove me wrong?
I guarantee you won't see the President bring up race at all.
I suggest you grow up.

[TheLearnedEnglishDog]

Do you live on planet earth?

[PlayTOE]

All of the pollsters are lumped into the same chart, but don't use the same standards of methods. This gives a picture you can cut logs with.

If we take each polling company and chart their results in a color coded way, with separated colors for each pollster, we may see trends far more quickly.

[Chris Weigant - Huffpost Blogger]

PlayTOE -

Whew! See comment within article about "my limited knowledge of charting".­..

There was one site which did chart polls in this fashion during the election -- you could tell by looking at the charts (even state-by-state) which poll said what -- but they don't seem to be doing anything with the site right now. If I find anyone doing this, I will point it out in the future.

-CW

[IndieBlue]

The numbers tend to be lower for LVs, as LVs are less engaged than RVs (hence the LV status). People are less engaged, typically, because they have a negative opinion of politicians in general.

[Principaldad]

IndieBlue, I am confused by your post. Likely voters ( LVs) are a subset of registered voters (RVs) who are considered most likely to vote. You can't be an LV unless you are an RV. As such, LVs are MORE engaged in the political process.

[letsbepeaceful]

That's true only if the likely voters ARE actually a subset.

It's entirely possible, if the question is, "are you likely to vote in the next election?" for a voter not to be registered now, but plan to register and vote.

It's hard to figure out polls unless you know EXACTLY how and who they were polling, and what the questions were.

[DoTheMath]

Chris Weigant,

Your math, your analysis, and Mark Blumenthal's comments reinforce my suspicion that the "likely voter" category has become a tool for subjective manipulation of a process that is supposedly objective and scientific. Polls don't just reflect opinions; they influence opinions. Shaving points off of President Obama's popularity by selecting an ill-defined sub-category of poll responses to report has political consequences. People use poll results to bolster their arguments about the level of popular support for the president's policies.

In my opinion, President Obama's policies would benefit the vast majority of Americans; therefore, a minority of Americans view those same policies as a threat to their disproportionate wealth and power. Given my opinion, the idea that a decisive majority of Americans would support the president and that a powerful minority would seek to muffle that decisive majority support is completely logical.

Where do the pollsters fit in, in terms of motivation? I don't know, but if I wanted to investigate, I would start by finding out who their customers are. After all, Standard and Poor, Fitch, and Moody all gave high ratings to subprime lenders even though, according to their own standards, they had no basis for rating these "innovative products" at all. Their customers are the institutions they rate.

[Principaldad]

First I should ask, where do you get the determination that this is not likely a partisan discrepancy? The data you linked shows only one poll of likely voters (LV). For that poll the Rep. and Ind. results are skewed negative compared to other polls from the same time. For the Dem. results, though, there is little difference. This would seem to counter your own argument (although a single poll is not definitive).

In reality, the trend you describe is likely due to a combination of reasons. As Blumenthal points out, Rasmussen results skew the numbers. As someone else pointed out, the young Obama supporters may not show up as likely voters. Also, there is the reality that people are more likely to identify as LV if they are dissatisfied. Other factors may be in play as well.

All of this helps to expose our overreliance on polls. Polls provide very useful information, but that information is easily misused. For example, comparing polls from different organizations is a very tricky business, since minor differences in questions or methodology can skew numbers. We also accept the margin of error too easily. When we hear that the MOE is 3% we assume the results can only vary 3%. In reality, that is the 95% confidence point. That means 1 out of 20 polls will be OUTSIDE the 3% MOE. In the end, polls are best used to determine trends rather that to identify exact percentages.

[Chris Weigant - Huffpost Blogger]

Principaldad -

You're largely right -- I hadn't identified the Rasmussen polls as being the large influence it is (there are a few other LV polls, but not many), which is why I asked Mark if I could include his links to point it out to everyone.

And I didn't have much of a chance to interpret the partisan breakdown polls, as they had just appeared on Pollster.com the day I was writing the article. So I admit, I just took a quick look at them and discounted them. I may have been hasty in doing so, and if so, I apologize. The one thing I did note was the small size of the partisan samples (only a few hundred) which should raise the MOE to at least 5% if not more.

-CW

[Whatashame]

Amount DEMS Obama aprroval rating is 94%.

[realpolitic]

Yes, the numbers are interesting, but Obama's fate likely rides with the economy. If the economy is improving and Americans feel positive, I think Obama has the oratorical skill to defeat almost any opponent. However, his numbers do matter for what he gets passed in Congress, so it is a good point!

[Dovespeace]

My response is just plain simple. It isn't about the poll. It is about reality. President Obama inherited a mess from hell on all fronts. He has been in office a little over fifty days. For eight years we were manipulated by lies from our White House and the majority held Senate and House. We were raped of so much of what made America great. We were manipulated by main stream media who fell in cadence with Bush and his cohorts. They preyed on our vulnerability after 9/11. What they did was criminal as is coming to light more and more. President Obama is moving with change and transparency to the people. It feels that one of the greatest battles we are fighting is here at home between what has been and a new light at the end of the tunnel. The party of "NO" would have us fail. If we cannot move with change what will happen to us? If we buy into the constant smear tactics of the GOP what will happen to us? We all have eyes to see and hearts to feel the truth. Now is not a time for polls. Now is the time of great opportunity for us to stand strong for our beloved country, ourselves, and each other. As long as we are filled with hate and fighting against one another we will be doomed. There are those who are depending on it.

[AnnfromCA]

I believe that LV numbers are often cited as the reason one poll or another was way off, particularly in elections.

[Chris Weigant - Huffpost Blogger]

AnnfromCA -

It's an interesting phenomenon, because polling LV in the first place was supposed to make the election polling better. By just polling LV, it is supposed to weed out the people who (a) have an opinion, but (b) don't vote. In normal years, excluding them does make the poll better predict election results (that's the theory, anyways), but Obama proved you can indeed bring others into the process.

But those are polls done to predict the election. These are "public opinion" approval/d­isapproval polls. Which raises a very good question: shouldn't ALL the public's opinion count, or should just the opinion of those who vote count, in such a poll?

I'm not being sarcastic, your comment made me think of this and I think it truly is the key question here.

-CW