01/19/2010 01:45 am ET Updated May 25, 2011

Do Polls Underestimate Dems in Blue States?

This is the first of two posts that will wrap up my thoughts about the Massachusetts Senate race. This first one covers an admittedly narrow point. It is provoked by an email I received reacting to Nate Silver's post from Sunday -- "A Statistical Ray of Hope for Coakley" -- from Pollster reader Harry Enten (aka commenter Poughies):

[Silver's] piece makes the point that the margin between Martha Coakley and Steve Brown could be overstating Brown's lead. Silver points out that polling in close (margins of 10 or less in the polls) Senate elections since 2000 in deeply blue states (as measured by the Cook Political Partisan Index) has by an average of 3.4 points underestimated Democratic candidates' margin of victories. In deeply red states, on the other hand, polling has underestimated Republican margins by 1.9 points.

Though he used a average in 2008 and Real Clear Politics average in 2006 and 2004, Silver used a simple average of all non-partisan polls conducted in the final two weeks compiled by in 2002 and 2000. Perhaps, he did not know, but a Real Clear Politics average is also available for 2002 and 2000.

I was interested what, if any, effect substituting the Real Clear Politics averages in 2002 and 2000 would have on Silver's results. Therefore, I decided to create a new dataset modeled after Silver's, but using the data in 2008 and Real Clear Politics average* in 2006, 2004, 2002, and 2000.

Using these new rules, the underestimation of Democratic margins in blue states stays the same at 3.4 points. This result is not surprising, as the blue states part of the dataset is small, and only three results are available from 2000-2002. The underestimation of Republican margins in red states drops from 1.9 to 0.9 points.

The overall average of underestimation drops from 2.3 to 1.5 points. In only 7 of 23 contests was the underestimation above 3 points. In no contest did the polling average incorrectly predict the winner due to underestimation of Democratic candidates in blue states or Republican candidates in red states. The only incorrect winner chosen was in the South Dakota (a red state) 2002 race when the polls predicted a victory by Republican John Thune.

The bottom line is that perhaps the polls are overestimating Brown's margin in Massachusetts. The limited data involving closely polled elections in blue states suggest that Coakley might do better than the polls suggest, but when you look at the larger dataset of red states, Coakley should not expect a bump.

*In some cases (such as Alaska 2004), no Real Clear Politics "average" existed. I just averaged all the polls listed on Real Clear Politics (and in the case of Louisiana 2002) conducted in the final two weeks. I, unlike, Silver use internal polls in these cases... modeling myself after's inclusion of them.

P.S. The Real Clear Politics web pages can be hard to find [but are available at these links] for 2002 and 2000.

He also passed along a table (in Excel and PDF formats) with the numbers.

It's also worth taking a closer look at the six races -- there were only six -- that are the basis for the conclusion that polls understate support for Democrats:


Most interesting are the three races (Maryland and Rhode Island in 2006 and New York in 2000) that produced the biggest errors. Of these, the Rhode Island example is partly the result of an RCP average based on just two surveys conducted over the final weekend of the campaign. We used a simple last-5-poll average for Rhode Island that year, which showed Whitehouse winning by six, just one point off the actual margin. Just swapping the Pollster and RCP averages for that one race would cut the average variance to +2.5. Whatever the challenges of polling in Massachusetts, greater random error due to a shortage of final polls is not one of them.

That leaves two big errors affecting Ben Cardin in Maryland in 2006 and Hillary Clinton in New York in 2000. What's interesting about the errors in both contests was that the final round of polls had the percentage for the Republican candidate about right (within a point), but understated the support for the Democrat by about five percentage points. Even if we assume that a similar pattern will apply in Massachusetts tomorrow, the problem is that the six of the last seven surveys estimate Brown's support at 51% or greater.