05/12/2010 02:55 pm ET Updated May 25, 2011

On Small Samples and Statistical Noise

Three new polls out this morning on Pennsylvania's Senate primary all lead to the same conclusion: The race between Senator Arlen Specter and Congressman Joe Sestak has closed dramatically in the last month and the three most recent snapshots of the race suggest a very, very close contest. That much is crystal clear.

Maybe I'm feeling grumpy today, but aside from that obvious big trend -- the movement toward Sestak over the past month -- I'm seeing a lot of web chatter explaining day-today patterns that are mostly statistical nose. So I want to pass along a few warnings and a thought or two about the numbers before us today.

The latest Muhlenberg College/Morning Call track, based on interviews conducted Saturday through Tuesday, shows a dead even result (45% to 45%). The new survey from Quinnipiac University, conducted from last Wednesday through Monday night has Specter "ahead" by a two-point margin (44% to 42%) that falls well within the survey's +/- 3% margin of sampling error. And among just 150 likely Democratic primary voters surveyed by Franklin & Marshall College from Monday last week through Sunday, Sestak has a statistically meaningless two-point advantage (38% to 36%).

Two quick thoughts about the Franklin and Marshall poll: I have mixed feelings about their decision to release results for just 150 likely Democratic primary voters. On the one hand, there is nothing inherently wrong with a sample of 150. It just has a larger margin of random sampling error (+/- 8%) than a sample of 400 (+/- 5%) or 1,000 (+/- 3%). Does crossing the threshold from +/-5% to +/-8% take us over some magic line? I don't think so.

On the other hand, I do not understand the logic of sending out an email to journalists touting the 38% to 36% result, with a 31-page report attached, yet omitting from both documents the undecided percentage on the vote question likely voters, the results when undecided likely voters were asked how they lean and the fact that they interviewed just 150 likely voters (though the report does include the +/- 7.9% margin of error for that subgroup). We got the n=150 number by reading the poll article in the Scranton Times Tribune.

Even more curious is the decision to include a set of cross-tabulations in the report (Table A-1 on page 9) that appears to break out those 150 interviews into even smaller demographic subgroups. How many interviews were conducted among typically smaller subgroups such as the non-whites, veterans or 18-34-year-olds among the likely voters? That they don't say.

So much for transparency.

As long as we're focused on sample sizes, I also want to pass along a tip for those of you who, like me, have gotten addicted to the daily tracking fix from Muhlenberg University. Their rolling-average design has important advantages, but one often overlooked drawback: We tend to get seduced by the false sense of precision inherent in those smoothed out daily numbers. In tables like the one below (reproduced from the most recent Muhlenberg release) they present something of an optical illusion. Do you notice a few apparent "trends" that last, oh, about three or four days before receding? We notice that smooth movement and forget that its an artifact of the rolling averages. More importantly we also forget that +/- 5% margin of error applies separately to every number in the table.


Let's consider the same data a little differently and look just at the non-overlapping samples (as in the table below). With the margin of error in mind, most of the differences between the three columns are indistinguishable from statistical noise. The only clear trend is the 10-point increase in Joe Sestak's favorable rating over the 12 day span of the tracking. Yes, it's tempting to read great meaning into the small fall and rise in Specter's vote and favorable percentages in the table, but neither change is anywhere close to statistically meaningful.


(I'm most struck by the lack of trend in Sestak's low UNfavorable rating -- remarkable given that the Specter campaign has reportedly been pounding Sestak with attack ads for the last two weeks. But I digress...)

So what do we know? There is no question that the most recent results -- from Muhlenberg, Quinnipiac and even Franklin & Marshall -- represent a big change from where the race stood just a few weeks ago, but any real trends over the last few days are just too small to be visible amidst the statistical noise.

Update - A reader emails:

I also liked this part [of the F&M report, p. 4, emphasis added]:

The May Franklin and Marshall College Poll shows Joe Sestak with a narrow
advantage over incumbent Senator Arlen Specter among those Democrats who are
most likely to vote, 38% to 36%, with about one in four likely voters still undecided.
When undecided voters who are leaning toward a candidate are allocated, the pool of
truly undecided voters is about 15%

So they push leaners to get it down to 15% undecided, but they never tell you how those leaners break? Also, they list RV and LV results for the Republican Governor primary matchup, but they never tell you the LV sample size anywhere

[Typos and grammar corrected in original post].