03/27/2009 04:48 pm ET | Updated May 25, 2011

About Those Conflicting PA Polls

The contradiction between two polls on next year's Pennsylvania Republican senate primary in (conducted by Quinnipiac University and Franklin and Marshall) is the sort of thing I usually write about. Fortunately, in the midst of a week of light blogging, The Washington Post's Jennifer Agiesta Jon Cohen provides a very helpful analysis:

The two surveys offer a similar read on Specter's standing in the hypothetical contest: about three in 10 registered Republicans said they would vote for him in both polls. Specter is the better known figure and, as the incumbent, the one on whom voters are passing judgment at this early stage, so it makes sense that his standing would hold steady across polls.

The variation is all on Toomey's side of the coin. Despite his prior run for Specter's seat, Toomey is largely an unknown quantity - nearly three-quarters of Republicans in the Quinnipiac poll said they hadn't heard enough about him to have an opinion - suggesting his support in the two polls is driven more by question wording than strong sentiment for or against his candidacy.

So why did Toomey get 41% on the Quinnipiac survey but only 18% on Franklin and Marshall's poll? The most likely reason, Agiesta Cohen argues, is that the latter included the option to say, "or aren't you sure how you would vote."

That final phrase makes all the difference. Quinnipiac's poll offers voters what we in polling call a forced choice, mimicking the one they would face in the voting booth (unlike Nevada, Pennsylvania does not allow voters to cast a ballot for "none of these candidates"), and as such is a great measure for predicting voter behavior. Franklin and Marshall's question makes it easier for voters to say they haven't made up their minds, a perfectly valid response more than a year before anyone has to cast a ballot.

Some have made much of the fact that the Franklin and Marshall poll had a relatively small sample of Republicans (n=211) than the Quinnipiac poll (n=423). Yes, a smaller sample allows for the potential for more random sampling error than a larger sample, but this is the one sort of potential survey error we can quantify. In this case, the so-called margin of error is +/- 4.8% on the Quinnipiac Republican sample and +/- 6.7% on the Franklin and Marshall Republican sample. A bigger potential for error, to be sure, but the notion that anything less than 600 (or some other arbitrary number) is unreliable is a myth.