WASHINGTON -- Is the presidential race between Barack Obama and Mitt Romney shaping up as a "virtual tie" in Tennessee? Is the likely Republican nominee leading the U.S. Senate race in North Dakota?
Headlines making both assertions have appeared in the last week, but both depend on reading too much into poll results based on populations being sampled either too broadly or too narrowly.
In Tennessee, a poll of adults sponsored by the Nashville Tennessean and conducted by Vanderbilt University's Center for the Study of Democratic Institutions showed President Barack Obama running just a single percentage point behind former Massachusetts Gov. Mitt Romney (41 percent to 42 percent). The result for adults led the Tennessean poll story and the headline chosen matched that tally: "Obama closes gap with Romney." An Obama fundraiser quoted in the story proclaimed Tennessee to be "a toss-up state."
Less prominently, the Tennessean story also included some far more relevant results, however. Among the self-identified registered voters interviewed for the poll, Romney led Obama by a statistically meaningful 7 percentage point margin (or 47 to 40 percent). The article also reported that John Geer, a Vanderbilt political science professor who helped direct the poll, considered Romney's lead among registered voters the "more likely outcome in November."
Geer was right to focus more on the registered voter subgroup, as those results are a far better gauge of the potential electorate six months before an election than a cluster of all adults. As the National Council on Public Polls argued in its "20 Questions A Journalist Should Ask About Poll Results," the people chosen for the base of calculations is critical. The council recommended looking for pre-election results based on registered or "likely" voters rather than all adults.
Otherwise, the Tennessean poll had no obvious methodological issues. It was conducted by a highly respected call center, reached subjects on land line and mobile phone numbers and was overseen by a bipartisan advisory board. And the results among registered voters mirrored those of a poll conducted in February by Middle Tennessee State University that showed Romney leading Obama by 6 percentage points (47 percent to 41 percent).
Of course, both polls suggest a closer race in Tennessee than the 15 percentage point margin by which John McCain defeated Obama there in 2008, but both new polls assign Obama about the same percentage of the vote (40 percent to 41 percent) that he received in Tennessee four years ago.
In North Dakota, a poll by the Fargo-Moorhead Forum on May 17 showed Republican Rep. Rick Berg leading Democrat Heidi Heitkamp by 7 percentage points (51 percent to 44 percent), but Democratic critics charged that this poll had too narrow a sample.
In this case the poll, conducted by Iowa based Essman/Research, screened for "likely voters" but for the wrong election. The poll of 500 likely primary voters had been designed to measure preferences in North Dakota's June 12 primary, not the November general election, but included a question about the likely general election contest between Berg and Heitkamp.
Democrats immediately criticized the survey. The Heitkamp campaign put out a statement, calling out what it described as the poll's "deeply flawed methods." Via Twitter, Heitkamp pollster Mark Mellman attacked the poll and argued that "ethics require retraction."
The Forum's reaction was largely defensive. On May 18, it published an article about Democratic criticism of the poll, quoting several political handicappers about whether the results seemed right (most of them did), but largely ducking the substance of the criticism. Another article the next day reviewed the history of Forum polls, stating that they "have often hit their mark. " Finally, on May 19, Forum editor Matthew Von Pinnon published an editorial bemoaning the partisan attacks, standing behind the poll and asserting with confidence that it "was conducted fairly and independently."
Von Pinnon's editorial also addressed the substantive argument that the poll had sampled only likely voters for a primary election in which "Republicans have two contested federal primary races and Democrats have none." That sampling would skew "the political profile of the poll respondents," he wrote, summarizing the Democrats' argument, since "more Republicans are likely to vote June 12."
But Von Pinnon rejected that theory, claiming that the "nonpartisan measures" on the June 12 ballot will make turnout "similar to a nonpresidential general election." The ballot measures have North Dakota election officials "bracing for a record primary turnout," one that's "similar, in fact, to a general election," Von Pinnon wrote.
There are two problems with that argument. First, Heitkamp and Berg will face the voters later this year, in a presidential general election, not in an off year. Second, it would take quite a record turnout in the primary to match the 321,133 voters who voted in the 2008 general election. Primary elections in North Dakota typically draw about only a third as many voters. According to election statistics published by the North Dakota Secretary of State, the high since 1980 was 146,867 in the 1992 primary election. Over the last 10 years, primary turnouts have ranged from a low of 92,066 to a high of 128,519.
The issue is not whether the poll was conducted fairly or independently or whether handicappers think the results seem right, but about whether the poll sample should be characterized as representative of the general electorate. Von Pinnon argued that it is "reasonable" to think that the answers provided by primary voters "might mirror those of the general electorate." They might.
But without measuring both -- as the Vanderbilt poll did with adults and registered voters in Tennessee -- how would we know for sure?
CORRECTION: An earlier version of this story misspelled the name of Vanderbilt University.