UPDATED BELOW: MSNBC tells The Huffington Post its "suburban women" results fall "within the margin of error."
***
Patience is not typically regarded as a virtue in political reporting. Next to accuracy, being first is king. But sometimes the two goals are in tension, as shown by this week's rush to make sense of Sen. Clinton's exit from the presidential field. Over the last few days, pollsters have been racing to gauge the first reactions among women voters to the all-male general election match-up between John McCain and Barack Obama.
Gallup hit the news cycle first on Wednesday, announcing their discovery of an eight-point swing among women toward the Illinois Senator since Clinton left the stage. Then NBC and the Wall Street Journal came out with joint polling results that were broadly similar, but with a caveat that represented a potentially troublesome hole in Obama's female support.
The NBC-WSJ poll showed that while Obama had increased his lead among women overall (52-33), "suburban women" still favored McCain by six points, 44-38, while a hypothetical Clinton candidacy would beat the Arizona Republican.
Since that poll was released Wednesday, MSNBC has been reporting the "suburban women" finding often, sometimes hour-by-hour -- perhaps because it reinforces residual doubts about Obama's viability in the 'burbs. The question led Chris Matthews' "Hardball" program at 5pm on Wednesday with a graphic that read "Woman Trouble?"
But how solid was the NBC-WSJ poll's conclusion about those voters? With only 1,000 total respondents in the poll, and no guarantee that the sub-group of "suburban women" was balanced nationally -- meaning that these suburban women polled were drawn from a balanced cross section from America's vast suburbia -- a group of polling experts from across the ideological spectrum told The Huffington Post they viewed the findings with some suspicion.
"I am skeptical about results for smaller subgroups like 'suburban white women,'" said Emory University Professor of Political Science Alan Abramowitz. "There is more random 'noise' with smaller subgroups. How many of these 'suburban white women' were there in the NBC poll out of the 1,000 total registered voters? Figure about 750-800 whites, close to 400 white women, so maybe 150-200 suburban white women. A shift of a small number of voters would change the outcome."
Republican pollster Tony Fabrizio, who you might expect to celebrate the poll, also cast doubt on its findings regarding a preference among suburban women for McCain -- if for no other reason than that any one poll's margin of error increases as the reduced sample size of a "subgroup" becomes smaller and smaller. "If you have a thousand samples, maybe your margin of error for your overall sample is 3.1 percent [the margin cited in the NBC-WSJ poll]." Assuming approximately half of the poll's respondents were women, he said, "your margin of error [when considering them alone] goes up to 4.5 percent. Then take females and segment them among rural, urban and suburban [subgroups]. ... You've doubled your margin of error in that group." [See update below: MSNBC revealed the margin of error tripled for its "suburban women" subgroup.] Fabrizio also said that disrupting the national distribution of a sample by looking at subgroups can throw any analysis out of whack by the same proportion, adding, "there are other vagaries that can go on to influence the sub-sample, too."
Democratic pollster Anna Greenberg agreed with Fabrizio, saying "I'm not sure I believe" the NBC-WSJ numbers on suburban women. She also said it was "bizarre" to single out the suburban women numbers as a statistically significant finding, given the more robust evidence that Obama was competitive with McCain among white women overall. "They missed the big story among women, in my view," she said. "It's not like white women are a 'gimme' for Democrats. Obama is doing significantly better among college-educated white women [than Sen. John Kerry in 2004]."
Republican polling magnate Frank Luntz also predicted the real breakdown among women voters would prove to be age, not location. "Obama will benefit from the usual Democratic advantage among women -- with one caveat," he said. "He is not as likely to win over older women compared to the traditional Democratic presidential candidate. His age, his inexperience and, frankly, his apparent association with people outside the mainstream will definitely frighten older women who would normally support the Democratic nominee. Even the crowds at his rallies raise questions. The aspects of his life and his language that is so appealing to younger women simply does not generate a similar reaction among older women. However, he will win an unusually large share of the younger female vote -- a very potent voting bloc in this election."
In the published 33-page breakdown of the NBC-WSJ poll's results linked to on Pollster.com, the following phrase appears on the front page: "NOTE: The results contained in this document reflect results among the national crossection of voters ONLY." Given that the "suburban women" findings are not included in that document, it appears the Hart and Newhouse polling firms, which conducted the poll for the two media organizations, recognized that its subgroup findings on "suburban women" were not representative enough to include in the breakdown. So far, that hasn't stopped MSNBC from turning those numbers into big news. Emails to NBC's political unit asking for a numeric breakdown of "suburban women" in their poll were not immediately returned.
UPDATE:
MSNBC has now provided The Huffington Post with more information on its "suburban women" finding showing a 44-38 McCain lead over Obama. "This is within the margin of error of 9.34 percent based on a sample size of 110 within the larger poll," an MSNBC source wrote over email. (That's three times the margin of error for the entire poll.) This means McCain's 44 percent figure of support among suburban women could actually be as low as 35 percent, while Obama's 38 percent figure could rise as high as 47 percent -- assuming a 95 percent confidence interval (for the stat wonks in the house). Alternatively, McCain could be leading Obama 53-29. While those distant outcomes are less likely true than NBC-WSJ's 44-38 finding, that broad variance raises questions about the statistical usefulness of this one particular crosstab, as opposed to the rest of the NBC-WSJ poll on the whole.* [Added later]
SECOND UPDATE:
NBC Political Director Chuck Todd defends the use of the "suburban women" crosstab in an email response:
Here's what I can tell you on our crosstabs. We never use one unless we have more than 100 incidents; in this case, the incident rate is approx. 140 interviews. It's always easy to shoot a pollster but the track record of the NBC-WSJ poll is undeniable; campaign operatives on both sides of the aisle believe it's the gold standard for media polls.