Few things are more perennial in the survey world than partisan attacks on high profile polls that produce a result that partisans don't like. More often than not, those attacks involve the issue of party identification. One of my first posts as a blogger five years ago involved an argument about party weighting (Democrats thought the polls were skewed too Republican) and a subject has come back like crab grass ever since. Yesterday, the object was the ABC News/Washington Post poll, and the complaints came from a Republican pollster and former Republican Speaker of the House Newt Gingrich.
The complaint from Gingrich drew a response worth reading from ABC News polling director Gary Langer, so I'll start there. The tempest was probably triggered by findings highlighted in the last two paragraphs of the Washington Post's front page poll story:
Only 20 percent of adults identify themselves as Republicans, little changed in recent months, but still the lowest single number in Post-ABC polls since 1983. Political independents continue to make up the largest group, at 42 percent of respondents; 33 percent call themselves Democrats.
The wide gap in partisan leanings and the lack of confidence in the GOP carries into early assessments of the November 2010 midterm elections: Fifty-one percent say they would back the Democratic candidate in their congressional district if the elections were held now, while 39 percent would vote for the Republican. Independents split 45 percent for the Democrat, 41 percent for the Republican.
Gingrich responded by telling a Salt Lake City radio station that he considered the ABC/Post poll "deliberately rigged." According to Langer, when asked about the finding that only 20% of Americans consider themselves Republican, Gingrich replied:
Well, it tells me first of all that the poll's almost certainly wrong. It's fundamentally different from Rasmussen. It's fundamentally different from Zogby. It's fundamentally different from Gallup. It's a typical Washington Post effort to slant the world in favor of liberal Democrats.
Langer produced the following table, showing that the ABC/Post estimate falls roughly in the middle of other recent national surveys of adults:
Now as both Langer and PPP's Tom Jensen point out, Gingrich was conflating two issues: Concerns about party identification and about the so-called generic house vote. In the same radio interview, for example, he pointed to a recent Gallup poll showing a closer margin in the national House vote and, according to Langer, emphasized the generic vote in a second interview later in the day,
Here Gingrich is on somewhat firmer ground: The polls that he cites -- especially Rasmussen and Zogby -- report on "likely voters" that tend to look more Republican than the adults sampled by other pollsters. While neither pollster provides much information on how they currently define likely voters, the universe of actual voters in 2010 will likely be far more Republican than the full adult population. Keep in mind, though, that the generic ballot is a blunt instrument that tends to produce wide variation among pollsters even in the week before the election when all are doing their best to identify or model the likely electorate. Bottom line: While the generic ballot is a useful measure (see these posts on efforts to use it to model the outcome) it is not infallible.
Back to the argument about party ID. Why does Langer leave out Rasmussen and Zogby, two of the three pollsters that Gingrich cites as looking fundamentally different? The implied answer is that ABC News considers neither pollster "air-worthy" (they explain their standards here), but even if you quarrel with that judgement, there are good reasons why such a comparison would be foolish.
Let's start with Zogby. First, Zogby reports results among "likely voters," not adults. Second, they do not -- as far as I know -- report their party ID results to non-subscribers (if readers can point to recent examples to the contrary, or to results from behind their subscription wall, I will gladly correct this post). Third, unlike the organizations listed in Langer's table, Zogby weights every survey by party ID, usually to match the estimate from exit polls in a prior election. So there is no point in comparing Zogby's weighted party ID numbers to those from recent polls of adults, even if they were available. It's like comparing an apple to fruit cocktail.
And what about Rasmussen? Like Zogby, Rasmussen typically reports results among likely voters and weights by party. However, Rasmussen does a service by routinely releasing their monthly party identification numbers among all adults, weighted only by demographics. Their results for September do look a lot different from any of the other pollsters: 37.5% Democrat, 32.1% Republican, 30.4% other.
Why might Rasmussen's party ID results look so different? It might be because of the kinds of people they sample as compared to other polls, but there are two other huge differences to consider. Rasmussen calls with a different mode (automated rather than live interviewer) and asks a different question. Other pollsters begin by asking respondents what they "consider" themselves to be, prefaced by the phrase "generally speaking" or "in politics, as of today," with the options typically Republican, Democrat, independent or "something else." Rasmussen simply asks:
If you are a Republican, press 1. If a Democrat, press 2. If you belong to some other political party, press 3. If you are independent, press 4. If you are not sure, press 5
If you believe that party ID is like eye color, that we are all either Democrats, Republicans or something else and that we will always provide the same answer under any circumstances, even if shaken awake during a deep sleep, well...it probably doesn't matter how the pollster measures it. But there is a ton of evidence that although the aggregate party ID numbers change very, very slowly, at the individual level all sorts of things can alter the answers that respondents give, especially if they are borderline between independence and identifying with a party: the wording, when the question is asked, what questions come before, how hard the interviewer pushes for an answer, and so on. So it is quite possible that people are willing to report their party identification differently when asked by an automated recording rather than a live interviewer, especially when the text of the questions differ. Comparisons between Rasmussen and other pollsters on this score prove little.
Was the ABC/Post poll right that Republican identification among adults is lower than ever? There we do have some pretty convincing evidence. As our Party ID chart (above) shows, their result was far from an outlier. Even with the sensitivity set to low (to diminish short term statistical noise from "house effects), the downward trend in both Democratic and Republican identification is evident. Click on the red dots below (to connect-the-dots for individual pollsters) and you will see that almost every pollster shows a decline in Republican identification over the course. The clear rise in independent identification may not help us forecast the results of the 2010 elections, but the trend is nevertheless undeniable. The notion that ABC and the Post "rigged" their result is laughable.