It is time -- actually long past time -- to summarize the returns from the Pollster.com "Disclosure Project." Back in September I declared my intent to request disclosure of key methodological details from pollsters doing surveys in Iowa, New Hampshire, South Carolina and the nation as a whole. I sent off the first batch of requests to the Iowa pollsters, and then began a long slog, delayed both by other activity and, frankly, by a surprising degree of resistance from far too many pollsters. The result is that now, nearly three months later, I can report results from Iowa only.
I should note that many organizations (particularly ABC/Washington Post, CBS/New York Times, Los Angeles Times/Bloomberg, the Pew Research Center, Rasmussen Reports and Time/SRBI) either put much of the information into the public domain or responded within days (or hours) to my requests. With others, however, the responses were slower, incomplete or both. A few asked for more time or assured me that responses were imminent, yet ultimately never responded despite repeated requests. Sadly, such is the state of disclosure in my profession, even upon request.
So while the results described below are far from a complete review of all the polls in Iowa, they do tell a very clear story: No two Iowa pollsters select "likely caucus goers" in the same way. Moreover, each pollster has a unique conception -- sometimes radically unique -- of the likely electorate.
This post is a bit long, so it continues after the jump...
I am going to start with some of the wonkier design decisions that pollsters make about their surveys, and work progressively to more intuitive information: How these mechanical differences affect the kinds of people sampled.
Sample - One of the first decisions an Iowa pollster has to make is about how to draw the sample, and as the table below shows, Iowa pollsters diverge on their preferred method. Six start by sampling from the universe of all working landline telephone numbers in Iowa using a random-digit-dial (RDD) method, four sample from the lists of registered voters compiled by the Iowa Secretary of State, one (University of Iowa) samples individuals in listed telephone directories and one (CBS/New York Times) uses a hybrid approach that combines both registration list and RDD sampling (see my post from July about the trade-offs between RDD and list sampling last summer).
Another issue is whether the pollster makes any special effort to reach voters with mobile phones who live in households without a landline phone. Two pollsters (Register/Selzer and Time/SRBI) identified cell phone numbers on the registered voter lists and dialed them separately. The CBS/New York Times poll drew a special supplemental sample of cell phone numbers and dialed it separately.
Finally, most of the Iowa pollsters use live interviewers, but two -- Rasmussen Reports and Public Policy Polling (PPP) -- use the "interactive voice response" (IVR) method that calls with a recorded message that asks respondents to answer questions by pressing on their touch tone phones.
Selecting Likely Caucus Goers - After choosing a sampling method, the pollster must decide how to select likely caucus goers from their sample. That task is not easy. The historical high for turnout in the Iowa Caucuses was 5.5% of adults for the Democrats in 2004 and 5.3% of adults for the Republicans in 1988. That means, in theory at least, that a pollster starting with an RDD sample would need to screen out nine out of ten otherwise willing adults in order to interview a combined population of Democratic and Republican caucus goers strictly comparable in size to past caucus turnouts. This daunting task is one big reason why polling for the Iowa caucuses is so challenging and why pollsters differ so much in their sampling methods.
Those that sample from lists do so partly so they can more accurately and efficiently reach known registered voters (although caucus rules in both parties allow for walk-in registration, so list-based polls will miss the small percentage that register on caucus night or just before caucuses). One pollster (Iowa State University) inexplicably samples only from registered Democrats and Republicans, throwing out registered independents (or more accurately those with "no party") even though caucus rules allow independent to participate by changing their registration status on caucus night.
A few pollsters (such as PPP, Garin/Hart and McLaughlin and Associates) use the actual vote history on the registered voter list to narrow the sample universe before dialing. In other words, the registered voter list includes a record -- for each individual voter -- of whether they voted in each recent election. So pollsters can choose to select, say, only those who have voted in a Democratic primary election in the last four years (or any other permutation you can think of). The internal polls conducted by the campaigns (and the two public polls conducted for the One campaign by Peter Hart Research and McLaughlin and Associates) typically make use of such vote history data for the past caucuses. The Democratic vote history is only available to those willing to pay a reportedly high fee to the Iowa Democratic Party, and none of the public pollsters has made use of it.
However, for the heavy lifting of screening for likely caucus goers most of the public pollsters rely on self-reports about intent to participate. I have reproduced the pollsters' descriptions of their screening questions (and, in many cases, the verbatim text) at the end of this post. No two ask exactly the same questions, but most pose a scale that includes options like "definitely attend" and "probably attend," using those categories to classify respondents as likely caucus goers.
The CBS/New York Times survey has so far been the only one to use a more complex model than simply screening based on vote history or self-reported intent to participate. That poll uses a variant of their classic likely voter model that weights respondents based on their probability of voting, using their past voting behavior (presumably from the registered voter list), their self-reported intent to participate, their expressed interest in the campaign and other questions (according to descriptions in the Times and the CBS release) "other questions." The less likely a respondent is to participate, the smaller the weight they receive (see my 2004 post for more details on how this weighting model works).
So what does all this wonky variation add up to?
Percentage of Adults Represented - The differences in the way these polls sample and select "likely caucus goers" adds up to some very different conceptions of the likely electorate. The simplest measure is the percentage of adults that each poll represents. As the following table shows, that percentage has varied widely, from a low of 5% for the Democrats and 3% for the Republicans (on the PPP survey in May) to a high of 39% for the Democrats and 29% of the Republicans (on the Los Angeles Times/Bloomberg survey in September).
[Correction: The original version of the table above incorrectly labeled data from the August University of Iowa polls as applying to their October survey. The updated version includes data for both polls].
The good news is that roughly half of the pollsters clustered around a median value of 12% of adults represented by the Democratic surveys and 10% represented by the Republicans. This finding raises a few questions. First, is such a screen too loose, given that the historical high turnouts have been 5.5% for the Democrats and 5.3% for the Republicans?
Not necessarily. First, as should be obvious, none of these screens can predict who will participate with great accuracy. Second, no one knows what the level of turnout will be next month. Records have already been broken in terms of the dollars spent, and no one will be surprised if turnout records are broken in 2008. Third, we should probably assume that all of these surveys have some non-response bias toward likely voters. In other words, we should assume that non-voters are more likely to hang up on a survey about political issues. So screening down to the one Iowan-in-five that might participate in either the Democratic or Republican caucuses seems reasonable.
On the other hand, it is questionable, at best, to report results representing 51% (University of Iowa in August October) or 68% 70% of Iowa adults (LA Times/Bloomberg in September) as representative of "caucus goers" or "caucus voters," as the populations were labeled respectively in those two surveys (but see the response from the LA Times at the end of this post).
And also notice the way the portion of adults represented by the Newsweek surveys grew from a reasonable quarter of Iowa adults in September (17% for the Democrats, 10% for the Republicans), to 40% on the survey released over the weekend (24% for the Democrats, 16% for the Republicans). If nothing else, we need to understand that the most recent Newsweek poll represents a slice of the Iowa population nearly double that of most of the other Iowa polls on the table above.
Percentage of Past Caucus Goers - How tightly the pollster screens is just the beginning, because two polls that represent the same percentage of adults can end up selecting very different kinds of people.
One of the most consequential indicators, especially for the Democrats, is the percentage that report having attended a previous caucus. As the table below shows, the variation on this measure is huge. For the Democratic samples, the percentage of first time caucus goers among Democrats ranges from 2% (for the Hart survey for the One Campaign) to 43% (for ARG). For the Republicans, it varies from 18% (Iowa State University) to 62% (University of Iowa).
Notice that the variation is huge even among polls that screen to the same percentage of Iowa adults. Among the Democrats for example, two polls that represent 12% of Iowa adults report first time caucus goer percentages that range from 8% (Time/SRBI) to 43% (ARG).
Among Democrats, at least, this variation matters a lot. Virtually every survey that asks the question shows John Edwards doing better among past caucus goers, and either Clinton or Obama (or both) doing better among first-time caucus-goers (something I also noted back in August comparing the results from the three polls with very different percentages of first-time caucus goers).
Demographics - Finally, we reach the measures that are easiest to understand but typically hardest to compare. The demographics of the likely caucus-goers -- how they compare in terms of gender, age and education -- are the data that pollsters are typically most reluctant to release. Also, several measurement issues make these data difficult to compare. For example, pollsters use and report different age breaks. Their questions on education or party affiliation may use different language or response options that complicate comparisons. So the table below has many gaps yet still shows considerable variation across pollsters.
Consider the table above, which shows demographic results for the Democratic samples. Gender is relatively consistent, with the percentage of women varying between 51% and 62%, but age and education are comparatively more varied. Comparisons by age are the most difficult, given inconsistent age breaks, but the percentage of 18-29-year-olds in the Democratic samples varies from 2% to 20%. The percentage of 18-44-year-olds varies from 19% to 33%, under 50 varies from 29% to 50%. The portion with reporting a college degree varies from 35% to 56%.
Keep in mind when looking at these results that in the Democratic race, most surveys show Hillary Clinton doing better among women than other candidates, and they typically show Barack Obama doing better, and Hillary Clinton worse, among voters that are younger or college educated. So surveys that differ in their composition by gender, age or education will likely produce varying levels of support for these candidates.
The demographic composition of the Republican samples shows similar variation. The percentage of women varies from 36% to 56%. The differences in those of retirement age are also huge, with the share of 65-or-older respondents varying from 13% to 45% of the respective samples.
The point here is not so much to guess at which poll has the "right" demographic mix, the right percentage of first-time caucus goers or even about how many Iowans will caucus. Again, knowing with precision who will show up on January 3 is all but impossible. The point is that the Iowa polls have varied widely in terms of the kinds of people they sample. Put another way, no two polls share the same conception of the likely electorate, and some of the differences have been enormous. Yet those of us that consume the data compare results across the various polls as if they all measure the same "likely caucus goers." They do not.
Finally, if you consider the differences described above jarring, consider what we do not know. The polls profiled above represent just a fraction of the Iowa polls released to date. We know almost nothing about the likely voter methodologies used in Iowa by Zogby, Strategic Vision, Research2000 and Mason-Dixon. All four ignored our disclosure requests (though the McClatchy Newspapers did, to their credit, post filled-in questionnaires that include limited demographic profiles for their recent surveys conducted by Mason-Dixon and co-sponsored by MSNBC). Others have been incomplete in responding. For example, the American Research Group has disclosed how it selects likely caucus goers but tells us nothing about the demographics of its samples.
So why did we go to all this trouble? As should be obvious now, the differences in the way pollsters measure "likely caucus goers" in Iowa are huge, not just in how narrowly they define the electorate but in the kinds of voters pollsters select as "likely caucus goers." But these issues are not unique to Iowa. In 2004, 21 states held Democratic primary elections with single digit turnouts (as a percentage of adults), and only New Hampshire had a turnout that topped 20%. Over the next year months, results from hundreds of polls will be released, polls that will set expectations and drive media coverage, and yet those of us that consume the data will know very little about how tightly the pollsters screen and the kinds of voters they select. If we want to be educated poll consumers, we are going to need to do something to change that. We need to push toward greater routine disclosure of methodological details.
***
Note: Knowing that the percentage of Iowa adults represented by the September LA Times/Bloomberg survey stood out considerably from the rest, I contacted the LA Times pollsters for comment and received the following response:
The LAT/Bloomberg Iowa poll in September was not a survey of likely caucus voters. We did not frame it, nor report it, as such. As we all know, determining who is a likely voter is problematic, and particularly so when it comes to Iowa caucuses. It has long been the practice of the LA Times Poll to ask questions of the entire pool of qualified party voters in any state while the actual election date is still months away. In this case, our intent was to take the measure of all of Iowa's possible party caucus voters and so we made our screen as inclusive as possible. We included anyone who was a registered voter, or who said they planned to register in order to participate in the caucuses, and who told us they planned to attend a party caucus. We excluded no one on the basis of vote history, nor did we ask how strong was their intention to vote.
We will, however, be including a likely voter screen in the next Times/Bloomberg survey in Iowa, as it will be much closer to the date of the actual caucuses.
The two stories published by the LA Times in September are no longer available online, but they reported the Iowa results as part of a "poll of registered voters who planned to vote in the three early primaries or caucuses." The listing in the LA Times poll archive currently describes the survey (which included parallel samplings from New Hampshire and South Carolina) as a poll of "caucus/primary voters."
Appendix: How The Pollsters Select "Likely Caucus Goers"
What follows is the information that pollster provided (or included in public releases) about how they screen for likely caucus goers on Iowa.
ABC News/Washington Post
Asks three screen questions. Likely Caucus goers are self-reported registered voters who say they will absolutely or probably attend the caucuses. The text of those questions is as follows:
Are you registered to vote in Iowa at your present address, or not?
I'd like you to rate the chances that you will attend one of the Iowa presidential caucuses that will be held in your precinct in January. Are you absolutely certain you will attend, will you probably attend, are the chances 50-50, or less than that?
If the caucuses were being held today, would you attend the (Democratic Party caucus) or the (Republican Party caucus) in your precinct?
American Research Group
Asks four screen questions:
They ask whether respondents are registered to vote, and whether they are registered as Democrats or Republicans. Non-registrants are terminated and not interviewed.
They ask registrants how likely they are to participate in the Caucus "a 1-to-10 scale with 1 meaning definitely not participating and 10 meaning definitely participating." Those who answer 1 through 6 are terminated and not interviewed.
They ask unaffiliated registrants ("independents" registered as neither Democrats nor Republicans) whether they plan to participate in the Democratic or Republican caucus. Registered Democrats and independents who plan to caucus with the Democrats get the Democratic vote question; Registered Republicans and independents who plan to caucus with the Republicans answer the Republican question.
After asking vote question, they asks the question that appears on the web site: "Would you say that you definitely plan to participate in the 2008 Democratic presidential caucus, that you might participate in the 2008 Democratic presidential caucus, or that you will probably not participate in the 2008 Democratic presidential caucus?" Only the definite are included in the final sample of likely caucus voters.
CBS/New York Times
Began with the self-identified registered voters who said they plan to vote in a Democratic or Republican caucus in 2008. They “weighted respondents by past voting behavior, how much attention they said they are paying to the presidential campaign and their self-declared likelihood of voting in the Democratic or Republican primary in New Hampshire.”
Peter Hart Research (for the One Campaign)
Began with a sample drawn from a list (maintained by the Iowa Democratic Party) of registered voters that had either previously participated in Democratic caucuses or had registered to vote since 2004. New registrants were included if they said they are “likely to attend a Democratic presidential caucus.”
Iowa State University
Sampled registered Democrats and Republicans (but excluded those with “no party” registration) from a list of registered voters and interviewed those who said they would definitely or probably attend caucuses in their area.
Los Angeles Times/Bloomberg
“In Iowa we asked three questions to try and get all of the caucus goers we could in the sample. The 3 questions are:”
Q1. Some people are registered to vote and other people are not. How about you? Are you registered to vote at the address where you now live, or are you not registered at your current address?
(IF REGISTERED) Are you registered as a Democrat, a Republican, or do you choose not to be registered in a specific party?
Q2. (ASKED OF SAMPLE 1- IOWA RESPONDENTS WHO ARE NOT REGISTERED TO VOTE / NOT SURE / REFUSED) I realize that the Iowa caucuses are months away, but do you plan to register to vote on or before the January caucuses in order to participate in them, or aren't you planning to attend a caucus this time?
(IF GOING TO REGISTER AND ATTEND) Will you attend a Democratic or a Republican caucus? (IF NOT SURE WHICH CAUCUS, ASK) Well, if you had to make a choice right now, would you lean more toward attending a Democratic caucus or more toward attending a Republican caucus?
Q3. (ASKED OF SAMPLE 1 - IOWA RESPONDENTS WHO ARE REGISTERED TO VOTE) I realize the Iowa caucuses are months away, but do you plan to attend either a Democratic caucus or a Republican caucus in January, or aren't you planning to attend a caucus this time?
McLaughlin & Associates (for the One Campaign)
“We merged the list of voters who voted in the 2000 Republican Caucus with the list of voters who have voted in at least 2 of the last 3 Republican Primaries (2002, 2004 or 2006). We included more recent super primary voters because the last Republican Caucus was in 2000 and we did not want to exclude potential new likely caucus voters. We only asked for and interviewed the specific person on the voter list.
“Even though we used a merged caucus/super primary voter list and specifically ask for the person on the list, we still ask screening questions to confirm that they are likely to vote in the Republican Caucus. Below are the actual screening questions.”
1. With which political party are you registered?
Republican (CONTINUE)
Democrat (TERMINATE)
Independent (TERMINATE)
All other responses (TERMINATE)
2. As you are aware, very few people actually vote in Republican Caucuses – How likely is it you will vote in Iowa’s Republican Caucus for President next year? Would you say very likely, only somewhat likely, or not likely at all? If you are not going to vote in the Republican Caucus for President, just say so.
Very likely (CONTINUE)
Somewhat likely (CONTINUE)
Not likely at all (TERMINATE)
Not going to vote (TERMINATE)
DK/Refused (TERMINATE)
Newsweek/PSRA
They begin by identifying Democratic and Republican “voters:” Democratic voters are registered voters who report their party ID as Democratic or lean Democratic; Republican voters are registered voters who report their party ID as Republican or lean Republican.
Likely Democratic caucus goers are Democratic voters who say they will definitely or probably attend a Dem caucus where they live in January. Likely Republican caucus goers are Rep voters who say they will definitely or probably attend a Rep caucus where they live in January.
Pew/AP
Respondents were registered voters who say they will definitely or probably attend either a Democratic or Republican caucus.
Text: Thinking about the upcoming presidential caucuses in Iowa on January 3rd, how likely would you say it is that you’ll attend the presidential caucus in your area? Will you definitely attend, probably attend, probably not attend, or aren’t you sure at this point?
Public Policy Polling (PPP)
“We sampled only voters who had voted in at least 3 of the last 4 primary or general elections, and we had a screening question and only accepted those who answered ‘very likely’ to participate in the caucus.”
Des Moines Register/Selzer & Co.
Sampled voters “registered as Democrat, Republican or no-party” from the list of all registered voters in Iowa and interviewed those who said they would "definitely" or "probably" attend the caucuses.
How likely is it you will attend one of the caucuses scheduled for January 3rd of 2008? Will you definitely attend, probably attend or probably not attend? Will you attend the Democratic or the Republican caucus?
Rasmussen Reports
For the Iowa caucus poll, we then asked three screening questions.
First, we ask about voting history and only include those who say they vote all of the time or most of the time.
Second, we ask how closely the respondent has followed the Presidential debates and other campaign activities. Those following the campaign Very or Somewhat Closely are included.
Third, we ask specifically about how likely each person was to participate in the caucus. Those who are Very or Somewhat Likely to participate are included in the final sample.
Time/SRBI
From their release: The sample source was a list of registered Democratic and Independent voters in Iowa provided by Voter Contact Services. These registered voters were screened to determine their likelihood of attending the 2008 Iowa Democratic caucuses.
Likely voters included in the sample included those who said they were
* 100% certain that they would attend the Iowa caucuses, OR
* probably going to attend and reported that they had attended a previous Iowa caucus.
Text of Questions:
Q1 To start with, many people haven't had a chance to register to vote yet. Are you registered to vote in the district where you live now, or not?
Q3. As of now, how likely are you to attend your party's caucuses for the 2008 presidential party nominations? Answer categories: 100% certain to attend, probably will attend (other categories omitted).
Q4. If you do attend a caucus, which party's caucus would you attend the Democratic Party caucus or the Republican Party caucus?
University of Iowa “Hawkeye” Poll
The original version of this post included only the following information from their August poll release: "Caucus-goers are self-identified. A respondent who answered 'very' or 'somewhat' likely to a question about attending the 2008 caucus is classified as a 'likely' caucus attendee."
The summary of their October poll (as corrected on 12/14/07) includes the following information:
Respondents were asked whether they were very likely, somewhat likely, not very likely, or not at all likely to attend their party's caucus in 2008. Among all registered voters contacted, 37.2% said they were "not at all" likely to caucus, while another 21.3% said they were "Not very likely". These two groups were not considered "likely caucus goers". The remaining 41.5% said they were "Very Likely" (24.1%) or "Somewhat Likely" (17.4%) to caucus. A second screen then asked which party's caucus the voter planned to attend. Of the initial screen of likely caucus goers, 4.4% could not name a party, and were dropped. Approximately 35% of the original registered voter sample is thus classified as "likely caucus goers". Of the total original registered voter sample, about 19.1% are likely Democratic caucus goers and 14.4% are likely Republican Caucus Goers.
Via email sent after this post initially appeared, the University of Iowa pollsters also provided more detail on their the full text of their screen questions:
1) First we ask people their likelihood of caucusing in January 2008 on a 4 point scale, from VERY, SOMEWHAT, NOT VERY, and NOT AT ALL. We classify respondents who are "very" or "somewhat" likely to caucus as LIKELY caucus goers.
2) Second, we screen for which PARTY the respondent plans to caucus for. Anyone saying DON'T KNOW or some party other than Republican or Democratic are excluded. If they can't name the party, they are probably not caucusing.
PRTYCAUC If you do go, which party's caucus do you plan to attend?
1 Democrat
2 Republican
8 Don't Know (VOL.)
9 Refused (VOL.)