The odds are very good that shortly after President Obama completes his health care address this evening, at least two television networks will release results from "instant reaction" surveys. Others will likely stage focus groups in which selected participants react to what they see and hear. The odds are almost as good that pundits and partisans will grossly misread or spin whatever those surveys and focus groups produce. Here is a primer for those hoping to make sense out of whatever new poll data we see over the next 24 hours.
1) Instant response polls measure only speech-watchers. While the methodologies vary, the most important thing to remember that these surveys aim to sample only those who watch the speech and, as such, are are not intended to represent the views of all Americans. The pollsters will hopefully provide some before-and-after comparisons of the speech audience -- showing how viewers felt about health care reform before and after the speech -- but those comparisons will involve only the sample of speech viewers. Thus, no one should take any of the numbers they see tonight and make comparisons to full-sample results from previously surveys of all adults or all "likely voters."
2) The audience is usually skewed toward the President's fans. Remember, not all Americans watch presidential addresses. Between 52 and 63 million Americans watched the debates last fall and roughly 53 million watched President Bush's address on the economic crisis last September.** Those are huge audiences, but plenty of Americans still tune out.
Since admirers of the president are usually disproportionate among those who tune in, the overall instant reaction numbers can be deceiving. Like debates, presidential speeches usually reinforce the opinions people held before the speech, so it will be important to look at cross-tabulations by party or by pre-debate attitudes about health care reform (for more details, see my posts from just before and after the 2006 State of the Union address).
3) Instant impressions can be fleeting. Generally speaking, reactions on instant response polls tend to be a lot more positive immediately after the event than on surveys taken days or weeks later. The sampling problem may explain much of this phenomenon, or it may be, as ABC's Gary Langer wrote earlier this year, that viewers "get caught up in the moment, [so] a single speech in and of itself is very highly unlikely to change any fundamental attitudes." The point is, be careful, instant reactions may fade.
4) Some pollsters have reservations about instant reaction polls. To understand their skepticism, let's review the methods of the polls that are likely to get the most attention:
- CNN and their pollsters, the Opinion Research Corporation, typically recontact respondents by telephone that they called a few days earlier who say they are planning to watch the speech. They call as many as they can, as quickly as they can, to get an immediate reaction.
- Gallup uses essentially the same method, not surprising since until 2007 Gallup produced surveys for the partnership of CNN and USA Today. The main difference has nothing to do with methodology: in recent years is that Gallup typically held their results for the following day rather than rush them on the air immediately.
- CBS also conducts panel back surveys, although theirs is conducted online using the nationally representative Knowledge Networks internet panel. Since all respondents can immediately log in and complete the survey online, CBS can gather their data very quickly.
They important point: All three surveys use what pollsters call a "panel back" design, a classic and widely used survey research technique. The strength of this approach is that it allows relatively quick, efficient recontact of likely speech watchers contacted earlier through more rigorous methods. It also allows for before-and-after comparisons with results gathered from the previous survey. The downside with any such "panel-back" survey is that (a) some polled the first time do not respond the second time which may create a bias that simple weighting does not resolve and (b) the experience of having been interviewed may indirectly change attitudes (respondents may pay more attention to the news after the first interview).
I should stipulate that I have no advance word on which networks will be conducting surveys tonight, but CNN and CBS News conducted instant response polls following all of last fall's presidential debates, and Gallup joined joined CNN and CBS in conducting a one-night survey following Obama's address to Congress earlier this year.
SurveyUSA, a company that polls with an automated, recorded methodology, sometimes conducts post speech polls using fresh samples in media markets in the Pacific time zone (where the speech ends before 7:00 p.m.). Their automated method allows for a lot of simultaneous dialing of a fresh sample.
But again, not all pollsters are enamored with these methods. Back in 2004, Republican pollster David Hill wrote a scathing assessment that described the traditional panel-back design as "worthless":
Considerable scholarly research demonstrates that simply being interviewed renders an otherwise normal voter abnormal. After being polled, voters are much more likely to seek out political information through the media, discuss politics with others and eventually to vote. The known effects are so great that in the earliest days of polling, voters would be screened at the outset of an interview to ascertain if they had ever been interviewed before.
ABC News stopped conducting instant reaction polls a few years ago, although as Gary Langer explains, their reservations were about more than the challenge of panel-back sampling:
Good sampling's a bear in this kind of thing, but there are two equally basic problems: Speech watchers tend to be favorably inclined to the speechifier in the first place (those who can't stand him are unlikely to watch); and speeches are crowd-pleasing (even platitudinous) by design (e.g., let's cure cancer).
5) Focus groups have value, but they are not surveys, and should be treated with far more caution. If past history repeats itself, we are likely to see Fox News, CNN and MSNBC conduct some sort of focus group. The Democratic party affiliated Democracy Corps frequently conducts its own focus groups. Regardless of the sponsor, the focus group usually involves a group of 20-40 adults gathered in a central location selected by some hopefully-representative-but-not-random method. A moderator talks to the group before and after the debate. Sometimes the members of the group provide their feedback throughout the speech using a dial, with the aggregate scores of the dials appearing as rising and falling lines on the television screen.
Ideally, focus groups provide "qualitative" insights that are tough to glean from standardized "quantitative" survey questions. The problem is, they are not random samples, and the discussion is sensitive to the "group dynamic" set by the moderator or the most verbose participants. I have long been skeptical that the reality-TV feel of turning participants into pundits in a television studio produces much of value. Also, while the squiggly line of the dial group chart may help campaign consultants and television producers spot the most memorable sound bites, the value for the rest of us is pretty negligible.
6) Will the speech make a lasting change in attitudes on health reform? That's the really important question, but to answer it, we will need another round of more rigorous national surveys that we will see over the next few weeks. Your favorite blogs are probably already humming with predictions and speculation about what might or might not change. For a useful reality check let me recommend three: John Sides passes along the work of political scientist George Edwards, who reviewed polling around presidential speeches and found that "statistically significant changes in approval rarely follow a televised presidential address." Sides also charts all of the Clinton job approval polls from late 1993 and finds "no increase in Clinton's approval immediately after [his 1993 health care] speech." Gary Langer recalls that Bill Clinton swayed opinions in an ABC poll conducted immediately his 1993 address that had largely dissipated two weeks later. Finally, its worth reviewing Charles Franklin's two-year-old post that illustrates the "mostly nil" affects of State of the Union speeches on presidential approval.
So my best safe but boring advice is to wait and see. For all the similarity to 1993, this speech provides its own new and hard to predict "model." The speech is coming much later in the debate, and for all the evident passion of opponents, many Americans remain both interested and confused. As this week's Pew Research Center New Interest Index poll shows, virtually all Americans (93%) consider health care reform important, 73% say it affects them personally, 40% to 49% say they have been following the issue "very closely, but a huge number (67%) also says the subject is "hard to understand." That sounds like a recipe for a large audience that is potentially more attentive and persuadable than for most previous presidential addresses.
But we shall see.
Update: Brenden Nyhan agrees with Sides and Langer that the speech is unlikely to move public opinon: "The reason is simple -- the president's message is typically offset by
that of the opposition. In the aggregate, the effects tend to cancel
out and the numbers don't move."
**Clarification: The original version of this post also cited the number of television households (113 million), which is a bit of a non-sequitur since the other statistics are counts of individual viewers.