My National Journal column for the week explores two recent surveys of physicians that produced widely divergent results. One was published in the New England Journal of Medicine (NEJM), conducted by two researchers at he Mount Sinai School of Medicine in New York and funded by the Robert Wood Johnson Foundation. The second was conducted by Investors' Business Daily (IBD) and the TechnoMetrica Institute of Policy & Politics (TIPP).
The column focuses on the huge difference in the quality of disclosure between the two -- the NEJM study tells us much, much more about its methods and questions than the IBD/TIPP survey. The latter fails to meet even minimal standards for disclosure mandated by the National Council on Public Polls and the American Association for Public Opinion Research.
Late Update - One of the subsequent stories published by Investor's business daily on their survey of doctors now includes the following information not disclosed in their initial report:
The questionnaires were sent out Aug. 28 to 25,600 doctors nationwide. The sample was purchased from a list broker, Lake Group Media of Rye, N.Y. One hundred of those responding were retired, and their answers were not included in the final results.
If it was there last week, I missed it, but this new information tells us two things: First, the IBD/TIPP survey did not use the same sample frame as the NEJM survey. Second, as reader Franzneumann points out, the response rate is significantly lower: Even if they had included the 100 retired doctors along with 1,376 reported interviews, their response rate is only 6%. Compare that to the 43% response rate reported by the NEJM survey.
Update1: So why do the two surveys produce such different results? As I argued in the column, without even minimal disclosure from the IBD/TIPP poll we can only speculate, but here are the most likely theories:
1) Different questions. The NEJM study asks a three-way, forced choice between a private only option (involving "tax credits or low income subsidies to buy private insurance"), a public only option (that would "eliminate private insurance and cover everyone in a single plan like medicare") and a combination offering a choice between public and private plans. It makes no reference to current legislation, the Democrats or President Obama. We do not know exactly what the IBD question asks, although it appears to be simply about "the proposed plan" or possibly about a "proposed government expansion."
We have seen some pretty big differences on samples of all adults between surveys that simply ask about health reform "proposals" being debated in Congress (without further definition) and those that attempt to describe a "public option." For example, in August, CBS News found 57% of adults in favor and 35% opposed to "the government offering everyone a government administered health insurance plan -- something like the Medicare coverage that people 65 and older get -- that would compete with private health insurance plans." At about the same time, the Pew Research Center reported 38% of adults in favor and 44% opposed to "the health care proposals being discussed in Congress."
The three-option question that the NEJM survey reported also produces a bigger percentage supporting a public option, alone (10%) in in combination with private plans (63%), than an up-or-down, favor-or-oppose question. The NEJM questionnaire actually asked about each of the three proposals separately (see the column for complete wording), then followed up with a three way choice.
They have not yet reported on the individual results, although Alex Federman, one of the two medical researchers at the Mount Sinai School of Medicine in New York who conducted the survey, tells me that 65% said they supported a choice between public and private options when asked about it separately. They plan to publish the remaining results from the survey in future articles. Federman added that they considered the three way choice most appropriate, both because of what they learned from questionnaire pretesting and because they wanted doctors to make the same choice facing legislators between a public-only option, private-only options or a choice between the two. In pre-testing, Federman says, they learned that Doctors had all sorts of opinions on health reform, but "a lot of people were more mixed" between their views of public and private options.
2) Different samples? I put a question mark on this theory because, as reviewed in the column, we know almost nothing about how the IBD/TIPP poll sampled "practicing physicians." As Nate Silver points out, they do not even define what they mean by "practicing." However, if the IBD/TIPP used a different sample frame than the AMA Physician Masterfile (a comprehensive list of all physicians, not an AMA membership list), it might well explain some of the difference in results.
[Clarification: See Late Update above -- The IBD/TIPP survey did use a different sample frame although for now we know only where it came from, not how it differed. They also imply that "practicing" means non-retired].
[One dissent worth noting: This morning, a public health research used Twitter to take issue with my observation of that the AMA file is a "very accurate" list. "Best out there?," she asked. "Yes, but problematic."]
3) Response bias? Even if the the two lists (or "sample frames") were identical, it is possible that different kinds of doctors responded to the two surveys. For example, did the IBD survey prominently identify their survey as sponsored by Investors' Business Daily? Those who know or subscribe to it know that its editorials tend to be more conservative, and its editorial criticism of the Obama health care reforms have drawn sharp rebuke. As such, reform supporters might be less likely to participate in an IBD survey. Of course, we do not know anything about how the IDB/TIPP survey recruited respondents.
[Clarification: See the Late Update above -- The IBD/TIPP survey has a response rate of roughly 6% compared the a 43% response rate on the NEJM survey].
According to Federman, the NEJM survey identified itself as coming from the Mount Sinai School of Medicine and prominently identified its support from the Robert Wood Johnson Foundation. The headline of the initial postcard they sent identified it as "The National Physician's Survey on Health Care Reform" (emphasis in original). The subhead: "Congress wants to hear from doctors on health care reform."
Some relatively recent research has shown that "interest in the survey topic, reactions to the survey sponsor, and the use of incentives" are the three most critical variables in the linkage between the response rate and the potential for non-response bias. Perhaps those interested in "health care reform" enough to do a survey are more supportive of reform than those opposed?
4) Different survey dates. The surveys did have different field periods. The first NEJM questionnaires were sent out on June 25 and they started analyzing data on September 4. The IBD poll, published on September 15, says only that it was "conducted by mail the past two weeks."
While it is possible that the field dates contributed to the difference in results, since opposition to reform has increased over the course of the year, it's unlikely. First, as Charles Franklin noted on Friday, opposition to health care reform among all adults has been mostly stable since early July, about the time when the NEJM survey started. Second, according to Federman, they sent out their questionnaires in multiple waves, analyzed the results by wave and found no trends over the course of the field period.
Again, much of this commentary is just speculation. The larger point is that when one survey discloses its methods and the other does not, we are left guessing.