CDC: Vaccine Study Used Flawed Methods

(NOTE: My original post on this topic mischaracterized the 2003 CDC vaccine investigation as an "Ecological Study," which it was not. I am reposting this piece to reflect that information accurately, but also to point out that many of the weaknesses identified in the CDC's data and methods apply to the published 2003 "retrospective cohort" study, as much as they do to any future "ecological" ones. I regret and apologize for the error.)

A new report (PDF) that CDC Director Dr. Julie Gerberding has delivered to the powerful House Appropriations Committee casts new light -- and new doubt -- on the data and methodology that the CDC used in its landmark 2003 study that found no link between mercury in vaccines and autism, ADHD, speech delay or tics.

Gerberding was responding to a report from the National Institute of Environmental Health Sciences (NIEHS), which evaluated the strengths and weaknesses of the CDC's vaccine database, and showed how the weaknesses, in particular, would have to be addressed in conducting further studies of thimerosal and autism.

These weaknesses included: uncertainties in case ascertainment, heterogeneity of business practices within and across HMOs in the database and their systematic changes over time, misclassification of exposure status, and the inability to control for temporal changes in awareness, diagnostic practices and potential confounding factors.

Many of these weaknesses should be taken into account when moving into the future, but they also apply to CDC studies that have been done in the past, including the methodology that was employed in the CDC's flagship thimerosal safety study of 2003.

To begin with, the NIEHS panel had "identified several areas of weaknesses that when taken together reduce the usefulness of the project for conducting an ecologic study design to address the potential association between exposure to thimerosal and the risk of autism."

Ecological studies are large, epidemiological analyses of risks and trends using data from large populations without making efforts to link outcomes to actual individual patients. The 2003 CDC study was not, strictly speaking, an "ecological study," but rather a "retrospective cohort study."

CDC researchers did go back and review some of the charts of the children diagnosed with the outcomes under study -- though this accounted for less than 1% of all children enrolled in the study.

Dr. Gerberding said in her letter that thimerosal studies done by the CDC have not been "ecological," because they utilize "medical chart reviews, neurological assessments, and parent interviews." But in the 2003 study, chart reviews were not done on 99% of the study population, and no assessments or interviews were conducted to make sure that the diagnoses were accurate.

The NIEHS report was largely focused on the feasibility of conducting an ecological study of the database, but many of the weaknesses identified are also applicable to the 2003 CDC study of the Vaccine Safety Datalink (VSD) which contains the records of hundreds of thousands of HMO patients.

In that investigation, CDC officials conducted at least five separate analyses of the data over a four-year period from 1999-2003. The first analysis showed that children exposed to the most thimerosal by one month of age had extremely high relative risks for a number of outcomes, compared with children who got little or no mercury: The relative risk for ADHD was 8.29 times higher; for autism, it was 7.62 times higher; ADD, 6.38 times higher; tics, 5.65 times; and speech and language delays were 2.09 more likely among kids who got the most mercury.

Over time, however, all of these risks declined into statistical insignificance, statistical inconsistency or else outright oblivion: The relative risk for autism plummeted from 7.62 in the first analysis, to 2.48 in the second version, to 1.69 in the third round, to 1.52 in the fourth, and down to nothing at all in the fifth, final, and published analysis printed in the Journal Pediatrics in November of 2003.

Vaccine officials attributed the steady drop to the elimination of "statistical noise" from the data through due diligence and the endeavor for excellence in governmental statistical analysis.

Indeed, the VSD study was the main pillar of a hugely influential 2004 report by the Institute of Medicine, which also concluded that there was no evidence of link between mercury, vaccines and autism.

To this day, public health officials routinely point to five "large epidemiological studies" representing the "highest quality science," none of which found any link to thimerosal.

In fact, the American VSD study has long been held up as the best and brightest of them all (the others were in Sweden, the UK, and two in Denmark, which WERE ecological studies, and presumably subject to some of the same weaknesses identified by the NIEHS). This reputation has stuck in the minds of medicine and the media.

Curiously though, even the study's lead author -- Dr. Thomas Verstraeten, an employee of vaccine maker GlaxoSmithKline -- protested that the VSD study "found no evidence against an association, as a negative study would. In fact, he said that additional study was needed, which "is the conclusion to which a neutral study must come."

That's when Congress stepped in.

In 2005, a group of Senators and Representatives headed by Sen. Joe Lieberman wrote to the NIEHS (an agency of the National Institutes of Health) saying that many parents no longer trusted the CDC to conduct independent minded studies of its own vaccine program. Lieberman et al asked NIEHS to review the CDC's work on the vaccine database and report back with critiques and suggestions for future investigations.

Among the official tasks given to the NIEHS panel were to "Identify the strengths and weaknesses of the VSD for evaluating the possible association between exposures to thimerosal-containing vaccines and AD/ASD," and had nothing to do with "ecological studies."

In her letter to the House Appropriations Committee, the CDC Director responded directly to many -- though not all -- of the most important critiques and recommendations contained in the NIEHS panel report. The weaknesses she noted apply to Verstraeten as much as they do to any future studies, ecological or otherwise.

For example, the NIEHS report had said the VSD data failed to account for other mercury exposures, including maternal sources from flu shots and immune globulin, as well as mercury in food and the environment.

"CDC acknowledges this concern and recognizes this limitation," the Gerberding reply says.

The NIEHS also questioned why CDC investigators eliminated 25% of the study population for a variety of reasons, even though this represented, "a susceptible population whose removal from the analysis might unintentionally reduce the ability to detect an effect of thimerosal." This strict entry criteria would likely lead to an "under-ascertainment" of autism cases, the NIEHS reported. Again, this would have been an issue in the Verstraeten data.

"CDC concurs," Gerberding wrote, again noting that VSD data are "not appropriate for studying this vaccine safety topic. The data are intended for administrative purposes and may not be predictive of the outcomes studied."

Another serious problem is that the HMOs have changed the way they tracked and recorded autism diagnoses over time. Gerberding said this would be a problem going into the future, but did not mention that the same principal might apply to past studies.

I hope everyone will read these documents, including the recommendations to make the VSD better, and the CDC's agreement with all of the suggestions. Hopefully, this data can still be used in some effective way.

As questionable as the US thimerosal methodology was, "it was an improvement on other studies, including the two ecological studies in Denmark, both of which had serious weaknesses in their designs," Dr. Irva Hertz-Picciotto, Professor of Public Health at UC Davis Medical School and Chair of the NIEHS panel, told reporter Dan Olmsted at UPI.

That leaves little for the CDC to go on in terms of proving that thimerosal and autism are not associated in any way.

Yes, there is always the study of disability services data from California -- which seem to be rising among the youngest cohorts of kids, who presumably received little or no mercury because thimerosal was largely removed from childhood shots.

But California is an "ecological study" as well, with problems of its own.

"Although (this) information is often used by media and research entities to develop statistics and draw conclusions, some of these findings may misrepresent the quarterly figures," cautions the website of the California Department of Developmental Services (DDS). "Increases in the number of persons reported from one quarter to the next do not necessarily represent persons who are new to the DDS system."

Even the CDC admits that "there are several limitations" with linking a VSD study design with the California data, Gerberding wrote to Congress, because, among other things, California only counts "persons who were referred to and/or voluntarily entered" the disability system."

It will be interesting to see how the House Committee -- and the mainstream media -- react to this report by the CDC, which does seem to want to conduct the best vaccine-autism science possible (see Gerberding's replies to NIEHS recommendations for improving the VSD: CDC officials are currently conducting in- depth follow up studies with VSD patients).

POSTSCRIPT:

This revised piece does raise two new questions, I think:

1) If the VSD is not necessarily appropriate to help determine the effect of reducing mercury levels in vaccines, are taxpayers getting their money's worth?

2) If studies done in Denmark, Sweden and California were also "ecological" in nature, are they subject to some of the same weaknesses and limitations?

Comments

[amandastories]

This is good news--not as good as it could be, but getting rid of lies is a step towards getting the truth out. Mercury should never have been in vaccines; it causes brain damage. Thank you David Kirby.

[Olerist]

Thanks David

any chance of getting a scan of the letter to congress posted so others can see?

Others responding , especially those claiming to be scientists, should keep in mind that the CDC has only done epidemiological studies. They have not looked at individual cases, cases studies or toxicology studies when drawing conclusions. When others have, there are associations.

Epidemiology alone will not answer the question. Nor has the govt been willing to share the data of these epidemiological studies - I would think that issue alone should make others cautious about what the CDC and NIH have been claiming - again especially among those responding claiming to be scientists.

I do not think the govt wants others looking at the VSD at all. We as consumers of these mandated goods should demand it be turned over and vaccine safety scrutinized for the good of the people and the program.

[ThoJ]

"They have not looked at individual cases, cases studies or toxicology studies when drawing conclusions."

Actually, the in-press paper by the CDC looks at individuals. According to the document that David Kirby cites, it should be published in September, but these things can get delayed.

It isn't much of a leap of logic to figure that
a) since the CDC knows the results of that study and, yet,
b) still concurs with the 2004 IOM report,

then, the study must show no relation between thimerosal and autism.

This is discussed in greater detail by a blogger here: http://leftbrainrightbrain.co.uk/?p=891

[ObjectiveAutismDad]

I think you missed the point. The CDC was asked, by Congress -- almost 10 years ago, to open up the VSD to independent researchers and to share the data that was used for their initial studies. To date, only one pair of researchers has been granted "access". Access that was hindered from the word go, and then pulled before they could complete their work. Yes, you are right about one thing: "these things can get delayed". But we've had enough of that.

It's no wonder people are starting to not get vaccinated. The government and the main stream medical community have no one to blame but themselves for the distrust that they've earned.

[ThoJ]

So,

how about a comment about the how you, Rick Rollens and so many others misrepresnted the CDDS data in your arguements? It isn't like there weren't people out there pointing out these flaws the entire time.

Those are the sort of tough questions you have been getting for years. Only now, as you advertise your speaking engagements do you you ask for tough questions.

""Increases in the number of persons reported from one quarter to the next do not necessarily represent persons who are new to the DDS system.""

You, Mr. Kirby, have been informed of this flaw repeatedly. And you continued to rely on those data to draw incorrect conclusions.

If you are going to ask for someone to step up and admit mistakes, please take the lead. This blog is a great place for you to point out that you have ignored repeated calls for you to recognize the limitations of the data you were using.

[agrund]

"That leaves very little for the CDC to go on in terms of proving that thimerosal and autism are not associated in any way."

Wow. Read that sentence again. This is logic turned on its head. So far there has been no link shown between autism and mercury . So far, several studies indicate a lack of linkage. If there are flaws in these studies, these can (and will be) addressed in future studies. The CDC and other medical research groups aren't evil or trying to cover up anything. Mercury has been obviously shown to be harmful, but so far has not been implicated in autism, and I don't work for the CDC (I am guilty of being a scientist, however).

If there is a link, it needs to be demonstrated scientifically. So far that hasn't been done. Nobody knows the cause of autism (if there is a single cause, which may not be the case), but most recent evidence points to genetic rather than environmental origin. Be that as it may, this constant demonization of the CDC is misguided to put it politely.

[kellianndavis]

"Mercury has been obviously shown to be harmful, but so far has not been implicated in autism,"

What about being implicated in a neurological injury? (Makes sense considering it's a known *neurotoxin* don't ya think?)

My son suffered neurological injuries shortly after rec'ing a boatload of mercury from his vaccines during the early to mid 90's.

What about that? You got a clever play on words to try and explain away that fact?

Kelli Ann Davis

[bluevalentine]

Kelli,
I hear you. My son, also, has neurological problems..­.although, they do not add up to Autism, they do add up to CP. Bad enough imo. My son also has a list of many less severe problems than the CP. That being said, I still feel like I am in the twilight zone on this topic. Thimerosal is a 49% mercury. Mercury is a neurotoxin. Second only to plutonium in toxicity. Why is there an argument AT ALL as to it's need to be COMPLETELY gone. And I'll add that it shouldn't be given to adults, either. Maybe not for the same severity of immediate concern...but simply for cummulative dose.
Oh well,
Thanks David!

[CraigWilloughby]

Ah, but since the CDC has been using thse studies as the cornerstone of all of these reports and "studies" that you claim prove that there is no link, it calls both the validity of the studies and the credibility of the people doing these studies into question. This isn't just one study...these are several studies with these admitted flaws. It's funny to me when these studies that the vaccine-thugs hold to so religiously are threatened, they get nervous and anxious.

[prunesquallor]

Craig,
"...these are several studies with these admitted flaws."
Did you read the post I recommended to you yesterday? I respectfully recommend that you go back and read the new post on the same blog. You will see that neither the NIEHS Report, nor the CDC Report, said that there were flaws in these "cornerstone" studies. Indeed, none of the studies were even discussed in the reports. That wasn't the purpose of the reports.

[ObjectiveAutismDad]

"...most recent evidence points to genetic rather than environmental origin..." But it's impossible to have a genetic epidemic. Most recent evidence points to genetics because it's a good distraction -- and not true, BTW. Most recent evidence, if you look at all of it, points to a genetic susceptibility that is triggered by an environmental insult or insults.

You don't go from an incidence of 1:10,000 to 1:150 in a generation because of genetics. (And, yes, I know there are still those who cling to the hope that there really isn't an epidemic and it's all just better diagnosis; but they are in serious denial -- See David's other post on the missing adult autistics and many other bits of evidence against those "theories".)

[Mikal]

It is inconceivable to me that the medical epidemiologists and statisticians at CDC are capable of making the study errors outlined by Mr. Kirby. They are basic design and sampling errors that only a novice would make. If I am incorrect, then those responsible need to be fired. Mr. Kirby's article did not describe Dr. Gerberding as having told Congress that she will look into the incompetence of her staff. Therefore, I assume that Dr. Gerberding is covering up for the political influence she allowed to occur that made the study findings unusable.

[Aasa]

Gerberding and political influence?
More like political interference!
http://www.scienceprogress.org/2007/10/redacted-testimony-of-cdc-director-julie-l-gerberding/