Newsflash From APA Meeting: DSM-5 Has Flunked Its Reliability Tests

The whole purpose of having a manual of psychiatric diagnosis is to promote diagnostic agreement. The great value to the field of DSM-III was that it established reliability and preserved the credibility of psychiatry at a time when it was becoming irrelevant because it seemed that psychiatrists could not agree on diagnoses. Everyone knew that the reliability achieved in DSM field testing far exceeds what is possible in clinical practice, but DSM-III took the major step of proving that reliability could be achieved at all. Until now the DSMs have facilitated communication across the clinical/research interface, promoted research, and provided credibility in the court room.

But bad news was recently reported from the annual meeting of the American Psychiatric Association in Philadelphia. The hard-won credibility of psychiatric diagnosis is compromised by the abysmal results reported by the DSM-5 field trials. This failure was clearly predictable from the start:

1. The writing of the DSM-5 criteria sets was far too raw and imprecise to be ready for the rigors of field testing. The ambiguity cried out for expert editing, without which reasonable reliability is impossible.
2. The design of the field trial was byzantine in complexity and could never be done on schedule.
3. Constant delays in starting and completing Stage 1 of the study forced DSM-5 to cancel the planned Stage 2, which was meant to clean up the poorly performing criteria sets identified in the first stage.
4. With Stage 2 cancelled without explanation, it looks like even the worst diagnoses are being given a social pass.
5. Most absurdly, the design was totally off-point, failing to ask the only question that really counted: the impact of DSM-5 on rates.

The results of the DSM-5 field trials are a disgrace to the field. For context, in previous DSMs a diagnosis had to have a kappa reliability of about 0.6 or above to be considered acceptable. A reliability of 0.2 to 0.4 has always been considered completely unacceptable, not much above chance agreement.

No predetermined publication date justifies business as usual in the face of these terrible field trial results, which are even more striking given that they were obtained in academic settings with trained and skilled interviewers, highly selected patients, and no time pressure (the results in real-world settings would be much lower). Reliability this low for so many diagnoses gravely undermines the credibility of DSM-5 as a basis for administrative coding, treatment selection, and clinical research.

What can be done to salvage this deplorable mess?

1. DSM-5 has never had anyone on board who could write a clean, consistent, unambiguous criteria set. DSM-5 appears to have received either no editing at all, or amateur editing at best. Getting the words right is certainly not enough, but if you can't even get them right, nothing else can ever be safe.
2. For DSM-5 to retrieve credibility, it must complete the second planned stage of its field testing. If doing the job right must delay publication, so be it. Public trust must trump private publishing profits, and it is self-defeating for APA to publish a book no one can trust.

I have been consistently pessimistic and critical about DSM-5 since my first piece on it three years ago. The sad thing is I can still be so surprised. Each step of the way I predict it will fail in one or another way. But then I discover that DSM-5 has managed to fail in ways that go beyond my poor imagination. This assault on reliability was predicted, but its scope exceeds even my jaundiced fears and creates a DSM-5 emergency.

Allen Frances is a professor emeritus at Duke University and was the chairman of the DSM-IV task force.

 

 

 

Comments

[Kittyburger]

10:24 AM on 05/14/2012

It's already well-documented that the gender and sexuality workgroup is hopelessly corrupted. DSM-5 is unsafe for gender and sexuality minorities, pathologizes non-vanilla sex simply for being non-vanilla, and generally steps backward in time toward a more regressive area - it's not surprising to me nor I suspect to anyone involved in that mud pit that the rest of DSM-5 is just as much of a mess.

[flipper99]

05:19 PM on 05/10/2012

I diagree that prevalence rates are the "only thing that mattered." DSM-IV was characterized as strong in reliability but weak in validity. DSM-V has failed to address that which is truly the most important data, the validity of the psychiatric syndromes, and whether they are actually "real." Under Dr. Frances's leadership...at least DSM-IV was reliable.

[Dr Joanne]

11:42 PM on 05/08/2012

The APA is going to incite a mental and emotional well-being and health revolution.