When Minnesota Teen Challenge (MNTC) responded to my recent blog entry about their anti-drug program, they cited a "study" to back their claims of being an effective treatment for addiction.
What this paper actually shows is how easy it is produce good looking numbers. In that way, it's actually quite instructive for anyone who wants to understand addiction research--or wants to avoid being taken in by exaggerated outcome data.
Even more intriguingly, MNTC failed to mention that the success rate they proudly cited is for adults--but buried in their own study is a much more dismal picture for teenagers.
Their uncontrolled research includes the classic statistical ploys used by drug programs for decades to inflate success rates. At first glance, a claim of 74% of graduates abstinent without relapse for six months sounds pretty good. OK, it's only six months--but often, if you can make it six months at a time without relapsing, you are doing pretty well.
Let's look a little closer at that number and how it was generated, however. Start with the fact that MNTC participants are not just "addicts off the street." They are seeking treatment--either because they have to in order to avoid prison or because they have decided they want to stop using.
Most have been through a detox program to help with withdrawal--and many will have dropped out before completing that. MNTC participants have also consented to attend a highly religious rehab--or had their parents consent for them. This suggests that we have already eliminated many of the addicts who aren't motivated to recover before they even set foot in the door.
Because there is no control group, all of those facts already mean that any success in the program we see could be due to pre-existing motivation: not to anything special about the rehab. Only with a control group of similar people who get no treatment or attend a different rehab can we really tell what works and what doesn't.
The second important clue to pick up is that word "graduates." This means that no one who started but dropped out of the program was counted in the success rates. Since the people most likely to relapse drop out of treatment quickly--and since long-term rehabs like Teen Challenge typically report 50-70% drop out rates-- this means that this 74% does not come from a representative sample from the start.
When you do a clinical trial, you have to include drop-outs as failures to make your results as applicable as possible to the "real world." Why? Well, let's say you have a drug that looks like a fabulous antidepressant in rats. You give it to 100 people and one person stays in the trial and is no longer depressed - but 99 drop out because the drug also causes an intense itching sensation all over the body. Not gonna be a blockbuster--not a useful drug, period. Your stock's gonna tank!
But by MNTC's measure, there's a 100% success with that one person who didn't get the side effect!!!
The executive director of national Teen Challenge has admitted to a 35-40% drop out rate to the New York Times.
That is unusually low for a long term program--so let's use the low end of the typical drop out rate and say that 50% quit without graduating. (MNTC has also said that it uses different practices than the the national organization).
That cuts their 74% success rate in half to 37% if we look at this as what is called an "intention to treat," study and count the dropouts as failures. That already puts them around average compared to other programs. Your average drug program can get about 30-40% clean for six months--but most of their studies have the same problem of having selected for motivated patients before the research even begins.
And with MNTC, there's another wrinkle. When they surveyed the graduates, only 55% agreed to participate in the follow up study. This is another red flag. If you think about it, who is most likely to be available to participate in a follow up on rehab success--the guy who is employed and off drugs or the one in jail or in the basement smoking meth?
Who is most likely to want to talk to a representative of a rehab--the woman who got clean or the one who is now depressed about failure, particularly if it's a religious rehab where failure means return to sin? Even if the surveyors state their independence from the program, it's hard to know whether addicts will actually believe this.
We can't assume that all of the 45% of the graduates who don't respond failed--it's just much more likely that the responders will be successes than failures. So, if we again give them the benefit of the doubt and assume that only half of these have relapsed (again, a very generous assumption), we can say that roughly 31% of the total sample has stayed off drugs for six months. If we count all of the nonresponders as failures, that yields a 20% abstinence rate at six months for graduates.
Which is pretty close to the 15% who get clean, on average at any given time, with no treatment whatsoever! (Of course, if you wanted to lie with statistics, you could say that you "improved recovery rates over no treatment by 33%!!). The extra five percent, however, could well be a selection effect due to the motivated patients being the only ones who started treatment in the first place.
In addition, there's an even more frightening statistic hidden in the MNTC report (and not mentioned by the program's defenders when they complained about my reporting). The 74% six month abstinence rate is for adults.
Only 37% of the teen graduates reported having been abstinent for the most recent six months--and only 29% of teens reported having had no drugs at all since they left the program. And all the same self-selection issues apply to the teen sample--so the real rates are probably below the natural recovery rate.
To be fair, however, the natural recovery rate is probably lower for youths because most people who stop using without treatment do it when their peers are stopping or cutting back, which tends to be in the 20's or later. And, to be even more scrupulous, only 15% of their sample was teens - so these statistics could rely on a sample size too small to be reliable.
Nonetheless, there is absolutely no basis for this program to make any claim of superiority to other treatments-- some of which do demonstrate greater success without playing with data and without the risks carried by being outside of mainstream medicine.
Follow Maia Szalavitz on Twitter: www.twitter.com/maiasz
It just doesn't make sense to me why someone would attack a program that does so much good. There is so much bad going on in the world, and Maia chooses to focus her attack on something that is changing lives. Even IF there numbers are off, couldnt you find a better way to make a living? It is sad and pathetic to think that you made money by tearing into a program that is changing lives.
And four years later, I am. I'm 25 years old. I spent 8 years addicted to drugs. Since TC I've completed my AA degree in college, making the dean's list every semester. I've been accepted into a private college and am waiting to find out if I have been selected for a full academic scholarship worth almost sixty thousand dollars. I have also worked the same full time job, which is the first job I have ever had. I dont smoke, drink, or do any form of drugs. I speak to youth groups about addiction. Attack all you want, but at the end of the day there is me, thankful to the point of tears because of the chance TC gave me at life. There is hope through Christ alone. If your idealogy blinds you from seeing the reality of lives changed, maybe you are the one beyond hope, not me.
These are folks who unabashedly toss around phony number to trounce upon those they perceive as their competition - namely “Alcoholics Anonymous”. I see the totally fabricated "2-5% “success rate” used for AA when AA HAS NO "success rate."
AA cannot be rated and compared to alcoholic treatment programs because AA's objectives and goals are different from those other institutions.
I know of an AA HomeGroup for example where 83% of those who have walked into that Homegroup and adopted the 12 Step Program recovered from alcoholism with the desire to drink completely removed within thirty days. THAT is my experience. Those who didn’t make it and returned to drinking and getting drunk either never needed the recover to begin with, weren’t qualified or just plain dropped out because it was too hard and opted instead for very expensive but much easier to do month-long stay at a "Rehab" facility. If they were not true alcoholics it seemed to have worked for them and if they were true alcoholics they are either now dead or still drinking.
Peace,
Danny S - RLRA
Real Live Recovered Alcoholic
http://recoveredalcoholic.blogspot.com/
Something that is almost never said in the media is that virtually all treatment is religious. The bulk of it is 12 Step, groups like Alcoholics Anonymous, Narcotics Anonymous, Messies Anonymous and Al-Anon. The remainder is mainly Fundamentalist Christian groups and Scientology.
Peace,
Danny S - RLRA
Real Live Recovered Alcoholic
http://recoveredalcoholic.blogspot.com/