Photo: "Stephen Hawking.StarChild" by NASA
"The greatest enemy of knowledge is not ignorance, it is the illusion of knowledge." -- Stephen Hawking
Readers of this blog are aware that I am enthusiastic about the EDGAR definitions of "strong" and "moderate" evidence of effectiveness for educational programs. However, EDGAR also has two categories below "moderate." These are called "promising" and "based on strong theory." I never mention these, but I thought it's time to do so, because they come up in current policy debates.
Advocates for evidence in education want federal policies to seek to increase the use of proven programs, and I have frequently written in support of encouraging schools to use these programs when they exist and building them when they don't. But some would expand the definition of "evidence-based" to make it too broad. In particular, in areas in which very few programs meet at least the "moderate" level of EDGAR, they want to allow "promising" or "based on strong theory" as alternatives. "Promising" means that a program has correlational evidence supporting it. With the right definition, this may not be so bad. However, "based on strong theory" just means that there is some explanation for why the program might work.
In a moment I'll share my thoughts about why encouraging use of programs that only have "strong theory" is a bad idea, but first I'd like to tell you a story about why this discussion is far from theoretical for me personally.
When I was in college, I had a summer job at a school for children with intellectual disabilities in Washington, DC. Another college sophomore and I were in charge of the lowest-performing class. We overlapped by a week with the outgoing teacher, who showed us in detail what she did with the children every day. This consisted of two activities. One involved getting children to close their eyes and smell small jars of substances, such as garlic, cinnamon, and cloves, and say what they were. The other involved having children string beads in alternating patterns. These activities were directly from a theory called Psychomotor Patterning, or Doman-Delcato, which was extremely popular in the 1960s and early 1970s. In addition to sensory stimulation, advocates of Doman-Delcato had children with intellectual disabilities crawl and do other stylized exercises on the theory that these children had skipped developmental steps and could catch up to their peers by going back and repeating those steps.
In our school, my partner and I started off dutifully continuing what the teacher had shown us, but after a few days we looked at each other and said, "This is stupid." We knew that our kids, aged perhaps 11-15, had two potential futures. If they were lucky, they could stay at home and perhaps get a job in a sheltered workshop. Otherwise, they were likely to end up in the state hospital, a terrible fate. We decided to drop the patterning, and teach our kids to tie their shoes, to sweep, to take care of themselves. We began to take them on walks and to a local McDonalds to teach them how to behave in public.
One of our children was a sweet, beautiful girl named Sarah, about 12 years old. Sarah was extremely proud of her long, blond hair, which she would stroke and say, "Sarah's so pretty," which I'm sure she'd heard countless times.
I was working especially hard with Sarah, and she learned quickly. I taught her to sweep, for example, starting with balled-up paper and moving to smaller and smaller things to sweep up.
One day, Sarah was gone. We heard that her parents had taken her to the state hospital.
For some reason, the parents brought Sarah back for a visit about a month later. Her beautiful hair was gone, as was the sparkle that had once been in her eyes. She stared at the floor.
A few years later, in another school, I saw teachers working with teenagers with Down Syndrome, having them crawl around the classroom every day. Like Sarah, these kids had two potential futures. This school had a sheltered workshop housed in it, and if they could qualify to work there, their futures were bright. Instead, they were wasting their best chances crawling like babies.
"Based on strong theory" may sound academic or policy-wonky to many, but to me, it means that it is okay to subject children to treatments with no conclusive evidence of effectiveness when better treatments exist or could exist. In particular, "based on strong theory" all too often just means "what's fashionable right now." Doman-Delcato Psychomotor Patterning was a hugely popular fad because it gave parents and teachers hope that intellectual disabilities could be reversible. When I was a special education teacher, "based on strong theory" meant that my kids had received years of useless sensory stimulation instead of learning anything potentially useful. Perhaps Sarah was going to end up in the state hospital no matter what my school did. But I cannot set aside my memory of her when I hear people say that "strong theory" might be enough when actual evidence is lacking.
From a policy perspective, it would be useful to have federal and state funding support programs with strong or moderate evidence of effectiveness. In areas lacking such evidence-proven programs, government might fund research and development, perhaps, but should not encourage use of programs that are only supported by "strong theory." Allowing weak categories into the discussion waters down the entire idea of evidence-based reform, as all programs could probably meet such definitions. Even worse, encouraging use of programs based on strong theory could lead schools to use the current fad. And if you doubt this, ask Dr. Doman and Dr. Delcato.