It's not unusual to read that a new study has failed to replicate -- or has even reversed -- the findings of an earlier study. The effect can be disconcerting, particularly when medical research announces that what was supposed to be good for us turns out to be dangerous, or vice versa.
Qualifications and reversals also show up in investigations of education and human behavior, but here an interesting pattern seems to emerge. At first a study seems to validate traditional practices, but then subsequent studies -- those that follow subjects for longer periods of time or use more sophisticated outcome measures -- call that result into question.
That's not really surprising when you stop to think about it. Traditional practices (with respect to teaching students but also to raising children and managing employees) often consist of what might be called a "doing to" -- as opposed to a "working with" -- approach, the point being to act on people to achieve a specific goal. These strategies sometimes succeed in producing an effect in the short term. Research -- which itself is often limited in duration or design -- may certify the effort as successful. But when you watch what happens later on, or you look more carefully at the impact of these interventions, the initial findings have a way of going up in smoke.
Consider three such traditional practices: using rewards to change people's behavior, making students do additional academic assignments after they get home from school, and teaching by means of old-fashioned telling (sometimes known as direct instruction). What happens in each case when you look at short-term results, and what happens when you then extend the length of the study?
1. An ambitious investigation of various types of preschools looked specifically at children from low-income families in Illinois. The two educational approaches that produced the greatest impact on achievement in reading and arithmetic were both highly structured, one of them a behaviorist technique called Direct Instruction (or DISTAR) that emphasizes the use of scripted drill in academic skills and praise for correct responses.
Most studies would have left it at that, and the press doubtless would have published the findings, which suggests that this model is superior to more child-centered preschools. (Take that, you progressives!) Luckily, though, this particular group of researchers had both the funding and the interest to continue tracking the children long after they left preschool. And it turned out that with each year that went by, the advantage of two years of regimented reading-skills instruction evaporated, soon proving equivalent -- in terms of effects on test scores -- to "an intensive 1-hour reading readiness support program" that had been provided to another group. "This follow-up data lends little support for the introduction of formal reading instruction during the preschool years for children from low-income homes," the researchers wrote.
One difference did show up much later, however: Almost three quarters of the kids in play-oriented and Montessori preschools ended up graduating from high school, as compared to less than half of the direct-instruction kids, which was about the same rate for those who hadn't attended preschool at all. (Other longitudinal studies of preschool have found similar results: The longer you track the kids, the more likely that a drill-and-skill approach will show no benefits and may even appear to be harmful.)[1]
2. For various reasons that I've reviewed elsewhere, there's reason to doubt that requiring children to do homework has any meaningful academic benefit. In elementary school in particular, there isn't even a correlation between doing homework (vs. doing none) or doing more homework (vs. doing less), on the one hand, and any measures of achievement -- even such conventional (and, I believe, dubious) measures as grades or standardized test scores. But one prominent researcher -- who does place stock in these measures -- noticed something interesting when he reviewed 48 comparisons in 17 published reports of research projects that had lasted anywhere from two to thirty weeks: The longer the duration of the study, the less impact that homework had.[2]
This researcher speculated that less homework may have been assigned during any given week in the longer-lasting studies, but he offered no evidence that this was true. So here's another theory: The studies finding the greatest effect were those that captured less of what goes on in the real world by virtue of being so brief. View a small, unrepresentative slice of a child's life and it may appear that homework makes a contribution to school achievement; keep watching and that contribution is eventually revealed to be illusory.
3. Many people who are concerned with promoting healthy lifestyles assume that it makes sense to offer an incentive for losing weight, quitting smoking, or going to the gym. The only real question on this view is how to manage the details of the reward program. In an experiment published in 2008, people who received either of two types of incentives lost more weight after about four months than did those in the control group. (Unfortunately, there was no non-incentive weight-loss program; subjects got either money or no help at all.) At the seven-month mark, however, the effect melted away even if the pounds didn't. There was no statistically significant weight difference between those in either of the incentive conditions and those who received nothing. This result, by the way, is typical of what just about all studies of weight loss and smoking cessation have found: The longer you look, the less chance that rewards will do any good -- and they may actually do harm.[3]
4. Belief in the value of rewards is, if anything, even stronger in the corporate world, where it's widely believed -- indeed, taken on faith -- that dangling financial incentives in front of employees will cause them to work harder. Conversely, if workers are provided with such an incentive and it's then removed, their productivity would be expected to decline. An unusual occurrence in a manufacturing company provided a real-world opportunity to test this assumption: A new collective bargaining agreement for a group of welders resulted in the sudden elimination of a long-standing incentive plan. The immediate result was that production did indeed drop. But as with the preschool study, this researcher decided to continue tracking the company records -- and discovered that, in the absence of rewards, the welders' production soon began to rise and eventually reached a level as high or higher than it had been before.[4]
5. Sometimes a different result emerges when a new study is done better as opposed to merely lasting longer. The topic of homework provides a striking example. One of the most frequently cited investigations in the field was published in the early 1980s by a researcher named Timothy Keith, who looked at survey results from tens of thousands of high school students and concluded that homework had a positive relationship to achievement, at least at that age. But ten years later, he and a colleague took a closer look -- this time considering homework alongside other possible influences on learning such as quality of instruction, motivation, and which classes the students took. When all these variables were entered into the equation simultaneously, the result was "puzzling and surprising": Homework no longer had any meaningful effect on achievement at all, even in high school.[5]
6. Finally, what happens when a second researcher comes along and does a study that's both longer and better than the original? Consider a report published in 2004 that showed third and fourth graders who received "an extreme type of direct instruction [in a science unit] in which the goals, the materials, the examples, the explanations, and the pace of instruction [were] all teacher controlled" did better than their classmates who were allowed to design their own procedures. Frankly, the way they had set up the latter condition wasn't representative of the strategies most experts recommend for promoting discovery and exploration. Nevertheless, the finding may have given pause to progressive educators -- at least in the context of elementary school science teaching.
Or, rather, it may have given them pause for three years. That's how much time passed before another study was published that investigated the same issue in the same discipline for kids of the same age. The two differences: the second study looked at the effects six months later instead of only one week later; and the second study used a more sophisticated type of assessment of the students' learning. Sure enough, it turned out that any advantage of direct instruction disappeared over time. And on one of the measures, pure exploration not only proved more impressive than direct instruction but also more impressive than a combination of the two -- which suggests that direct instruction can be not merely ineffective but positively counterproductive.[6]
Despite their diversity, these six sets of studies hardly exhaust the universe of research that forces a reevaluation of what came before. Still, any observer willing to connect the dots may end up not only waiting for replications to be performed before accepting any preliminary conclusion -- a reasonable posture in general -- but more skeptical of studies that seem to support traditional practices in particular.
1. Merle B. Karnes, Allan M. Shwedel, and Mark B. Williams, "A Comparison of Five Approaches for Educating Young Children from Low-Income Homes." In As the Twig Is Bent . . .: Lasting Effects of Preschool Programs, ed. by the Consortium for Longitudinal Studies (Hillsdale, N.J.: Erlbaum, 1983). For a summary of other research on early-childhood education, see www.alfiekohn.org/teaching/ece.htm.
2. Harris Cooper, The Battle Over Homework. 2nd ed. Thousand Oaks, CA: Corwin, 2001, p. 16.
3. Kevin G. Volpp et al., "Financial Incentive-Based Approaches for Weight Loss," Journal of the American Medical Association, 300 (December 10, 2008): 2631-37. For a review of other research on the (lack of long-term) effects of financial incentives on weight loss and smoking cessation, see www.alfiekohn.org/miscellaneous/healthresearch.htm. That review also appears in Alfie Kohn, Feel-Bad Education...And Other Contrarian Essays on Children and Schooling (Boston: Beacon Press, 2011), pp. 199-204.
4. Harold F. Rothe, "Output Rates Among Welders: Productivity and Consistency Following Removal of a Financial Incentive System," Journal of Applied Psychology 54 (1970): 549-51.
5. Valerie A. Cool and Timothy Z. Keith, "Testing a Model of School Learning: Direct and Indirect Effects on Academic Achievement," Contemporary Educational Psychology 16 (1991): 28-44.
6. The original study: David Klahr and Milena Nigam, "The Equivalence of Learning Paths in Early Science Instruction: Effects of Direct Instruction and Discovery Learning," Psychological Science 15 (2004): 661-67. The newer study: David Dean, Jr. and Deanna Kuhn, "Direct Instruction vs. Discovery: The Long View," Science Education 91 (2007): 384-97.
Follow Alfie Kohn on Twitter: www.twitter.com/@alfiekohn
Dr. Mariappan Jawaharlal: Why Is STEM Boring?
Stella Moga: Is That Really a Head Start?
Wray Herbert: Are Parental Decisions Driven by Reason or Emotions?
I was always 'that kid.'
Rotten social skills. Yet I was right on track with kids a year or two my junior; the IQ marginally outpaced the EQ and for this I was a lifelong pariah. Lousy homework and study habits. (These days they say I'm ADD and feed me stimulants. I don't necessarily disagree with the diagnosis, but I am skeptical that the condition is in fact pathological so much as it is a poor fit for factory work.) Set the curve on every test. Even in graduate school I was stellar on exams -- couldn't bring myself to LOOK at homework assignments, toward the end, but got perfect exam scores in the rather unforgiving field of computer science. Obviously the material was never the problem. The environment was.
I don't claim to know what the solutions are but I know in cases like mine, the status quo is a recipe for a suicidal adult. That's the one field I can really consider myself an expert in. But that's always been my problem, hasn't it? Putting theory into practice.
What I saw during that week of shadowing was the most horrifying display of teaching one could dream about. To stay on topic I'll discuss the homework policy of this teacher and school.
There was a 'Homework Policy' school wide. Ten minutes per grade level, 1st grade was 10 minutes a night, 2nd 20 mins, 3rd 30 mins, 4th 40 mins, and so on.
On my fourth day of shadowing, I was put in charge of reviewing homework. The brightest student in that classroom, and one of the brightest students I've worked with, came to my table bawling. She couldn't talk she was crying so hysterically. She forgot to do the homework the night before, and the teachers punishment for this was detention.
My first day as lead teacher I asked for somebody to retrieve the specially designed homework detention slips. I broke up the pile and gave each student a stack of about twenty each. I placed three trash cans around the room. I crumpled up the first detention slip and tossed it like a basketball to the 'hoop' in the far corner of the room, it banked off the wall and scored. You would think I just drained the winning shot of the NBA finals. The next fifteen minutes was the funnest those students had had in that classroom all year!
And because of that you cannot standardize teaching. Nor can you standardize evaluation of learning or of teaching effectiveness. Short term doesn't equate to long term. And Americans in particular are very impatient for results.
Most of all, you need to take the politics out if it. Education needs to be run by teachers. Not politicians or would be politicians. Not lawyers or accountants or housewives, used car salesmen or travel agents just because they happened to get elected to a school board. In fact, the first thing that needs to go is school boards.
As soon as teaching is relieved of the wrongful notion that children are widgets, there can be standardization and there is such a thing as an average, teachers can teach again and students will learn. Let the experts do what they know how to do.
Trust them to get the job done. Pay them with respect and a living wage. Treat them as the professionals they are.
In my state you must have a bachelors degree and then spend an additional year doing post graduate work both studying education pedagogy in the classroom and doing student teaching under the supervision of a master teacher. Both your education dept professor and your master teacher observe you and evaluate you. If they don't pass you, you don't get a teaching credential. There are student who decide after a year of student teaching that teaching is not for them. There are some that decide in the first three years that teaching is not for them and leave before being offered a permanent contract. There are teachers that after becoming a permanent employee opt to leave the profession in the first five years.
Many teachers who continue teaching go back to school to get masters degrees. Even those that don't participate in mandatory professional development.
Poorly trained is some other state but not mine.
Either name names or don't use generalities. In my state, teachers are professionals and perfectly capable of making decisions.