Stanford Economist Rebuts Much-Cited Report That Debunks Test-Based Education
When the National Research Council published the results of a decade-long study on the effects of standardized testing on student learning this summer, critics who have long opposed the use of exams as a teaching incentive rejoiced.
But Eric Hanushek, a Stanford University economist who is influential in education research, now says the "told you so" knee-jerk reaction was unwarranted: In an article released Monday by Harvard University's journal Education Next, Hanushek argues that the report misrepresents its own findings, unjustifiably amplifying the perspective of those who don't believe in testing. His article has even caused some authors of the NRC report to express concerns with its conclusions.
The question of the effects of testing has long plagued education. Few dispute the need to have some way to take stock of what students learn from their teachers. But critics assert that an emphasis on test-related incentives, such as grades for students and grade-based funding for schools, has tamped down on creativity in the classroom, treating kids like identical items on an assembly line -- without the product. Disagreements over the role of testing to shape school outcomes emerged as a crucial flash point in recent discussions about the reauthorization of the No Child Left Behind act. Congress passed NCLB a decade ago, putting into place a system that sanctioned schools based on the scores of its students.
The 112-page-long NRC study came at a critical point during the NCLB discussion -- and it read as a manifesto against the use of testing as a tool to promote learning, Hanushek claims. The report found NCLB to be the most effective test-based policy, but even then, it found that the law's programs moved student performance by eight hundredths of the standard deviation, or from the 50th to the 53rd percentile. Other more low-stakes tests were found to show "effectively zero" effects on achievement. According to the NRC report:
Test-based incentive programs, as designed and implemented in the programs that have been carefully studied, have not increased student achievement enough to bring the United States close to the levels of the highest achieving countries.
"This is an extraordinarily serious and contentious policy issue," Hanushek told The Huffington Post Monday. "I am quite taken aback by people who read the report and said that testing policies don't produce learning. The evidence that they provide indicates that accountability has provided significant positive impacts."
In response to the report, Hanushek titled his article, "Grinding the Antitesting Ax: More bias than evidence behind the NRC panel's conclusions," and jazzed up its first page with a man in overalls, well, grinding an ax. Hanushek concludes:
The NRC has an unmistakable opinion: its report concludes that current test-based incentive programs that hold schools and students accountable should be abandoned. The report committee then offers three recommendations: more research, more research, and more research. But if one looks at the evidence and science behind the NRC conclusions, it becomes clear that the nation would be ill advised to give credence to the implications for either NCLB or high-school exit exams that are highlighted in the press release issued along with this report.
The committee that produced the NRC report formed about a decade ago, in the wake of the implementation of NCLB, the strongest federal test-based accountability law ever passed. The National Academies -- a group of four institutions chartered by Congress to consult on various issues -- launched the committee in 2002, and since then, it tracked the effects of 15 programs that use tests as teaching incentives. According to the report, its members were chosen to represent a balanced mix of view points due, in part, to the "tension between the economics and educational measurement literatures about the potential of test-based accountability to improve student achievement."
Its 17 members included economists such as Duke's Dan Ariely and Boston University's Kevin Lang, educational experts like Harvard's Dan Koretz and Stanford's Susanna Loeb, in addition to a former superintendent, a psychologist, a sociologist and a political scientist. The committee also saw presentations from various experts, including Hanushek himself.
According to Hanushek's analysis, the panel's thorough examination of multiple studies is not evident in its conclusions.
"Instead of weighing the full evidence before it in the neutral manner expected of an NRC committee, the panel selectively uses available evidence and then twists it into bizarre, one might say biased, conclusions," Hanushek wrote.
The anti-testing bias, he says, comes from the fact that "nobody in the schools wants people looking over their shoulders."
Hanushek, an economist, claims that the .08 standard deviation increase in student learning is not as insignificant as the report makes it sound. According to his calculations, the benefits of such gains outweigh the costs: that amount of learning, he claims, translates to a value of $14 trillion. He notes that if testing is expanded at the expense of $100 per student, the rate of return on that investment is 9,189 percent. Hanushek criticized the report for not giving enough attention to the benefits NCLB provided disadvantaged students.
The report, Hanushek said, hid that evidence.
"They had that in their report, but it's buried behind a line of discussion that's led everybody who's ever read it to conclude that test-based accountability is a bad idea," he said. Hanushek reacted strongly, he said, because of the "complacency of many policymakers" who say education should be improved but that there are no effective options.
But Lang, a member of the committee who produced the report, said Hanushek's critique is misguided. "His objection is that he feels that we said stop test-based accountability," he said. "We very clearly did not say that."
Rather, Lang said, the report showed that test-based policies don't produce the effects claimed by their proponents. "What we said was test-based accountability is not having the kind of effect that the rhetoric suggests," Lang continued. "The rhetoric behind test-based accountability is the major force for education reform."
But Paul Hill, a research professor and director of the University of Washington's Center on Reinventing Public Education who also sat in the NRC committee, saw merit in Hanushek's critique. "The conclusions were more negative about the contributions of test-based accountability than his review of the evidence would suggest," Hill said. "That's well worth considering."
Hill said he was slightly concerned with the report itself, and that its tone was a product of a committee comprised of experts with mixed views on testing. "It said that test-based accountability alone won't raise achievement," he said. "I believe that. Test-based accountability, though, with reasonable supplementary policies … is a good idea."
The apparent anti-testing bias, Hill said, came from those on the committee with backgrounds in education.
"This is not a group of wackos," Hill said. "Inside the education profession, there's a lot of resentment against the use of tests."
Loading comments… 
First Posted: 11/23/11 11:05 AM ET Updated: 11/23/11 12:18 PM ET