NEW YORK -- Lawmakers should be more careful when using international test scores to drive education policy, argues a pair of researchers in a new paper for the left-leaning think tank Economic Policy Institute -- because the results aren't always what they appear to be.
According to a new paper released Wednesday, the average scores on international tests -- the numbers over which advocates and politicians do much public hand-wringing -- don't tell the whole story of America's academic performance, and inferences based on those averages can be misleading, Stanford education professor Martin Carnoy and researcher Richard Rothstein argue. They found that contrary to popular belief, international testing information shows that America's low-income students have been improving over time.
"Policymakers are quite sophisticated about domestic test scores," Rothstein said in an interview. "We know that student achievement varies by social class. We were concerned that all of this publicity about international test scores reports simply average national scores." The study touches on a question that has long driven education policy debates: What effect does poverty have on students' abilities to learn?
When latest results of the international exams known as TIMSS and PIRLS were released in late 2012, U.S. Secretary of Education Arne Duncan released a public statement expressing his disappointment in the performance of U.S. students. Carnoy and Rothstein argue that his statement is based on the misleading data.
Since in-depth information on TIMSS and PIRLS was not available at the time Rothstein was conducting his research, he looked at a different test: the PISA, which is administered by the Organisation for Economic Co-operation and Development. Rothstein's analysis looked at three top-performing countries (Canada, Finland and Korea) and three average-performing countries (the United Kingdom, France and Germany) and sought to determine whether test scores correlated to students' socioeconomic backgrounds, and to assess how those groups did over time.
But Andreas Schleicher, the creator and administrator of the PISA, contests their findings. After Carnoy and Rothstein invited him to participate in a press conference on the paper, he declined. "My view is that this paper contains too many fundamental misunderstandings and misinterpretations of the data for us to be joining a press call on this," Schleicher wrote to the authors, according to an email provided to The Huffington Post.
Rothstein found that the U.S. is more unequal in social background, so he wondered whether differences between the average U.S. scores and those of its competitors were driven by that inequality. Rothstein said he was not surprised by his findings, given that the achievement gap between rich and poor U.S. students has always been large. "Higher social class students have higher average scores than lower social class students," he said.
But compared to other countries, Rothstein found that the income achievement gap in the U.S. is actually smaller. "You cannot draw valid conclusions from a single test, and you cannot draw conclusions from average test scores," Rothstein said. "Bad policy flows from it. The achievement of disadvantaged students is rising and it's higher than post-industrial countries."
When asked for a specific example of such a policy, Rothstein pointed to the general focus on students on the low end of the income range. "If you look at these international test scores, it suggests that our problems are with our advantaged students," he said. "It does not compare favorably with the achievement of students in other countries who are advantaged. Disadvantaged students are going up phenomenally."
The study found that if U.S. high school students had similar social class distributions to peer countries, reading scores would be above average, and math scores would be about average. Disadvantaged students would perform better than comparable students in other countries in reading, and the same in math.
"The performance of the lowest social class U.S. students has been improving over time, while the performance of such students in both top-scoring and similar post-industrial countries has been falling," Rothstein and Carnoy wrote.
However, Rothstein said that he was surprised by what else he found in the PISA scores: sampling errors. According to his analysis, PISA made a sampling error by overrepresenting students from low-income families in the U.S.:
A sampling error in the U.S. administration of the most recent international (PISA) test resulted in students from the most disadvantaged schools being over-represented in the overall U.S. test-taker sample. This error further depressed the reported average U.S. test score.
In a written response, OECD's Schleicher contested this assertion, calling it "simply incorrect and unsupported in the paper." Schleicher followed his claim with a few PISA graphs and quotes from the new paper that appear to substantiate his claim, showing that the inference of an overrepresentation of low-income families comes from two sources "that, for one reason or another, are not consistent."
After reading Schleicher's response, the authors suggested that his points don't address their main argument.