I have always had my suspicions about international comparisons, but people in the U.S., and especially in the educational research community take them hook, line and sinker. I was disappointed to see PISA results quoted as if they were gospel in Democracy at Risk from The Forum for Education and Democracy. No TIMSS (on which we look better). No PIRLS (on which we look good indeed). Just PISA. Having just perused part of a book of chapters critical of PISA, all written by Europeans, I can see why: For critics here, PISA is just a general cudgel. As one of the authors points out, "it only confirms what was known from earlier international studies -- that American students don't do very well in such tests compared to the students of many other nations." In parts of Europe governments and other agencies have tried (erroneously) to draw grand curricular and structural conclusions from it.
From the introduction to this book:
"What has emerged is a picture not unlike the behavior of large companies when they encounter a potential scandal" (the section is titled "PISA -- The Contergan of Educational Research?" Contergan was the firm that made thalidomide).
"If some critique is voiced in public, the first response seems be silence." (People affiliated with PISA were repeatedly invited to contribute to the volume. None would.)
"If that is not enough, the next step is often to raise doubts about the motives and the abilities of those who are critical of the enterprise.
"The next step seems to acknowledge some problems, but to insist that they are very limited in nature and scope, not affecting the overall picture. Alternatively, it is pointed out that these problems are well known within large scale survey research of the kind like PISA, and even unavoidable when working comparatively. Of course, this claim does not reduce the impact of these problems on the validity of the results.
"Finally, there is the statement that the criticism does not contain anything new and this claim is often accompanied by references to opaque technical reports that only insiders can understand, or to unpublished reports."
Sound familiar?
From a chapter from a Joachim Wuttke at the Forschungszentrum Julic in Munich: "Some codes [for right answers] are kept secret because national authorities want to prevent certain analyses."
"Only Luxembourg has scanned and published some student solutions to free-response items; these examples show that students sometimes misunderstood what the item writer meant to ask."
Although PISA permits 0.5% exclusion for organizational reasons and 4.5% for intellectual or functional disabilities, the decisions about exclusion depended only on "the professional opinion of the school principal or other qualified staff," a completely uncontrollable source of uncertainty.
"Items that did not fit into the idea that competence can be measured in a culturally neutral way on a one-dimensional scale were simply eliminated. Field test results were never published. This adds to Olsen's observation that in PISA-like studies, the major portion of information is thrown away."
This from Svein Sjoberg at the University of Oslo:
An observer noticed Taiwanese parents gathered with their children before TIMSS testing. The director of the school gave and address exhorting them to do their best and the national anthem was played as the children entered. (Archie Lapointe reported a similar phenomenon in Korea--as each child's name was called, he or she stood up to wild applause from the rest of the class. Such an honor to perform for the country!.)
"In this country [Singapore], only one thing matters: Be best--teach to the test."
As the test nears in Singapore, stores have special offers: At the bottom of a shelf are painkillers and at the top, collections of exam papers for parents to buy for their children (a picture of same accompanies the text).
This from Stefan Hopman at the University of Vienna, talking about NCLB:
The core of accountability is narrowly focused on student achievements measured by 'academic standards.'" Other functions of schooling (such as the role school plays for locale communities or in shaping society) are hardly mentioned, and if at all they are constructed as minor problems...The starting idea of the "basic principles of curriculum and instruction (Tyler, 1949), which was embedded in the methodologically much broader approach of the Eight-Year Study (Aikin, 1942) which asked for a comprehensive understanding of schooling as a social and local institution, has dwindled to a concept of measurable yearly progress."
Advertisement