THE BLOG
10/02/2015 05:51 pm ET Updated Dec 06, 2017

Are the PISA Data for Shanghai Flawed?

Introduction
Cross-national comparison of educational performance has become an important building block for national educational policies (Martens, Rusconi, & Leuze, 2007; Meyer & Benevot, 2013). The PISA data, collected by the OECD, has become one of the main tools for this cross-national comparison, and the number of participating countries outside the OECD has increased strongly since the first PISA wave of 2000.
The participation of Shanghai in the PISA measurement since 2009 has opened a debate whether the published outcomes for Shanghai are representative for the 15 year old pupils living in Shanghai. Some authors (for instance Loveless, 2014) suggest that only pupils who are legally living in Shanghai are sampled and thus a very substantial proportion of the 15-year-old pupils without Hukou (permission to migrate from the Chinese countryside into cities like Shanghai, and which is obligatory for entrance into high school) are omitted from the sample. It is clear that Hukou is still relevant in modern China and that is influences educational and occupational opportunities of immigrants from the Chinese countryside into the cities (see for instance Zhang & Treiman, 2013). But the relevance of Hukou in modern China is no prove of the omission of pupils without Hukou from the Shanghai PISA sample.
This research note adds to this debate by comparing the educational performance and social background of pupils of first and second generation immigrants from China mainland into Hong Kong, Macao en Shanghai. All three cites have substantial number of these pupils. Immigration into these three cities from mainland was and still is difficult, due to economic en legal obstacles to establish oneself in these cities. But an immigrant needs also a Hukou as an additional requirement for a legal settlement in Shanghai, while a Hukou is not needed for Hong Kong or Macao. If the PISA sample of the Shanghai pupils is representative for the whole population of 15-year-old pupils, one would expect minor differences in educational performance and social background, especially of first generation pupils (= born in mainland China, thus not born in Hong Kong or Macao), because it is equally difficult to enter each of these cities, migrating from the countryside. If the PISA sample of the Shanghai pupils contains only the pupils, who lives legally in Shanghai, one would expect major differences in educational performance and social background of first generation pupils (= born in mainland China, outside test-city) and second generation pupils (=parent(s) born in mainland China, outside test-city), and these differences would be in favor of Shanghai pupils.

Data
I use the PISA 2009 wave and selected only those pupils in the test-cities Hong Kong, Macao en Shanghai who were born in mainland China outside these three test-cities or of whom at least one parent was born in mainland China outside these three test-cities. If both parents are born in one of the test-cities, the pupil was excluded from the analysis. First generation means that the pupil is born in mainland China and not in the test-city. Second generation means that the pupils in born in the test-city but at least one of the parents is born in mainland China outside the test-cities. I follow the method and data-coding more elaborately described in Dronkers & Kornder (2014).
Reading and math scores are the averages of the five plausible values at the reading and math test. 500 is the average score of the test of the OECD pupils, with a standard deviation of 100. Mixed marriage means that one parent is born in the test-city and another parents outside the test-city. ESCS is the index of the economic, social, and cultural status of the parents. This variable represents a composite index created in the PISA dataset of the parents' occupational status (Ganzeboom, de Graaf, Treiman and de Leeuw, 1992), the parents' educational level (United Nations Educational, Scientific and Cultural Organization, 2006), and the presence of any material or cultural resources at the pupils' homes. This combination of the parents' occupational status and educational level together with resources at home produces the strongest indicator of parental environment (OECD, 2010). The ESCS score was standardized such that the OECD average was set to zero, with a standard deviation of 1.00. Native reading or math scores. These indicators are the average PISA score of the total native population of the three test-cities. This variable serves to approximate the quality of the destination country's educational system.

Results
Table 1 shows simple descriptive statistics. First generation migrant pupils from mainland China in Shanghai have substantially higher readings and math scores (546; 589) than first generation pupils from mainland China in Hong Kong (515; 526) or Macao (489; 520). These Shanghai pupils hardly come from mixed parents (9%), in contrast to those in Hong Kong (24%). The socio-economic background of these Shanghai pupils is much better (-.38) than those in Hong Kong (-1.40) and Macao (-.87).
The same holds even more for the second generation. Second generation migrant pupils from mainland China in Shanghai have substantially higher readings and math scores (570; 619) than second generation pupils from mainland China in Hong Kong (545; 565) or Macao (489; 529). However, these second Shanghai pupils come very often from mixed parents (61%), in contrast to those in Hong Kong (33%) or in Macao (16%). The socio-economic background of these Shanghai pupils is much better (-.44) than those in Hong Kong (-1.04) and Macao (-.81).

2015-10-02-1443811503-8999200-Table1.png

Table 2 tests whether the background variables have the same impact on the educational performance of migrant pupils from mainland China in Hong Kong, Macao and Shanghai.
The impact of parental socio-economic background on the reading score is stronger in Shanghai (30,9) than in Hong Kong (17.1= 30.9-13.8) or in Macao (12.6=30.9-18.3). The same holds for one-parent born in test-city (16.8 versus -7,3 and -5.9) and only for Macao the second generation (18.0 versus 24,3 and -0.8). The results for the math scores are analogous.

2015-10-02-1443811708-1126950-Table2.png

Conclusion
The PISA migrant pupils in Shanghai originating from mainland China have substantial higher educational performance and a better socio-economic background than comparable migrant pupils in Hong Kong or Macao. Also the impact of the background variables on educational performance of migrant pupils is substantially stronger in Shanghai and in Hong Kong or Macao, which means that the PISA migrants in Shanghai fits better within the local inequality structure than the PISA migrants in Hong Kong and Macao. On average the impact of background variables is smaller for immigrant pupils than for native pupils (OECD, 2010).
Given that is was and still is difficult to migrate to these three cities from mainland China, these differences suggest unmeasured selectivity of the PISA Shanghai migrant pupils. The importance of possessing Hukou for establishing oneself in Shanghai and thus being allow to attend better primary and secondary schools might be a sufficient and economical explanation of this unmeasured selectivity of the PISA Shanghai migrant pupils. The surprising high percentage of mix marriage for the second generation supports this Hukou interpretation: marrying a local with a permit is the golden way for any illegal immigrant to settle legally after migration.
This unmeasured selectivity of the PISA Shanghai migrant pupils does not mean that Chinese migrant pupils do not score on average high on the PISA tests. Also outside Asia Chinese migrant pupils are high performers (see Dronkers & Heus, no year). But the unmeasured selectivity of the PISA Shanghai migrant pupils makes also the high reading and math scores of the native Shanghai pupils (555; 599) in comparison with the scores of the Hong Kong native pupils (537; 562) and the Macao native pupils (480; 522) questionable. It might be that third generation Chinese migrants without Hukou are not included in the Shanghai PISA sample in contrast to Hong Kong and Macao.
This unexplainable selectivity of the PISA Shanghai data does not invalidate the whole PISA data collection, but it should make us aware of some important limitations of the PISA data (Mortimore, 2009). Moreover, governments and interest groups use these cross-national data to promote their own interests and the OECD as intergovernmental organization might have other goals than only scientific research.

References
Dronkers, J. & Heus, M. de (no year). The higher educational achievement of Chinese pupils, inside and outside of Asia: the higher transparency of Chinese numbers or a higher value of learning within Chinese culture? Unpublished manuscript. www.eui.eu/Personal/Dronkers/English/Heus3.pdf
Dronkers. J. & Kornder, N. (2014). "Do migrant girls perform better than migrant boys? Deviant gender differences between the reading scores of 15-year-old children of migrants compared to native pupils" Educational Research and Evaluation: An international Journal on Theory and Practice, 20:1 44-66.
Ganzeboom, H. B. G., de Graaf, P., Treiman, D. J. & De Leeuw, J. (1992), "A standard international socio-economic index of occupational status". Social Science Research, 21, 1-56.
Loveless, T. (2014). Lessons from the PISA-Shanghai Controversy. Part I of the 2014 Brown Center Report on American Education. Brookings Institute, Washington D.C.: www.brookings.edu/research/reports/2014/03/18-brown-center-report-loveless
Martens, K., Rusconi, A., & Leuze, K. (Eds.) (2007). New arenas of education governance - The impact of international organisations and markets on educational policymaking, London: Palgrave
Meyer, H.-D. & Benevot, A. (eds.) (2013). PISA, Power, and Policy: the emergence of global educational governance, Oxford: Symposium Books.
Mortimore, P. (2009). Alternative models for analysing and representing countries' performance in PISA. Brussels, Belgium: Education International Research Institute.
Organisation for Economic Co-operation and Development (2010). PISA 2009 Results: What Students Know and Can Do. Paris: Organisation for Economic Co-operation and Development
United Nations Educational, Scientific and Cultural Organization (2006). ISCED 1997: International Standard Classification of Education. Re-edition. Paris: United Nations Educational, Scientific and Cultural Organization
Zhang, Z. & Treiman, D.J. (2013). "Social Origins, Hukou Conversion, and the Wellbeing of Urban Residents in Contemporary China". Social Science Research 42(1):71-89.