Did Shanghai cheat on PISA?

BY VALERIE STRAUSS

• December 15 at 12:13 pm

•  

 

Shanghai       (By Carlos Barria/Reuters)

Everybody knows (or should) by now that Shanghai’s 15-year-old students ranked No. 1 on the most recent Program for International Student Assessment for the second time, which, if you believe the recently released results, means that a representative sample of kids got the best scores in the world in reading, math and science. And 15-year-olds in the United States performed no better than averageof 65 countries and education systems, which is pretty much what they always do on these international tests.

There are a lot of reasons to disbelieve that the results mean what PISA enthusiasts say they do — that the average performance of U.S. kids means the future of the U.S. economy and national security is at stake (here’s a good explanation) — but new questions have been raised about whether the Shanghai students who took the 2012 PISA really are representative of the city’s population of 15-year-olds.

The Organization for Economic Cooperation and Development, which sponsors PISA and released the 2012 data earlier this month, has said that the schools that were used in the sample do represent the city’s 15-year-old population and that people who say otherwise don’t understand changes in Shanghai policy that have allowed migrant students from rural parts of China to enroll in the city’s schools.

But one researcher has called out the OECD on this subject, and some China experts say that migrant children are still routinely excluded from schools in Shanghai, which is wealthier than the rest of China and is not itself representative of the rest of China (though OECD officials have said that unpublished results of some rural areas of China from where the Shanghai migrants presumably came show good performance.)

Tom Loveless, a senior fellow at the Brown Center on Education Policy at the Brookings Institution, who is a former teacher and Harvard University professor, wrote a scathing piece on the Brookings Web site titled, “Attention OECD-PISA: Your Silence on China is Wrong.” He says that even though official Shanghi policy changed in 2008 to ostensibly allow migrant children to enroll in city schools up to the age of 14, most of them are still excluded anyway — and it is nearly impossible for migrant students to enroll in the city’s high schools. This is a result, he said, of China’s hukou system, which assigns what amounts to every Chinese a registration of sorts in a home city or village. Migrants into a city are barred from that city’s public services unless official policy changes to allow participation. Loveless wrote in part:

Shanghai is portrayed as a paragon of equity in PISA publications. A 2010 OECD publication, Strong Performers and Successful Reformers in Education, highlights model systems that the world should emulate. Shanghai is singled out for praise. One section on Shanghai is entitled, “Ahead of the pack in universal education.” The city is described as an “education hub,” and the only discussion that even remotely touches upon migrants is the following:

 “Graduates from Shanghai’s institutions are allowed to stay and work in Shanghai, regardless of their places of origin. For that reason, many  ’education migrants now move to Shanghai mainly to educate their children.’ “[2]

That description is surreal. PISA’s blindness to what is really going on in Shanghai was also evident in the official release of PISA’s latest scores. The 2012 data appear in volumes organized by themes. Volume II is entitled, PISA 2012 Results: Excellence through Equity, Giving Every Student the Chance to Succeed. Shanghai is named as one jurisdiction where schools “achieve high mathematics performance without introducing greater inequities in education outcomes (p. 28)” and one with “above average socio-economic diversity (p. 30).” In the 336 pages of this publication on equity, the word “migrant” appears only once, in a discussion of Mexico. The word “hukou” does not appear at all.

Andreas Schleicher, the OECD deputy director for education and skills and special adviser on education policy to the OECD’s secretary general, wrote a post on the organization’s website entitled, “Are the Chinese cheating in PISA or are we cheating ourselves?” that counters such criticism about Shanghai. He writes that policies that keep migrants out of Shanghai’s schools have “long changed,” and that “resident migrants were covered by the PISA samples in exactly the way they are covered in other countries and education systems.”

Still, it seems to be easier to cling to old stereotypes than keep up with changes on the ground (or to read the PISA report).

True, like other emerging economies, Shanghai is still building its education system and not every 15-year-old makes it yet to high school. As a result of this and other factors, the PISA 2012 sample covers only 79% of the 15-year-olds in Shanghai. But that is far from unique. Even the United States, the country with the longest track record of universal high-school education, covered less than 90% of its 15-year-olds in PISA – and it didn’t include Puerto Rico in its PISA sample, a territory that is unlikely to have pulled up U.S. average performance.

(Puerto Rico isn’t part of the United States so, in fact, it wouldn’t make sense to include it in a U.S. sample, but let’s move on.)

I asked Post reporters who have covered China about Shanghai, and they said that despite the 2008 change in Shanghai policy, things have improved slightly for migrant families but not very much. Children who do not have Shanghai registration can now in theory get into Shanghai public schools, but only through very complicated and often arduous means that require either a brilliant kid, valuable inside connections or bribe money. (See this Washington Post story that talks about how Chinese parents pay bribes to get their children into top schools.) And while  the government recently announced that the hukou registration system is going to be reformed in the next few years, this does not mean families can suddenly get registered in big cities such as Shanghai.

This Sept. 10, 2013, story in the South China Morning Post is revealing about attitudes toward migrant children in Shanghai. Titled “School place for migrant pupil sparks ‘locust’ row in Shanghai,” it says that some some Shanghai residents accused a 6-year-old girl who got into a Shanghai school of wasting school resources. It says in part:

Shanghai residents, many of whom are also parents, were angered when Diao Qianwen, a six-year-old girl living with migrant worker grandparents in a wet market, was finally allowed to enroll in a local primary school after her plight was aired on a state TV news show. Some angry Shanghainese even copied their Hong Kong counterparts and called Qianwen a little “locust.”

The girl, originally from Shandong, has been raised by her grandparents, who work as vegetable vendors in a local market. Her mother had abandoned her at birth and her father is serving a sentence in a Shandong jail, according to reports.

Diao had been turned down by local schools prior to the show. Besides lacking official papers to prove they were the girl’s “legal guardians,” schools say the elderly couple had not been contributing to the city’s social security fund — a prerequisite for all new pupils. Her grandparents, after reaching a dead end, grew demoralized as the new semester approached.

After the girl’s plight was revealed on television, she was allowed into the school, but many Shanghai residents complained because legally registered students often have a hard time getting into city schools, the newspaper says.

Schleicher is correct when he wrote that “international comparisons are never easy and they are never perfect” and that the PISA exams “are administered under strict and internationally comparable conditions.” Anyone who “really wants to find out,” he wrote, “can review the underlying data.”

The question isn’t whether the OECD tried to make valid international comparisons. It is whether it succeeded — and whether international comparisons can ever be “apples to apples” given the very different education systems, cultures, school enrollment policies and data-collection processes that exist across the globe. And the question that Loveless raises about whether Shanghai cheats by its exclusionary admissions policies still begs an answer.