Last month I ran a post looking at the troubled Program for International Student Assessment, better known in the education world as PISA, which has come under fierce criticism in recent years. PISA is given every three years to 15-year-olds around the world in reading, math and science, and the results are taken as proxies by many governments — including the Obama administration — as important reflections of the state of their education systems. Critics say the tests are flawed in various ways and the results should not be taken as solemn pronouncements on the success of a school system. (Incidentally, the United States has always tested average at best on the exams.)
Last year, dozens of researchers and academics from around the world wrote an open letter to Andreas Schleicher, director of the Program of International Student Assessment, urging him to suspend administration of PISA until a new exam can be created. (You can read about that here.) The post I ran in March looked at the controversy around PISA and asked whether it should be saved, concluding that it should. It was written by Pasi Sahlberg, visiting professor at the Harvard Graduate School of Education and the author of “Finnish Lessons 2.0″, and Andy Hargreaves, professor in the Lynch School of Education and the Carroll School of Management at Boston College and the coauthor of “Uplifting Leadership.”
The following post is a different look at the problems with, and the possible futures, of PISA, written by Alma Harris and Yong Zhao. Harris is a professor of educational leadership at the Institute of Education at University College London. She is currently working at the University of Malaya and conducting a comparative study of system performance across seven countries. Yong Zhao is presidential chair and director of the Institute for Global and Online Education in the University of Oregon’s College of Education, where he is also a professor in the Department of Educational Measurement, Policy and Leadership. He is the author of numerous articles and books, including “Catching Up or Leading the Way: American Education in the Age of Globalization.”
By Alma Harris and Yong Zhao
The last few months have seen large cracks appear in the veneer of one of the most prominent international comparative assessments. The Organization for Economic Cooperation and Development’s (OECD) Program for International Student Assessment (PISA) is under the white-hot spotlight with much contemporary critique and counter critique. The latest piece on the future possibilities of PISA, written by Professors Andy Hargreaves and Pasi Sahlberg on The Washington Post’s Answer Sheet blog, takes a balanced view of both the positives and negatives of this global comparative measure. The authors argue that PISA may be leaning but it is ultimately worth saving. While we concur that PISA has raised the bar on comparative analysis, we argue that this is an insufficient reason for keeping it as the global barometer of educational performance.
While comparative assessments offer useful measures of relative performance, they are simply just that, a measure. Unfortunately, the complex world of schooling cannot be reduced to a single score or position on the league table of global educational performance, even if this is very attractive, particularly to policymakers. If the measures of educational performance are narrowed to three tests, as in the case of PISA, it must be questionable firstly, what exactly is being measured and secondly, for what purpose? While it is interesting to see how different systems fare in this global beauty contest, major methodological flaws remain — as many experts have highlighted — that pour doubt on its analysis and conclusions.
Interestingly, the critiques of PISA have not been marketed, popularized and vocalized as much as the supportive voices. If they had, it is debatable whether the credibility of PISA would have remained intact for so long. Without question, PISA has become big business and highly influential. Hence it is unlikely to relinquish its grip on the global educational market place any time soon. The stakes are too high.
Also, many of the overly romanticized accounts of the “top-performing” countries would carry zero weight if the PISA measure were in serious doubt. Consequently, the critiques of PISA have been largely drowned out by a global hype that has mobilized mass attention towards the issue of ranking and ‘top performance’ rather than the substantive matter of measurement.
So in reality, how secure are these rankings? Looking at other data sets and indicators such as those assembled by statisticians in the World Bank and UNESCO, it is interesting to see quite different representations of educational performance and to note that information about education gaps by income, gender and area for the “top-performing countries” are often incomplete or missing.
Furthermore, it is interesting to reflect upon the exact moment when PISA shifted from being a comparative measure to policy prescription. As a large-scale assessment process, its data offers a useful basis for diagnosis and discussion but PISA has moved a long way from its original role. As Hargreaves and Sahlberg note, there are now clearly defined linkages between PISA, the OECD and commercial organizations. Andreas Schleicher has openly advocated the merits of the Common Core in the United States, stating:
“They are actually modeled on the top-performing education systems. If they are actually done in classrooms, they are going to get the U.S. pretty much upwards.”
In short, PISA is not a neutral source of comparative data or information. It is now a powerful and pervasive policy instrument advocating certain approaches to educational reform over others.
There is also an interesting paradox in the appraisals of PISA. On the one hand some critics say that PISA may not be 100 percent trustworthy as a benchmark of global high performance but simultaneously draw upon the PISA result to argue that systems like Finland, Singapore and Ontario far outperform others. Undoubtedly, insights from the “top-performing” countries have been useful and even entertaining. In some cases however they have resulted in faulty accounts and gross misinterpretation.
The “success” of Shanghai is attributed to hard-working students and teachers. While this is undoubtedly a factor, less is said about the fact that in China the only way to succeed in life is to pass the tests. There is no choice. Hence parents in China pay vast amounts of money on private tuition and young people spend all of their waking hours prepping and practicing. The same is true of other education systems but this powerful cultural explanation is overshadowed and overlooked, in favor of more palatable and exportable explanations.
Hargreaves and Sahlberg touch upon the methods and approaches used to elicit such exceptional outcomes from the young people in these systems but the whole area of culture and context is left totally unexplored. PISA takes little, if no account, of the cultural and contextual influences that fuel the performance of different systems, it is as if every country in PISA is devoid of historical, social, economic and cultural heritage. These influences are forensically airbrushed out in favor of neatly wrapped causal attributions that can be conveniently turned into policy solutions and commercial packages. This homogenizing is not only devaluing but it is potentially dangerous. It suggests that there are “universals of excellence” that can be copied from certain countries, irrespective of context or culture. It encourages policy borrowing from systems that may be high flyers in the global rankings but low performers on issues of equity, diversity and innovation.
So what are the ways forward? Here are four different perspectives.
Firstly, dismantling PISA, if it were to happen, might mean that it would be simply be re-created or re-constituted in some other form or fashion. It could even be more commercialized; such is the international appetite for comparing education systems and basing policy decisions on a simple analysis of performance.
Secondly, dismantling PISA runs the risk that country specific blinkered accounts of performance re-emerge allowing education systems to ‘believe’ that they are performing well. Comparative analysis is therefore important but it has to include cultural and contextual factors.
Thirdly, PISA can be “fixed” if those who lead it accept the methodological weaknesses and attempt to adjust and revise accordingly. So far, this is not a path the architects of PISA have chosen.
Finally, we simply move away from one test or measure to many measures. This means broadening out assessments of educational performance to more than a set of numbers. It means assessing how far young people in different education systems are innovative, creative and entrepreneurial rather than assessing how proficient they are at passing tests.
Is PISA waving or drowning? Is the appetite to retain it, for whatever reason, stronger that the desire to let it sink? On the one hand, some would argue that PISA is focusing on the right things; more skills-based assessment and the need for 21st Century skills. But we would say that it is also focusing on the wrong thing. The PISA measure has become far more important than the process it attempts to measure or quantify. The neat narratives of high performance and the romantic accounts of success fuelled by the PISA league tables have dumbed down educational reform to little more than copying and pasting from the ‘best’ based on rhetoric rather than reality.
So, it is time to press the pause button on PISA and to ask what is it really doing and for whom. Rather than debate whether PISA is waving or drowning, we need to really ask ourselves is PISA actually serving young people. If not, it is time to think again.
