Turning the global league tables

Last week’s publication of a new global education league table by the Economist Intelligence Unit and Pearson raised some eyebrows with its claim that the UK’s education system now ranks sixth in the developed world.

After all, on the same day, the Chief Inspector, Sir Michael Wilshaw was using the data from the OECD’s Programme for International Student Assessment (PISA) to make a case that English schools must do better if we are to match global competitors in the future.

The UK scores around average in PISA for reading and mathematical literacy, and a little above average for scientific literacy, based on tests of 15 year-olds. This places UK schools 25th for reading, 28th for maths and 16th for science, out of 65 countries.

Pearson has aggregated this PISA data with other studies from the Trends In International Mathematics and Science Study (TIMSS), which measures international trends in mathematics and science achievement of 9 and 13 year-olds, and the Progress In International Reading Literacy Study (PIRLS) which just focuses on the reading achievement of 9 year-olds – those in fourth grade.

The Pearson Learning Curve report also includes some UNESCO data to create a ranking of countries that looks at both cognitive skills and educational attainment. For cognitive skills, they use PISA, PIRLS and TIMSS, and for attainment they use UNESCO data on adult literacy and OECD data on graduation rates at the end of secondary school and at university.

PISA and TIMSS/PIRLS measure different things and do so at different ages. The primary difference is that TIMSS/PIRLS looks at what you been taught in a particular subject and how much you have learnt, whereas PISA looks at what you are able to do with the science you have been taught. PISA is more about the practical application of your knowledge.

The Learning Curve report also draws wider conclusions and lessons for education policymakers, including the importance of good teachers, a strong pro-education culture and on the best ways to engage parents.

But how does it reach so different a conclusion from PISA on the comparative health of the UK education system? It does so quite easily, in fact, and it is all in the weightings. And in looking at how it reaches the conclusions it does, there are also lessons on how one should use such league tables.

First, on PISA, the Pearson/EIU index has fewer countries than PISA in its list, so removing those countries and only using Hong Kong for China (PISA includes Shanghai and Macao too) elevates the UK four places higher in reading and science and six places higher in Maths. The new index is more interested in a country’s relationship to the mean than its ranking, so bunching around the mean in PISA would also reduce ranking differentials.

Second, on PIRLS and TIMSS, England and Wales score significantly better on these tables (some of which exclude other major developed nations such as France, Finland and New Zealand) than on PISA. Although Scotland scores lower, the UK average remains strong. Because TIMSS has both Grade 4 and Grade 8 tests, the combined value of PIRLS and TIMSS is stronger than that for PISA.

And finally, although the UK is rated 6^th overall, it is only ranked 11^th for cognitive skills – those measured by PISA, TIMSS and PIRLS – and its higher rating on the overall table owes more to adult literacy and graduation rates.

Add in all these factors, and look at the weighting given to each factor in the report. The default weight for the Index is two-thirds to cognitive skills and one-third to educational attainment. Within the cognitive skills category, the Grade 8 tests’ score accounts for 60% while the Grade 4 tests’ score accounts for 40% (Reading, Maths and Science all account for equal weights).

 So, the PISA reading and Maths scores combined, where the UK is weakest, only account for 20% of a country’s ranking, and PISA science another 6.7%. But because PIRLS and TIMSS are available at Grade 4 and Grade 8, they will be worth 26.7% for the Grade 4 and 13.3% for the Grade 8, a total of 40%. The HE and adult literacy scores are worth a further 33.3% between them.

Perhaps a bigger issue than the rankings is what the tables do not reveal about education in the UK, particularly the absence of any measure in the Pearson/EIU table of the attainment gap or of social mobility. One recent OECD report, for example, said that only Russia and the Czech Republic had a more socially segregated schools system.

The big gaps in attainment between pupils on free school meals and their peers in GCSEs are another important indicator – other countries have narrower gaps in attainment, as we demonstrated at our social mobility summit in May.

And the rankings should also look at how well countries perform with their most able students. Sutton Trust analysis earlier this year showed that, in maths, just 1.7% of 15-year-olds attained the very highest PISA level (level 6), compared with an OECD average of 3.1%, placing England 26^th out of 34 countries.

The new index is an important step forward in consolidating international data. But any such league table is dependent on the quality and range of inputs. As the Economist Intelligence Unit and Pearson develop the index, they should consider adding measures of mobility and the achievements of the most able to give a fuller picture of the success of national education systems.

This posting first appeared at the Sutton Trust blog. It was also quoted by Anne McElvoy on the Economist Blighty blog.