Turning the global league tables

Last week’s publication of a new global education league table by the Economist Intelligence Unit and Pearson raised some eyebrows with its claim that the UK’s education system now ranks sixth in the developed world.

After all, on the same day, the Chief Inspector, Sir Michael Wilshaw was using the data from the OECD’s Programme for International Student Assessment (PISA) to make a case that English schools must do better if we are to match global competitors in the future.

The UK scores around average in PISA for reading and mathematical literacy, and a little above average for scientific literacy, based on tests of 15 year-olds. This places UK schools 25th for reading, 28th for maths and 16th for science, out of 65 countries.

Pearson has aggregated this PISA data with other studies from the Trends In International Mathematics and Science Study (TIMSS), which measures international trends in mathematics and science achievement of 9 and 13 year-olds, and the Progress In International Reading Literacy Study (PIRLS) which just focuses on the reading achievement of 9 year-olds – those in fourth grade.

The Pearson Learning Curve report also includes some UNESCO data to create a ranking of countries that looks at both cognitive skills and educational attainment. For cognitive skills, they use PISA, PIRLS and TIMSS, and for attainment they use UNESCO data on adult literacy and OECD data on graduation rates at the end of secondary school and at university.

PISA and TIMSS/PIRLS measure different things and do so at different ages. The primary difference is that TIMSS/PIRLS looks at what you been taught in a particular subject and how much you have learnt, whereas PISA looks at what you are able to do with the science you have been taught. PISA is more about the practical application of your knowledge.

The Learning Curve report also draws wider conclusions and lessons for education policymakers, including the importance of good teachers, a strong pro-education culture and on the best ways to engage parents.

But how does it reach so different a conclusion from PISA on the comparative health of the UK education system? It does so quite easily, in fact, and it is all in the weightings. And in looking at how it reaches the conclusions it does, there are also lessons on how one should use such league tables.

First, on PISA, the Pearson/EIU index has fewer countries than PISA in its list, so removing those countries and only using Hong Kong for China (PISA includes Shanghai and Macao too) elevates the UK four places higher in reading and science and six places higher in Maths. The new index is more interested in a country’s relationship to the mean than its ranking, so bunching around the mean in PISA would also reduce ranking differentials.

Second, on PIRLS and TIMSS, England and Wales score significantly better on these tables (some of which exclude other major developed nations such as France, Finland and New Zealand) than on PISA. Although Scotland scores lower, the UK average remains strong. Because TIMSS has both Grade 4 and Grade 8 tests, the combined value of PIRLS and TIMSS is stronger than that for PISA.

And finally, although the UK is rated 6^th overall, it is only ranked 11^th for cognitive skills – those measured by PISA, TIMSS and PIRLS – and its higher rating on the overall table owes more to adult literacy and graduation rates.

Add in all these factors, and look at the weighting given to each factor in the report. The default weight for the Index is two-thirds to cognitive skills and one-third to educational attainment. Within the cognitive skills category, the Grade 8 tests’ score accounts for 60% while the Grade 4 tests’ score accounts for 40% (Reading, Maths and Science all account for equal weights).

 So, the PISA reading and Maths scores combined, where the UK is weakest, only account for 20% of a country’s ranking, and PISA science another 6.7%. But because PIRLS and TIMSS are available at Grade 4 and Grade 8, they will be worth 26.7% for the Grade 4 and 13.3% for the Grade 8, a total of 40%. The HE and adult literacy scores are worth a further 33.3% between them.

Perhaps a bigger issue than the rankings is what the tables do not reveal about education in the UK, particularly the absence of any measure in the Pearson/EIU table of the attainment gap or of social mobility. One recent OECD report, for example, said that only Russia and the Czech Republic had a more socially segregated schools system.

The big gaps in attainment between pupils on free school meals and their peers in GCSEs are another important indicator – other countries have narrower gaps in attainment, as we demonstrated at our social mobility summit in May.

And the rankings should also look at how well countries perform with their most able students. Sutton Trust analysis earlier this year showed that, in maths, just 1.7% of 15-year-olds attained the very highest PISA level (level 6), compared with an OECD average of 3.1%, placing England 26^th out of 34 countries.

The new index is an important step forward in consolidating international data. But any such league table is dependent on the quality and range of inputs. As the Economist Intelligence Unit and Pearson develop the index, they should consider adding measures of mobility and the achievements of the most able to give a fuller picture of the success of national education systems.

This posting first appeared at the Sutton Trust blog. It was also quoted by Anne McElvoy on the Economist Blighty blog.

Reading behind the international surveys (2)

Coming a week after the PIRLS reading survey, today's PISA results appear to confirm a trend. It is not, as the more excitable right-wing commentators suggest, that our students have dipped significantly in their performance because of central government policies - in fact on reading and science, only seven countries score significantly higher than England (though, our maths performance is rather less good); rather, it is that some other countries that have introduced strong reforms are improving faster. As I argued around PIRLS, the intensity of the strategies around literacy and numeracy (and this is as true at Key Stage 3 - age 14 - as it has been at Key Stage 2 - age 11) was lost between 2001-2005. In primary schools, the focus is being re-energised; in secondaries, the requirement that English and Maths results are included in the five good GCSEs recorded in the league tables is starting to have a similar effect (it is noticeable that far more pupils get a GCSE C+ in English than Maths (See Excel Table 7)). But the lesson of these international surveys is not that we should abandon testing, leave teachers alone or give up. Have a look at the dismal performance of Wales - against which 16 countries do significantly better on reading - to see what that approach brings. It is instead that - as the 2000 and 2001 surveys showed - when government has a clear overall strategy, schools are clear about their priorities and public accountability is strong - then pupils will achieve not just at the average level of their international counterparts, but significantly above average.

Reading behind the international survey

Today's PIRLS international study (pdf) of reading literacy standards - the ability to understand and use those written language forms required by society - raises some interesting issues. On the one hand, English pupils do better than the average for their international counterparts, outperforming countries like New Zealand, Scotland, Spain and France. On the other, English readers fell back to 19th in the world in 2006 (though a raft of Canadian provinces appear individually this time, distorting the rankings), compared with 3rd in 2001. There is more to this than whether youngsters enjoy reading, or indeed whether the brighter or weaker youngsters are dragging the results down. The 2001 survey came after a period of rapid improvement in primary schools, as a result of national tests and the literacy strategy. But the improvements have been slower since, while other countries have clearly improved more rapidly, not least places like Germany which were startled by their low performance in international studies.

Much of the increase in the proportion of children achieving the expected standard – level 4 – in English from 49% in 1995 to 80% this year took place by 2000, when the figure was 75%. (The proportion doing so in reading is now 84%.) The same was true of the top pupils - the proportion reaching level 5 rose from 12 to 29% from 1996 to 2000, and is now 33%. This test improvement was also reflected in Ofsted inspections: the literacy and numeracy strategies nationally had a dramatic effect on the quality of primary school teaching between 1996 and 2000 – only 43% of lessons for 7-11 year-olds were deemed good or better in 1996. By 2000, this had risen to 72%. That figure fluctuated a little until 2005, when it stood at 74%. Under a tougher inspection system, it has again started to rise.

There is no doubt that we need to regain the momentum of the period before 2000. The government is now rightly promoting synthetic phonics – using letter sounds to build up words - reflecting the growing evidence that it is the most effective way to teach children to read. Jim Rose's review in 2005 set this in train, and phonics has been the expected method since September 2007, as a result. This will need the sort of intensive rollout that accompanied the literacy strategy, with Ofsted inspecting how effective schools are in teaching reading, and whether they are using phonics. And it must be pushed hard before falling back on Every Child A Reader, which should be used as a recovery programme for the few who don't learn through phonics, not to provide literacy help for the many.