Tuesday, 5 August 2008

A statistics test

Your statistics question for 10: Which is likely to be more accurate?

The test results of 600,000 11-year-olds published today in aggregate form, and showing a modest improvement, with the insistence of the scrupulously independent chief national statistician and the independent exam regulator that their 99% sample is a fair reflection of the overall national test results for 2008, despite some delays in their marking.


A survey by a right wing think tank, which has never knowingly supported anything associated with the Labour government, that quickly questioned 107 out of 9600 secondary school heads of maths, science or English (about 1%, with a sample skewed towards heads of English) about the quality of SATs at the height of the fuss about this year's tests (problems which actually affected Key Stage 3 English tests most) telling us that they have no confidence in the tests.


oldandrew said...

Ooooh, this is quite annoying. It's best not to get you mathematical methods wrong when commenting on tests that include mathematics. And I've just heard Jim Knight do the same thing on the radio.

Anyway, sampling is about probabilities. It does not depend for it's accuracy on the proportion of the total population sampled. So, for instance, if you threw a coin 107 times and got heads 90% of the time then you would think the coin wasn't fair regardless of how many times you were planning to throw it again later. You certainly couldn't say that the more you are planning to throw it later the less reliable your present judgement of the coin is.

Accuracy in polling depends only on total sample size not proportionate size. 107 is not an accurate sample size and the margin of error is likely to be around 10%, far more than a conventional opinion poll. However, as in the coin example, when the response is 90% saying that SATs weren't reliable that still tells you a good deal i.e. it tells that if the sampling was fair then the true proportion is unlikely to be below 80% but could actually be as high as 100%.

The latter possibility you could look into yourself. Do you know any teachers who think the tests are accurate?

Teaching Blog here
Latest entry: 3/8/2008

Conor Ryan said...

I do know quite a few teachers who believe pupils are arriving in secondary school better educated in the 3Rs now than in the early nineties. However, it is not just the size of the Cititas sample that is problematic: as you say that in itself means a ten point margin of error. The sample was further skewed because of how they arrived at their 107, as a look at their report shows; they didn't reach two thirds of their original 356 schools; they didn't weight it by subject teacher; and their assertion is not that 90% of the tests are inaccurate, but that around a third are too generous and a tenth not generous enough. All of which enhances the probabilities of error.

oldandrew said...

I do know quite a few teachers who believe pupils are arriving in secondary school better educated in the 3Rs now than in the early nineties.

That's hardly the same thing.

Anything with a small sample also has a risk of errors in sampling. But even if there were enough errors and inadequacies to increase the margin of error to 20% we'd still have between 70% and 100% of teachers thinking that SATs weren't accurate. That said if you look at the issue of bias towards subject you will discover that this has increased, not decreased the number of teachers who think them accurate, as maths is over-represented but has the most teachers who think SATs accurate.

If you break it down by subject we still have the following saying SATs accurately reflected abilty last year:

Science: 4% with a 19% margin of error.
Maths: 21% with a 14% margin of error.
English: 0% with an 18% margin of error.

And from personal experience I'm surprised it's even that many.

Teaching Blog here
Latest entry: 3/8/2008

Anonymous said...

Not inclined to take seriously the view of anyone who thinks QCA and Government Statisticians are independent.

Conor Ryan said...

I always find it hard to take seriously people who respond to blogs anonymously myself.