Fat pigs and assessment reform

One hoary old saw in education says that “you don’t fatten a pig by weighing it”, which is true enough, and I was riffing on this when I tweeted this morning that “Trying to raise standards by changing the exam is like trying to fatten a pig by buying new scales.”

This was in response to some not-very-good soundbites I’d heard around the proposal – yet to be unveiled – to change GCSEs (the exams most 16-year-olds take in the UK England and Wales). But it’s also relevant in the wider, international debate around high-stakes and standardised testing.

Everyone (well, nearly everyone) wants educational change, and everyone (well, nearly everyone) agrees that assessment really matters. Alas, there is less agreement on how education should be changed, and still less on what should happen to assessment. There’s even less agreement on what “raising standards” means in an assessment context: Increased average marks? Mean or median? More students passing? More accurate exams? More precise discrimination between scores? More reliable or consistent exams? Fewer students failing? More students getting top marks? Fewer students getting top marks? More students learning more and better? More valid exams?

Buying new scales, or weighing more often, isn’t going to fatten the pig on its own. I seem to recall that the “you don’t fatten a pig by weighing it more often” argument was deployed when the UK system was changed in favour of more continuous assessment. But what you measure and how is crucially important in what the educational system as a whole ends up achieving. If your scale is broken, or measures the wrong thing, or measures it badly, you’re going to struggle to improve.

This is one of the themes I harp on about in a learning analytics context (including in my LAK12 paper The Learning Analytics Cycle, now available open access form!). If you make a system that optimises for a particular metric (say, exam results, or completion rates, or whatever), but that metric doesn’t reflect what you actually value, you’ll be making things worse, not better.

If what you actually want is more lean meat (or, heaven forfend, happy pigs), then optimising your farming system for the pig’s weight alone is not going to help you: you might end up with bigger pigs, but more fat and less muscle. Maybe you don’t actually want to eat pork anyway.

What’s on the exam, and how things are examined, is crucially important. Teachers and learners are not daft, and generally want to do as well as they can. They’re also under increasing external scrutiny and pressure to do so. Changing what’s assessed, and how, will not on its own improve learning. But assessing what you don’t value is likely to make things worse, not better.

I’m really not convinced that the most important thing we want sixteen-year-olds to have learned is the ability to get good marks in a single-shot closed-book hand-written three-hour exam with no consulting of references, sources, notes or colleagues. But I think I might have to wait a while before that view is reflected in the formal assessment system – though hopefully not until pigs fly.

–
This work by Doug Clow is licensed under a Creative Commons Attribution 3.0 Unported License.
No further permission needed to reuse/remix (with attribution), but it’s nice to be notified if you do.

Author: dougclow

Data scientist, tutxor, project leader, researcher, analyst, teacher, developer, educational technologist, online learning expert, and manager. I particularly enjoy rapidly appraising new-to-me contexts, and mediating between highly technical specialisms and others, from ordinary users to senior management. After 20 years at the OU as an academic, I am now a self-employed consultant, building on my skills and experience in working with people, technology, data science, and artificial intelligence, in a wide range of contexts and industries. View all posts by dougclow