Category: mine-rants

Educational research has two cultures, but unlike the two in C.P. Snow’s famous talk, they do overlap and they do talk to each other. One is fiercely qualitative, concerned directly and immediately with the lived human experience of learners and teachers, in all its ethnographic complexity, subtlety, and sophistication. The other is determinedly quantitative, concerned with what can be counted and known in more regular, repeatable, transferable ways.

My sympathies lie with both, but my inclination is definitely towards the empirical, quantitative, and generalisable. But that mustn’t be at the cost of losing sight of the human perspective. With the rise of learning analytics, and more and more quantification of learning – of which I’m a small part – it’s easy to be sucked in to watching the numbers and forgetting the people.

Two things have come to haunt me recently. Both are about predictive analytics: using all the data we have about learners, and previous learner’s success or failure, and using that to predict the success (or otherwise) of learners who haven’t finished yet – or even signed up. The OU is doing a lot of work in this area at the moment. The ethical issues are complex and difficult. I think both the quantitative and the qualitative perspectives are needed to guide our policy development here.

cc licensed ( BY ) flickr photo shared by Photo Mery

Continue reading “The numbers are people”

“Prediction is very difficult, especially about the future.” – Niels Bohr, Mark Twain, Yogi Berra, or perhaps someone else.

I think the nascent field of learning analytics is facing its first serious, defining issue.

It seems the claims for the effectiveness of Course Signals, Purdue University’s trailblazing learning analytics program, are not quite as robust as it seemed. You can get a quick journalistic overview from Inside Higher Education’s story (which has good links to the sources) or the Times Higher Education’s version (which appears identical, except the useful links have been stripped).

This post is a bit long. As Voltaire, Proust, Pliny or (again) Mark Twain may have said, my apologies, but I didn’t have time to write a short one. It’s important, though.

Continue reading “Looking harder at Course Signals”

One hoary old saw in education says that “you don’t fatten a pig by weighing it”, which is true enough, and I was riffing on this when I tweeted this morning that “Trying to raise standards by changing the exam is like trying to fatten a pig by buying new scales.”

This was in response to some not-very-good soundbites I’d heard around the proposal – yet to be unveiled – to change GCSEs (the exams most 16-year-olds take in the UK England and Wales). But it’s also relevant in the wider, international debate around high-stakes and standardised testing.

Everyone (well, nearly everyone) wants educational change, and everyone (well, nearly everyone) agrees that assessment really matters. Alas, there is less agreement on how education should be changed, and still less on what should happen to assessment. There’s even less agreement on what “raising standards” means in an assessment context: Increased average marks? Mean or median? More students passing? More accurate exams? More precise discrimination between scores? More reliable or consistent exams? Fewer students failing? More students getting top marks? Fewer students getting top marks? More students learning more and better? More valid exams?

Buying new scales, or weighing more often, isn’t going to fatten the pig on its own. I seem to recall that the “you don’t fatten a pig by weighing it more often” argument was deployed when the UK system was changed in favour of more continuous assessment. But what you measure and how is crucially important in what the educational system as a whole ends up achieving. If your scale is broken, or measures the wrong thing, or measures it badly, you’re going to struggle to improve.

This is one of the themes I harp on about in a learning analytics context (including in my LAK12 paper The Learning Analytics Cycle, now available open access form!). If you make a system that optimises for a particular metric (say, exam results, or completion rates, or whatever), but that metric doesn’t reflect what you actually value, you’ll be making things worse, not better.

If what you actually want is more lean meat (or, heaven forfend, happy pigs), then optimising your farming system for the pig’s weight alone is not going to help you: you might end up with bigger pigs, but more fat and less muscle. Maybe you don’t actually want to eat pork anyway.

What’s on the exam, and how things are examined, is crucially important. Teachers and learners are not daft, and generally want to do as well as they can. They’re also under increasing external scrutiny and pressure to do so. Changing what’s assessed, and how, will not on its own improve learning. But assessing what you don’t value is likely to make things worse, not better.

I’m really not convinced that the most important thing we want sixteen-year-olds to have learned is the ability to get good marks in a single-shot closed-book hand-written three-hour exam with no consulting of references, sources, notes or colleagues. But I think I might have to wait a while before that view is reflected in the formal assessment system – though hopefully not until pigs fly.

–
This work by Doug Clow is licensed under a Creative Commons Attribution 3.0 Unported License.
No further permission needed to reuse/remix (with attribution), but it’s nice to be notified if you do.

The conference dinner for LAK12 was at the Vancouver Aquarium, in Stanley Park. The highlight of the evening was a dolphin show.

For me, it raised lots of interesting issues linked to themes emerging in the conference. Dolphins are legendarily clever, and I was reminded powerfully of Douglas Adams’ Hitchhiker’s Guide to the Galaxy on relative intelligence:

For instance, on the planet Earth, man had always assumed that he was more intelligent than dolphins because he had achieved so much—the wheel, New York, wars and so on—whilst all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man—for precisely the same reasons.

(Which also reminds me of how far we’ve come with inclusive language.)

Continue reading “Learning from dolphin learning”

There’s a serious danger, I think, in Learning Analytics, of producing and processing ever more data in ever more exciting and attractive ways, but which aren’t connected up to making things any different for learners.

As Marx famously put it in his Theses on Feuerbach,

Die Philosophen haben die Welt nur verschieden interpretiert; es kommt aber darauf an, sie zu verändern.

(Philosophers have only sought to interpret the world in various ways; the point is to change it.)

This is nowhere (!) more true than in learning analytics. Analysis without action is wasted effort – which we can ill afford now. Those of us with a penchant for data, sums and code (I know, I know, but we have our uses) can get terribly excited about new sources of data and new analyses. It’s great to come up with new ways of understanding and interpreting what’s going on. But if that effort doesn’t translate in to better action – it’s pointless.

On the other hand, action without analysis may be making things worse – you just don’t know. (A point sometimes lost on misinterpreters of Marx who focus only on the ‘change it’ bit. You can say many rude things about Marx – and I sometimes do – but not that he was short on quantity of analysis of capitalism.) We’re certainly going to be seeing a lot of dramatic change in universities in the coming years. If we don’t have the right evidence supporting decision-makers in that process, not only will the decisions be made in the dark, we won’t even know how wrong they are.

That’s why there needs to be a learning analytics cycle:

The process from learners to data to metrics/analytics is the core of what learning analytics is. But those outputs need to feedback to learners if the process is to be of any real value. (As Tony Hirst mentioned in his Learning Analytics Pre-Workshop, in technical terms, we’re talking a closed-loop control system.)

For me, this means engaging with all the frustrations of internal systems, processes and decision-making. If I was just interested in research outputs, I could just dive in to the systems I can get access to, grab some data, do some analysis, and go. It’s much harder to arrange and endure the endless meetings, papers, persuasion, and frustration that is what happens when you try to make a substantial change to a large bureaucratic organisation like a university. But that’s where the real value and potential lie.

Learning analytics starts with the learner, and their ‘data exhaust’. We need to make sure it gets back there again.

 

NB For clarity (thanks anonymous tipster!) that last sentence is meant entirely figuratively and I am emphatically not advocating gassing students. That would be Evil and Wrong. I’m trying to make things better for them.

(This post brought to you by my desire to bolster my resolve and enthusiasm for helping to work up an internal project along these lines. Will say more if and when there’s things to say.)

–
This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.