LAK16 Thu am Paul Kirschner Keynote

Liveblog notes from Thursday morning at LAK16 [full list of blogging]: Paul Kirschner keynote: Learning Analytics: Utopia or Dystopia.

Dragan Gašević welcomes everyone. How many in the audience are new to LAK? [overwhelming majority] Welcome!

Sets out the schedule for the day, but also including a celebration of Erik Duval’s life at 4pm, and calling today an Erik Duval Day.

Hendrik Drachsler takes over to introduce Paul Kirschner, from OUNL.

Learning Analytics: Utopia or Dystopia

Professor Paul A. Kirschner, Open University of the Netherlands

Paul remarks that Hendrik is special issues editor of JCAL.

Starts with thanks to conference organisers for inviting them. But also specifically Grace Lynch, for her caring work. Also thanks organisers for challenging him to go in to a field that he was busy with on the side, but never in depth. This was out of my comfort zone, it was really nice to be forced to do that. Don’t stay in your comfort zone.

Normally I’d ask you not to use your phones and tablets, but do please turn off your ring tones.

Adjust your expectations: this is what this keynote is not about. Privacy, ethics, legal, commercial interests, technology. Like Bill Clinton: It’s the learning, people! Mireille’s presentation was cartoon-driven, mine is image-driven.

We’re here in Edinburgh. Charles Dickens was here, in Canongate Kirk yard, Ebenezer Scrogge grave. Text analysis went wrong, but Christmas Carol was born. The world of LA is more like a Tale of Two Cities, inspired by Thomas Carlyle. Opens with “It was the best of times, it was the worst of times … it was the season of light, it was the season of darkness”

A clip from my favourite TV series, Black Mirror. British TV anthology, dark speculative fiction, unanticipated consequences of new technologies. Martha and Ash move to a remote countryside house. Ash checks his phone for social media updates a lot. The day after, Ash is killed returning the van, maybe texting while driving. Martha accesses a new service that lets you contact dead people, a virtual replica. [Video clip … cuts off just as she starts to talk to him.] It’s on Netflix!

Obligatory slides. Data is growing, it’s really incredible, but that’s not what I’m talking about. Hype cycle, learning analytics is at the top of it. David Wiley tweet – if you hear “learning analytics” try substituting “statistical analysis”. Moore’s Law, submissions for this conference also follows that pattern.

Why am I here? Carolyn Rose contacted me, I asked what she expected. Two tasks: First, to shake it up, be your controversial self. Second, break it down, explain what learning and learning sciences are. I would rather be infamous than famous. I accepted the challenge.

First aspect, what is learning? 640 million results for defining learning. A month ago, 570 million. 45 million new pages in the last month on what learning is. Simplest is Wikipedia definition. I use a much simpler one – a permanent change in one’s long-term memory. That was in an article I wrote, much cited.

Technical, neuroscience slide. Different parts of the brain are specialised for different memory. Prefrontal cortex/frontal lobe for short-term, etc. Emotional memories are amygdala. So maybe that definition is not so simple if we look at it.

Second, what are the learning sciences? Interdisciplinary field to further scientific understanding of learning, and design of learning. Fields like cog sci, comp sci, ed psych, anthropology, neuroscience, more. Learning sciences as plural. How different fields contribute to the understanding of how people learn.

What is learning analytics? Campbell DeBlois & Oblinger 2007 Educause paper – five steps, Capture, Report, Predict, Act, Refine. Rebecca Ferguson covered this well. Cyclical model, go back to the beginning. George Siemens 2013 model. Application of big data in educational settings. So far so good.

Question arises, what to learning sciences have to do with LA? Just about everything. ICLS workshop editorial – LA at the intersection of the learning sciences and computational data capture and analysis.

Now time to shake it up with dystopian futures. [dark slide] What practitioners are doing are educational analytics. Learning and education are different things. Learning happens within the learning. Education is the system for instruction, especially in school or university. Teaching aims for formal learning; teaching not the same as learning. Most of major uses of LA are more academic or institutional analytics and not very much about the learning.

Looking closer at George’s model. Purposes clear. Learning in fourth place. (!) Strange. Data team – someone missing? It’s the learning scientist. Why’s this a problem? Emphasis role of theory – difference between digging and mining. Mining companies don’t just dig and hope they find something. The company isn’t like Jed Clampett from the Beverley Hillbillies. These companies use theories, geology, and gather and analyse specific data about land, make a well-formed guess as to whether what they’re looking for can be found where they’re looking. The learning scientist is the crucial missing factor. Operates on the basis of theory.

Theory has several functions, listed on this slide. John Sweller – no damn power, no damn point. Don’t read your slides, redundancy effect. Ending with potential generalisation of results. It’s not a one-way street. Exploratory analysis of large data is really good, Baconian sense, C18th/19th antiquarians filling rarity cabinets, thinking how that happened. Yesterday, Tim O’Shea, spoke about James Hutton here who did the same thing. He was a farmer, for 15-20 y, came up with tectonics, is father of modern geology. Nothing wrong with finding patterns, associations across modalities, over time, microgenetic level. If you look at it that way, it’s exploratory. Any good learning scientist, ed psych, from exploratory research you really can’t draw good causal conclusions about how to intervene. You get ideas about what relates to what. If you develop a theory, can explore [the causal relation and what it is]. Otherwise it’s just digging. Like a toddler in a sandpit. The first challenge is to build strong connections with the learning sciences – Rebecca Ferguson.

Dystopia 2 – Theory free theory poor LA. Not only true of the US elections. “Unencumbered by the thought process” – slogan from the 80s. The current one, you really understand what I’m talking about. LA as highly data-rich but theory poor. Second dystopian future deals with this. They make use of what there is, and not what’s needed. It’s like the joke about the drunkard, or the engineer, who lost their car keys, and is looking for them under a streetlamp. Police officer helps, then asks if certain they were dropped here. No, it was across the street. Why look here? The light here is much better. This is the streetlight effect. Observational bias, people only look where is easiest to look. In certain respects, many of us doing LA are like this. We have lots of data, looking there for answers. We’re looking at the data we have, not the data we need.

Keynote yesterday – what we don’t or can’t capture is not part of what is analysed. Without proper theory, we might be looking at the wrong things.

Third dystopia – looking at wrong or invalid variables. Unicorns on the slide – visual/auditory learner. Learning styles – it’s pure hogwash. I’d use another word in other places. Motivation influenced by success and failure, not the other way round. Engagement, shown to have little or no effect on learning. One poorly-informed group of researchers hailed value of learning styles such as SRL. Using learning styles myth, but no idea that SRL is a metacognitive process, not a learning styles. Zimmerman!

Dystopia 4 – Seeing correlation as causal. XKCD cartoon about correlation and causation. LA researchers have a difficulty with correlational and causal relationships. He talks through the definitions.  More clouds, darker sky, more likely it is to rain – positive correlation. Causal relation is if the first causes the second. Cause, effect. Correlation does not imply causation. If there is a causal relationship, they must be correlated. I know that all heroin addicts, rapists, Donald-Trump-Mexicans “not their best people” [??], drank milk when they were young. Possibly also rode bicycles and went to school. You could say, let’s stop giving them milk or letting them go to school, if you didn’t know that.

Spurious correlations – e.g. copies of video games vs drivers killed in collision with heavy vehicle or bus. Higher test results and stress, reported that lower stress gives you better test results; could also explain it that getting better test results lowers your stress. Great website that generates spurious correlations. e.g. worldwide consumption of mozzarella cheese vs civil engineering doctorates awarded.

In an article, test predictive model by giving it to advisers and students to see if it gave actionable information, were people called who wouldn’t get one otherwise. Tells us how it worked, but doesn’t tell us if the students IDed at risk were at risk. The idea is to first ID the at-risk students, the validity was measured on whether they got a call or not.

Dystopia 5 – Robert Burns – best laid plans of mice and men gang aft agley. Unintended consequences. Wrote in C18th. It’s about a mouse who made things for her family, then the ploughmen came and it went awry.

A recent study, a good one, student motivation in context of an LA in a summer bridge programme, an American thing about extra tuition. Changes in academic motivation over that program. How an LA one. Based on goal-orientation paradigm – mastery/performance, approach/avoidance. Results, mastery orientation decreased. Exposure to dashboard that compared their activities in the model to grades of their peers, negatively predicted the change. Resulting tools can affect students interpretations and subsequent academic success. Tell a mastery-oriented student that they’re not as good as the others, it tends to decrease their mastery behaviour! The idea of learning is to have people learn more deeply. Deep learning has quite a lot to do with mastery approach. So people who would normally learn in a deep or effective way change to shallow learning so they are as good as the others.

Pigeonholing, profiling, stereotyping. Learning labels may be tacked on to learners. Intentions noble, help to guide most appropriate instruction. Can instead pigeonhole them in to a certain track of courses, profile. Once a label has been given, it’s difficult to remove. As with credit checkers, learner has no inkling of the label that’s been applied.

In LA, there are instances of learners branded from their previous history. Labelling bias. Learner profile biases the instructors and institution perceptions. Reinforcement of discriminatory attitudes and actions, profiling based on race, gender etc. Also in the demographic data. Why do you want to know it? What’s in your framework that makes race, gender important? No problem using it, the question is why? Often to massage data. Extraordinary rendition of data, torturing it until it says whatever you want. If you send JCAL stuff where you used race or gender, need to know why.

LACE project Visions of the Future “teacher-student relations would be undermined to the detriment of learning … could be greater social exclusion”.

Intermezzo

The complexity of the ecology of learning and education. It is complex. There are systems, and systemic. A system means things are connected in a network – learners, educators, tech and media, in an environment. In that respect, the learning ecology is a system. It’s also systemic. Any changes in one part affect all the rest of the system. Capra, transaction process, multiple component. Systemic properties destroyed when the system is dissected. The nature of the system as a whole is always different from the mere sum of its components. Michael Pollen on food. You can add all the ingredients that make up a carrot and it doesn’t make a carrot.

Slide about Afghanistan COIN dynamic – complex.

Frontispiece of Thomas More’s Utopia. We can only speak of potential of LA. Done correctly, can lead to five utopian futures. Caveat, or warning, these can only be achieved if we keep in mind the dystopia. Success factor and failure factors. You can have all the success factors, but if you miss a failure factor [you fail]

Utopia 1: It’s tough to make predictions, especially about the future – Yogi Berra. Knowing what will happen (and when and why). More Yogi Berra quotes. LA could help us make predictive models that may indicate likelihood of success/failure to learner or instructor. Lots of research on this. Examples are IQ, course performance/grades, logins, prior academic history. Are these really the most relevant variables, or just the ones we have easy access to? Drunk and car keys under the streetlamp?

EARLI, SIG 27 Online measures of learning processes. Finding more objective and sophisticated markers than can be used as predictors. Try to come to winter meting in Oulu, Finland.

Utopia 2: Custom tailored learning and instruction. Adapt/personalise? Learner is often not equipped to adapt their learning behaviour. Two examples. Explaining something to yourself, works really well; re-reading and highlighting don’t work. Most students do the latter. Slide of proven/unproven strategies.  Variability of practise, much better than block. Not aaabbbcccddd, but abcdabcdabcd. Leads to better transfer. Check out book, Ten steps to complex design, soon in 3rd edition. LA can help guide learners to effective, efficient strategies. Could also relate to the level of learning.

Example adapted from Barbara Wasson’s group. Algorithm in the instructional model. What tasks student should do next. Not just whether an answer was correct and how long, but the cognitive load/mental effort. If you needed a lot, shouldn’t take a large step, but if did it quickly, maybe do more. An expert uses much less effort to solve a problem than a novice. Low effort + correct, much better idea of how large the next step should be. How many use measures of effort? Not very many.

Watch out for dystopia about myth of learning styles. Also stereotyping, pigeonholing.

Third utopia – The right thing for the right learner at the right time. Recommend/advise/intervene. Recommender systems. Beware the proper variables are used, not just the obvious ones. We’re getting better models, but are more on item hierarchy than model-based. Classification systems, advice from paper by Hendrik.

Utopia 4 – Enlightening the learner. Heavy reliance on real-time feedback to the learner. More effective or enjoyable. Zone of desired effects at 0.55. Feedback is nearly the most important thing for a large effect. LA uses a limited range of feedback possibilities.  Hattie & Timperly 2007.

Utopia 5 – Simply the best. Better learning environments. It gives us things to improve courses, grouping students (what are good students?), planning and scheduling, more things.

But! The information uncovered through good learning activities can reveal things that are otherwise opaque – quote from Erik Duval.

Some nice futures. Example, SLAM project – http://www.slamproject.org/blog – using different types of physiological data, visualising relevant SRL data. [Note to self: Point Ralph at this!]

The problem with worst case scenarios is that they actually occur.

Learning isn’t a means to an end; it is an end in itself – Heinlein. The opposite sometimes looks true in LA. LA is not the place we want to be, it’s a tool to get there.

Advertising!

Check out JCAL, special issues on LA in massively multi-user environments. JCALLAK16 free code! Be quick, only 300 can use it.

Second – to avoid myths, check out my book onUrban Myths about Learning and Education. If you hate me, write me an email, don’t troll me on Twitter.

Twitter is full of disagreements and challenges.

Questions

Tim O’Shea: Brilliant keynote. Comparing it to Paul Dirac rehearsing arguments for predicting antimatter. Argument was a puzzle. You referred to Hutton, he very interestingly was worried, driven by two puzzled. Why are seashells rocks above the water, and why is the rock in Arthur’s Seat so hard? You’ve made us all humble, it’s more complicated than we think. To go forward, are there puzzles you would recommend we should work on? That might give a dramatic move forward?

No. Dramatic move, no. It will be incremental. The most important puzzle, is why people study LA without knowing much about learning. [laughter] If you want a great leap forward, not everyone should be a learning scientist, but the biggest conundrums, lsci always the persons who ask those questions that you don’t want to be asked. They make you think. I don’t know if there’s anything I can think of off the top of my head. If you want to make great leaps forward, you need to work on a good theory, number one. And stop making use of all of these incredible self-report measures that people use. If anything has been proven wrong, it’s how people think that they learn. There appears to be almost no correlation between what they say they do and what they actually do. If you do that for learning styles, you’ll be even worse.

Jeff Grann: LA is a step-child to educational programming, large scale. Dependent variable is often great, available. How about it as a proxy for learning?

It’s a proxy for what you wanted people to learn, not for learning. Also, not a disgruntled ex-patriate. The educational system in the US is highly grade-inflated. You can’t tell a student they’re doing wrong. Everyone gets a prize for just trying. You’re making your measurement unreliable just by the way you’re dealing with that measurement. It’s not a very good proxy for what’s learned. It’s low-hanging fruit, it’s there. But if that’s the thing you should want to be studying? I’ll do it with the simplest LA. David Wiley presented it. The pheromone analytics. Like ants leave when they set up a trail, attract partners. The successful route of the ant is emphasised by second ant, third ant, release an agent followed by the other. Routes that lead to nothing, precipices, aren’t followed, and the trace becomes less. Maybe easiest way to do it, look at students that came before, look at the students who were successful, what route did they take? At the OU, no set order of tasks or order. Maybe simplest thing, look at the pheromone trail of the ants. Maybe best way is not to look too deeply, but look for success routes. Maybe that’s how you should advise them. Not saying that’s correct. Not using a proxy for success, he used success.

 

–
This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.