LAK12: Tuesday plenary 7 – EDM

Liveblog notes from Tuesday’s closing plenary session 7 – Educational Data Mining – at LAK12.

Simon chairs – Ryan Baker sends apologies. Will segue from a short paper to a panel.

George Siemens and Ryan S.J.D. Baker
Learning Analytics and Educational Data Mining: Towards Communication and Collaboration

Paper didn’t make it in to the proceedings, is available here bit.ly/lak-edm

George passes on Ryan’s apologies. After George is done, will step off stage because expecting a bloodbath.

Two communities: SoLAR first conference in LAK2011. IEDMS – first even EDM workshop in 2005 at AAAAI. Publishing JEDM since 2009.

Shared interest in big data, and what it means for learning. Shared vision to emphasise discovering new ways of understanding learning, impact of interventions. Want to tease more out of the assessment model. Aim for better real-time support for learners.

EDM defined at IEDS website. Key techniques in Baker & Yacef (2009) and EDM Handbook.

Emphasis on prediction – topical aspect in EDM discourse. Bit uncertain the level you can predict complex social activities and their outcomes. It’s humans. We face this too. Clustering, relationship mining, distillation of data for human judgement – but also emphasis on discovery of and with models.

More recently, latent knowledge assessment (semantic analysis) – knoweldge structure discoverery. Or Romero & Ventura paper for state-of-the-art.

Learning analytics – LAK11 definition! Variety of methods. Benefit of analytics isn’t in providing answers, but providing better contexts for questions. Generally tied to some sort of intervention model, bit more prominent than in LA.

These are unique spaces, but a lot of cross-tool work. Methods used as an overlap. LA comes from SNA, semantic web, systemic interventions. EDM from educational software and student modeling.

Modes of discovery subtly different, and role of human judgement and classification. EDM slightly more reductionist,. Not just a semantic difference. LAK is more systems as a whole, in complexity, holistic approach. EDM stronger emphasis on reductive, analytic approach to understand individual elements.

LAK about informing and empowering educators and learners; EDM more on automated adaptation.

Beneficial to have both (we think). Options for future: winner and loser, merger, friendly competition.

1. Win/lose, often default. Doesn’t have to be the outcome. Inefficient. Tribalistic, don’t have openness. Focus on political efforts rather than scientific efforts.

2. Merger. Not clear yet. Why pick one set of values and standards? Would either community be happy?

3. Friendly competition. We favour this model. Continual efforts to learn from each other; bi-lateral links; work together where appropriate. Sign of success would be citing each other, learning from each other. Intent is to do quality research; collaborate on capacity development.

Questions

Jon Dron: Intruiged about the third community. Have seen faces here at adaptive hypermedia. Open adaptation, social adaptation. Have we thought of that community, that goes back to the mid-90s.

George: That has come up in the past, good question. Haven’t thought about it. Shirley Alexander, Simon have mentioned there are multiple tributaries feeding in. SOmeone for LAK13 should put in a paper integrating the communities.

Dan Suther: Both communities have interest in big data. I see learning analytics as informing educators, is also appropriate for small data, but strongly computationally-supported decision-making. Data mining is only useful when you have a whole lot of stuff. But LA could include automated microanalysis.

George: Small data often overlooked. Often at point of the end user. Informed by analytics developed from large datasets, which inform the metrics. Has to be a diversity. For drawing distinctions, we’re labeling things in boxes which don’t belong in boxes. Need to acknowledge the fuzzines.

Rebecca Ferguson: Question for the audience – how many here been to an EDM conference.

[two hands, four if you count the panel]

George: We have 200 people here, that’s a good percentage.

RF: I’ve spent time looking at bibliographies of papers at LAK, and reviews of EDM. Identified top-referenced works in EDM in 2005 – things with 110 references. Don’t see those in anybody’s work here. Worry that by dividing the two communities, we’re ignoring a large body of relevant work. People say yes, we’re related to EDM, and I’ve found this handy review I’ll reference. It’s tokenistic, don’t get sense of people going back to what’s referenced. Want to encourage people to see these overlaps, and learn from EDM community. We’re replicating some of the work.

Simon welcomes John Stamper, into the lion’s den.

Educational Data Mining meets Learning Analytics

Ryan S. J. d. Baker, Simon Buckingham Shum, Erik Duval, John Stamper, David Wiley

John Stamper

John starts. SoLAR is much cooler a name than IEDMS.

Overview of EDM. Community grew out of the AI field, intelligent tutoring field, cognitive science field.

AI people – a few. Cognitive scientists – a few. Education – much more.

Seems to be a lot more education folks in the LA community. Baker & Yacef (2009) paper is key definitional.

Prediction. 60% of last EDM conf were prediction models, focused on learning. Does a student know a skill? At LAK, my brain hasn’t hurt as much as at EDM conferences. Mathematical, statistical modeling – whether students are gaming, detector models.

Clustering. Papers here generally applying existing clustering algorithms; EDM make new ones. Usually use clustering when nothing is known about the structure of the data.

Relationship mining. Not explored much in EDM world. Big place of overlap.

Discovery with models. Particularly predictive models, go in to ITS.

Distillation of data for human judgement. Automated tagging is big. Using AI to understnad, lessen amount of human input needed.

Knowledge Engineering – creating/improving models by hand rather than automatically fitting.

Pittsburgh Science of Learning Center (PSLC) – NSF funded, in 8th year, pivotal in creation of EDM community. They have a DataShop – many large-scale datasets, both public and private. Open, anyone can submit data. Now people want to get their data in. Have about 330 datasets, representing about 75m student actions (e.g. mouseclick, prompt filling), 200,000 student-hours of work. Corporates (eg. Pearson) don’t think that’s much, but for academics struggling with IRB that’s a lot. Some are public, some private – DataShop acts as a conduit to connect to PIs to get permission. Also has tools.

In EDM, have tried to engage the greater data mining community – KDD/ICDM. Had some hesitation – education data’s not interesting, too applied, not big enough for eScience. Learning analytics is even more applied. Then a 2010 KDD Cup. KDD is ACM’s most prestigious conference on data mining. Goal was to predict student responses given tutor data (big set) provided by Carnegie Learning. 655 teams registered, 130 submitted predictions, 3,400 submissions. National Taiwan University won. Only one entrant in top 10 was EDM. Website still there.

How do EDM and LAK fit? I want to find out. EDM believes in open proceedings, we do our own publications, the journal is open, you never have to pay for them. I’m chairing EDM this year, a PhD student was worried – I submitted my final version and couldn’t find the copyright release – told them you don’t need to. Kinda scary that rest of the world has PhD students looking to give away their copyright.

LAK may be more applied – visualisations, social interaction graphs. EDM more foundational, exploring algorithms. LAK could be useful at the beginning and end of many EDM projects – perceived risk of not understanding the data (problem with KDD community too).

EDM 2012 conference in Crete – invites us to go along.

Erik Duval

Has been here for about half an hour. Starting with principles. First, never allow panelists to use slides. Second, don’t encourage them too strongly to work together, or worry about it too much.  Third, I’m typically way too nice. Could’ve introduced this by saying so isn’t the reality that the data mining people don’t like EDM so much, maybe you need a new friendly home.

For Erik, EDM is part of learning analytics. LA encompasses EDM. Or – EDM as autonomous vehicle; LAK as dashboard.

LAK about empowerment, user centred design. Awareness, of people not algorithms. Help people to think, make sense. With EDM, try to build algorithms. Modest computing (Dillenbourg) – try to use people for what they’re good at, let computers do what they’re good at (number crunching).

Share at least two problems, we struggle with badly and don’t know how to solve. How do we capture meaningful traces? And how do we evaluate what we do? Maybe we can start there.

David Wiley

Slides will be at http://www.slideshare.net/opencontent

Own personal view of gospel according to David.

There are lots of shiny LAK 12 objects! HLM, Bayes, LSA, SNA, sentiment analysis. Bit like taking an educational research article with a graph in it, put it in your LMS, and you can replicate it any time. Don’t think it should be like that. Can’t lose our focus on learning – socially responsible research.

Bloom’s 2-sigma (1984) – studied multiple grade levels, subjects, compared instructional methods. One was one-on-one tutoring, or one-on-two. Called 2-sigma, found average student in tutoring group was 2-sd better on final assessment than average of those in classroom setting.  Not a theoretical number, a repeated empirical finding.

But is ridiculously expensive – we can barely afford 1:35, never mind 1:2. Results – educational research golden ratio: delta outcome / delta cost. Research should take account. Bloom’s (nearly 30yo) challenge: Can see this additional capacity to learn – want to bring that out, without taking the extra resource.

Analytics and the numerator? Through practice that’s adaptive, recommendation, feedback, remediation targeted; on the denominator – enable strategic tutoring, and decreased marking time.

Analytics and pedagogy are tightly related; the ‘do something’ is with students.

Analytics on the cheap! Open High School of Utah – online charter school, accepts by lottery. Uses OER, leverages analytics on the cheap. Khan Academy – standing assignment to go there until you achieve mastery, the analytics will figure out when that is. Strategic tutoring, driven from the gradebook – let the data tell you when and what kids need tutors for. Could do this very sophisticatedly – but here using the current grades. Teachers spend 4h/d in one-to-one, driven by this. This is often very simple, done on the cheap. Spending $8k/course to assemble materials and fill in gaps. Not at 2-sigma yet, but proficiency levels +0.84 vs state. At lowest per-pupil funding anywhere in the US (or Utah?)  <$5500 per student. Still chasing Bloom – get there in 5 years?

Plugs Open Education conference 2012 – openedconference.org

Questions/discussion

Someone 1: Learning a lot here. Key distinction is about the dashboard vs engine, front- vs back-end. Automated, or inform human decision-making. Value in these two approaches is in combining them. My sense, mixed expertise requires people to mix. Could have every other year conferences. Maybe other ways. Top-down effort?

John: The computers are only as good as the algorithms. The understanding of the algorithms is really important – can’t tell you how often I see e.g. more advanced tools used incorrectly.

Simon: We tend to pull stuff of the shelf, these guys put it there.

Someone 2: Last year there wasn’t a single ITS paper at LAK. EDM would be a shocker, where at least half do. Tutoring systems comes from Anderson in the 80s, prompted by Bloom’s paper. Attendees – different people here than at AIEd, EDM. Adminstrative positions – if I could have my Provost show up, or Director of a Learning Centre, which one should I want? A more fundamental, or are the both applicable at that level?

Simon: If Provost likes maths and equations.

John: EDM would be a tough sell for most Provosts. Appreciate the intro where you talk about policymaking. I don’t think the EDM community is on that level, don’t anticipate being on that level. Would be great if our work was translated in to that policymaking.

David: It’s great that there are so many educators here. At BYU, this Fall, teacher prep, introducing courses on data-driven decision-making – help them adopt a mindset when they think there might be data that might help them decide what to do. EDM to LAK – how many translations before a teacher in a high school can use it. Until it’s in a place where normal teachers can pick it up and use it, it’s interesting, but not impacting learning at scale. Hope we have more teachers involved. Hope more people will take this approach.

Someone 3: Was co-author on EDM/LAK brief SRI did for Dept of Ed, wanted to be a practitioner’s brief, speak to district, but hard to make the translation down. Responding to Erik on evaluation – wonder what community thinks about whether criteria on which we’re evaluating is always about learning. Many papers worried about retention, whether students graduate – it’s different things. Is it always about learning?

Erik: Are you think about other things yourself?

Someone 3 (Marie): These questions were universities, our criteria for whether these methods are effective si whether students at-risk complete – that would be a measure of success. That’s not necessarily about learning but completing the course.

Erik: Collecting meaningful traces. Results on standardised tests may not be meaningful traces of learning. (!)

Someone 3: Tying the two together might be more important.

Erik: It’s about many more things.

Someone 4: EDM gets a low-level things. At LAK, about retention, grades, mentoring, preventing disengagement. Don’t hear a lot about learning. What are the things that could be collected, what are the important ones, who’s looking at what those issues are? So these LMSes and dashboards will give advice to people that will help them promote learning, not simply help promote retention and grades which are the same thing.

Simon: Essentially a +1 on the previously. LAK focus on learning, not simply retention. How political should LAK be? Are we jumping to the needs of administrators?

Erik: It’s obvious many dashboards show rather uninteresting data. But doesn’t mean building a dashboard is a bad idea.

Someone 5: I’m an educational psychologist. Potential issue. Meaningful traces. The data gathered provide a model. What consideration for the data that are mined, analysed; consequences for the interventions that can follow.

Simon: We’re having trouble processing that one. Let’s pause there, the conversation continues. If an EDM paper come in to LAK, should we send it there – or does it belong in LAK because it has X criteria?

–
This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.