LAK13: Wednesday afternoon (4)

More liveblogging from LAK13 conference – Wednesday afternoon, fourth session of the day.

Multi-disciplinarity of LAK

Multidisciplinarity vs. Multivocality, the case of “Learning Analytics”

Nicolas Balacheff, Kristine Lund

Both presenting. Nicolas starts.

Technology enhanced learning is multidisciplinary. Two possible origins – within the field, or from the outside, borrowed from other domains.

Is TEL just reinventing the wheel every year. Learning analytics, educational data mining. New discourse. Two responses. Pick up the words, see if there are commonalities. There is a consensus that LA was born close to 2009, first conference 2011. EDM longer history.

Picked up keywords relating to learning analytics, from what the communities produce. Used open archive on TEL, has about 2,000 papers, but a bit old. Took the grand challenge problems from the STELLAR network of excellence. Looked within those texts, to find LA and EDM. Found keywords related. There’s a cluster of terms close to LA, and another cluster closer to EDM, but no direct contact in terms of the lexicon. The contacts come from shared ideas – possible boundary objects, within LA, but also between LA and EDM.

DataTEL workshop, and Productive Multivocality at Alpine Rendez Vous 2011. Almost no shared vocabulary, apart from cognates of ‘learning’. Great difference in terms of scope.

Data is at the core of both communities, but in different ways – no evidence it means the same thing. One is on improving algorithms to treat data; the other focuses on interpretation of shared data. What data means could be important.

Kristine takes over. Presenting as outsiders.

Productive multivocality project.

Sharing corpora on small-group educational data. Book coming out – Eds Dan Suthers, Kristine Lund, et al. Coming from different traditions, with own dataset. ‘Pivotal moment’ as a boundary object.

Multiple disciplines can be good or bad. Multidisciplinarity – no integrations of theory or results – approached from different angles. Interdisciplinary – integrates theory, concepts, methodology. Claim is that interdisciplinary is better. Multivocal research is productive, even when from different epistemological and methodological frameworks working on the same data. Making the positions explicit important. Defining the conditions under which learning occurs.

Argument: LA community is much like the CSCL community: multidisciplinary but potential for interdisciplinarity. Their version of multivocality is closer to interdisciplinarity than to multidisciplinarity.

Example of multivocality tending to interdisciplinarity. Example on fractions in a Japanese school classroom – folding 3/4, 2/3 of paper. Three researchers worked on the corpus, the ‘pivotal moments’ are of differing lengths; made conceptions explicit.

Integrating method, or theory. First, no convergence. Two researchers compared pivotal moments, progressed in problematiques – Chie found breakpoints in frequency of new ideas; Shirouzu reconceptualised them in his framework. Not much convergence, but a discussion had happend. Second, with convergence. Two other researchers. One extended their definition of an analytics concept (Bakhtinian ‘voices’), incorporated gestures – analytic integration on a conceptual level.

Deeper theoretical integration is more difficult. Don’t always aspire to it, tension can be productive.

“Data” as a boundary object for learning analytics and EDM. Call to analyse the nature of data, and the problematiques.

LA and EDM have definitions. Set out four questions – Are the LA tools sufficient? Should all student activities be learning data? Isn’t LA reductive to academic success – limits the problematique? What’s the specific contributions of LA?

Nicolas again. Reviewer said would be more significant if analysed the publications within LAK. We started that. Saw a discrepancy in this corpus – learning analytics, and educational data mining. The words that are related are different – significant words. LA more learn, learning, knowledge, activity. EDM more teacher, student, classroom, teaching, instructional. Two in common: tool, and model.

Questions

Someone: Do think learner model is now implicit? In ITS had a model in advance. Not having that leads to difficulties?

Nicolas: Issue is what it means, what learning means in both communities – it’s not clear. Learner modelling – from the plenary, two talks today – the vision of making sense of the learner understanding and the nature of the learning outcomes, the way the learner learns, this is at the core. Explicit proposals within LA community for models, visualisations, to understand. The words are present. Now we have to look at what type of data, what type of model. One problem – to have automatic interpretation of the outcome of the learning process. Could we have a powerful representation of the learner, the learning. This is a model of the learner.

Someone: Why important to find similarities or differences between these two fields?

Kristine: For me, it’s evident, they’re fields that are similar. We want to avoid having two communities that don’t speak to each other but are working on the same issue. If come to agreement on the terms, what kind of learning, what we’re focusing on. This conversation allows people to come together and make progress as a larger community.

Nicolas: Difficult to respond. Big difference suggesting to a student that they have to make a choice because of what we see of their trajectory they would be better to go in a particular direction. Don’t need accurate view at the epistemic or cognitive level. At the same time, you may need to have fb to the learner at a finer grain. For example, this morning, the cognitive relevance of the environment – you need to have feedback to target the misconception. It’s different feedback than working with 10 years history.

Dan: We should stop worrying. We have to deal with these issues. The experimentalist, the ethnographer, the data miner. We have to have some boundary objects to support that discussion. Not enough to use the same data – they come out with completely different ideas. So use common ideas – e.g. pivotal moments. Want to make productive interdisciplinarity.

Someone: One of the challenges is you’ve disambiguated between two research communities. LA has e.g. provosts, presidents, Educause, CIOs – they use it the same way as some of the researchers. The LA community is bigger than the research community. Not sure same is true for EDM. Would be interesting to consider this. Look at broader communities.

Kristine: Really good point. It’s true we’ve framed this in terms of researchers from communities using the terms, and the concepts behind them. Other stakeholders are using the same kinds of terms – what are they putting behind them? Equally important to look at them.

Alyssa Wise: With multivocality, the data wasn’t enough as a boundary object – need semi-compatible lens to make the analyses talk to each other. What are the sorts of things that might serve as the additional framing of the boundary objects?

Kristine: That’s the next question! Data put forward, but that’s not enough – needed ‘pivotal moments’ to have similar lens. Also tools and models – maybe those, if we ask more specific questions, could enable the discussion in deeper ways.

Panel: Educational Data Scientists – A Scarce Breed

Simon Buckingham Shum, Ryan S.J.D. Baker, John Behrens, Martin Hawksey, Naomi Jeffery, Roy Pea

(Roy Pea sadly unavailable, replaced by Taylor Martin from Utah State U.)

Ryan Baker – President of EDM Society. John Behrens, Director at Peasron. Martin Hawksey, JISC CETIS in UK. Naomi Jeffery, OU Scotland. Taylor Martin stepping in for Roy Pea.

McKinsey report predicted a shortage of talent necessary for organisations to take advantage of big data. “Data Scientist: The Sexiest Job of the 21st Century” (Harvard Business Review). We do need people who can do this sort of thing.

Strange new creature emerging, need to understand it. Are they about the rule the world, as geeks have been taking over in the last one.

Panel position paper and submitted questions.

Ryan Baker

We Need More Education Data Scientists. They’re in demand, sexy. Who in industry has hired recently? A lot of hands. Many in academia. Many hands again. Amazing compared to two years ago.

Educational data is special. It’s not the same as other kinds of data. There’s a role for specific bioinformatics. Masters Programme at TC on Learning Analytics. Learning Sci and Engineering at CMU, similar at WPI. Much going on

John Behrens

I’ve been to EDM and LA. Pearson sponsors both. Both groups are really nice people.

Educational data scientists should be deep-thinking philosophers too.

Quiz – mystery picture, raise hand when you know what it is. We think things are data. But in what way?

In the context of making stuff, have to think clearly about what you’re doing.

Things that make him made:

• “Looks cool. What does it mean?” “I don’t know” – means you’re just a technician.
• “Looks great at a high level, how have you explored the assumptions?” “What assumptions?”
• What are the relevant background issues? I’m just the data person.
• Does it make sense? Dunno, it just came out that way.

Be a philosopher, don’t let your schooling interfere with your education.

Many communities not represented here. Data analysis community, Exploratory data analysis – John Tukey led. Visualisation, robustness. Think of data scientist as new – but Tukey’s student wrote an article about it in 2001.

Question assumptions!

Martin Hawksey

He’s employed as a learning technologist, doesn’t see himself as a data scientist. Fallen in to this world recently.

Electronic distance measurement to educational data mining (EDM to EDM).

IBM’s CEO says Bid Data is like oil, enterprises need help extracting the value. (article). With commoditisation comes innovation.

Pioneer in electronic distance measurement – on an oil platform he used to work on. In South Shields, production platform, was being refurbished, lived under the helipad. A big tangle of pipes where all comes up, work on it, then back to shore. Very complex, many critical systems. Using lasers to measure things very accurately.  Graduated with degree in structural engineering.

Then worked for Data Converters Ltd. Converted Word documents (learning material) in to HTML. Very exciting stuff. Taught him to look at the source.

Went back to university, multimedia and ?adaptive systems. Midway through the course, tutor emailed everyone the current grades – anonymised using only matric numbers. I got the code – put the numbers in to webmail, it tells you who the person is. Did that, did conditional formatting, merged all the data together – saw he was at the top of the class, graduated with distinction.

Job at Document Intelligence. A scanning company – scanned paper, indexed it.

Job as learning tech adviser at university. Coal face stuff. Pit pony style. Seeing how staff respond to pressures from above and below. Project redesigning eassessment. Got interested in innovation. In 2009 came across Tony Hirst, was blown away by what he was doing with data – making it do tricks and flips. Now ‘find the feed’.

Google Spreadsheets does interesting stuff, feeds from social networks, combine it, tell people things that are interesting. Used Gephi – looks great in presentations. Now started on ocTEL – Open Course in Technology Enhanced Learning. Reinvented Google Reader inside a WordPress install. Bubble diagrams.

Requires creativity, curiosity, playfulness. Make mistakes in the open.

Naomi Jefferey

Good at organising data, not so good at organising herself (she says).

I’m working in administration at the OU, been there 5 years in different roles but same task: answering questions people have with data about our students. Always looking for software that’ll help visualise data. At the moment, networks – as our knowledge expands beyond just passing modules to progressing, thinking more about pathways – gets very complicated very quickly. Network visualisation software is what I’m on the look for.

Big Data – concerns about how it’s used. Really exciting, but also really frightening. Sometimes no underlying robust model. Data scientist crosses from exploring the data, what you can see then down through LA in to the educational data mining, building robust models you can then communicate back to people. Both students and teachers.

Background in statistical modelling, modelling pass and progression at the OU. Exciting work presenting this to students about their learning; keen to draw through from robust models built on cognitive theory to present to students things that are meaningful to them.

Taylor Martin

I’m not Roy. Maybe it’s starting to happen. Not representing his beautiful elegant philosophy.

Our work, future of educational data science. Talking to students creating their own data science careers. Cognitive psychologists learning different methodologies, like her.

We have cognitive, contextual, design aspects of the environment. Often online or computerised. Will be an interdisciplinary career. Big topic at Strata (big data big conference). Ask good questions first, otherwise hard to get good answers at the end. Can now answer new questions.

We have to play nice with some people who might not get represented at conferences, like academics. Need a clarion call for people who generate programmes to develop collaborations that will build next generation data scientists.

This year Big Data Summer Camp – Learning Analytics Summer Institute. All sorts of people, building from cognitive neuroscience who got together every summer – doctoral students, professors, taught each other. It’ll be like that. Look forward to seeing many of you there.

Questions

Chris Brooks posted a question – role of scientists, multidisciplinary thing.

Chris: One question is interdisciplinary versus multidisciplinary – where are data scientists? Should we be creating this new breed, with new techniques – or building teams?

Ryan: Both! Computer scientist, info vis person. I’m not a computer scientist any more. Not really just a cognitive scientist. Multiple specialisations that blend in to something new.

John: Previous session, different interpretations of the disciplines, is important to keep in mind. Not clear what an educl data scientist is – if someone says they are I want to unpack it. It means different things to different people. Computational biologists, computer scientists, all sorts – but if someone says I’m only this, that’s a problem. Even info vis people approach from e.g. comp sci, communicative, art perspectives. All part of the ecosystem.

Naomi: A background in science is important. It frightens me a lot of work on data is coming out with very pretty things, but what’s the meaning? It looks great, but is it repeatable? Is it robust? Need a good understanding of data analysis to be in a team of edl data scientists.

Taylor: Plug for trip to NRC to change IRB. Whatever teams people put together, tend to forget this line unless people have a psychology background in running experiments. Often commercial activity published in e.g. blogs so not an IRB violation. (But problematic.)

Simon BS: Martin, you don’t think of yourself as a data scientist?

Martin: No. I don’t like being classified. Don’t like what I do being classified as a particular thing, I just do it, because there’s a need to do it. What are the answers you’re trying to come up with? Sometimes it needs a big team, sometimes just one person with the curiosity to go off an explore it.

Xavier: A lot of educational researchers ask what is this new data research – we’ve been doing this for years. Collect data, apply analytics, come to conclusions. What is different? Because we use data mining not stats? What’s the difference.

Martin: Nothing. It’s fashion. MOOCs – we’ve been doing online courses for decades now, but the terms get fashionable. E.g. Big Data – it’s just data.

John: My original tradition is educational statistics. Big changes going on, so Ok to have different names. One thing, there’s a digital revolution going on, world changing dramatically. We need to have place for those discussions as they relate to methodology. An example: assessment. In the last century, it was expensive to collect data from students. So built artificial environment, score at scale – that’s how we got the tests we use, especially multiple choice tests. In the current century, the work product comes from games, homework practice, lots of different places. If approach it with your last-century, digital desert conceptualisation, will be blind to it. That’s why educational stats community hasn’t come in – they’re still teaching ANOVA from 1920s, invented when computing limited what you could do. Fisher wouldn’t do that now. Most multivariate stats invented in 1939 and practiced in 1960s when the computing allowed it. The computing’s becoming automated, the data extraction is automated. The methodology has to evolve to reflect these changes.

Taylor: A lot of what we think of is what are good standards. We don’t have the equivalent for most of our stats of e.g. p<0.05 so it’s good. How do we know we’re not chasing our tails?

Naomi: I think there is a difference. Educational data scientists are cool. Sexy. That’s important. We’re starting to be involved in places we weren’t before. Need to think about analysis before you create the data. Higher profile, more chance of being involved from the start.

Someone: Big data in the US, also programs created in predictive analytics – e.g. when consumer will buy things. Similar – lots of data. The predictive part in LA – of failure, doing an intervention – but where is the socio-cultural aspect? The motivation of the student and socio-economic status will predict outcome, and even lifetime earnings. How is it different? Do we know more about learning?

John: Some of my favourite courses were in business – probabilistic methods and others. What’s the nature of the analysis in the human activity we’re undertaking? Example – predicting how highly-related, on average, the socio-economic status (e.g. books in parental home) predicts outcome. For individual student the trajectory can be very different. Many were outliers in that model – me too – I was predicted to be working in a factory. Need to tease that out. How individual feedback helps the individual. Second example, in business stats, decision analysis – it’s easy because they’re almost always optimising for money. We want to say this cost/benefit decision analysis we should bring in, understand it for what it means in education. What’s the risk of making a wrong decision to a student? Recommending, if you recommend Harry Potter vs 50 Shades of Grey, that may not affect your life course. But if you have them spending time that’s less efficient, that’s a different level of outcome. We need education in data science.

Ryan: This field draws off many. Business is one. Medical diagnostics is one. Bioinformatics. Lot of variables are the same. E.g. Bayesian modelling.

Yalta?: The practitioners: if there’s more analytics for K12 schools, needs to be understood. Need teachers, principals, special needs coordinators to understand too. Who on the ground, in real life, should be equipped with more knowledge in how to read data and  interpret it?

Taylor: As the teacher ed person here. Have taught for many years. We’re working a lot with that question. There’s a lot of dashboards. Administrators have to have it. Has to be incredibly clear. But no time in teacher ed programs. We have to make those representations for teachers. The teacher can see what the kids did at the end of the day, linked to a set of standards that I’m responsible for. We’re working on that, but not enough research on it.

John: Great contribution slowly moving in to education is the tradition of HCI and analysis there. Old model was, psychometricians made test, score report, legal implications, and it just went to the teacher. The whole idea for HCI is an important set of analyses – Activity Theory, interaction design. Different stakeholders have different needs.

Martin: Related to HCI – have seen dashboards emerging now, I worry – if you have to educate a practitioner on what the dashboard means for half an hour or more, you need to work more on the dashboard. We’ll see more personalisation in how people control how data is presented to them. Looked at Thomson Reuters stock exchange data – very much individualised, for brokers to pull out feeds most useful to them, in most useful way, in configurable dashboard. Likely to see a lot more of that in the future.

David Shaffer: Jump back a question. Economics and education, and medicine. Would you comment on idea that in education, the outcomes are less well specified than in business. Pain scale arbitrary in medicine, but QALYs for treatment options. But in education, what outcome we want to achieve is up in the air, and that makes the analytics different.

John: It’s worse than that. Because of implementation issues, politicisation of the educational system, we measure things that people agree on, not what they act on in the political sphere. Small slice of measurement. Ask the educational system to feed the kid, feel like a human being, many other things we’re not measuring. Expectations huge, measurement tiny. That’s not Pearson policy, by the way!

Jan Plass: Accountability in US schools. Students and schools accountable in terms of learning outcomes. Unidimensional construct might be reversed by data mining to e.g. textbook producers. That is a huge shift – who is accountable for what, how the materials are under more scrutiny now, efficacy trials.

John: Project Taylor, Ryan and I are working on. A large implementation of online interactions. In this product, teacher and students have a lot of choices in what they do with it. We’re looking at how variation in e.g. how teacher assigns things, how much students interact affects the learning, and feed that back. Teachers are still the ones in control, students have to make decisions. I agree in large electronic systems there’s huge potential for data mining to improve the learning and how to optimise it.

Taylor: Trivial point. Teachers are bombarded by 5000 versions of ‘here’s what your kids learned’. Would love to bring in more, but problematic. Teachers don’t know what it all means. Need a common framework, or a set of APIs or something, agreed and used across diverse systems.

–

This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.