Liveblogs from Tuesday morning at LASI14.
LASI Locals report
Dragan introduces the session in Harvard. Three LASI Locals – Egypt, Netherlands, UW Madison, reporting via videoconference.
First, the Netherlands.
Hendrik Drachsler reports from the Netherlands. More technical stuff today. Yesterday they were diving in to privacy and ethics. Co-located with computer assisted assessment (CAA) conference, discussion about LA and that working together. Have made recordings that will be online later on. Projects like LACE. Joint paper session with CAA conference, on the research agenda for learning analytics. Keynote yesterday about workplace learning, matching labour market to student particulars. Presentation beyond the obvious analytics, what you can do with more advanced analytics. Finished with ethical and privacy discussion, led by someone from TU Delft. Rights as European citizens, what’s allowed, what that means for learning record stores and so on.
Dragan: Interesting to have the link with assessment. Now turning over to Madison.
Kyle, Madison: Theme for yesterday was strategic value of learning analytics not only for UW Madison, but throughout UW system. Talks from CIO, Director of Academic Tech, Chief Data Scientist. Building relationships. Foundation for following panels. One from Registrar, on value of data governance in regard to data access, use, policy, security, privacy. Another about LA from institutional research perspective, IR has done some form of LA. In afternoon, session from UW Milwaukee, and Oshkosh, about their use of a solution called MAP-Works, looking at at-risk students, predicts future risks and gives info to advisers and aggregate view for administrators to allocate resources to improve retention. Another panel from IRB folk, legal services department, on fuzzy line between evaluation and research. Issues legal services have seen relating to FIRPA. More data-driven research at the institutional level. Another great set of panels later on today. Looking forward to reporting back on those tomorrow.
Dragan: Thanks. Sounds like a strong institutional and policy perspective. Turning over to Cairo [but there are connection problems]. Open to audience here in Cambridge.
Phil Winne: Madison mentioned FERPA (Family Educational Rights and Privacy Act), ethics and privacy. What happens when it goes international – different nations have different regulations and expectations.
Kyle: Scholarly interest in the privacy of LA. Yes, the US level, we have to abide by FERPA as it will evolve. Have a lot of learn from different national contexts – moral rights, responsibilities. FERPA here, it’s the floor, not the ceiling – what we have to do, not all that we should do. Look to France, NL, what they’re doing, there’s a lot that we can learn. Obligations are LA researchers to share the frictions we have and what resolutions we’ve tried that have worked or not.
Chris Brookes: For the NL. Co-located with a computer science perspective. Have you talked about the complications of LA as an interdisciplinary area, tensions between CS technical and the teaching and learning perspectives.
[But the NL have their video and sound turned off.]
Dragan: Maybe they don’t want to be friends with us any more. Does Madison have questions for us here.
Madison: No, we’re good at our end.
Grace: Other LASI Locals. We do have Cairo. 30 people attending there. 30 people in Hong Kong. 30 people in Madrid. 60 in Madison, 50 in Utrecht. Yesterday, we had 115 participants here, 130 registered. At the public event, had an additional 100 people from the local, largest public event ever held at Harvard (!). Over 200 people last night. And 200 people online through the 5 other sites. We do have some slides and loops from Hong Kong and Madrid, we’ll play them in the break.
Dragan: Announcement – small special interest groups for dedicated lunchtimes. One is about trying to matchmake people who have data, and people who want to analyse data. Also support to early career faculty members. Phil Winne and Carolyn Rose and [someone else] have offered their time with early career faculty or new postdocs. Tomorrow Caroline Haythornthwaite and Anatoliy Gruzd.
Caroline Haythornethwaite: Happy Canada Day! Welcomes Phil Winne, at Simon Fraser University. Canada Research Chair in self-regulated learning technologies. Interests in self-regulated learning, metacognition, adaptive software. Impressive publication record. h-index of 50.
Keynote: Learning Analytics for Learning Science When N = me
Expanding scope about learning in the wild.
Phil Winne (Simon Fraser University)
Personal pronoun throughout. Photo of his location – his house, and his office. Thanks his funders.
Why learning analytics must succeed. “If we have data, let’s look at data. If all we have are opinions, let’s go with mine.”. Jim Barksdale.
Learners have poor memories for what they do. Overcome this opinionated perspective learners have about how they study.
Waste in online learning. I like to let you read, I talk less. We’re wasting data by not capturing it and using it.
Overview. Focus on N=me, and software called nStudy, a super online annotation system. Emphasising this notion about me. I’ve italicised ‘productive’ in self-regulated learning. Everybody constantly self-regulates, but not all the ways we do are productive. Key is finding what works, and how they can make it more satisfying.
Traditional learning science. What comes out of classical studies in ed psych, they don’t speak directly to me. They differ dramatically from what I experience as a learner. I have a profile of variables that are ignored in the mean used in determining whether an intervention has an effect.
This (on replication) may surprise some. About effect size and replications. (Stanley & Spencs 2014, Perspectives on Psych Sci.). Even if it was perfectly designed, try to generalise, using the same sample, you get a very wide replication interval. It’s risky for me, and for everyone els.
Skepticism about RCTs. I’m not the average. A lot of issues matter when you use RCTs to give a recommendation to an individual person. Named moderators really causal factors? Fixing mean and variance – if the names don’t really matter, you have a mis-specified model, and generalising from that is really hazardous. Little information about this. Example is sex. There’s only about one major aptitude that differentiates males from females, a form of spatial paper-folding; for most things sex doesn’t matter. Why are we required to always identify sex as an important variable? Other factors are not explored.
My learning online is quite different. Common challenges in generalising to me from RCTs. We’re on soft ground in extending from the study to me. Shouldn’t throw out these studies. But moving from them to individual recommendations is really chancy.
Hope LA becomes less authoritative, and more open, engaging with learners.
A definition of learning analytics – from SoLAR. At Learning Sciences conference – ‘raw data is an oxymoron‘. Whenever we design an instrument, it’s based on theories. Gender, sex; socio-cognitive construct. Raw data are built up from a foundation of assumptions. Need to keep that in mind. When we find data, we’re enforcing ideas about how they were constructed.
Triad of exchanges between me as an operator on information, and what happens in my mind. How do I study? I view/re-view items, tools, features. I output information, respond to prompts, search for / filter information. If you ask me what I’m doing, that’s how I’ll talk to you. [It’s all quite practical, operationalisable things – I think I do that too, but if you ask me what I’m doing I’ll probably say ‘I’m trying to get to grips with [this topic]’]
The easy language to use is the language of the interface that provides a medium for their work.
nStudy, designed to collect everything observable about what they’re doing when they study online. It’s a browser addon for Firefox and Chrome, logs data server-side. Anything in .html or PDF. Each learner self-regulates learning – they decide what to do. They make the decisions. Not just honour them, but help them become clearer about the bases for their decisions. Has a dual-purpose: offering tools, and gathering data. Two of us are interested – me as a learner, you as analysts. Not just a microscope looking at bacteria; they don’t care, but the learner does. Want to paint a full picture for both of us.
Example of study space. Left-hand panel with interface, right-hand with the resource you’re visiting, with popup menus to do things like annotating, quoting, noting, bookmarking. Chat space, with prompts, from CSCL, small-group work lit. Suggest roles, e.g. summariser, and that gives you some scaffolding about what you might say. Report-writing tool. As you’re doing research for e.g. a term paper, can begin drafting your notes, edit it, move toward a final product. All-items view shows everything.
The data can be exported to CSV. Everything timestamped. Finer-grained than keystroke level. User story, more human-readable account of what went on. How do we couple this with learning science? Bridge from observable activities, and what I like to do as a learning scientist?
The idea of a trace, it’s valuable. It should describe how I operate on info, what I operate on, and implicit evidence that I was motivated. Must remember that how we examine them is infused with our understandings of what learning is. SMART – search, monitor, assemble, rehearse, translate. Shortened lexicon for speaking about what learners are doing on nStudy. Not complete, but works for a lot of what we’re interested in.
When I make a quote, I’m metacognitively monitoring it for a match to a kind of information – important, on the test, enthused about. Why else would I colour it if I didn’t plan to review it? Can check that hypothesis. Quoting and annotating using a note template; you have a way to scaffold learners, and record what learners enter in, and where they got it. Copy and paste are similar. Several others. All preceded by monitoring, that’s the trigger.
Some of these activities have come along ‘for free’ – when it’s displayed for the learner, you can assume that learners get a quick rehearsal just by popping up the template.
Can include something as a child of another. Constructing larger packages of information, organising it in ways that I think will be useful for me. Search.
What do we do with this data? What do I want to know about me? Yet to design a query for the db so students can say, what notes did I make including term X, or what were my chats with Al about why biology is silly. What can nStudy tell me that I don’t know? Human memory unreliable. What can I do with this information? If I’m quoting that never include the disciplinary building blocks, now what?
Reiterate the interface. That’s what makes sense to me. Get some support, give guidance when filling out a form – or can leave that out.
The reports we generate, will take much work. It’s like advertising for learning science. Some basic notions about statistics – counts, workflow, tendencies (conditional probabilities), effects. Life is not as simple as ‘if I see a bold term, I quote it’ – have to ferret out by looking at that information to see what is the learner using as a standard for metacognitively monitoring what they do and don’t quote. And then are they ways of reviewing that that makes sense to the learner.
If have a sequence of events, can organise them in a transition matrix. Becomes a graph. It’s a picture of a learning strategy. Want to investigate is my strategy more or less complex or effective than yours? Are they congruent? Can we show learners pictures like this, and perhaps some other infovis, so it makes sense to them – so this is how I go about scanning, studying in greater detail. If I make them diverge, is there some benefit? Can think about how these patterns become structured.
Terms, and TermNets. Step down from where we want to go, to string-matching. Terms are building blocks of knowledge. Intro Biol book, 1000 pages, glossary has >800 terms. Biology is two things: learning a new language and learning to write sentences in that language. Can we make sense of the fact that there’s a term like mitosis, and inform learners about that. So nouns, verbs, adjectives, that you’d expect learners to not know when they start studying. Meaning is a particular configuration of these terms. Three are easy to detect. In-terms-of, sentence co-occurrence, paragraph co-occurrence. Can make graphs of these, compare them, and see what isn’t capturing it.
Can we invite learners to think more thoroughly about the building blocks of their study just by showing them these pictures? Have node diameter showing how many times I’ve operated on a term, on grounds that the more I operate the more likely I am to recall it. Weights of lines indicating sentence co-occurrence.
I do the studying. Targets – things I think we might want to work towards in learning analytics. I carry out operations, calculate the frequency, examine patterns, so to nudge me to monitor differently, could recommend new standards, help me identify information. Could we build a classifier that says, this paragraph makes an argument, this I think has been done.
A TermNet, show me the paragraphs connected to what I’m looking at at the moment. Remind me about terms define in terms of others.
I’m suggesting we nudge people toward thinking about what I’m doing, how I can extend the information I’m working with, in ways that correspond to networks, help me ID features of text that I’m not operating on enough, attending to, assembling. Patterns of operation. Major issue in 1980s about reading comprehension. Easy to teach comprehension strategies, hard to get them to continue to use it when released. Can we remind them about it? You thought it worked well, try it again?
Part of LA definition is about optimising learning. Translating information, you see a term – e.g. quadratic. What’s that mean? A graph affords useful information you may not realise, it has a point of inflection, the sign of the coefficient tells you which way it’s oriented. Do I remember those things when I see quadratic in the text? When talking about physics and scales, talking about decibels, do I understand what that means? Could do that, based on simple techniques.
More information may be better – offer adjuncts to web pages and pdf articles. Learning objectives. A learning skills course at U Victoria. People don’t really set goals when they study – more open the book, Chapter 6. When you specify a goal, setting metacognitive standards.
Big data is essential – random in RCTs yields average not like me. Idea is, my complaint about RCTs is the mean is a statistical construction that doesn’t apply to anyone. A person who scores at the mean has a set of moderating variables, but it doesn’t mean they don’t matter for them. Let’s use all those moderator variables, not just the demographics, but the ways that working with it they’re doing. Construct a model of how I study what I study, use in a post-hoc fashion, to find learners just like me but for a little difference, now have a basis for suggesting to me about here’s something I might nudge you towards.
Build systems that gather big data about studying ‘in motion’. Objective: help me join you as a member in the community of learning scientists. I want to learn better, and I need to be a learning scientist. Work in partnership, sharing data, explanations. I theorise we can help one another.
April: Looking at nStudy tool. Seems dedicated to learning declarative knowledge. What needed to scaffolding more procedural knowledge?
In the browser panel, anything on the internet you can work with. If you have simulations, or fixing values for e.g. pendulum motion, simulation records those. Record the table of the simulation, have them examine that table like a text, quote, record. Possibility for e.g. computations, is available.
April: Thinking more of, e.g. intro stats class, they’re doing analysis in SPSS. There’s a lot of declarative knowledge, but they often get stuck in interpreting a graph, or making it in the first place.
I teach multivariate stats course. I start with fractions. Most people don’t realise that the denominator is the ruler in inferential tests. Thinking about a pie chart, three things on top. Give them a sense for the effect size. Can describe that in text. Calculations done in whatever you like. Moved from mechanical calculator to electronic one. Reason that ANOVA model was popular is that it’s hand-computable, but matrix inversion isn’t. Let the computation happen as a black box. Teach the concepts, like fraction. That’s do-able in nStudy. Might have to collaborate to figure out how to make it work for you.
Tiffany Barnes: We need to instrument the steps of procedural learning – I’ll be talking about that tomorrow morning.
Regina Collins: Any assessment components? I may think I’m learning, but I may do poorly on the test. Any way you judge the quality of what the students are doing?
Can use completely different service. We have our own in-house one. Want to see if can nudge people about how they quote, highlight. Cloze test presentation of web pages. What they need to pay attention to is context. Yes, it’s possible to coordinate that and connect it up to what I talked about.
Amy: Zach yesterday talked about emotional states, bored, frustrated. Are they something that you can use to annotate your process here?
If you have other instrumentation you want to use, sync all that data, make a bigger multivariate profile of me. Interesting thing about motivation. A student looked at self-report instruments. Constructed text with hyperlinks, allowed learners to tag. ‘click here for interesting details’, ‘click here for info on the test’ – mastery vs achievement orientation. Traced what they clicked. Behaviour didn’t match self-report, self-report worse predictor than what they clicked. The information was identical. What people report about themselves is a statistical average. Same thing true about emotions, but we don’t know. When you study statistics, do you get anxious? Sometimes frustration is a good thing. What’s the real story? Have yet to find out. You can match up nStudy with any other instrumentation you want.
Elle: Average level for whole population, versus individual learner. What if, for a given population, does it ever happen that average level is closest to optimal level, and students want to achieve that level? Students who are not yet fully self-regulated learners.
We don’t know. Some cases where something that’s average might be best – e.g. inverted U for arousal. These are questions to be investigated. Is average the right thing to think about, or is it more about fit of one model to another?
Andreas Konstantinidis from KCL: Is it a limitation that it’s only online text-based resources?
Yes! You can use video or audio. Can bring up a menu. Don’t get a quote, yet. Simulations, again if you sync up the data with nStudy, can put them together.
Andreas: The recommended actions to learner are based on previous?
Not necessarily. Describe case when I’m studying because I’m interested. Or can use it in regualr RCTs.
Andreas: Maybe not average, but top students?
Something to explore. Don’t want to throw away the lit we’ve generated on RCTs and approximations to them. It’s a place to start. Can we tailor those findings in ways to match individual learners who aren’t at that average.
Andreas: Learners don’t set goals, do you think the learner is the most appropriate person to set the goal, shouldn’t that be the tutor?
Both cases are true. I set goals in my psych class. You should be setting your own goals. History report, write a paper – how much direction? Students need to learn about goal-setting as well. Environments have constraints and affordances.
This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.