Liveblog notes from Wednesday morning, 16 March 2016, at the two-day expert workshop on “The implications and opportunities of learning analytics for European educational policy”, at the Allard Pierson Museum, Amsterdam, organised by the LAEP project and the LACE project.
Summary of yesterday
Garron and Jenna summarise. Garron – one point was the need for presence of teachers in the field, how to connect to their problems. Another thing was around data privacy and considerations to take in further efforts.
Jenna – a lot of talk about the human element in learning analytics, to serve processes, about more than just numbers. What do we mean by education? How does LA add to that? Student data goes beyond simple activity data, rich datasets needs. Debate – if all you have is activity data, can you still gain valuable insight? A question about what is it that we want to know, and what data do we need. What we did not talk about is also important. A lot of the focus on tools, practices, teachers. Big demographic we’re missing, the students, the people LA is supposed to serve. Big question for today, how is LA servicng them, how will they gain. Secondly, much focus on HE, a lot of us come from that so perhaps natural. It also focuses on [other areas], we should keep that variety in mind. Finally, activity in the afternoon brought good, broad suggestions, but work today on explicit policy suggestions. How do we get from ideas to policy?
Konstantin Scheller
From the Commission, DG Education and Culture, funding this project. Some context about why you’re here, and what will happen with the outcomes. In 2013, EC had policy communication called Opening Up Education, first to mention LA. Most member states didn’t have it on their states. Now, some countries were. It was already a field, we noticed at that time there was little understanding of how this could be in practice, no attempts to get to policy. This research task exists to provide input to policymakers. It will be a document, which will come to member states – e.g. Estonia, Denmark – what you say will reach the ministries, the Ministers, the message will be there, and go on to Commission policy. How can we bring that vision to reality, what are the policy issues we should address, what are the big questions. We should not get involved at all? Perhaps cause more harm than good if no deep understanding. You’re not just talking to researchers, you’re talking indirectly to education systems about what they should be doing. High-level messages. What should policymakers have in their minds? Usually they don’t know so much. We are all not experts, you are the experts. Important for us to understand what concerns you, what might be coming in a few years. All the issues that you see, that need to be addressed at a high level – privacy, standards. Thanks for helping us to make good policy.
Visions of the Future
Doug Clow (me) gave a presentation on the LACE project’s Visions of the Future study.
Planning Exercise
What are your current concerns over the next 3 years in moving LA forwards in your own practice, in your own environment? [small group work]
Simple solutions? What if we focused on simple things, like e.g. happiness, people are willing to share things like emoticons on social media. Walking before we can run. How do you convince people that LA is a good thing? Teachers, students, policymakers, others?
Young kids and parents – they’re very involved at that stage. Anything with connotations that don’t go down well. But being on top of what the child are doing, keeping a close eye on their progress, parents are very supportive of that.
Subtle differences making a big one, between being cared about, and being surveilled.
Procrastination can be an important thing! Need to be allowed to play your procrastination card today. Don’t want to be reminded, just to gorge on Netflix.
We could do stuff technologically, but is that going to put people off? A friend had a heavyweight spreadsheet about everything they do – right down to e.g. sex?! With LA, we can have knowledge like e.g. you’re not going to do well in this class. That’s already sending a signal not to make the effort. It should be a prompt to get more help. We have to find a way to use the analytics in a way that’s more human, more responsive to the needs of the person.
Two things: not going too far, but also about the tone.
User experience stuff is important. As researchers, we’re interested in the detailed nuances, but that’s only interesting to other researchers. So e.g. standing tables, people who are tall use them differently to people who are short – uninteresting to people apart from furniture designers.
Must be some parents who are more controlling than others? Absolutely. End user experience, for the child, or the parent. A 3 or 4 yo, has limited ability to agree to how their data is used. Have to bring in another stakeholder.
Supporting good pedagogy, formative assessment, what teachers have always done, but enhancing that, and it’s not too onerous.
Near future?
A: Last 10 yo, schools required to have a learning platform. Now fragmented school system, change hard in England. Leaders in the school system influence, self-improving school system. My school is a national support school, so a chance to roll out what we do to other schools. Ofsted still there. Networks of schools, no local government. Now headteachers who are system leaders, messy, slower to effect change.
B: Creating systems that support small group work around design engineering tasks. To have a system that can support interest-driven work by students. So the role of teacher is more of a coach, students have more of opportunity to reflect on how the group work is going. Ask why did our project not work, create documentation on the fly to create their reports, learning portfolios. Lots of those tools missing. Create tools to support students, give them the chance to reflect. We’re known for thoughtful interaction design, to teach people be more reflective practitioners.
C: From policy perspective, will be increased demand for services like learning analytics, educational systems are under pressure to cater for refugee crisis, shift of demand in skills in the labour market, shifting funding regimes. Solutions lie in scalable systems, relieve the teachers from some of their more tedious duties. Efficiency perspective on this is important from policy perspectives. Those aspects will be very much sought after in years to come. Funding will be less explorative, less quality improvement of learning, more on efficiency and easing the burden on the systems as a whole. It’s kind of a bleak message. Looking for leap from structured subjects (STEM) to e.g. language or history teaching? There’s some.
D: Assessment processes – formative assessment. MCQs is easy. But more qualitative, essay writing, we can do more to support humans. It could be more efficient if supervisors get help from text analysis. It should be reliable.
E: Interaction could be a thing – face readers to train communication, it’s already used. e.g. for security guards. Also in sales. A program called Communicate, learning interaction, moving to face readers. One of the modules is on negotiation. How to be a better person!
Plenary feedback
(a) There are question marks. Europe has shown that privacy is important. Definition of personal data will be larger. New category of data, anonymous and also pseudonymous data. Third thing, role of the processor, may change in 3y, controller has more obligations. Helping parents, users understand who is responsible for what. A bad scenario would be organisations that are privacy-sensitive are cautious, meaning slow, organisations that are not will be first on the market, will be easy and compelling to end user, so worst scenario with companies offering software that is not really privacy-compliant.
We already have this worst-case scenario. We have this in the US, adaptive smart courses, but there’s no data protection. That is why they’re successful, they don’t care about [data protection]. We must be careful not to have regulations that force us to fall too far behind.
Putting ourselves at a competitive disadvantage … but maybe we should? Playing out at the moment with LMS features?
(b) Three themes. Bringing people on board. Going to teachers, students and staff in institutions. Issue of partners working with teachers, employers, trade unions. Second, issues around legislation. New rules, guidelines from EU perspective. And navigating existing legislation – is what we do compliant? Third, resource creation and developing and exploiting opportunities. Building new ones, new settings, or developing what we already have, getting benefit for individual users.
Quality!
(c) We also had three. First, institutional support. Teachers are experimenting with small-scale projects. To scale up, need institutional support to scale up and experiment. More time for teachers to spend on these. Second point, more evidence. If we have opportunity to experiment, get good practices. We should see whether it works. Will help to convince people. Should be explicit part of grant – include evaluation phase, not just design of a tool, where you evaluate and share results with the community. Maybe should also include one or two people who spend more than 10% of time, so less scattered project team. Critique of proposals, for smaller startup experiments, you need lots of paperwork and time passes before you start. Once everything is settled, a few years to start. If you could start from day 1 would have results more quickly. Maybe something in the bureaucracy.
SoLAR had Open Learning Analytics framework. Need this framework, monitor how you cover that, and that’s missing. Lots of universities are building dashboard. If we don’t have a framework … we can’t distribute the grants.
In H2020, require that things will be open, no intermediate step. If you go for commercial, you have to have business plan away.
Academic project often just build it. But that’s changed in H2020, looking for product, not algorithms.
Maybe an EU framework?
Is quality the same as empirical evidence? It’s about assessing quality in general.
(d) 3y is one generation of students, and 0.1 y of teachers. We have to socialise the idea with [all stakeholders], currently building silos to protect themselves. “My data” could be a problem. What are the sentiments coming in? Scary developments. Frank Pasquale’s book on the Black Box Society. Nobody knows what these are doing. Demands for safe spaces – students must not be frightened, shocked, scared. Will LA have to deal with that? If that happens, we have to distinguish ourselves from that. We are not Fb manipulating your emotions, we’re not Google exploiting your data. We’re not unaccountable algorithm, we have humans. We need a different story from the big data. The wave of sentiment created by them will sweep up away.
I disagree. We’ve been doing large-scale implementation of predictive analytics. Have interviewed 10 tutors over time, they’re saying I don’t trust the data. Later say, yes, the 10 were at risk they dropped out. Then later say, yeah, 10 new people identified. There has to be some human interpretation. We’re losing students because people are not acting on the data, even though we know it’s accurate?
This is a critical point. One thing is data, stats, this student is at risk. What can I do? What is realistic? Is there any guidance about what I could do within the resources I do have? What are my chances to reach this person? These are the critical questions for the next year. Superficial information, from red flag to this is what you can do.
My reading of the policy to students is that what they do with red flag is go to your tutor who knows e.g. you have just moved house. At the moment the OU is explicit that there is human mediation.
The purpose of red flag is not the concern of the student, but ROI. Natural churn rate with adult education. Is this the right use of LA, to ID someone at risk, because as a teacher we’re concerned. Or is it the OU interested in maintaining higher profit.
Not trying to defend the OU! We know there’s 2-week time window we flag students at risk, they eventually drop out. Thousands of students. Even if 10k at risk, don’t have enough to call on them. We are already at at stage to use LA, but how do we implement it without the structures?
DeepMind is look for a job now!
Institutional interest in higher retention. Also a strong issue for the student. If they’re leaving on a mature, active decision. If they fail and didn’t realise they were in trouble, that’s different. So LA could help them make adult decisions about how they are working.
It’s an example of how we have the data, don’t know how to apply it. Should be, we know what to apply, then what data do we need.
(e) Finding resources, bringing people on board. Many of us developing tools, trying to implement it. At some point, our project ends, what then? I’m already involved, our digital portfolio product, what do we want to do when the project is done? Did I do a lot of effort to develop tools, and then …? Another issue is related to emotional or social aspects. We have emphasis on WBL, you have to be confident in the workplace also.
There’s a gap between little projects, then bigger. Role for Commission in talent-spotting after project finished?
At the end, you have to do a sustainability plan, would like to write it will be used in our teaching system. But we only have it in our system because the project is there.
Other side of the coin, problem of everything is either open or in a business plan. Was supposed to be a solution.
It can be open-sourced, but the system still has to be adapted, fine-tuned to change our program.
In European research funding, it’s balancing on an edge. Project duration 3y, 4y. First develop prototype, go in to schools. Doing some research, but also real life of the school. If you want to engage with them, to use what you have, you need a professional product, and it costs to realise that product. Have to support that each day. Make sure they don’t delete their databases. Easy to use configuration tool. It’s a difficult balance between real life. Then the project is over, and we don’t have resources to support them like a company does.
Also SMEs, they want to have profit, they do not want to give a licence for free. Have a strong collaboration, but what happens next. Example in the WATCHME project, we’re working through it. We have concerns how this will end up. We have a strong sustainability plan, data market analysis, but who will take responsibility for that?
Reference model idea. Do reference implementation based on the best of those, give a basis for companies to compete.
(f) Motivational and policy effects at many different levels. IT department having control over the data, institutional policy, not allowing faculties to have access to complete student data. Institutional policies on continuous assessment as a motivational factor to drive other experimental analytics. Government and agency policy considerations, e.g. how student funding is affected by levels of completion, examples from Norway about effect of introduction of continuous assessment. Use of different learning models, PBL, systematic change of institutions.
Commercial vendors are already there with quality assurance, it’s a real pressing issue.
DataShop example, anonymised data.
Best practice guidelines, evidence, should be packaged to the level where decisions are made.
(g) Our own settings, but general themes: When thinking about tech, future, important to consider the danger of getting carried away and doing too much too soon. Restraint! Not doing too much. Restraint in the level of LA. There are things we can do, but are they going to encourage broad take-up. Perhaps more focus on shallower depth across more institutions, rather than some racing ahead at high levels. Restraint in tone of LA. End user experience of the learner – react to automated messages if you’ve been red-flagged. Nuance in learners – in my setting, age of pupils, the parents are also in play. Teachers as well as end users. Marking, how possible it is for technology to mark essays. If you are a student, compare feedback from a machine versus human. From teacher perspective, nice to have machine marking.
Underlying message: wait for LA to mature?
At a low level, that maturity is already there. Not being distracted by the possibilities – running before you walk.
Counter-argument, leadership in UK has to take more LA. If we’re restrained and we’re the experts, not in line with the increase on the throttle.
Different to summarise different settings. My setting, I’m a headteacher in a fragmented school system, it’s becoming more fragmented. Key part of the hierarchy is now gone (local government). Not sure the message from the Government in England is clear.
The Australians, NZ, have policy downwards. We may say restraint, that’s a competitive disadvantage compared to others.
I see a system, looking across institutions, shallow, but everyone’s involved, but alternative where some institutions at very high level, very sophisticated, but very patchy.
Interlude
So in 2025 … that’s not so far away, but in technology terms, it could be.
Case studies
We have nine case studies. Provocative, really weird. We want you to think out of the box. Hope for a heated discussion. One row on scenario 1 – one focus on role of the learner, one on teacher. Another row on scenario 2, learner, learner, teacher. Until lunch.
[This group scenario 2: Tayla Özdemir, Jan Zoetemelk.]
This is a very weird scenario. A bit dystopian. Think it’s supposed to be. Very big brother watching you. There’s potential for LA to help Tayla develop her Dutch skills. Over the years, problem here is borders between countries, qualified in one country and moving to another. We want LA to be international, free movement of people and skills, can be used wherever they are. Movement of skills and data? Would need legislation as well. The data has gone in this scenario. But if it’d been held centrally in one country … except here they’re outside the EU. Could create a training program for her, assuming willing to pay off the chip over time, could work with a social worker. LA could evaluate if she has the skills to go forward. In Germany, they have community translators, you can work as e.g. a certified medical translator. A program like that she could apply for, become a trainee to be recertified. This happens today! Recertification is a C20th idea. This is a case of someone having the skills, not the certification. In 2025, maybe there’s the previously-wired skills tests, so we can test that to be admitted, do these courses. This is already happening, with competency framework in Dutch. Language skills, she can’t afford the translator.
Self-testing, then recommender systems. Recognition of the value of diplomas. But can’t run the tests without the language. Wouldn’t it be cheaper to just give her the app? If she works as a trainee, can pay off the app, do the self-testing. Microfinance for skills assessment and translator tool. You would still want a live evaluation, field experience. Or peer evaluation.
How about increased demand for social workers, have them train in Dutch but also introduce to the system, for more credits. An intensive language course, is solving the recruitment crisis. Quite targeted language courses, need special vocabulary. A lot of field work. A local community language is then a strength.
- Analytics – used for self-testing, recommend resources, field experience with Dutch peers.
- Main barriers – funding, but if LA can identify likely success, can be more confident of state outlay on her training. If for religious reasons she will refuse implant, will have problems getting a job. Moral or attitude training towards accepting it. [!]
- Policy – in the training programs.
Plenary feedback
[after musical chairs exercise]
[the scenarios will be online later …]
Scenario 1: Learner – Jack Wood
We found it overwhelmingly grim. Potentially quite realistic. We went surreal in our answer. Being creative, bringing in spying agencies. It almost feels like LA is irrelevant. So many core fundamentals just wrong. It highlights the inequality, evidence to say this is unfair, student is falling behind because. But a few points – the LA could detect that this student always responds, could maybe adapt the materials. There are other things you could be doing.
I thought it was quite relevant. If the secret service have the algorithms and data, that could be outsourced and useful for the whole community. If it’s done anyway, we should embrace it for wider use. NSA have not saved lives with their data.
Policy, you could see people with poor devices, give them better. But should take steps back and think of overall policy. Access to devices, is that as core as access to heat, water etc. It may not be LA, but access to tech becomes almost a right.
Scenario 1: Teacher – Jane Philips
We got really depressed by this scenario. Someone must before class average!
Teachers could be subjected to this, but we’ll act subversive, do something else that they would like to do. Their roles are reduced here.
Not unrealistic – tutors paid at e.g. University of Phoenix about results. You can measure this stuff.
It was too much about LA, with a teacher should not mention LA, it should support what they’re doing. We should not even bother them with that.
If teacher doesn’t understand why she should be doing that, she will try to game the system.
Using marketeer language, there’s a real risk.
Scenario 2: Learner – Tanya Özdemir
Chance for policy to be changed to let people learn more widely.
Option 1, LA could be used to short-circuit recertification process putting them in realistic scenarios, comparing data with benchmarks of approved Dutch people through that, see specific areas needing training. It might take less than training from scratch.
We discussed the same route. Concept of stealth assessment. As a teacher, we do assessment, but it’s not the goal, want to train them to be good, then do some assessment. Can see how she adapts to the cases. If she does have her history, she will adapt quickly. If not, it will just take more time.
There are highly talented people coming from across the world to Europe, and we’re not using their skills.
Language is a way of thinking, they look at concepts in a different way. Not learning Chinese any more will not help us. Like Hitchhiker’s Guide to the Galaxy Babel Fish.
Scenario 2: Employer – Jan Zoetemelk
The employer has no more money, great demand, rising tensions.
Could watch behaviour in action. Problem is she has to take the implant [for translation].
What about policy?
How much different is this from smartphones now? Some have it, some don’t. It’s just not in your head.
It is fundamentally different, it’s a physical violation, you don’t need to do it anyway, there are other ways to contact your brain.
More about if we want to regulate this, why don’t we regulate it today? It’s just better quality data.
We do need to regulate. If we don’t, society will make its own mind up, people are already turning off their phone in shopping centres because of unregulated tracking. If you regulate by societal trust, you end up with extremes.
Now you have free choice to disconnect. It’s difficult to do. But the choice is the same here. It’s not a fair choice. My niece and nephew have to have a device, it has to be an iPad, at age 12. This is already happening.
What has all this to do with LA?
Analogy of Facebook – some employers ask for access to employee’s Facebook. Not about prohibiting it, but what can you use that data for. That’s can the employer request that data.
That’s an ethical question. For me personally, I care about ethics, I’m thinking about Tayla that she would love to work as a social worker but she fails in language. The first problem is how to teach her Dutch so she can interact, given the resource context. How can learning technologies support me in this attempt in teaching her?
Ethics is important part in it. We need to see the cultural aspect. Language is also about how people behave, how they think, you approach people in a different way. You would not get that from a language translator. That’s about education, social behaviour. This is not a viable solution for the problem we have right now, with refugees. We need to approach this at a different level.
At the end, we’ll discuss the scenarios, we’ll ask you to vote which ones are likely, and which ones you would hate.
–
This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.
Doug Clow's Imaginatively-Titled Blog 
2 thoughts on “LAEP2016: Weds am”
Comments are closed.