Patrick McAndrew: Becoming more agile researchers: experiences from researching Open Education Resources

Open society. Private option not available any more! As the OU we’ve been working in open education resources for the last 8 years. But it’s all accelerated recently. Coverage wide – but not to The Sun. Interest in MOOC over time (Google Trends) – exploding up from 2012 – but has now peaked!

It’s a CALRG conference – so need an activity theory triangle. Used to map out activity.

Looked at Research 2.0 (paper with Steven Godwin and Andreia dos Santos, 2009). Web 2.0 characteristics (O’Reilly) mapped on to research approach.

Then OLnet, Evidence Hub, OpenLearn, OER Research Hub. Spotting interesting things, finding the evidence.

Used various tools – Claims Garden (OLnet) – looked at claims, get feedback from people. Produced various visualisations to get messages across. OER Research Hub – exploring hypotheses (have a list). Setting out things agreed with the funder, interested in the policy side – issues we can explore. How do we do that? With OERchery – get collective input.  Two scales – important/not important, provable/not provable – ask to rate on that scale.

Working with range of collaborations in different areas – HE, Schools (K-12), Informal learning, Community Colleges.

Looking for mixture of eureka moments – we know this. Talking to people. Have to balance that with testing the ideas, collective input.

Shows image to inspire – small rodent called Ziggy who’s a bit agile and gets his reward.

Agile Development/ eXtreme Programming. People don’t know what can be done until you’ve done it. Do it very quickly, not spending a lot of time in planning.

OU Camden, FutureLearn developers actually are in the basement. Good exemplar of people following that process. Lot of turnover in staff. Clear on vision, what they’re trying to achieve. Language is about sprints – two-week targets. Have a spring board – sticky notes, classified by idea, not started, started, blocked, finished. Also do a sort of task-size bingo – how big is it. Or T-shirt size – S/M/L/XL, XXXXL. Doing a scrum – stand-up meetings, object passed around, person says 3 things, what did yesterday, what doing today, any issues. Really interesting to observe. E.g. someone saying they couldn’t progress today, so could progress. Burnup, aim at 100 points of work over 2 weeks, draw a graph.

So decided to do this approach for OER Research Hub team. Deliverable -> Spring backlog. Meetings -> Scrum.  Pretended every 20 minutes was a day – amazing how much progress they made. Built a shared, quickly-built collection of what they knew about it.

(Note to self: tell Patrick about Trello if he hasn’t seen it.)

Finding ways to find pointers, directions for the work we do. Being a researcher is changing. It’s becoming a more public job. Martin has good talks on digital scholarship. There’s an element of how we change working practice. Not just an individual rebel, but changing as a group, as an organisation.

Questions

Martin: As someone there, it was really interesting doing the evidence dump. Once you start, you think, could use it over here. Could produce a course in two weeks like this. Doesn’t work like this all the time. Longer-term stuff as well. Mix up with sprints.

That came out. Sprints tire you out. Occasional sprint, or whether it’s a permanent change in methodology – which it is for the software world.

Mike: I’ve seen advantages and disadvantages. Advantage, moves quickly, everyone up to speed, as new people come in; very structured, not just random group of anarchists. Difficulty is about user engagement. The problem with FutureLearn is that they’re still developing things, but not trying them out with users, being rather secretive, so not really feeding in from real users yet. So the user engagement isn’t working the way it should. Doing it for research, how do we do it? Users don’t always know what they want, have to inform with deep thinking.

If you look at approach, each sprint should have a product, delivered to the people who need it. FL is treating you and a few others as if we’re the people who need this, but we’re not. Haven’t made these public outputs. We have a blog about it. Need to do the last bit. This will give us something that can feed on collective input. An open sprint, do it as an open process. We have 11 hypotheses, we’ve worked on 1, need to work on others- ask world for input. A divisible process.

Tim: My experience is that in general there are two ways of doing things. Very slowly and carefully, the OU is famous for courses taken 2-3y to do properly. Or you can do it very very fast. I can think of things that people tried to change at the OU that still haven’t changed, similar at Edinburgh, complex network of stakeholders. Once in a while – e.g. in our case we did the MOOCs very fast. The six MOOC teams turned in to one, with a very high level of trust. If don’t have community, trust, it’s immensely difficult to do things at a fast lick – get lots of ‘just a minute, need to consult, has this committee discussed it, what does the pro-vice-chancellor think’. Do things either take a fortnight or two years?

We can’t afford to always live in the two years.

Tim: Most things at OU took 2-3y.

Josie: Intriguing thing is what’s the tipping point – for some things, like CBM, emergence of learning design as a method was the trigger. Not sure about Student Support Teams.

Tim: For us the tipping point was if we want to be first.

A move towards the more rapid approach. Actually the tools have changed. Tools for working in public are definitely there.  You end up with an impossible list of people to check before you do something. Case to do something in the right way, so people can check it while it’s working.

Tim: Trick is to say ‘it’s on the Senate wiki’.

Karen: Trust is really important, that’s when things can move quickly. Links in to something, you had mix of online an physical. Agile in software is very physical.

Almost a rebellion against teams in India, US, UK – more let’s get a team physically together in one place.

Karen: More and more we’re not physically in the same place.

The simple bit is, what did you do yesterday, what will you do today, share with a group of people – not including your line manager (!). (laughter)

Martin Weller: Surviving the Day of the MOOC

This is a Riff a GIF of David Kernohan’s Day of the MOOC poster for the Horror of the MOOCs assignment on #ds106.

#H817Open. Ran open course in Open Education – 7 weeks, informal, OpenLearn. Existing OU students taken out of comfy VLE to mix with informal learners. Had blog aggregator, was the best bit.

Loads of different tools. MailChimp to send out weekly email – this was the killer app. Cloudworks for badges. OpenLearn, Cloudworks, manual blog adding to aggregator.

cMOOCs, xMOOCs – H817Open was somewhere in the middle.

Wasn’t too big – 239 syndicated blogs, 911 posts, 4,689 new visitors. 49 prospectuses requested, 4 registrations on other OU courses.

Survey evaluation, just com ein this morning – completion rates are low for MOOCs, survey rates are low, so this is very small. Satisfaction 78%, MOOC average more like 74%. Peak age 45-54. Big peak of visits in the first week, tailed off over time.

Tech issues with Drupal talking to Moodle – 30s for page loads! Also not clear to enrol. Too much screen real estate used for other stuff, navigation not clear.  ”I was trying to make a monkey be a donkey.” Badges in Cloudworks was nice, but required multiple registrations. Had to add the blogs manually, hunting for RSS feeds – some are very well hidden.

Learner issues – didn’t know who to connect with, information overload. What’s the right metaphor? Like a teacher with 50 essays – no, dip in, think of it as a stream you dip in to. Put aside 30 minutes and do something.

What worked – mixing informal and formal learners. Blog aggregator. Weekly email – sending one per week, Downes is daily, that’s too much, adds to pressure; sets the tone for the MOOC. Allowing flexibility, can catch up. Fairly independent, activities don’t build on each other. People liked badges – even for simple things like just give me an answer that’s not trivial. Live sessions good too – not many turned up, but the ones that didn’t liked to know it’s there.

What I’d do differently? The platform – not what OpenLearn does weel. Twitter didn’t work so well, but Google Plus did; so do more work to connect people on Twitter. Advice on reciprocity/engaging. More live sessions. Limit blog options – Blogger, Tumblr, WordPress – because RSS discovery automatable. Update content, but not much, largely the same. More multimedia solutions – many text-based, but the multimedia ones (choose your own tool) went really well.

What I’ve learnt: MOOCs are hard work. MOOCs are scary. MOOCs are fun. I believe in open. There’s a song there. I like learners owning their own solutions, in their own blogs.

Finally .. you don’t get poems like this at the end of most courses.

Key things:

• Support. If you take the support out of teaching, of course you can do it cheaply. Peer 2 peer, community champions, paid moderators, platform based, encourage self-help, hashtag it.
• Scale. At <1000 can create community. At >1000, content overwhelming. How do people find each other? How do they cope with overload?
• Motivation. Get many different motivations – nosy, leisure, people turning up explicitly to moan: free trolls! Key achievements along the way; don’t want to damage people along the way. Highlight vital activities, pathways – don’t have to do everything, allow catchup. Hardline deadlines can work, but dropout. Encouraging messages, assessment and recognition – badges, certificates, etc.
• Identity. Is the community the identity? DS106 is. Is it the academic? Does it go beyond the MOOC? Explicitly experimental? Don’t be scared of being publicly shamed. Be upfront. How would you craft an opening email?

MOOCs didn’t come from nowhere. Weren’t invented in 2012. Interlinking map of e.g. Open Education, OER, Connectivist MOOCs.

Battle for openness – open is not the same as free. Coursera MOOCs are closed. Announcement that Coursera will be an on-campus elearning provider – blended learning, oh wow. Retreating from open – e.g. FutureLearn T&Cs fuss last week. If claim to be open, people will take you to task. What winds me up: MOOCs as a solution to ‘broken higher education‘. There are narratives excluding higher education from the process, whole Silicon Valley exclusive focus.

Many interesting questions raised by MOOCs – openness, flexibility, smaller achievements, bridging informal and formal learning, technologies, building more automatic support.

MOOCs are our friends so long as we can answer these questions. It’s a mistake to ignore them, but also a mistake to panic. Make it clear that supported learning worthwhile. Understand that this is a crucial time in what it means to be open. We’re at the forefront of that.

Questions

Tim: The first third of your talk maps my understanding of where Sian Bayne and colleagues are. They use Coursera but are doing a cMOOC. Were using OERs, no talking heads, particular story. Fabulous one of Sian’s students produced a love letter. Can see the emotion. That leads to an important question – what is the identity? If studying a number of MOOCs, what’s your identity? Haven’t got our heads round that. Qualitative work by Sian. Users have really powerful issues of identity. The Open University is not free, never was. Not open as to people with no money. (Actually I think we are so long as you are in the UK.) But OER is free. VCs putting in 8-figure sums, this is not philanthropy. How does open relate to money? Maybe FutureLearn is a huge act of philanthropy.

John Pettit: That poem, reminded me of tutoring early MAODE module here. 1999. One student, at the end – was a high intensity module, lots of forums. Student left a song about her sadness at leaving. Doesn’t matter whether it’s a MOOC or a more open environment – the innovation, encouraged creativity. I did your MOOC for a few weeks but dropped out because life took over. I’d like a badge for dropping out, I did learn something, it wasn’t a wasted experience even though not picked up in any of the surveys. I entirely agree about the platform, would link that to the metaphor. Thought it was a course where you’re supposed to do stuff.

Eileen Scanlon: Informal learning and science communication

Hope that conversations will go on through the conference. Expected talks would overrun, so planned a shorter talk. Full talk at ICALT2012, on ORO.

Modelling problem-solving in physics, types of CAL that make a difference to science education. Then in a different world. Working to think through how the public might be engaged with science through technology. Few case studies of learning in informal settings. Reflections on issues, especially models of learning in use.

Engagement – participation in science, not the deficit model – ‘doesn’t mean turning everyone into a scientific expert, but enabling them to fulfil an enlightened role in making choices, which affect their environment’.

The oldies are the goodies – informal learning, complex area, hard to clarify – but e.g. Bruner 1966 Towards a theory of instruction. Get student ‘to take part in the process of knowledge-getting. Knowing is a process not a product.’ Also Bruner 1960 The process of education: interest is the best stimulus; aim to arouse interest.

Recent paper (Falk & Needham 2013, Factors contributing to adult knowledge of science and technology, J Res Sci T). Telephone survey, n=1018. People learn a little bit about science from all sorts of activities. Self-report. Informal experiences account for 39% of total variance – the plurality. Interactions between all, so suggests synergistic effects.

Groups that engaged with inquiry-based games at the Exploratorium significantly outperformed control groups in the quality and duration of several inquiry skills.

Different groups/modes of engagement. Indifferent. Concerned. (and two more).

Bevan 2011, four-phase model of interest – how interest developed.

Two brief things. Reporting contested science. Rick and I got some money looking at reporting of contested science. Example of being agile and sideways thinking in research. We were interested in what impact new findings reports would have on general population. One that everyone remembers – finger length and sexuality.

Looked at representations of contested science, exploring media influence. Bruno Latour – ‘ready made science’ vs ‘science in the making’. Stuff that’s already known, vs stuff that’s changing.

Worked with prompt material, small groups, write tabloid articles. Nice data showing the process of how these news products were developed. More interestingly, have the actual artefacts produced. They were quite creative. Creative artefacts can be very powerful, often overlooked, give us a handle on understanding learning.

When asked people where they got ideas – said they hadn’t seen anything on TV or newspapers, but when you had the artefact, you could trace it back.

Incidental learning – MASELTOV. Project looking at recent immigrants to the EU. Have an extremely complex way to map this – an Incidental Learning Framework. Informal learning of this type is massively complex to examine. Look at features of the model – the place, the tasks, the tools, the social support, the learning outcomes, the time.

How to measure informal learning is an unsolved problem, not least because of the complexity. We could work quite hard on thinking more broadly about evaluation activities around MOOCs. What counts as a learning episode? They learned something sometime that we managed to dip in to. When do you have to start measuring? What’s the baseline? What counts as learning? If saw examples in artefacts – e.g. things they’d picked up about modern genetics, but also about tabloid journalism. What counts as success? How good does a service trying to help you have to be? E.g. achieve proximal goal (e.g. catch a train), or more broad ones about increasing integration. Also hard to attribute causality. Our users will have access to many other sources.

Questions

Tony: A zillion! (Only allowed one.) Fascinated by MASELTOV. Who decides what people need to learn, how do they make that decision?

From two sides. Has to be the user of the service that decides. Has to be pull rather than a push. Could have a system that sends you everything it knows about train transport when you walk in to a station. But it has to be with the user of the service.

Tony: Massive implications. Who decides which bits of science the public want to learn about?

Interesting privileged position we’re in in universities. Who decides what we should research too? Talk more outside the room.

Patrick: All hooks up with the MOOCs thing. I’ve done three. I’ve ended up studying things because they’re there. Doing Coursera machine learning, with Doug. MOOCs are good at professional development. We’d like them to do something else. Careful not to miss what they’re good at – offering interesting challenges to people who want interesting challenges. But not what the OU does to develop people as learners. MOOCs are getting me back to programming, maths, rather than TV. Cross to social learning is a real challenge. A menu of things you might like to do is what it’s quite good at.

–

This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.

Stumpled Upon (cc) by Ian Sane, on Flickr

Eileen Scanlon: Opening Remarks

The new Regius Professor of Open Education, Eileen Scanlon, welcomes everyone to this, the 34th CALRG meeting. Thanks Canan for organising. To celebrate the award of the Regius Chair, to the OU, to IET, and to herself, the programme focuses on open education.

Tim O’Shea: Flourishing after Forty Three years in the wilderness

Professor Sir Timothy O’Shea, Principal & Vice Chancellor, The University of Edinburgh. Founder of the CAL group in the early 70s.

Everyone’s thrilled about the Regius Chair. Started PhD 43 years ago, in computer-assisted learning. Educated in Essex – Royal Liberty, Sussex (AI), Leeds (PhD Comp Sci), Texas (practical work), Edinburgh (4y postdoc), OU, PARC, OU, Birkbeck, Edinburgh; 19y at OU; Edinburgh 15y.

Machine code – 1964 – if everybody was forced to understand machine code, we would have order and understanding. For A level (experimental) wrote a maze learning program in Algol 60. CBLU at Leeds. Self-improving systems 1970 – discovery method for learning quadratic equations. Learning with Logo 1974. 1976 modelling subtraction errors. 1978 OU CAL Group. Advises (tongue in cheek) people to do as he did and listen carefully to the director of IET and then do otherwise. 1979 Micros for Managers. 1982 Micros for Schools. 1986 Smalltalk at PARC. Series of failures – We’ve failed to get everyone to learn to program in Logo, in Smalltalk, in BASIC. 1989 Shared Alternate Reality kit. 1997 Crystallography at Birkbeck. Good stuff there in geography and others, and crystallography -open source, RasMol. 2002 Edinburgh eLearning. 2007 Online Masters. 2012 MOOCs at Stanford.

False starts – intelligent tutoring systems, student models, natural language understanding, programming for all (maybe we’ll go there with Scratch – Uruguay is the only country that’s implemented one laptop per child – exploring all of them how to learn Scratch in Spanish, currently looking simpler than thought.), CYCLOPS (OU system – remote screen drawing, annotation), video tunnels (nicer than Skype! – but big wooden box; less high quality but ubiquitous now), touch screens (in 1974, had big boxes with touch screens). All prescient, but within universities we didn’t have the capability to get them out. Speech input, UK eUniversity, NCET (then BECTA) – good idea but abolished, NDPCAL, UK Subject Centres, International eUniversities (many started but failed), 50 IETs (originally every university would have one – Surrey, OU, King’s, CBLU at Leeds, sort-of at Heriot-Watt; now just the one left. At the minimum you can be the museum (!)).

Reasons for failure: too early, tech not tobust, to hard to ‘solve’ AI, platform dependent, person dependent, assessment unresolved (big permanent issue), funding inadequate. Ministers get excited, say here’s some millions of pounds, usually a factor of 10 too little for the goal – same applies for MOOCs.

Three case studies – innovation is technology driven’ austerity is a poor driver. Learner enhancement good. Tech and scale my focus. MOOCs aren’t going to save us a lot of money.

Micros for Managers

It was nice, Intel 8049 simulating itself. 25,000 users. First OU home computer kit, went out 1979.

Micros for Schools

OU 1982, 10,000s users. 5 microcomputer platforms, simple languages which turned out ephemeral. OU Home Computing Policy started in 1987 – students weren’t required to have them but could use them to study.

Making journals available – JISC

Helping SOAS to student enrolment.

Edinburgh –

Principal’s eLearning Fund - FY 2003-8, $6.3m, 64 projects from £20k to £150k, selection. Then investment in online Masters – building on the capability. £4.5m investment over 5y; all 22 Schools having 100 person online masters. Most successful one in surgery. Also equine health, law, elearning. Currently 31k conventional; aim 10k students online PG.

Now in to MOOCs.

Arrived in Stanford the day after MOOCs hit the news, signed up with Coursera. Went for very different courses, six. Stopped people who wanted to build their own platform – some not pleased. Edinburgh has the capability to build one but we didn’t.

Why? Reputation as early adopter of educational technology, has been fabulous. Exploration of new space to inform practice. Wish to reach as widely as we can with our courses. 100k started on philosophy course! Many sixth-formers, found ourselves doing widening participation work. Sharing experiences with peer universities. Fun! Very positive at Edinburgh Senate meeting about MOOCs.

Who? Limited data. Mainly 18-35yo, most >83% (?) have a degree qual already. 2/3 not US/UK.

Next steps – offer credit, accept credit, partnerships. Integrating MOOCs in to core first degree curriculum. We’ll be doing a lot more, looking at conversion. Equine Nutrition – that’s the baby MOOC, only 23.5k started, but has best retention and completion of any Coursera MOOC. They’re serious. If 1% go on to PG course, that’s lots, even 0.5% would double the enrolment.

For us reputation enhancement strong. Not so for Georgia Tech’s one. High risk game. Few times in my career as a manager I went leading edge; normally I say let 25 others do it first. Not free to produce (academic staff costs mainly) but not expensive. Spent £200,000 to get 308,000 hot sales leads, that’s a good deal.

Trans-national education in the UK – why not overseas campuses? 3% is done by overseas campuses. Online or distance 28% (mostly OU), validation of local programmes 51%, other (joint degrees, flying academics (18%). Only growing one is online or distance.

Cheery bit: New dawn. Online postgrad courses, partnerships. MOOC momentum, tech is cheap, ICT convergence complete, hybrid normal (online/f2f), platform independence (if you don’t, you’re nuts), statistical AI. When I did machine learning it was all symbolics, now all Bayes’ Law – Bayes was an Edinburgh alumnus. We can do it now because we have the numbers. More recently saw learning analytics using Bayes’ Law on MOOCs. Listening to Roy Pea talking about the big project – quote from my PhD thesis, which was impossible in the 70s. With 10,000s of learners it’s possible, can have statistical student models.

Technology is a driver, Moore’s Law still true. Off to CERN tomorrow for next big collider launch. Good for another 8-9 years. Metcalfe’s Law still untrue – group-forming networks scale 2^N; important that it’s not the case or the Internet would fall over. Cloud 40 years late but works. OU, Jisc and MOOCs scale. OU and JIsc sustainable; perhaps MOOCs are sustainable. Radical change is possible.

Acknowledges Jeff Haywood and Sian Bayne.

Questions

John: Tell us more about credits and MOOCs – is it Edinburgh M-level credit?

That’s what we’re thinking about. Our MOOCs are heterogeneous. If you look at it from our point of view, we offer 600 different undergraduate degrees. We are the most comprehensive university in teaching terms apart from Birmingham. Scottish 4y degree, online, with routes, is complex. An online masters in AI, equine health, it’s easier. There are aggregators – people taking stuff from our MOOC, others, aggregating for larger course targeted at African universities. My supposition is we’ll start doing it in small amounts at the PG level. We do a ladder – Cert, Dip, M – so do e.g. 1/4 of a certificate. They’ll still pay for the other 11/12ths. We will beef up the assessment side. Coursera doing good stuff on authenticating learners. We’ll do it piecemeal. For Edinburgh, professionally oriented PG courses. Online masters in Forensic Science, Palliative Care, with international partners. So used to give credit, accept credit. Assessment is the big issue.

Patrick: The world is 43 y behind you. What that you were working on, do you wish would still happen?

It hasn’t all happened. Have real emotional bond for young people programming. Was part of my own development, at 14, it was just so exciting. Woke up at 14 about thinking about setting registers. Incredibly exciting, most exciting thing between 14 and 18. Firmly believe that with IT/comms convergence, to understand modern world, and don’t program, it’s very hard. The Seymour Papert project of learning programming … We failed in Logo. Smalltalk. Basic. Don’t know if there’s enough in Scratch. I’d see that as helping young people. Gives more control to young people. Program or be programmed. That’s the big one.

Leigh-Anne: Noticed that bridge building charity launched a MOOC with EdX. Saw any future in MOOCs as a partnership between universities and NGOs?

It’s happening. Difficulty is philanthropist says everyone needs to understand this issue, here’s half a million pounds, produce the MOOC that enables them to understand. It’s risky. It is happening. The MOOC providers are banging their head on the wall. Issue with funders. Real issue – who do you take the money from. People say here’s 20,000 pounds, get everyone to understand ?diet. Will cost more than that. Easy to say give us the money and we’ll sort you out. Nervous about history on people working to get computers to ameliorate autism, dyslexia – but didn’t. A moral issue there.

Ann Jones: Open digital resources for informal language learners: a case study of Welsh learners’ practices

I’ve taken Tim’s suggestion – I’ve ignored issue about logos and attributions. Tim has been talking big, I’m talking about work in progress that’s very small.

Partly inspired by talk in CREET, Clare Crouch, founding director of Berkeley language centre. Heritage language learning – spoken by previous generations of your family, but less so now. Motivation different, a connectedness, part of your roots. Project focused on heritage/minority language learning in the UK.

Another line – personal history. First language in English, grew up in most Welsh-speaking town in Wales; mother didn’t speak Welsh, didn’t speak it at home. Went to school at 4 with no Welsh – and it was a Welsh-speaking school! Negotiated talking in the street. Never taught grammar. In secondary school, got red marks on Welsh work, never got on with teacher. When I realised I’d lost my Welsh, one reason was to get back at that teacher to show her. After this talk I’m off to do an exam in Welsh, just to show her.

Clip of minority language in action – Tura Arutura – appropriating tools. Met his father at a seminar in SOAS, who said ‘my son raps in Irish’. And indeed he does.

Welsh – spoken by about 20% of population in Wales (562k speakers); shock in 2012 Census was that it’s going down, not up, as was thought. Emerged from Brythoneg in C6th, common precursor of Celtic languages. Minority language by 1911. 8% in Cardiff, 42% in Aberystwyth, 88% in Caernarfon. Small speakers, uneven distribution – ideal for web2.

Intersection of literatures – work on minority and heritage languages, language learning and OER, Web 2 in language learning, mobile devices. Relatively little on learner’s informal practices. ‘cultural nerve gas’, Krauss 1992. David Crystal – An endangered language will progress if its speakers can [use online]. Cunliffe reckons it can help.

Crowdsourced Welsh Facebook, now in use. Study exploring extent to which it was normalised. Looked at profiles, social spaces, groups. Found bilingual, Welsh-only. 400 groups (some small, some 13k members). Pretty much in use.

Mobile devices increasingly important role in the learner’s toolkit of resources. Use what they have.

Focus on use of digital techs to support learning Welsh informally – interview small N (10 so far) by phone, short survey. 8 live in England, about half have a connection with Wales.

Early ethnographic study – Prosser 1986 – ethnographic study learning Welsh to see process and community, acquiring a new identity. Motivation very different than for non-minority language. There were some monolingual people in 1970s, but not now.

Some surprising motivations. One went to the Eisteddfod, spoken to in Welsh, wanted to be able to respond. Another, I did not want to be a monoglot English speaker and lived close to Wales. ‘I wanted to learn another language, so picked Welsh, because I was close’ – now lives in Derbyshire and drives to where they can pick up Welsh radio. More often – Welsh fading out in family history, stops being spoken in the household. In South Wales, parents would say, we’re not speaking Welsh, English is better for your education – sometimes a sudden, instant stop.  Another had many Welsh links, started look around, and then found there were Welsh lessons available in Basingstoke (!) so went for it.

Google translate – Gwgl (Welsh search engine, doesn’t appear to be official Google property). Email, text, Facebook, Twitter, blogs, wikis. Many other resources – Memrise, SSIW (Say Something In Welsh – first lessons free, then paid) – meetings, S4C, CDs, Big Welsh Challenge, BBC Wales, new one Y Bont (the bridge).

TV programmes on S4C and Clic (=iPlayer). Youtube, Radio Cymru. Facebook used a lot to connect learners in England – ‘adopt a learner’. Others. Blogs.

Questions

Rebecca: Struck at mention of crowdsourced. I worked with a group doing listings for S4C and Radio Cymru. Excitement in 80s when Welsh atlas came out, had been no reference source for that before. Is there scope for big crowdsourcing projects?

Only one I’ve bumped in to is looking at what happened with Facebook site. Active group of techie Welsh people. Bit of a barrier if learned as a second language, terminology is recent. Worth having a look at.

Martin: Interesting analogy with tensions in Wales. Interviews with people in villages – people move in, do we make them speak Welsh? Welsh-only makes a silo. Or speak Welsh in the ordinary Facebook.

Both. Some people choose English because they want to reach out. A tension.

Anne Adams: Different use of video. Creatively used.

Tim: OU Catalonia, teaches only in medium of Catalan. Could imagine an OU of Wales that’s only in Welsh?

In some, it’s smaller, the inner bit, that will be your experience if you do a Welsh degree. Can I imagine that? Probably not. Catalan is different, quite a lot of speakers, though is still minority – minority doesn’t necessarily mean small.

Mike Sharples: Massive open social learning: a Pedagogy for FutureLearn

(cc) peasap, on Flickr

When Tim was talking about notable CAL failures – Cyclops, Logo – my past flashed before my eyes. You wait 30y for a major elearning success, and then suddenly they all come at once, which is great. We have the Open Science Lab, and we have FutureLearn.

The reason for the pedagogy – hope you all know about Futurelearn, MOOC platform set up by OU with 20 othter universities, BL, British Museum. Decided couldn’t compete on technology, but possibly we could on pedagogy. With people from OU, BBC, partners. Pedagogy very important.

Thanks Rebecca Ferguson, Russell Beale, Matt Walton for ideas and slides.

FutureLearn vision statement – which gives you pedagogy – open to all, social learning-through-doing. How do you enable social constructivist learning for the Futurelearn platform?

Long list of different ways that could happen – direct learning from others; knowledge sharing, vicarious learning, implicit learning, conversational learning, orchestrated collaboration, shared knowledge building, Zone of Proximal Development (ZPD). From this menu – which do we choose, how do we enable that at a distance, at scale (currently on iTunesU at 100,000s) – and all done in two weeks. That’s the great thing – in past, have 3y project, final report proposing some innovation. But here have to do it in 2 weeks because there’s a bunch of people in a basement in Camden who are waiting to implement what you say. New iteration of platform every two weeks. Has to embed excellent pedagogy. Exciting opportunity! Having said pedagogy needs to inform design for years.

How?

First challenge is the challenge of massive – the OU still largest user of Moodle. FL development team considered taking Moodle as platform. After weeks of debate, decided no. 1, it’s too monolithic, around centralised course provision module, doesn’t fit with eclective, modular approach. Also, 2, it just doesn’t scale – at 200,000 it’s badly struggling. If you have a forum, new learner comes in, too much flashing in front of their eyes.

One approach – link conversations to content. All notes and questions are directly linked to content. Content includes a time-sequenced part of a video – e.g. ask a question at a specific point in a video. So don’t see a generic forum, see notes and questions directly linked to the content.

Another approach – personal activity feed. Take that contextualised information. Issue there in seeing in general what’s happening, want to keep up to date. Each learner has their own feed of content. Don’t want it flashing, scrolling every half second, need to prioritise it.

So then organising the social learning – three ways: Following (other learners; Twitter model), cohorts (large groups, assisted by mentor, doing course together), activity groups (small groups in shared activity for a short time).

Following – new learner follows pre-existing friends, people who seem interesting. Activity stream prioritises posts from people you follow. Learner sees people who are following, can follow back. Links persist after and across courses.

Example – see someone, decided to follow them. Then can see either everyone stream, or people-you’re-following stream. Possible relationships – inspiration, study buddy, tutor, mentor. Can be paid e.g. tutor.

(All the ‘person’ pictures have dots for eyes and a round O for a mouth – look surprised, or vacant/stupid!)

[NB This next bit about cohorts of 200 is what they decided not to do!]

Within a course, learners allocated to groups of about 200, last for the duration. But in cohorts where activity has dropped off (e.g. to <80), they are collapsed together.  Can associate mentors with courses. Mentor can see top-rated activity from all cohorts, copy posts across cohorts. Can see everyone in your cohort. Can follow people, see selected posts from other cohorts.

Problems – this relies on mentors to spread ideas across cohorts. Possibly complicated ‘collapsing’ mechanism. Some cohorts may have more activity than others.

So, for next iteration – cohorts 2.0 – changed the idea. Take social learning and apply to large groups. Within a course, tag learners by interest, or by mentor, to form ‘cohorts 2.0′. Or perhaps tag automatically (?). Learners and Mentor see prioritised activities of their cohort. Learners see prioritised messages from their Mentor. Everyone sees other messages but lower down the activity stream.

(Note to self: Mockups using Lorem ipsum text can be pretty misleading when the point is to direct your attention to the content! Hard to make sense of it, very hard to understand how the experience will be for the real situation. Very very hard to show someone having a different experience of the same interface thanks to filtering etc.)

Third strategy – activity groups. Short-lived, form together to do something. E.g. general discussion, or e.g. a jigsaw learning activity, or a peer assessment. Form in to small groups for particular activities. Based on who’s active at the time – e.g. at 10pm on a Tuesday night, form them in to a small group to do that activity. Learners allocated to different groups for the next activity. Randomly, or by ability.

Three mechanisms – bottom up following, top-down cohorts, activity groups – basic approach to doing different social learning and collaboration.

In the first version at least, everything is public.

That’s the first basis for it. We think it’s going to scale. That’s what the people in the basement in Camden are beavering away at. They’re used to building large scale applications. As long as the mechanism is scalable, we have people who can implement it.

And one more thing … (shows some screenshots of the platform as it was last night)

Liz: Why 200 for the cohort? Dunbar’s number? (Actually 150.)

That was our first thought. About the right sort of people you could identify with. But we decided not to go with that. If spread across students, some will be less active, some more, and only see inside that group unless a mentor decides to percolate material across. Want benefits of grouping but without rigid segmentation

Karen Kear: Sounds complicated but interesting way of dealing with people. Highly dependent on behaviour – e.g. rating things – IME students not that great at doing it. And assigning people to small groups, if some don’t participate. Lots of potential problems.

Rating – assumption is that people will learn vicariously, see things associated with content. If have large numbers of students, there will be some activity with all content. No assumption you’ll be an active contributor, but see it, but then become more confident and make contributions, aligned to bits of content. Helps people who are shy initially.

Karen: Not being shy, it’s rating.

If you’ve got massive, you only need 1%, or 0.1%, to get useful ratings. We’re getting value from the massive. The small groups, organising people – it’s still a problem, we have to solve. We’re wondering if there’s something different to do there. If organise on the fly, they’re online at the same time, currently engaged in learning – not allocated at the start of the week. But then have to ensure they’re doing something, or you’ll sit there rather folornly.

Tim: Part of the folklore of the OU was that it was specially suited to the shy or misanthropic. Making strong assumption that all will turn in to social, happy bunnies. Or long term space for misanthropes?

Absolutely space for misanthropes. It’ll depend on the course. Astronomy and Moons, maybe not heavy social engagement. But will see things happening. If on creative writing, expectation you’ll be a bit more creative and that you’ll write. More expectation of learner contribution. That’s what we’ve got to organise. Not assumption that all learners on all MOOCs will be engaged – but make it possible.

Tony: Fascinating. OU also has history of abandoning relationships. DO a 9 month module, all those relationships chucked out. Incredibly irritating, against social pedagogy. Are you assuming only one unit on FutureLearn, or will relationships continue?

Absolutely will continue. Want relationship with Futurelearn. Will have a persistent, long-term FL identity. Following will persist across courses. All material searchable from outside (!!!). Relationship with the concept, rather than a particular course.

Someone: (I couldn’t hear)

Very good question, don’t have a good answer. A lot of these things, want to build in the opportunity – e.g. for tagging, can say I’m interested in this, this and this. And use that to prioritise the activities. How that works in practice for a particular course and particular learners is to be determined. What happens if they shift interests, or change, add new, want to support that. Once have mechanism in the platform, that’s where the learning design comes  in. When have the possibilities, need to think about how you design the learning to make use of these opportunities.

–

This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.

Most of the time I work on Macs, because one of the things I particularly like about them is that they’re very shiny, visual and easy to use on top … but underneath it’s basically  POSIX-compliant BSD Unix, so you can do the full-on GNU/Linux command line thing properly if that’s the best way to get a job done. Historically, one of the big drawbacks of Unix-related stuff – and legendarily GNU – was that getting hold of a new bit of software and making it work on your machine could be a bit … involved. Especially when to get X working you had to install Y and Z, and Z required P and Q, which both require Y, and so on. Never mind keeping it all up to date. But modern package managers do a great job of making it much, much simpler.

Alas, the Mac doesn’t come with a package manager – Macs have their own (generally much more user-friendly) way of installing software, which doesn’t work for GNU/Linux software. Happily, there are several ways of dealing with this.

cc licensed ( BY ) flickr photo shared by Steve Evans

This post is a techie log of what I did to make a Mac work a bit more like a GNU/Linux machine at the command line, for my own reference (I’m expecting to have to do this again soonish), and for anyone running in to the same problems I did.

My proximal goal was getting Image Magick (for some image tweaking) and Octave (for a Coursera Machine Learning MOOC) working, and several other useful bits and pieces, on an iMac running Mac OS X 10.7.5 Lion.

There are three main choices for package managers on a Mac: MacPorts, Fink and Homebrew. I’ve previously used MacPorts but had problems here and there that were too much bother to fix. But based on this useful StackExchange post on the pros/cons of MacPorts, Fink and Homebrew, some personal recommendations, and realising that the whole ‘it all (looks like it) lives in /usr/local’ approach made sense in my muddled old-school head in a way the other approaches hadn’t, and very much liking to minimise the use of sudo and root on principle, I decided to go with Homebrew.

Before doing any of that, I installed XCode, which you need for doing anything involving compilation on a Mac. Easiest way (for me) was the free install from the App Store. Once installed, start XCode, go to Preferences, then the Components tab, and install ‘Command Line Tools’. C was my first proper programming language (which you can probably tell if you ever catch sight of my code in any other language I try to write in), so I banged out a quick hello.c (with cat, so I could feel properly old school) and make hello and it worked fine.

Then it was time for Homebrew. The simple install:

ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)"

didn’t work for me – I got:

curl: (7) couldn't connect to host

I guessed this was a proxy problem for curl (I’m behind one on this machine), so I created a .curlrc in my home directory (~) with the single line:

proxy=http://my.proxy.url.com:80

Then the install worked fine. Checking with

brew doctor

gave me this warning:

Warning: You have a curlrc file
 If you have trouble downloading packages with Homebrew, then maybe this
 is the problem? If the following command doesn't work, then try removing
 your curlrc:
 curl http://github.com

Well, for me not having a curlrc gave me problems downloading packages, so never mind that. (And trying curl http://github.com worked fine and showed me the 301 Moved Permanently error which points you over to the https version.)

Installing ImageMagick then worked perfectly well with a simple:

brew install imagemagick

… which was very nice, and just the thing I wanted to be able to do. But trying Octave like this:

brew install octave

failed with:

Error: No available formula for octave

There were actually two problems here. I tried a few other packages to install and got the same. So I tried

brew update

and got

Initialized empty Git repository in /usr/local/.git/
error: Failed connect to github.com:443; Connection refused while accessing https://github.com/mxcl/homebrew.git/info/refs?service=git-upload-pack
fatal: HTTP request failed
Error: Failure while executing: git fetch origin

Aha! I thought. Git is having proxy troubles too. That was the first problem. So I did:

git config --global http.proxy http://my.proxy.url.com:80

and then ‘brew update’ worked fine, doing plenty of work.

For Octave, following this article (https://www.nesono.com/node/356) I did

brew install fltk

(I tried ‘brew install –HEAD fltk’ first but got ‘Error: No head is defined for fltk’ – which I fear may lead to trouble down the line.)

and then

brew install --with-magick-plus-plus GraphicsMagick

which both worked fine. In retrospect I might’ve got away without these two steps. The second problem is that Octave lives in the homebrew/science set of formulae, not the defaults, so I then did

brew tap homebrew/science

to include the homebrew/science library, then

brew install homebrew/science/octave

which did fine, merrily downloading and building things, until it complained that it needed a Fortran compiler. A Fortran compiler! In 2013! Actually that makes perfect sense, since some of the heavy-lifting numerical algorithms in Octave are written in Fortran. (And it started life as a project so engineering students could learn about doing numerical computation without having to learn Fortran.) Homebrew very helpfully suggested, and I went along with:

brew install gfortran

which worked fine, so I did

brew install homebrew/science/octave

again. Which then installed all the way without problems. Then I did

brew install gnuplot

… on the grounds that that it was probably needed for plotting. Octave started and simple sums seemed to work fine. But when I tried to produce a plot (from my course), I got:

gnuplot> set terminal aqua enhanced title "Figure 1" size 560 420  font "*,6"
                      ^
         line 0: unknown or ambiguous terminal type; type just 'set terminal' for a list

Another known problem with a known fix: I created a .octaverc in my home directory (~) with the single line:

setenv("GNUTERM","X11")

… ran Octave again, and all was well.

I’m not sure whether you need to install X11 manually or not these days (I suspect not) – you need it for Gimp, and I’d already installed that. There doesn’t seem to be an official Homebrew formula for Aquaterm (to get more Mac-native-looking plots), and Aquaterm seems quasi-abandoned since mid-2011, but it might be worth trying (maybe: uninstall gnuplot with brew uninstall gnuplot, install Aquaterm with the dmg, then brew install gnuplot again and remove the .octaverc diverting it to X11. But reading a bit of the Aquaterm mailing list about needing to recompile to get plot widths beyond 1000px wide suggested X11 will do fine for now.

Easy when you know how!

–

This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.

Analytic architectures

(cc) Scorpions and Centaurs on Flickr

From Micro to Macro – Analyzing Activity in the ROLE Sandbox

Dominik Renzel, Ralf Klamma

Dominik presenting. Motivation, increasing adoption of distributed learning services – PLE, hybrids. Responsive Open Learning Environments. Thriling that you can get an unprecedented insight in to learner behaviour on a really large scale. Some proprietary techniques are biased/have limitations, move towards standards.  But learner behaviour is evolving and inherently hard to model. Their approach to step back to proven standards – simple web logs. Because: standardised byproduct, wide adoption – every web server admin has these logs. Lowest possible degree of bias, low level protocol data. Generic low level but liftable to higher-level semantics – e.g. RESTful interfaces. Analysis on multiple ecosystem levels, from micro to macro. Also have highest level of data interoperability, allows cross-service analysis.

Focus and level of analysis from Ecological System Theory (Bronfenbrenner): student at the centre with 5 nested and interrelated ecosystems around – from microsystem, mesosystem, exosystem, macrosystem, chronosystem.  Existing work often focuses only on one specific system or subsystem. A comprehensive LA framework should allow analysis on all levels – not just learners, teachers but operators, politicians. May never neglect micro data, always apply aggregation careful – danger of washing out anomalies – and the anomalies are interesting and useful hints. It should work with historical and real time data so you can have targeted and timely intervention.

Data interoperability was a concern. Could create/use proprietary data ‘standards’ for monitoring learner activity in LA. Convenient analysis, but danger of losing anomalies. Standardisation processes are long, no guarantee of adoption – hard to combine data. Use of incompatible formats makes cross-service analysis complex. Why not build on existing web standards? Web logs and page tagging – profit from things that work from years on the web.

Example web log entry, from a widget-based PLE. Combine for a learning task, move the widgets, all covered by RESTful web services. Creates logs:

IP address (can get a geolocation roughly), date, operation, URLs (use the linking structures), can also get other things. Can pull in external data – like geolocation from IP address; from widget, can extract widget metadata. In ROLE they have widget metadata, which gives more insight of what learning is going on.

Real-time processing of web logs, simple shell scripts, process the data, clean it, pull in other data.

Analysis techniques. Single operations gives micro level. Careful aggregations can give you higher levels. Relations – IP address called URL, set up paths, get sequences of activities. Set up actor networks, people using tools, or tools used together – use SNA on people or arbitrary actors. Temporal information can be used for time series analysis. Semantic enrichment gives you muh more. Benefits to learning stakeholders (broadly considered) – community awareness by visualisation, recommendations of e.g. tools, people, resources.

Main argument: don’t neglect web logs! They have limitations but always available without further instrumentation. Simple to set up. Multiple techniques possible.

Next steps: More serious analysis on ROLE Sandbox dataset. >8m API requests from 3000 IPs, 500 widgets (!!), 1300 PLE spaces, 3700 shared resources. Transfer approach to other contexts.

Questions

Q Maybe I missed it, how do you identify the user, the session, from web logs.

That’s really a problem. With IP addresses, cannot do it precisely. Many people use the same machines every day. There are other techniques that up to this, e.g. page tagging used in web analytics, make the problem less serious. You cannot differentiate users. MOOCs – people can have different IDs, come in with one ID, try again with a different. Other approaches can fight this problem better.

Dan: In many of our logs, a single action shows up in 3 entries – bringing up the page, edit round, posting – have to aggregate in to meaningful actions. That’s where you do the work to move to a specific application. A level of representation that abstracts to something where the analysis is re-usable. Do you do that? Or analyse directly?

A little bit yes. The data is usable for many scenarios, you can lift from this data for different scenarios. RESTful web services have their own semantics, use HTTP to model higher level semantics. If have documentation for service, can model it.

Dan: If you write an analytics solution, can that be reused because there’s an abstraction layer from the web log?

We put that in the database, we do not do the analysis on the plain text from the log, we put it in a db schema. Use visualisation tools to query the db in a more convenient way.

Analytics of collaborative planning in Metafora – architecture, data, and analytic methods

Andreas Harrer

Also ‘tales of a ‘tooler’ about an interdisciplinary approach to reach the middle space’. ‘Confessions’ perhaps.

Co-designed with pedagogical partners a system to support collaborative learning. Build a system, report, reflect, feed back to hypotheses. Also developed analytics components they wanted to present to the students. Ongoing process.

PlaTO – an analytics agent for planning behaviour.

Pedagogical background of Metafora – self-regulated and collaborative learning. Secondary school, small groups (3-6). Complex learning challenges, extensive lessons – 20h. Planning as a key competence to learn to learn together (L2L2). System supports learners and teachers with intelligent analysis.

Physics microworld, constructivist approach, students manipulate their own model. Tool instances can interoperate – it’s a tool box, but a semantic level exposing this in to a discussion environment, so it can be explored by the learners and then come back to the tool.

Picture of the tool in use.

The main tool is the Planning Tool – learners self-organise activities using visual elements – shapes, cards that represent activities, attitudes (Critical, open, reflective), roles in organisation, and connectors. Learners can create and document their progress and status. Tool is a bit like LAMS or similar tools. Can click from card to the tool itself. Elements have state e.g. ‘finalise’ on the model when you’re finished.

Use wide range of web technologies, flexible architecture to extend system with AI components. Want analysis across several tools – do plans and actions fit together? So need comprehensible data formats on an abstraction level.

Analytic architecture diagram. The components don’t interact directly but via channels. PlaTO takes data from Planning Tool (via logging channel) and passes output on.

Range of indicators from PlaTO – create/delete plans, open/change maps, etc.

Dataset – 7 school challenges, in Israel, China, planned in Brazil. 1200 users, 550 groups, 1000 planning mapes, 279k user actions.

Don’t want to post feedback untested to the students. So first do observation procedure. Replay log files, expert sees activities by the user, like on a TV, can then get the analysis component feedback, see if it’s useful or not, which creates coding on the usefulness of these indicators.

What to do with these results? Connect to Alyssa’s talk. Small tools informing students and teachers – an embedded tool, folded away most of the time, teacher can invite students to fold it down and do some reflection. Inspiration for the design from visualisations summarising football games.

Dataset is available and research method.

Current activities – eye tracking study for effects of feedback messages. Contingency Analyser based on Suthers et al Contingency and Uptake analysis.

Questions

Dan: With both talks, to what extent are you offering a solution, an approach or platform for doing LA? Is it at the level of implementation you have, or at the architectural level? What should others take home?

Here mainly demand-driven by the pedagogical partners, interested in being aware of what’s going on. Influenced the definition of our indicators. Had good experience with legacy systems, also with newly developed tools. Now integrate tools semantically in to our architecture. Interesting to hook that up with other systems in the spirit of having an analysis toolbox or analytics app store.

Dan: That’d be the role of yet another standard?

This format pre-existed from a European Network of Excellence, it’s not very complicated. Some XML transformation are not that critical. Could be some interaction with web logs, we don’t use them at the moment.

Q More the contextualisation. Challenge: you may hear that it’s not really learning analytics it’s intelligently supported CSCL, 15 y ago. I’d say that constructively. We have to build our evolution. Open learner modelling for collaborative learning, connects in well with this and LAK in general.

I would follow you. I buy that with intelligently supported CSCL. Here we have a heterogeneous environment with specialised tools. Open learner modelling implicit – students get feedback from what the system collected about them. You can’t manipulate or correct the model, that might be interesting too. It’s not very skill-based or -oriented. It’s relatively unbiased still. If you start or finish and activity, or created an artifact. The loop inside the head is more important than the loop in the system.

Dan: To what extent are you building on the plan recognition literature?

Maybe the terms overlap. Don’t use regular techniques. We use workflow engineering techniques for analysis.

–

This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.

Affective analytics and SNA/vis

(cc) Jason A. Samfield on Flickr

Affective states and state tests: Investigating how affect throughout the school year predicts end of year learning outcomes.

Zachary Pardos, Ryan S.J.D. Baker, Maria O.C.Z. San Pedro, Sujith Gowda, Supreeth M. Gowda

Zach presenting.

What is the correspondance between students’ affect during the school year and their end of year state test scores. Affect – every second! – engagement, boredom, confusion, frustration; behaviour – off-task, gaming.

Predictions from room: Might assume that more engaged student would have a higher test score. Boredom – maybe, arguments both ways. Confusion and frustration similar. Off task and gaming probably less good. Smilies/emoticons to illustrate!

• O_O
• 
• ?_?
• >:o

Performance on the tutor and test outcomes has been investigated (Fend et al 2009), compared no feedback (just testing) and feedback – can predict just as well if you let the students learn, so maybe the state test isn’t so necessary. What part of the outcome story does affect tell?

What is this important? Affect is an important state that could tell the story of what interventions are right for a given state. E.g. if bored need condition A, if confused need condition B. Could give you a bigger effect size. If the VLE knew your affective state – e.g. engagement, arousal – could say e.g. you’re fading, take a break for 20 minutes and come back. Could be valuable signal for teachers, get info they can’t know all the time. Teachers are expert affect detectors, but they can’t look everywhere all the time. Why affect correlation with state tests? High-level: a lot of K-12 teachers, there’s a lot of teaching to the test. State standards, common core. A lot of concern there’s not enough focus at the policy level to help students in non-maths-skilled ways. Paulo Blikstein “We teach what we can measure”. If we can measure affect, and it’s a strong correlate of state tests, it’s Ok to teach to that.

Methodology: Measure during the whole year. Not observers in classroom the whole year. Code a sample of students with labels while using a tutor. Classroom field observations, human coded. Take those labels, find corresponding tutor log data, create features, then learn a function mapping from those features to the labels, then apply that to the entire dataset. Machine learn a function, apply to the out-of-sample students. Collect end of year state test scores, observe the correlation between raw affect and end of year scores. Two years.  Only measure during time using the tutoring system.

ASSISTments system. Primarily algebra and geometry. Two types of questions. Original questions – publically release state test items. Or scaffolded version – gives scaffolding that breaks question up in to steps (if asked for or student answers incorrectly). Content problem set types, while student is in the lab they get random items from the problem set. Also mastery and skill builder sets.

Coding the students. Two expert field observers chose classes, two schools, rural and urban. Code students one at a time for 20s intervals, and label the affective states. Inter-rater reliability tested – kappa of 0.72 for affect, 0.86 for behavior. 3075 observations of 229 students.

Baker at al EDM 2012 methodology. StudentID/Time/Label, use machine learning classifiers to map from tutor log features to the label. Stepwise regression, add features until it no longer improves fit, then take those features and include in the more complex classifiers. Example features – number of correct answers during clip, proportion of actions taking >80s to respond; whether student followed scaffolding with a hint request; how many of students’ previous five actions included same. Many others.

Gaming is trying to get the right answer without thinking – e.g. answering question wrong, getting sub-things wrong, until it tells you the answer. The teacher is alerted that they had the hint with the answer in.

Applied eight classifiers, with the idea that different classifiers might work better for each state. The average A’ was about 0.8, average kappa from 0.2 to 0.5 – chose on kappa.

Applied these tools to two whole years of ASSISTments log data – 639 students 2004/5, 764 2005/6 – a while ago because had state test scores available.

Goal to correlate affect with state test score but affect needs to first be summarised for each student. Average of each affective state calculated for originals and scaffolds, weighing affect in each skill equally. It’s not a simple average, it’s treating each skill (subtraction, addition, etc) equally. The state test samples skills equally, so more likely to see a correlation.

Results: Pearson correlations for affect vs original / scaffold problem types. None much higher than 0.4, many small. Engagement and gaming had strongest signals – engaged students had +ve correlation (0.44) – gaming had -ve correlation (about -0.44 for each). Boredom is negative if bored during original question, but positive is bored during scaffolding (0.3) – maybe you knew it and are bored of the help. Confusion – similar – if confused on the originals, that’s not good for state test correlation. But confused on the help, that’s not bad; in lit, confusion is good if it’s on items that resolve your confusion – which scaffolding is meant to be.

Plot summary of affect scores for different proficiency categories. Advanced to failing. Frustration scores are higher the more advanced you are. Confusion scores go down the more advanced you are – same with gaming.

Affect has strong correspondance to state test outcomes. Looked at difference between affect during the original and the scaffolding – e.g. confusion, boredom. Affect important in state test achievement.

Questions

Q: If student was frustrated and then solved the problem and moved in to happy, how was it coded if a state shift in the clip?

The first one was used.

Q Role of stepwise regression in the process? Stepwise is usually highly sensitive to outliers. Do any transformation or diagnosis?

Don’t think we did treatment of data for stepwise regression. In the methodology, was used for feature selection before the fivefold validation to choose best qualifiers. Could be improved by using a multi-state classifier, rather than separate for each. Might not be perfect – could still have different errors for each. Work to hone these models.

Q: Analysing the in-tutor time, take precautions to say don’t care about off-tutor? What was effect of out of tutor on final score?

We didn’t measure that, outside the tutor learning. Presumably there was lots. The tutor was once per week. It’s an open question how much affect in the modality of a VLE relates to in the classroom. It was a sampling of their whole time.

Phil: What was the operational definitions of the affects? How do you know if I am frustration?

Field observer: We used the BROMP protocol for affect coding. Holistic, not single signals, try to make a judgement based on posture, facial expressions, etc. You get double inter-rated reliability. People good at spotting someone’s bored, bad at saying why that is.

Chris: What about the classifier?

I don’t know!

SNA and visualisation

Considering Formal Assessment in Learning Analytics within a PLE: The HOU2LEARN Case.

Eleni Koulocheri, Michalis Xenos

Eleni talking. Hellenic OU environment – HOU2LEARN. SNA and activity metrics, looking at formal grades.

Focus on the course level and SNA. She outlines Social Network Analysis methods and terms – nodes, edges, betweenness centrality etc.

Used students – N=76 in MSc course PLH42 at HOU. Platform open to all. 6 assignments, 1 final exam. Aim to promote openness, to create and share content in the network. To communicate informally in a less stressed way, exchange experiences, content etc. To promote socialisation among members, endorsing ideas.

Front page of platform – has an activity stream prominently. Based on Elgg. From Sep 2010, research and educational purposes. It doesn’t support roles.

Learning 2.0 elements – group members, activities, blogs, networking. Profile interface – personal profile, user can create portfolio etc.

Looked at a range of SNA software – used NodeXL, Gephi, UCINET, TouchGraph, Graphviz, GraphiInsight. Linked to HOU2LEARN database schema.

Designed activity metrics (SQL queries) according to the course needs and set up – this is work in progress. E.g. topics each user has uploaded, new bookmarks, etc.

SNA visualisation from early on – many students entirely unconnected, and the most connected/largest node is the instructor. Then next shot after two months, the connections have increased, with still many students who ‘are introverts’ (!). But the instructor is no longer dominant in the network, there are several other nodes that are large too. Then in May, connections now around 400, can identify clusters among the users.

Then took in to consideration the final grades, with nodes size proportional to the final grades. The largest grade was not well connected (but not very badly). Then indegree centrality, popularity of the node – most popular node does not have the highest grades. Higher grade does not mean higher indegree centrality. Next – outdegree centrality – most outward pointing nodes, how many users you follow – the highest outdegree centrality had a pretty high grade. Finally betweenness centrality – brokering – top node was same as highest outdegree centrality, and higher grades.

Highest indegree centrality node had lower grade than average. Highest outdegree centrality had one of the highest grades and also had highest betweenness centrality. Students with high grades have to increase betweeness centrality.

Future work – run again, more development and combination, integrate experiments with groups.

Questions

Sheila: Did you share these slides back with the students, did they understand what action they could take?

Not yet. Want to consider anonymisation and ethics first. We work with their IDs, have to find methods to face, otherwise will feel they are monitored.

Doug: Did you do the correlations between the centrality measures and the final grades?

No, this is preliminary research.

Visualizing Social Learning Ties by Type and Topic: Rationale and Concept Demonstrator

Bieke Schreurs, Chris Teplovs, Rebecca Ferguson, Maarten De Laat, Simon Buckingham Shum

Bieke talking, from OUNL. Work is close to previous presenters. This place is special for me, I graduated 10 years ago in this auditorium, the ceremony was here. Was is a political and social scientist doing here? It’s the result of interdisciplinary collaboration.

Worked in EU projects on innovation in education, interested in CPD of teachers, 2y ago started PhD, moved to OUNL, supervisor Maarten de Laat. Question: How do teachers develop learning ties to develop in the workplace? Face to face.

Teachers fed up with questionnaires. Online reflective tool on reflective account of ties in the workplace. Tool helps us gather data, but direct reflection for them. Practice-based research. Linked up with IT person on visualisations, created the Network Awareness Tool presented last year. Want to visualise online interactions, link to Simon Buckingham Shum and Rebecca Ferguson, informal learning platform for learners and teachers – SocialLearn, plug in this tool and see what happens.

Social learning analytics are designed to support learning through social networks. Based on Networked Learning Theory. Interested in the structure of networks, position of people in a network adn the antecedents and consequences.  Also Social Capital Theory – looking at the content as well as the network structure.

Developed the NAT plug-in for SocialLearn. Multiple levels at once. Have theme cloud, the overall network structure. Then the network structure itself. Then the tie and its multiplexity – friends, followers, responding – for seeking advice, people tend to go to people they like rather than real experts. And ego networks – individual network relations per person.

Live demo!

Big network of entire platform. Nodes are people, ties are friends, followers and responding shown as yellow, pink, blue lines. Content represented as tag cloud. Can see some individuals unconnected, there’s a really popular person with lots of ties.

Smaller network – select from tag cloud, see that subnetwork. Can replot. Mouseover a node gives you list of topics, mouseover the tie, gives you list of topics shared. Can click through to an individual’s ego network.

Relfected with teachers on the tool, but haven’t reflected with learners. From experience with teachers, expect learners can see their own learning network. Many important roles in the network as well as being popular. Could be a block if you are very central. Have to have tutorial, guide to understand these visualisations.

Many directions for future research. Learners perceptions, does the content of ties influence the structure, semantic analysis on the tagcloud, dynamic analysis. Do students find more peers to learn from using the NAT plug-in? Potential tool for conferences to find co-researchers to do interdisciplinary research?

Screencast of NAT plugin in SocialLearn

Question

Q From network analysis, many things to pursue. Suggestions – looking for subcommunities, that would be very interesting here. Also if want to find different patterns, behavioural patterns, ? modelling.

Yes, definitely. This is for the teachers, want to offer for the users. But for us, more modelling is useful. Interesting from a research perspective. Find tacit, explicit knowledge, is it different structure.

Q Trying to get a community of PhD students, all sitting on an island. Would be able to use this tool to create communities?

Oh yeah, we’re using it in my institution also. It’s not related to an online community, but as a PhD student I say these are my interests, are displayed so that my colleagues can see and connect. It’s free to use.

–

This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.

Panel: Perspectives on Learning Analytics Research

(cc) wili_hybrid on Flickr

Paulo Blikstein, Stephanie Teasley and Alyssa Wise

Stephanie Teasley, U Michigan

New York Times articles – about a product called CourseSmart (“Teacher Knows If You’ve Done The E-Reading”, April 8). ‘The plan is to introduce the program broadly this fall’. If we don’t act, critique, improve – even though something just happened, we don’t know what it’s about, it’s being moved ahead.

Hyperbole or reality? Many terms and phrases to do with MOOCs, online learning – big data, game changer=personalised learning, disruptive innovation, etc. Administrators are taking the hyperbole seriously. It’s a call to arms.

At the NSF, message that the budget proposal had gone to Congress for an increase, had one area around cyberlearning and online education, inter-disciplinary, with enormous potential – and that’s up to us.

Quick thoughts about challenges. We are a new field. Only the third conference. Don’t fight old battles (quant v qual) or create artificial battefields (LA v EDM). Expect rigor – not another thing that gets hot but never realises the promise. Have rigor – not only by one standard, we’re interdisciplinary, being rigorous about the method we each work in. Identify which data is significant and more importantly be able to say why. We have a lot of data – but why do we care about it, and how can we take action? Aggregate knowledge across data types and datasets, take things that you know from your students and think about how we can put the datasets together. Many journals require data to make it available; issues with educational data. We should be able to figure out ways to deal with privacy to create some really large datasets.

Many many dimensions – picked out three. (Rubik cube visualisation.) But our goal isn’t only to talk to people like us.

We need to do it well.

Paulo Blikstein, Stanford

“We teach what we can measure.” If we don’t measure what we care about, it will never be taught. For the vast majority of K12 and a lot of HE.

I came in to LA because I wanted to teach things not easily measured by multiple choice tests. Key goal is to identify types of learning we care about – e.g. project-based learning. Extensive literature saying how, especially for kids who are not top of the class, having tangible materials, working in teams, is tremendously beneficial. These types of learning are more inclusive, help people learn better. Our responsibility to design analyses to show that those kinds of learning can be measured in an objective way. Open-ended learning – can you use different sensing, techniques, to produce dashboard to help assess that? So not just looking at what’s taught today and just optimising it and making it massive. This community could have a really huge impact. Overnight. Schools have huge incentives to adopt measurable programs.

The bias towards “cheap data”. Easy to collect, already available. We should work on data analysis, but also on new data collection techniques. Multimodal – video, biosensor. Careful not to be seduced by large easily captured datasets that might not have a big impact on changing education. Sometimes you have to spend a year designing a data collection technique, but it might have a huge impact in the future.

We need to push the system to embrace multimodality and the social aspects of learning. People get information in different ways, react to learning differently. Easy to collect data from people who are using a computer. But there’s more going on around – talking to other people, other interactions that we can’t completely ignore and imagine we’re picturing the entire learning experience of students.

Finally, there are big political and economic interests at play. The community is unique, we are in an interesting time, huge interest in education from companies and governments. When you see some unanimity you should be careful, everyone talking the same way about the same thing, I get nervous. Those interests might not be those of the students.

We need to be disruptors and public intellectuals. A lot of impactful things going on. Universities being dismantled for MOOCs. K12 systems being revamped to use blended or online learning. It’d be easier to say, we just look at data, we sit on our computers and run R scripts. That’s not what this community should be about. When we see the cliches, that are not research based, we should speak up.  We have a big opportunity and responsibility there. If you study emergent or complex systems, it’s not a linear thing.  If there’s an article in the NYT or CNN, it’s all suddenly true! Dismantling this smokescreen, do research – a big responsibilities for the EDM and LA communities.

Alyssa Wise, Simon Fraser University

The other two set up the context. I’ll do something a little different. Suggest a contentious and provocative vision that’s different from what we’re currently doing. My title would be “Do we really need another dashboard?”

Why am I not enamoured of them? There’s a place, they’re useful. But they’re separated out of the learning activity, they’re large and universal. Instead of big, overarching ones, we should have small systems tightly tied towards the learning system. I understand the attraction of scale, but maybe a different way to go.

It’s a space that’s moving fast. We have a dual challenge – being rigorous, quality thinking; but at the same time, at a pace that’s much quicker, it takes time to do this well. Trying to do something big we have to sacrifice some quality – not sure that’s a tradeoff to make. Smaller projects – targeting more specifically.

Like in my talk – embedded analytics, tightly tied to what the students are doing. The interface for the activity and the analytics are one and the same. We should be designing analytics-enhanced learning tools. A bigger demand, but a better one. Not building the one system, but a lot of systems that are good for the learning activity we’re focusing on. Can move more quickly providing an alternative to some of the large systems.

This is so important. We teach what we can measure. Our LA will drive our pedagogy. Think about consequential validity – what assumptions are embedded in our systems. Are people going to need to teach in particular ways? E.g. reducing incentive to change assignments.  Important value in diversity, and innovation.

We need to keep the human and the person at the centre of our focus. Data, algorithms, theory are important. But we are trying to help people learn. We shouldn’t be data miners, but data geologists. Or data archaeologists – we’re trying to tell a story. The easy-to-collect data isn’t capturing everything.

Many different methodologies we could deploy, but supporting human activity is very important. Agency – a lot of power involved in data. More excited about data used to empower people. Leave some power in their hands, some of the sensemaking.

Finally, about methods. Don’t play loose with theory or methods. Most people in this room people may be more excited about one or the other; we should be a little keener to play in the other half. Need to think of the hallmarks of quality LA. Some standards in methods. Looking at system as a whole, we don’t have the same things in mind.

Discussion

Q Inspired by your talk. Do you envision an app store – learning tools empowering the students. It’s still standardised and global but flexible and has small applications for enhancing your personal, human (context?)

Alyssa: That did occur to me. There could be an app store with diversity. I don’t know how much things are standalone pieces. Maybe there’s an analytics app store. There’s lots of different choices.

Q: We’re playing very safe here.  How many things blew your mind? We should be like children on an Easter Egg hunt, running around finding interesting knowledge. We should try to fail more. Let’s see if the heartbeat of the children has an effect on their learning. We should do more risky research.

Stephanie: I like it. Risk taking has to happen. Not sure I haven’t had my mind blown at all. Rather than play favourites, maybe other people would like to say, something that came close to blowing their mind, or something not particularly safe. The best paper here, that blew my mind.

George: I’m reminded of a book on religious experience – no much a big bright light as a slow increase in intensity. It’s been a progressive awakening. We have a dual role to play. We should ask provocative but practical questions. My experience is a progressive understanding of LA significance. Duty of care to honour the structure of learning.

Q: I come from the ed tech field. This is mind blowing to me. Learning about data collection and data analysis. Being able to add that in to the educational technology field that has been somewhat static. There are two areas that need to be developed within LA. One is information visualisation. Ways of representing and interacting with data, that’s an area where we need to move forward. Second, tremendous research in computer-based instruction. A lot is fairly old research, coming up again now. Ed tech has moved on a long way in terms of the pedagogy.

Greg: What blew my mind is that .. I’ve heard a lot of talks by computer scientists, there’s often a lot of handwaving and imaginary learners when you talk about education. It’s the extent to which everyone were grounded in theory about learning. That’s really exciting.

Stephanie: I love the concept of imaginary learners. Or extreme examples. As if they represent the general population.

Phil: I’m an educational psychologist. I’m also from SFU where Alyssa is. If you’re developing an analytic, a report, what should it change? I’ve seen presentations about new kinds of report, it’s unclear what would happen to a learner or teacher on being presented with it. Second, what cognitive or motivational or social process is reflected in the data that are analysed? Connected to the idea of cheap data. It’s fun to play in lots of data, lots of methodological affordances. Most are distant proxies for what you’re really interested in. Challenge to get closer to the cognitive, motivational, social processes. Every learner I’ve met – unless punished for it – underlines or highlights. They’re making a distinction to that information. What standards are they using in doing that? Do they promote learning? I don’t see enough of that. It’s an important challenge. Old garbage in, garbage out maxim is worrying.

Paulo: GIGO is a concern we should have. Great things you can do with numerical methods. But sometimes used to mystify what’s actually happening. Big methods in machine learning, people assume we know what we’re talking about. I’ve seen meaningless blind application of methods to data that’s not well known. It’s easy to be seduced by new tools, put stuff in RapidMiner and find lots of clusters, but what does it mean?

Alyssa: Often we start with methods and see what they can do with us. If we start with questions, the methods we use are those we know about. Want to know more about what kinds of methods help with what kinds of questions. Phil’s point about cognitive, social, motivational processes – some people do that, others come up with indicators that are predictive but may not have a meaning. We haven’t understood them – is that useful?

Phil: Discovery is a wonderful thing. We should pay attention to it. But need to dig in to it – what is it indicative of. Correlation doesn’t mean causation.

?Avi, Turkey: Mind blown yesterday by EDM presentation. His grasp of those parameters – talking about kappa like it’s his best friend. I liked that. I was working on signal processing through the 90s, all these techniques – classifiers and so on – we used in 70s and 80s in audio coding. Felt grasp of familiarity in that presentation. We should look at those parameters and see what they amount to. I’ve not seen intervention papers here. That’s critical, the way you use a piece of information makes all the difference. Whole field of behavioural psychology. Results of LA have to be coupled with behavioural psych.

Alyssa: Yes.

Someone in a startup in field of dyslexia: Driver’s seat, co-piloting – academic community and the entrepreneurial community. Numbers are staggering. Year ago first accelerator on learning tech in California. Since then, about 20 accelerators in US, London and elsewhere. So expect 400 to 1000 startups in education. This is a major accelerating force – maybe for good, or for bad. How should this community work with entrepreneurs? They are a fact of life, much more important than they have been until a year or two ago.

Paulo: In Silicon Valley, bump in to them all the time. They talk to us on campus. Everybody has the same idea – education is so efficient, why don’t we put them online. I’ve heard that 20 times this week. People need to think more out of the box, not just we’ll scale up X, other industries transformed by tech except education … I’ve heard that every other week. Entrepreneurs bring energy, speed to implementation that academia can’t offer. The relationship is a little weird, goes both ways. Entrepreneurs don’t want to listen to research that’ll crush their dreams. Hearing e.g. it’s been tried many times, not keen. The way entrepreneurs consume research should be different, maybe the collaborations should be more productive too. Problems the other way – academics saying how about a five-year study before we start. The numbers that you mentioned, those things are accelerating in a way I’ve not seen before. Adjusting how those collaborations happen is important. It’s also important to think about how we allow multiple forms of training to happen. Entrepreneurs want the basics of the field, not a PhD – there aren’t good ways for that professional development to happen. Same for teachers, principals. We don’t have good models right now.

Alyssa: We’re all concerned about what the future should look like. As entrepreneur more concerned about what it should look like. But tricky place for us to go, what we try to do here is uncontaminated by real world concerns, profitability. A tension we should be more willing to live with. Some of the companies, trying to help them do things a bit better even if it’s not how we would do it. Lot of energy and motivation in terms of impact. I’m concerned not with what should happen but what will.

Chris: At LAK11 and LAK12, mind blew by Course Signals, SNAPP from Shane Dawson’s group. Didn’t see things like that at this conference. The immediate scalability throughout the institution, could affect 20,000 students, or half the faculty and instructors. I’m less interested in these highly refined questions and theory, small groups. More interested in results tried on thousands or tens of thousands. A lot of ed tech communities – lots of them – often focus on small studies, with strong questions, or clear ones. I thought that’s one difference with this year’s LAK and others. What’s the new big tech that someone’s going to spin off that I can say to my provost – your interest in LA can spin off here.

Dan: I’m going to change what I was going to say. This has to do with what we’re about here. Is LAK about MOOCs mainly? The large scalability has been a motivation for LAK – a whole lot of learners throw off a whole lot of data. But on the other hand, objective here – big players are moving quickly, bring the learning science in to that. Maybe we succeeded too far in that direction. Is LAK going to be a conference where different stakeholders meet? A boundary conference? Or a core identity? Or both – a core/periphery structure. The core focus looking at new analytic methods, or understanding existing ones in how they inform learning in an actionable way – want a stronger emphasis on that. Hard to do. Eye tracking paper, poster looking at graph motifs, has potential. An issue there, once you have it nailed down, then it doesn’t belong here but at e.g. learning science conference. We rejected a good paper that was learning science using existing techniques to address a problem. Are we informing big fast-moving things with big paper? New innovative techniques?

Linda Baer: The core function of LAK compared to analytics broadly. We have to be aware of the more big data continues to impact education the more we need research to inform what is happening. Urgency - politicians are grabbing stuff to be more efficient and effective. How do we stay ahead of the game with this community of learners and researchers that care about analytics – do it right, but stay ahead of those who do it regardless.

Alyssa: I don’t know that we can be ahead of them. We can think deeply, but not be right at the edge. Getting ahead isn’t feasible.

Paulo: Every community needs both people. Some people are already doing that. Doesn’t have to be an explicit choice, senior people talking to the press. Communicating a message, making a connection between this community and the public. Sometimes more explicitly, strategically done, sometimes not. Select e.g. papers that are seminal in the field and have sexy results and have that ready to tell the journalists – e.g. MOOCs are not that great. Journalists and the press get excited about new things. After that peak, there’s an incentive for other journalists to debunk the myth and sell more newspapers. Detect those cycles, deliver a message that’s not the overhyped message.

Stephanie: I like you call to arms to be public scholars. It’s becoming more common for academics to be visible in the public press. Used to be inverse correlation of perception of seriousness and your public profile. We can be more public when things like this article come out, pointing out where there are unfounded and ungrounded statements. Many universities have publicity machines, always trolling for stories about research that’s going on in the academy. We should be pro-active not re-active.

Naomi: Present different students with different analytics over time. Students have a disjoint experience of study. How can we pull that experience together?

Alyssa: Good question. The analytics integrated with learning activity they’re a part of. In different courses it’s often quite distinct; merge analytics with that. How to bridge that across is also important. Not saying no room for overarching analytics. Some might e.g. more academic analytic. That’s not the only thing but that’s a point to take in to account.

Caroline: Linda’s comment about keeping ahead. I don’t think we’ll get ahead. But we’ve learned from systems development to go through that lifecycle to nimble, agile computing. Nimble, agile analytics – use what we have now in a more action research way, we have a better chance of keeping on. We’re swimming in it, can’t stand on the shore waiting. Embrace the change and go with it. I like this time, closing the loop – not just analysing, using theory, but bringing it back to the learners [Learning Analytics Cycle!]. Outcomes assessment – it’s a reasonable thing to do to see if your intervention works! Last thing, one of our problems is dealing with multiple stakeholders – who are we talking to? Our core here is talking among researchers, that’s a good niche. But outside we have to be public intellectuals.

Phil Long: Reminding myself – Dan made comment about boundary agents. We can be boundary agents among these stakeholders, provide a framework for the hype. Express thoughts by buying something, think you’ve solved it and moved on. As interdisciplinary group, work with other stakeholders, policy people. Inform learners of options that could improve the management of their own learning. Juxtaposition – do we need another dashboard? And yet for many learners the dashboard is how their quick-and-dirty interactions are mediated. What other mechanisms of real-time feedback are effective and useful?

Greg: Speaking as a techie who wonders how research in CS can be useful, really move things forward as opposed to doing something everyone’s done but in a new way. E.g. yesterday computational linguistics methods, network methods. How to channel energies so as to say – not building another method, but take this problem and try to create a computational method that can do better along some metric. Alyssa – share your tiny system. EDM had a competition – what other ways can we take your data and try to present it to learners in the 50,000 ways we can imagine and see if they work as well or beter.

Chris: Linda’s comment, the companies that provide learner analytics and their visibles. A few startup companies. Where are the Blackboards, Echo 360s, what is the relevance of this community to these partners who seem to have disengaged a little bit, what will that mean over a few years. Change those products and they get sold back to products. ITSes have 30y research but not broadly used. It’s stayed as a narrow focused research question, it’s used in K-12 but in specific domains, unlike the LCMS which is used everywhere.

Stephanie: What many of you have said is interesting, I’m excited to be part of this community challenging each other. Look forward to you bringing these ideas forward, and ways to address it, to the next LAK.

Paulo: I want to mention a couple of quick things. The learning sciences community, one fundamental value is a lot of people care about inquiry rather than scripted curriculum. A small thing, if you ask people, they say trying to make things more inquiry-based. That gave the community an identity, made a difference. For this community, we should make some of  those choices. E.g. between technology as a way to make education cheaper, or make it better. We might say make it better. Many choices like that we can make. We’re not like physicists locked in a lab. The ethical choices we can make are a really important thing. Pressures from private sector, government. There are always values at play, moral choices that we can make, which will make a difference. What are those choices? We’re changing the lives of children. This community has the power to decide how things will be through our research and our public voice. We should think carefully.

Alyssa: It comes down to taking the notion of what we think learning should look like, and engaging with the people saying what will it look like. It’s a tension, some compromises involved. How can we do something that we feel good behind, but not staying in our own sandbox. Ed tech work with strong foundations but not taken up widely. We need to create viable alternatives. That does involve compromise. Need strong sense of what values we need to hold on to. I was impressed in the mind-blowingness, there was a lot of diversity – there’s interesting and new things in the agglomeration of what we’re doing. Not breaking boundaries in the home discipline, but coming together in a unique way.

–
This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.

Panel: Recent and Desired Future Trends in Learning Analytics Research

(cc) NASA’s Marshall Space Flight Center on Flickr

Liina-Maria Munari (European Commission), Myles Danson & Sheila MacNeill (JISC), John Doove (SURF), Sander Latour (SURFSIG LA)

Erik Duval is chairing. Welcomes everyone. Aim to have dialogue between funders and researchers. Series of ten-minute presentations from the funders on the panel. Focus on LA research. Then discussion and questions. Better understanding is better for all sides.

Liina Munari (European Commission)

One of the project officers (=program manager in US terms). EU funding research in technology-enhanced learning TEL. EU slides traditionally full of text and boring. Have open call on LA closing on Tuesday – probably 90% of the Europeans in the audience are involved – but not talking about that.

Our current programme ends at the end of this year. Entire EU is in the process of drafting the next one – won’t be a Framework Programme but will be Horizon 2020.

European Commission funds FP7, CIP, think of in Brussels, but their unit is based in Luxembourg. Fund research on learning-related technologies, FP7, FP6, FP5 – 20 years. We’ve done it all, workplace, school, universities, networks of excellence. STELLAR, many you’ve heard of, we’re proud of them, we’ve pushed forward research in this field. There was a huge hype about what tech can do, practitioners not convinced, have kept it on the agenda to know and hope it’ll be included in the next 7y programme.

Funded 44 projects, EUR 185m total. Learning analytics is about EUR 6.6m, in the current call. The rest is something else. Went back to project file – Juxtalearn, WeSPOT, Go-LAB – which do have some learning analytics as a small part – from 2011-2012 call. Only calling it learning analytics very recently.

Commission is a large institution, other units fund other technology. What about neighbours who fund Big Data? One project - Linking web data for Education – a support action.

Data Value Chain vision – creating social and economic added value based on the intelligent use, management and re-use of data sources in Europe. This is LA! Big data ‘frenzy’. Focused services, drawing on big analytics, on fast data. All about speed and scale. What’s big data? When the size of the data is part of the problem, or large enough tat it cannot be processed using conventional methods. Do we have that in education? We have a boutique problem. If we want to convince the big data community, we have to show them the big data … but it doesn’t come out unless you use more technology for learning. Chicken and egg problem. Where do we position ourselves? Big data? Or very specific with links? Volume, Velocity, Variety.

Good if standards mature, and if there is sufficient interest and funding from national and state governments … need to define the location, and argue the value of this research.

Erik: My first job in 1989 was funded by the European Commission. We’re involved in some of those positions.

John Doove, SURF

I’m a project manager, responsible for activities including learning analytics.

SURF’s not a research funder. But a collaborative organisation for innovation. Only fund projects in HE in the NL, and they’re not research – more practical application. Want to improve quality of HE and research. Learning analytics important.

Their focus is transfer from research to practice, and policy in to practice. (Less interested in Policy to Research). And transferring good practice more widely, at national level. Special Interest Groups, people who are interested in a theme and want to advance it. The QR tags take you to their website about the LA SIG. They also inform policy, trying to influence directors and boards.

We like SoLAR. It’s a research community, but with an eye for putting it in to practice, and getting that relationship to policymakers.

Fund not just projects, but programmes. Find and ID gaps in knowledge they need.

Concrete example – one building block is experimenting, developing and encouraging use. Did that with 7 projects, only EUR 10k each. Interesting results. They were allowed to fail. They don’t have to get significant research results for a paper, or a major success story. Can fail, so long as it’s responsible and others can learn from it. There’s a video to show them.

Started with research expertise. So have an Expert Group: Wil van der Aalst, Hendrik Drachsler, Erik Duval, Paul Kirschner, Ton de Jong. Also reviewed results afterwards.

Research projects that would make them happy:

• Research driven by a question from educational practice – not a solution, or a dataset; practical applicability important
• Ongoing exchange between practical experience/validation and research focus – iterations, each iteration is valuable
• LA is multidisciplinary topic, strong proposals involve those multiple disciplines.
• Stakeholder involvement (service providers, users) throughout the process; vendors have picked up on at-risk identification, but little more comes out, not looking in to the learning process.
• Impact analysis – helps big time in reporting back to policy and decision makers
• Research that is transferrable, reproducible and broadly applicable as possible
• Research that makes use of related educational fields – not LA on an island. Digital assessment, adaptive learning, OER, MOOCs, etc.

Myles Danson (JISC)

Myles slides on Slideshare

JISC is similar to SURF, but in the UK. A bit bigger. We’ve funded about 1400 projects, all underpinned by technology. I’m a programme manager. Last year 40 projects in his purview.

JISC is a shared service across UK universities. We run the network that connects them together and to the internet – SuperJANET – fastest in the world. Couldn’t do that individually. Also content services – negotiate on behalf of the sector, cheaper. Then services helping with uptake and embedding, like John. Finally, the innovation group, where I am.

Within my 40 projects, looking at relationship management, from pre-admission to alumni. Had 4 projects looking at learning analytics without having namechecked them, finished last year. Had a piece on Business Intelligence. GBP 50k over 6 months. Two drifted in to LA.

Have also worked on activity data, engagement analytics. All similar fields; you get this. Got about 50k, Sheila did the JISC CETIS Analytics Series – big breadth, includes learning analytics. All this stuff available for free.

Early characteristics of the field:

• Newish field
• Beacons of excellence
• Narrow applications
• Promise of great things
• Little coordination of effort
• Little evidence of business cases (Myles special interest)
• Reliance on the implicit – articulate the benefits, bring them to the fore, quantifiable ones – useful for demonstrating need for funding
• More holes than net – Sheila’s work uncovered, much that could be done
• New terminology, be careful, can bamboozle & confuse people you want to influence
• New roles – data scientist vs put the right team together, more fluid
• Intra community excitement – lot of buzz, the SoLAR, LAK, good job
• Extra community confusion – publicity pieces, in the Times Higher, jargon-free but trying to lift the field so important folks get a grasp

Many opportunities. Early adopters, grassroots interventions, nurturing activity. Strong case for peer support – conference a good examples, workshops, SIGs, CoPs, Shared problem identification and solving. Educause, JISC and SURF have had videoconferences to coordinate internationally.

Challenge: Important part is the institutions and sector stakeholders. Requirements gathering/horizon scanning. Creating, updating and testing – lots of that, the research this conference is focusing on. Don’t see much synthesis and knowledge creation, could work on that. Which means not enough solutions, advice and guidance to go back to the institutions and sector stakeholders. JISC’s role is about embedding, changing practice, within institutions – uptake stuff. This is a cycle, it’s iterative.

Business Intelligence – roadmap for doing it, 1-6, from fragmented sources to reliable predictions. Use maturity frameworks, adapt, look at organisational maturity around analytics. Work through representative bodies. Shoot high – senior management team, policy, governance.

Need to show beneficiaries, and reality of benefit in the timescale, reality of sustaining the outputs, value to the sectore, and innovativeness.

Same benefits from BI can be demonstrated by LA projects. (Has a long list – see Slideshare above.)

Summary

Innovation/embedding cycle. Focus on benefits. Keen on co-design. Business case. Policy and governance; readiness issues (digital literacies). Keep up grassroots innovation!

Discussion

Chris: Interested in international partners that are working in synergy, through SoLAR. Would love to work with Simon, he could get JISC funding. How do we show our synergies could benefit?

Myles: That’d fall in to the evaluation criteria, track record and make up of the team. If Simon applying to JISC, benefits go to the UK, but if you’re adding the value through team makeup, expertise.

John: Easy way, I agree with Myles. It’s about the added value. If you deliver value, that’d be a good way to go. We are funders for the Dutch situation, but we do care about other places. If working in Canada, we’d evaluate that. Added value.

Liina: We’re a bit more generous. We can’t directly fund Americans, Australians, Japanese, unless you really justify why expertise is not found in Europe. But we have other means, can always be there as a formal partner without asking for funding. In the current call, asking for a coordination and support actions, organising events, networking things – counterparts can help, use the EU money in a more, larger context. We’re international, there to benefit the European Community, but EU research has links, not bound by continent. If in China or Russia we can fund you – ACP countries. But developed ones.

Sheila: There’s a role for community building here with project funding. Events important, maintain connection and open practice. Share our research, cite each other, project proposals cite relevant people we might work with or directly relates. While funding projects, we can’t fund people from the States, but can do community building. Moving the research community in to the practical application.

Sander: The SIG doesn’t get directly involved in funding. But do value collaboration, especially international. The LASI in Stanford, there’ll be in Amsterdam and England, try to get out of national borders and work together. Expertise exchange works through to next funding proposal.

Q: Compare with discussion after lunch, with the US community. One big difference was the venture capitalists, the money from industry in LA. What’s the position in Europe about this.

Liina: Everything we do, we try to boost growth and jobs, if can link it to that. We’re happy for industry partners. We have some instruments, the European Bank of Investment, that’s venture capital. Anything that would attract VCs, maybe the new programme, there’s be more instruments related to this. In TEL, looking for more VC to fund risky interventions. That’s not excluded. But institution sets a pace for a number of years. It should be highlighted if not happy with how things are funded now. EU is very democratic, but whoever shouts the loudest can lobby. Very pro any business initiatives. Even when a research programme.

Sheila: From JISC, we’re agnostic. It’s not part of our culture, we fund institution. There are small companies who’ve worked with our community. Tools developed in tandem, not vendors just coming in with a tool, but can say we need x and y. That could change over next couple of years.

Myles: Projects outlined strong cases to work with software vendors – not VC per se but certainly private sector. Consider benefits to the project, insist through licensing that outputs freely available to the JISC sector (UK HEI). Partnering with companies, suppliers. Obvious benefits – not developing software and then struggling to spin it out, but include in a commercial offer – with caveat to make available.

John: Get the stakeholders involved. Meeting for vendors on LA.

Erik: Great, we can still get funding if we have VC partners.

Myles: Wonder where the loop is from LA research community, going in to best of breed. Market share – Banner big for student record systems, developing their own tool. If influence those vendors, features IDed as valuable from research. Rather than vendors providing the wrong ones.

Erik: We are developing proposals to do that.

Myles: Standards are key, and interoperability.

Sheila: Moving in to a mixed economy. Cloud services, more challenges. Work with vendors differently. Get access to data. Manchester Met, using hosted services, small vendors built them some apps, testing with real users. The way we’re working is developing. Want to inform the development.

Q: Can we promote the work of researchers, on LA, in the CFP on educational practice, lifelong learning, to make innovation work. So we don’t start over again so all the researchers work is on the shelf.

Sander: Be a good community, share the knowledge. Large gap from where researchers stop and practice begins. We should move towards getting tools so teacher can use them. The tools are ready, but when try in practice,  have to read a lot and there’s no time for that. Finding research is one thing, SIG helps. But seeing where gap still lies, aiming at that is important.

Q: That gap is what you want to solve in your project. But lots of people don’t know what’s there. The researchers do not connect very well. How many people are here who are not researchers? What role could they have?

Sander: Role for SURF, JISC and my SIG. We’re trying to do, like SoLAR, have connection places. E.g. SoLAR Evidence Hub. Practical things – LASI in Amsterdam. Week before that, will organise a seminar to inform teachers and where that gap lies. You do that by communicating in the community.

Liina: Communicate among the researchers. Open access to research results. If you do something nice, that makes a difference, communicate, and in lay terms. If you have something nice – to capture the eye of the Commissioner, you need an article in say Times, this EU project, then we hear – did we fund this, can we fund more like this? Don’t forget the old means, press releases, talking to journalists. When in terms that are understandable.

Myles: When put your proposals together, look for a stakeholder map. JISC specifies will mark down no evidence of related work – often does come out in the evaluation process, stop reinvention. In JISC, have programme information management, can search all previous projects, outputs are all there. JISC repository with materials.

Sheila: We have a project db with tools and technologies in projects, outputs and links, completely open. Was Linked Data, may not be any more. Want you to do the research. People always want to reinvent the wheel.

Erik: Not reinventing the wheel. Does that mean you think LA is not old wine in new bags, or do you not think it is? Is it different? We’ve had close to 30y funding for TEL research, why do that?

Sheila: We’re refining the wheel, getting faster and quicker. There are elements of LA that are new and exciting. Common cultural problems in all TEL. Barriers to adoption, nothing to do with the technology, that’s communication. Same message in a different way. LA is a powerful set of tools to affect learning.

Liina: Same thing. Bit of a newbie in this field, but if you can instantly tell teacher or student this will help, we’re closer to closing that gap between adoption and research.

Sander: What makes LA slightly different, it’s useful much earlier in the research phase. Not develop entire theory in research environment. We have participatory design, set movement in the teacher community. A lot earlier than with many TEL movements. Vital to bridge that gap.

Myles: I also have a background in TEL, eAssessment systems, with a team of learning technologists. The cyclic nature of using things – instructors, academics, teachers would go to use the systems, then leave them for six months, a year – is a problem. Then the system had been updated, need further support to re-remember. Heavy lift done by the analytics, that lowers the bar, provide accessible visualisations, it’s a literacy people are picking up, embed analytics in practice, run these reports out, replacement for previous task, not an extra burden, few key differences there. [Doug's prize for most metaphors linked in a single utterance there  - good points though.]

John: I do not have a background in TEL! Group psychology. Never waste a good hype to use for a good cause. LA called hype for a good reason. More RoI, more online data, more open education.

Erik: Don’t waste a good hype, nor a good dinner! See you at the dinner!

Liina: Promotion: Commission uses experts to evaluate programmes. Come to our website, might need people for the forthcoming call. Always gaps to be filled. A good way to get involved. International experts.

Erik: It’s also a good way to learn how to write good proposals.

–

This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.

Discourse Analytics

(cc) Iñigo Elimay on Flickr

Analyzing the Flow of Ideas and Profiles of Contributors in an Open Learning Community

Iassen Halatchliyski, Tobias Hecking, Tilman Göhnert,  H. Ulrich Hoppe

Ulrich? presenting.

• Problem – characterising evolution of knowledge in knowledge creating communities.
• Approach – modification of ‘Main Path Analysis’
• Care – Wikiversity
• Technology embedded in ana analytics workbench
• Zone of Proximal Development – next steps.

Not an empirical paper. Development of a computational method.

All this is on open data – Open Source Software developers. Collaborative knowledge construction at KMRC. EU project SISOB on science analytics. And network analysis techniques.

Knowledge building – epistemic artifacts – Bereiter and Scardamalia 2003 – modality of learning is the same as in science. Main argument to look more deeply in to methods to analyse science production, transfer to the learning area. Re-uses scientometric methods to analyse knowledge building in communities.

Relations around epistemic artefacts – entities are actors and artefacts; links are  direct (create, cite, communicate); induced/mediated links [inferred?] e.g. between two documents  if there’s shared authorship.

Main path analysis: given a graph of dependencies between items, how to identify the main pathways through which the knowledge evolved. It’s not there in scientometrics; there’s a question of validation of these heuristics. In concrete terms, use citation links (references in publications), as relations between publication. Generally acyclic graphs, no cross references (mostly – except in e.g. chapters in edited books cross-referencing). Hummon and Doerian 1989 on DNA papers – derives a main line of development.

Citation networks are directed acyclic graphs. Links are inherently noted with time – sources are oldest, sinks are newest (have no successors). DAGs have at least one source and one sink. Add one virtual source and one virtual sink. Then can calculate edge weight according to a weighting scheme. Draw all trajectories from the artificial source to the sink. Count of times a link was passed; select the path with the highest edge weights. The paths may be multithreaded. Then cut off artificial source and sink – so could be disconnected.

Example from Wikiversity. Use the author names from the pages and their separate versions. 228 Pages in philosophy. Complication compared to citation networks: they’re not acyclic! Web pages often cross-reference bi-directionally. So no longer DAGs! So for each version of the page, introduce a new node – even when operation is only to link to a new page. That then becomes DAG again. Use swim lane diagrams, filter out redundant edges. Some redundancies. A link persists, but isn’t repeated. If links are deleted we’d need to do something, but we haven’t seen that; so we just accumulate them. Filter out redundant edges. Can then run main path analysis.

Medicine is rich on Wikiversity. Two different types of application of the method – one with minimal branching, only when ambiguity forces a branch; other times you might want richer threading and so allow for branching.

What insights do we get? A better understanding of the contributions in context. Show by example of user/author profiling.

Show user ID, with number of contributions  - three users are about equal. So can’t distinguish. But methods gives indicators to distinguish – number on thread of the main path (which can be threaded) – can see that some have a more on the main path.

Three descriptions – Connector, creates many contributions on the main path, receives many refs but not on the main path. Inspirator creates many contributions on the main path, receives many refs including on the main path; can be self-cites, but Ok if on the main path. Worker creates many contributions, fewer on the main path, very few references received.

Technology: SISOB Workbench.

Used for many different purposes – workflows. Visual representation, based on pipes and filters approach (Yahoo pipes). Filters as agents (mostly in R). General architecture for cascaded processing.

Looks very like Yahoo pipes. Have a data source, processing, outputs, transformation.

Next steps – Zone of Proximal Development.

Have used MPA as heuristic method. Question – what dataset do you use to find main paths – e.g. all CSCL, or AIed; there isn’t one main path, there’s thematic subnetworks. Need a specific seed. Can start from the end nodes, backwards fan, then from old to new, can get a more coherent picture. Want to explicitly manage the seed.

Then extending the range of the method – in Masters thesis, applying it to chat data. Replicating Dan Suthers’ analysis on Tapped In and added main path analysis.

Blend method with other network analysis methods – improve ‘link prediction’ approach in scientometrics (predicting future co-authorship, or future topics for authors); main path analysis could further elaborate certain conditions. Can interpret figures, adding main path figures, that’ll help you profile productivity, use as role indicators like from SNA (e.g. brokers, betweenness centrality).

Questions

Q Thanks for interesting presentation. In both examples, citation and Wikiversity – are connectors much more likely to be interdisciplinary folk? If viewing this, in citations they’re likely to be interdisciplinary folks?

Depends on your dataset. Here (in Wikiversity) it’s relating gynaecology and diabetes, so it’s interdisciplinarity within medicine.

Q Working backwards – it strikes me you’re more likely to miss out some small paths that die out quickly and go nowhere. Could work forwards to be sure you don’t miss out.

Usually you’d do both. Preselection problem. If you don’t preselect, you get lots of pathways, it falls apart in to very different pathways, each one a large branching graphs. If you want to ask more focused questions, that’s the seeding question, preselection. So work backwards or forwards.

Epistemology, Pedagogy, Assessment and Learning Analytics

Simon Knight, Simon Buckingham Shum, Karen Littleton

Simon Knight presenting.

Wrote this in first two months of his PhD.

Thinking about this middle space: LAEAP-ing the divide. LA buy in to ways of thinking about epistemology, assessment and pedagogy. For theoretical, practical and ethical reasons we should address that. Those considerations have practical application, and that’s where the middle ground is.

Introduction to epistemology, pedagogy and assessment.

A triad – foreground the relationships between them. What does it mean to know? How do I know someone knows something or not? And how do we get people to come to know? (to learn)

Truth, accuracy and fairness have a very practical purchase in the context of assessment. LA buys in to ways of thinking about this, flexibly enough that we may be able to move forward.

LA is data mining, on digital traces, deployed for pedagogic purposes. Thinking about LA as assessment – AFL – assessment for learning, formative assessment.

Example – Danish case

Video. New system in Copenhagen, taking an exam, allowed to use the internet. Can use Facebook, but not communicate beyond the classroom. If you send a question and get an answer back, that’s cheating. Educate students about that. Penalty explusion. Spot checks by invigilators. 28 students in a room, can’t watch all at once. Some will try, some will be caught, some will get away with it. Questions make it hard to cheat – not facts and figures, but analysis. Danish government proud to be first to allow it, other countries will follow. Is the UK system out of date? Stephen Heppell saying don’t cap them, arm them to be ready for the C21st, right now they’re not ready for the C19th.

The policy documents (mostly in Danish), they’re making epistemological claims. “If you allow communication, discussions, searches and so on, you eliminate cheating because it’s not cheating any more. That is the way we should think”. In university context, this would also allow auto-grading, area for LA.

Looking at student’s own judgements. Knowledge communicative, discursive. Picking out tokens of knowledge in decontextualised ways may not assess knowledge but something else.

Assessment should nurture knowledge development and practices (through AfL). Particularly where pedagogic assumptions on the subject, not the assessment practices. Schools teaching to the assessment.

Other side of the coin

That was theoretical stance. The other side is what students do, apply it to a context. Epistemology in action – what they do.

Interested in things in action, learning in action, what they do that. Analytics gives us unprecedented access to that. Shift from standardised assessment to knowledge in action, not in a vacuum.

The lens of epistemic beliefs or behaviours – how they conceptualise certainty, simplicity, source, justification for knowing. Information seeking behaviour is particularly salient. Not suggesting that probing behaviour unveils beliefs. In action is key, language to do things. Psychometrics are useful, but standardisation is problematic (and they’re poor in this area now, poor reliability).

Exploring various behaviours – how students choose sources, which experts they turn to, how knowledge flows between agents in a domain.

Probing it by looking at structured knowledge building – e.g. in OU Cohere tool, building on CSCL tool on making things explicit in structured environments like Knowledge Forum, Belvedere, etc. Students tag, make semantic ties; can analyse this.

The next class of those tools – that LAEAP the divide – are built on this E/A/P framework, emphasise dialogue around knowledge building (see DCLA paper from Monday). How dialogue is used as a frame for decisions. Provide a structured environment, with a time flow. Understanding how students co-construct the frame. Pragmatic stance. Will use NLP tools – MALLET, WEKA – exploring the discursive context. Discourse not just as a context, but as a creator of context. Subject domains, fluid property that discourse allows. The structured environments mean we can mark off sections and give feedback – e.g. say of a node, you might want to consider this, or use ideas from another node, etc.

SOCL – Microsoft social research platform, released the dataset. In SOCL a search creates a post or a node, you pin objects to it, in a public space. We can access the tags, what they’ve accessed, who they’re linked to – the usual social network data. There’s also commenting. It doesn’t allow us to see how they were framed prior to the search. Example of two brothers debating whether aliens built the pyramids. Will explore this dataset using exploratory talk, and accountable talk, to see how people build knowledge together. How they engage in interthinking – Littleton and Mercer term. That kind of talk has particular epistemic implications. It doesn’t preclude disagreement, but they must be accountable.

Analytics questions addressed: How do users interthink to construct and solve problems? How do they co-construct their information needs? (In structured and free-text environments) And how do they use each other?

LA and assessment regimes implicate epistemologies. Use of data for particular conversations/assessment/AfL is key; it’s not context-free. View of LA to start conversations, encourage learners to sense-make around their own data. Highlight some design implications of exploring that epistemological assessment, pedagogy triangle.

Question

Chair: Looking at explicit and implicit behaviours?

In the context of the structured platforms, get students to make explicit their thinking. Those kind of platforms encourage students to think in particular ways, prompt them towards accountable talk, or making particular sorts of connections, explicitly. Implicit, we now have access to some of that data, can trace that much more. Interestingly, in structured systems that ask to make thinking explicit, it’s harder to track, it encourages thought in particular ways and to interact with others.

Jim Simon(?): Reference frame on epistemic beliefs and behaviours. Environment that encourages certain kinds of collaborative or intentional interactions. My work has a model to encourage classroom to work as a community. Not uncommon in Toronto. Intrinsic interest in getting students to believe and act that way. Challenge walking in the door – they’re not coming with that epistemic set. Some want to beat another kid and get a better grade. Implications in this LA to help them adopt new epistemic positions to learn about why they might be coming in to the classroom and repurpose themselves in the learning narrative? Not so much how we measure epistemic beliefs, but how can we change them? Need to understand first or they are mistfit to the design. Why they think they’re in science class. If you have this measure, you should be able to use that as input in to an epistemic treatment – that’s what I’m after.

Answer depends on whether what we’re looking for is LA that are radically disruptive, or are evolutionary. There are answers for both. On the former, it could be that the assessment system is wrong – they come in thinking that way because the system is set up that way. Our job is to challenge that. I think that has big legitimacy, but others might not think that. Fits well with the triad. On the medicine, is it evolutionary side. Exploring providing prompts – e.g. you’re talking about this, have you thought about that; or previously, you engaged in this kind of discussion, maybe you could go for that. The lit around exploratory talk and associated dialogues – Rebecca may talk about this – it’s pretty strong as a pedagogic tool. Students learn better when their talk is accountable to each other, when they explain their reasons, they try to build common knowledge. That’s the case now within the current educational system. And system that’d support and encourage that more widely would be fantastic. Trying to encourage more dialogic education within the current system.

An Evaluation of Learning Analytics To Identify Exploratory Dialogue in Online Discussions

Rebecca Ferguson, Zhongyu Wei, Yulan He, Simon Buckingham Shum

Focusing on how dialogue, speech is used not just to convey knowledge, but to build and construct it.

We’re looking at discourse. Why? From a socio-cultural perspective, it’s crucial in the development of knowledge. How people engage, relate and explore ideas. Neil Mercer’s work, began in schools in f2f, now going online. They identified three main ways of talking:

• Disputational dialogue – don’t want to see it, no meeting of minds
• Cumulative dialogue – more positive, put ideas in, come away having learned something, quite happy to see
• Exploratory dialogue – most educationally valuable – sharing ideas but also challenging, combining, exploring them.
• How can you find exploratory dialgoue automatically, and how can you encourage it.

Paper to LAK 2011, took chat from 2d online conference in Elluminate. Manual analysis, finding cue phrases that might be indicative. We could distinguish some parts, but very labour-intensive. Not scalable, which created a problem. So talked to computational linguists, asked them how could move on.

They spotted three problems: The dataset is too limited. Text classification problems are typically topic driven, but this isn’t. Nevertheless, both dialogue features and topical features need to be taken in to account. It is topic-based, could have a good exploratory dialogue in context, but not useful for followup.

Need a self-training framework. One you can do self-training from instances – give examples to classifier, say go find things like this. It starts to go wrong. “I think …” is fine “I think that this is … a proof” vs “I think you’re stupid”. So need to use features. Look in more data, split in to unigrams, bigrams, and trigrams. Average across examples, get more detailed. More reliable. But not taking topic in to account.

Taking account of the nearest neighbours of every turn; it’s the ones with the most unigrams, bigrams and trigrams in common. Explore the k nearest neighbours. Calculate value for support for the pseudolabels generated.

Also look at cue phrases from pilot work – 94 – very precise, but recall bad (human coders missed examples).

Needed more data. Picked up chat data, 10,500 turns, about 9 words per turn, from MOOCs. Got to postdocs to code it. Label obviously exploratory, and obviously non-exploratory. And subcategories for exploratory. Subcategories not useful, they overlap, and it depends how far you’ve got in the conversation. Coders did agree on exploratory/non-exploratory (mostly); only took examples when they agree.

Applied method to that dataset: train initial classifier, apply the trained classifier to un-annotated data (swapping around through all combinations); use self-learned features to find exploratory dialgoue, use cue-phrase mathcin, k nearest neighbours etc – redid it repeatedly.

• Accuracy – Pilot 0.5, new 0.8
• Precision – Pilot 0.9 (manually coded), new 0.8
• Recall – Pilot 0.4, new 0.9

Fine in computational linguistics … but what does it mean in LA terms? We can take a big dataset – 2, 3, 4 hours of conference. Want to know where the interesting things are happening. Could act as a reflection tool for people involved.  Example vis showing exploratory/non-exploratory over time, bunch 10 turns at a time. At the end, the last 20 turns are not exploratory – that’s all the goodbye/thanks. Similar at the start – lots of hello/good to see you. Bumpy bit in the middle, then a long exploratory bit towards the end.

Another vis – individuals plotted exploratory turns vs total turns – shows who’s individually doing more exploratory dialogue. Issues though – may be discouraged, might be e.g. had issue with a microphone. Or saying I was the nominated welcome person, I did my job perfectly. There are nuances within this. So need to know how to uses this to motivate and guide, not discourage. Also need to build visual literacy. It takes some getting in to to understand them. Also participatory design – not just researchers making decisions but learners and teachers.

Working in the middle space. I come to this from discourse analysis and education, others from computational linguistics. We found this tough. It’s not just tough framing the questions, it’s tough to interpret what you’ve done. Finding out how to present that to a mixed audience is a challenge. It was difficult, but really worthwhile.

We’ve developed and tested a self-training framework. Built annotated corpus. Ideas for how to support educators with it.

Questions

Greg: I used to be at CMU, did a lot of research on this. Difficult to compare, very different methods. Hard to replicate. In paper have kappa for agreement, seems kind of low. What kappa did that get for the precision, recall figures, what kappa does that translate to?

The only kappas I know are in the paper. The linguists were happy to go for the exploratory/non-exploratory, but wanted it higher. Didn’t take all their examples, only where two coders agreed.

Greg: If take computer as a third rater, what kappa does it get?

I don’t know what kappas it’d get.

Greg:Like essay grading where computers are doing as well as humans because humans aren’t very good at it.

Yes. Could be like that. Room for improvement in the human coders.

Q: Using markers other than dialogue for classification? You might use language differently to me. Might be things about the way that Rebecca’s talk seems more exploratory, not just n-grams but phrasing. Vs tone of my messages might be very different. Considered those?

Yes, that’s a possibility for the future. When Neil Mercer and team have worked on this, was across several countries. Including Mexico. When worked on developing exploratory dialogue, grades do go up. Though it’s to some extent cultural, to some extent it’s about more. It’s in English at the moment. We’re pleased that often in language processing you have to take quite formal correctly-spelled language. But this has nonstandard spelling, usage, emoticons, but still gets significant results. But take your point.

Chair: In Mercer’s work, does he talk about how other sorts can lead to exploratory talk? Or are there elements in the others that lead to exploratory talk?

Neil would say cumulative is valuable in and of itself. Doesn’t have to progress. Happens a lot. I can sometimes see value to disputational dialogue, others can’t. I say sometimes you need disputational dialogue, others strongly disagree. Usually Neil’s team have found you need specific support to develop exploratory talk – a few guidelines can move learners and teachers on.

Chair: Automatically detecting those other sorts of talk?

Not for now.

–
This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.

Challenges

(cc) Schlüsselbein2007 on Flickr

Issues, Challenges, and Lessons Learned When Scaling up a Learning Analytics Intervention

Steven Lonn, Stephen Aguilar, Stephanie Teasley

Slideshare of their slides.

Idea is to scale it up – ‘on the road to scale’.

Working with STEM Academy, helping ‘at risk’ engineering students.

Originally, data from Sakai, hand-cranked using Excel. Massaged it, made it in to managable reports for advisers to see how students were doing on each given assignment. Took assignment data from gradebook, sense of how students are doing. Fine for first year but labour intensive. Want to automate.

Student Explorer. Not giving answers to advisers, but data to explore with students and have a conversation with them.

Hit a crossing point. As an academic unit, had to work with ITS who have a business leaning. Couldn’t always give them their whole attention. They want to hitchhike on an existing BI platform – BusinessObjects, with a new database Extract, Transform and Load (ETL) presentation layer – but that had itses own challenges.

BusinessObjects used widely. They found some usability gaps between previous Excel version and this one. It looks very similar. Had some new functionality initially. Previously, could only see the current view, but with this they could see weekly snapshots so could see how current status compares to history. One problem – in Excel, click on current view, and it replaced the window, then click back. But the BO system, it kept opening a new tab in the browser – quickly problematic for advisers. Another – paged view, for about 500 students. Hard to see what page you’re on – only one little view that said “1 / 1+. Also calculation gaps. Lots of nuance, edge cases with grades. Example of large chemistry class, that’s a shared class so the LMS has four sections of 100 students each. In Excel, massaged this by hand with long Excel formula. In automated process, couldn’t do that – so blank counts as null not zero. Functionality in gradebook where you can have an assignment graded that doesn’t count towards the final grade. So might put in attendance, helpful for advisers to know.  Also access gaps – to get in to the system, you need special authentication – including two-factor authentication, many advisers didn’t have them. Also delivered in an IFRAME, but it didn’t work so got ‘Invalid session’ periodically. Performance gaps, they want to work at scale. Polled the data, polled everything – which crashed the LMS for all 12,000 people. Need to load test. Automisation gaps – some manually massaged, have to send manual CSV file with cohort, students, advisers.

Now doing an alternate route. BO was helpful, but not all the way there. Partnered with another campus organisation, using similar UI but with some tweaks. Will leverage existing system used by 75% of advisers. IT have infrastructure that works for that, will load same db and ETL layer in to another tool.

Hope lessons from our road will help your road!

Questions

Stephanie: Details of the dashboard was last year at LAK, so you can see them in the paper last year.

Chris: Important to talk about taking academic work and rolling out out campus wide. The academic labs are producing and institution is consuming. One challenge we’ve had is that our IT organisation expects that research lab stuff is a bit crappy, doesn’t work, not load balanced – but they hire full time people to work with vendors whose products are similarly problematic. How did they look on what you did?

Stephanie: We had the opposite. Provost very interested in analytics, told IT org to do an analytics projects – the IT people came to them to find something.

Chair: Have you disseminated this. You have a STEM progamme, what was the feedback from the instructors.

Stephanie: The academic advisers have this data, not the instructors. It’s not a very intelligent system. We trust the advisers to have the intelligence to match to resources on campus, things we can connect you to. Do want to scale up with a dashboard for instructors. But some say they don’t need it. We have to change some of that thinking.

An evaluation of policy frameworks for addressing ethical considerations in learning analytics

Paul Prinsloo, Sharon Slade

Sharon presenting.

HE institutions have collected student data for a long time. Have policies in place. But they’re not keeping pace with learning analytics. So looking at how they should develop.

UNISA very large (?300k students), South Africa. OU in UK, 250k students.

Who benefits from LA and under what conditions? Need involve all stakeholders. Students should be active agents in the data process.

Issues around consent, de-identification and opting out. Massive sets of data, think poses no risk. By registering, they provide implicit permission. But when it triggers customisation, often based on unseen criteria, they have a right to know how it’s being used. Few reasons not to tell students: growing need for a system of informed consent. Students can opt out, and make the implication of that choice clear to them. Consent needs to be refreshed regularly. Need trust and data protection. Also responsibility for students to provide correct and current data.

Vulnerability and harm. To students and other stakeholders. Implicit and explicit discrimination – support based on ?random personal characteristics. Validity and impact of labels – treated as different from others. Potential of stereotype; student risk profiles. Identity is transient. It should be transparent and regularly reviewed. Need to let students re-label themselves. Ensure analyses robust and on suitable datasets. Reduces potential for allegations of vultnerability and harm if it’s transparent. But could be risk of student abuse. What about students providing incorrect information – perhaps to get more support? Many policies say can terminate registration if data wrong.

Collection, analyses, access, storage. Issues around data from outside the LMS, particularly external e.g. social network services. Need explicit consent, and extra responsibility around that data. Need to take care not to amplify error in the aggregation process. Avoid bias and stereotyping, acknowledge incomplete and dynamic nature of identity. Risk of perpetuating bias and discrimination. Students have right to be assured data protected against wrong access. Students should have overview of stakeholders granted access to their datasets. Ensure effective governance. Most important step is to create a comprehensive data governance structure. Need for resource allocation. Understand drivers for success – e.g. overriding policy of targeted recruitment and retention – and decide prioritisation – make it transparent.

Data policies – reviewed all policies in OU and UNISA. 26 policies! Including research, data protection, records management and some bizarre. Who benefits? No mention in UNISA, focus on protection (largely of UNISA). In OU, shared responsibility and Charter with nice-sounding aspirations. Alumni office has responsibility to update alumni and get donations – details held indefinitely.

Consent and de-identification. UNISA – they may monitor, surveille – about employees, not students. Students as consumers. Don’t have to say, may only opt out as research subjects. If research, anonymity and opt out possibility is mandatory.  OU data protection disclosure  = explicit consent, maybe be used to provide specific support.

Vulnerability and harm. UNISA – no mention on tailoring support. Disciplinary code about materially false information, about university not students. Don’t define misconduct about false personal information. OU highlights empowering students at key points, using data to personalise. Student responsible to ensure data is correct. Doesn’t discuss detail, or who makes decisions about ‘best interests of the student’.

Collection, analyses, access and storage. Much on general data processing. But UNISA only implied for student stuff. OU has clear data protection policy. Covers data transfer on need-to-know basis (but not provide), protected from unauthorised access. Made aware can get access to their own data. Neither have mention of methodologies to manipulate data not how student could learn about it.

Conclusion – policy frameworks largely on research, academic data. Have guidelines on data, links to legislation. Both policies have institution as main role player. Approach to LA is not very student-centric, primarily as objects of analysis rather than partners. Students only have to make their data correct and current. Relatively simple models for focusing student support. Current OU high-level report on learning analytics but focuses on things other than student participation.

Question

Q With education, I agree with a lot of things, I like the analysis. HEIs often deal with training, we need professionals to serve our community – doctors, engineers, lawyers. We have 50 seats in our medical college, 900 who want to come in. Is it appropriate to allow them to opt out of technologies that can help them learn or reduce workload. We need to scale.

Really interesting. Hadn’t considered that issue. We take students without educational qualifications (necessarily). We should be more transparent about what we’re doing with student information. People should have the right to find out more about what’s going on.

Aggregating Social and Usage Datasets for Learning Analytics: Data-oriented Challenges

Katja Niemann, Giannis Stoitsis, Georgis Chinis, Nikos Manouselis, Martin Wolpers

Katja presenting. Talk about data we collect about the learner, less the learner themselves.

We collect a lot of usage data, social data. Normally stored in different formats, hard to map them to create larger databases. Open Discovery Space portal, for teachers to connect, bring together other portals. We don’t have information on learning object metadata, but also portal data on social and usage data. Many formats – CAM format, NSDL formats, Organic.Edunet format.

First study investigated 3 datasets, tried to map to a single dataset. First was the MACE portal, learning objects in architecture stored in CAM. Data like clicks on, tagging, searching. Learning objects from schools. Organic.Edunet, from organic agriculture in own custom format.

Much detail on specific data mapping, and how different formats were transformed in to others. Decisions to be made, not simple mechanical mapping.

Created two merged datasets – A with user information, and B with all events from all datasets, but no user information.

There’s no one-size-fits-all solution. Have to investigate the domain. We created another new format, to contain the other data formats so they can all be mapped in – its less work. In future want to go deeper, do some integration. If two objects are the same, they have different IDs on different portals. Mapping and merging only really helpful if it can do this. Want to experiment with the datasets.

Questions

Chair: You have different datasets. How often is the harvesting process done, where does data reside?

Once a day, but incremental, only harvest new information. Have one central repository.

Chair: How can this be incorporated for e.g. educators, to find relevant information, or for creators to market their resources?

We have a lot of very interesting LA tools. You can only use them with data in that specific format. That’s a challenge to create one format where you can use all the tools for all the datasets. This is also a chance for the educators.

–

This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.

Communication and Collaboration

(cc) Haags Uitburo on Flickr

Analysis of Writing Processes Using Revision Maps and Probabilistic Topic Models

Vilaythong Southavilay, Kalina Yacef, Peter Reimann, Rafael A. Calvo

Topic is collaborative writing (CW). Essential skill. Majority of documents are authored by more than one person. Single-authored essay is a rare instance in professional life. CW combines the cognitive and communication requirements of writing with the social requirements of collaboration. It’s a complex process, has attracted cognitive psychology, applied linguistics and other research interest.

Coordination issues – many different forms – group single-author writing, sequential writing (amendment), horizontal-division writing.

22 masters students in the course EDPC5021 – topic learning sciences and technologies. Every two weeks (a cycle) students divided in to groups of 4-5 members, write about a topic jointly, 3000 words. Six cycles over the whole semester.

Used Google docs. Have the text, with comments on the right hand side. Gave template to students – a part A where they formulate their individual understanding, and Part B where they were more collaboratively to work on relating the readings to the big themes of the course or the field.  Google docs stores revision history – version number, author ID, timestamp. Some challenges in using Google Docs API.

Want to provide feedback on the finished product, the essay, but also give feedback and guidance on how to work together.

The easy part is the final product. There’s a set of essays, can be graded, appraised and give feedback using a rubric. There’s also the revisions from the revision history, which tends to be large and extensive. As a teacher, can’t go through all those revisions to draw conclusions – need help from computer scientists.

That’s the task – information on the process as well as the products to give students feedback.

Big table – group ID, number of revisions and other indicators. Mean 66 revisions per doc.

Visualisations – revision maps, topic evolution charts, topic-based collaboration networks.

Data flow – google docs, api, get version history to create revision map. For the other two, send via text comparison utility, topic model (topic evolution chart), author-topic model (topic-based collaboration network). Details in the paper.

Three kinds of mining activity: artifact analysis, process analysis, social network analysis.

Only one shared with students is the revision map. Works on the level of sentences and words. Students need guidance on how to interpret. Columns are revisions, rows are paragraphs. Columns are attributed to the author. Each cell is colour coded – green=words added, red=words deleted. [more red/green colour coding!] Yellow is balanced. White is no change in that revision. Right-hand column showing total change per paragraph; top column showing total change per edit.

This helps students answer – which sections had most and least action, when major edits happened, whether students worked sequentially or in parallel, and how many authors worked on each paragraph/section.

Critical point – deleting something someone else has written. Can be a sign of trouble, or of a very agreeable writing team.

Topic evolution chart.Depicting how topics evolved over time. Topic is a cluster of words that frequently co-occur in a revision. Used Latent Dirichlet Allocation, DiffLDA for writing processes – for extracting topic evolution within a document. Bayesian and Markov models, not trivial technically.  It’s not in an archive of different documents over time, but revisions within one document, requires a few changes in the processing.

Get a chart showing relative weight of the topics over time. Nice graphics. Can see when topics appear, change importance.

Topic-based Collaboration Networks – who contributes to the topics. DiffATM. Node is a student, connection/link showing that they wrote about same topics. Can quickly see sparse networks – which nonetheless got high grades.

[The revision map is rich, but takes some expertise to interpret.]

Have used these on small corpus, but the techniques are stable on larger corpuses. Not just in the background – using R, but want to automatise them to serve as continuous feedback or at least on weekly level.

Question

Q LDA is currently the method for analysing textual artefacts. We’ve done that, found one problem. These are topics, they’re keyword clouds. Sometimes the granularity is not fine enough to see the changes. Changes below the level of these clouds that LDA give you.

I cannot say the details. The sensitivity, if there’s not a lot of revisions, it’s an issue. Had 1000 iterations to identify the prior probabilities, ran it 200 times in Markov model for practical reasons. Sure there’s things we’re missing. Used for indicators.

Q If you want novelty indicator. Also looking at dynamics. This may not be so good.

Technical questions, address in the proceedings for first author who can answer those.

Learning Analytics for Online Discussions: A Pedagogical Model for Intervention with Embedded and Extracted Analytics

Alyssa Friend Wise, Yuting Zhao, Simone Nicole Hausknecht

Not universal LAs, but particular for a single context – learning through collaborative discussions online.

Learning context – online discussion. Data type – process data based on clickstream. Timeline – in-process learning events, short cycles of feedback. Interpretation/action – instructors and learners making local pedagogical decisions. Very human-centric model.

Three challenges. Capturing meaningful traces, presenting it meaningfully, supporting interpretation. Model of online speaking and listening, embedded and extracted analytics, pedagogical model of intervention.

Need a learning model, what we think is going on. We’ve always had lots of data. Model focuses on how students contribute and attend to the messages of others. Lot of other work looking at the messages, but less on ‘online listening’ to others’ messages.

Social constructivist perspective; two basic processes – speaking (Externalising your ideas), listening (taking in the externalisations of others). A metaphor!

Students have high control over timeline of engagement online. Can skim comments quickly, decide what to read and which order. Could be reflective. But students have trouble with this, managing their time, especially when discussions are prolific. So want to help learners to actively monitor and regulate how they attend to others. If’ nobody’s listening, it’s just a bunch of people shouting in a room.

Speaking is mechanism for sharing ideas with others – recurring, responsive, rationaled, temporally distributed, moderate proportioned. Speaking is visible, but not all qualities are salient. Huge work done on post quality – textual analysis, complex to assess.

Listening – it’s the invisible part of learning through online discussions. Value in listening that’s broad, integrated, reflective. Early research suggested they’re really bad at it, but recent stuff says they have different strategies – coverage (read everything), interactive, self-focused (only their own stuff and replies), targeted (look at e.g. people they know have good idea). Different students do different things; give them pointers about that they can better self-regulate.

Metrics – many! Range, number of sessions, percent of posts read, others. Percent of posts read – Listening Breadth – taking this as a key. Notion of a read is important.

Data processing – mySQL query merging log and post tables, Excel VBA macros to clean it up. Reading and scanning categorisation – view actions based on maximum reading speed of 6.5 wps (=about 400wpm), read if spent the time, scan if less.

Simple table format as visualisation! Had lots of amazing, brilliant things … that the students couldn’t understand. Not very innovative, but very effective.  Your data compared to class average.

Extracted vs embedded analytics. This table is an extracted analytic – pulled out and presented back. It’s separate from the learning activity. Want to feed back within the activity: embedded analytics. In the discussion interface – alter it to reflect that info. Used Visual Discussion Forum – can see your viewed/unviewed posts, own posts in light blue.

The spider-diagram/starburst type vis is actually the interface. Yellow starting post, with discussions spinning off from those. Can see what you’ve read, and what you haven’t, and in what parts spatially you’re contributing. Students are asked to take collective responsibility for the discussion. The red/blue colour for viewing, will be different from % posts read metric – if you just click through, you don’t get high % read, get different responses from students.

Supporting Interpretation. How do we make those visualisations integrated, part of the pedagogy, very actionable by students and teachers. Danger of rigidity of interpretation, lack of transparency about process, optimising to only that which you can measure, possibly impeding metacognitive self-development.

Six principles as a pedagogical framework for learning analytics intervention.

1. Integration with the learning activity. Part of what’s going on, the goals.
2. Diversity of metrics based on learning model, multiple pathways
3. Agency in interpreting meaning
4. Reflection happens in explicit space and time
5. Dialogue between students in interpretation
6. Parity between instructor and students.

Trial context: Blended doctoral seminar with 9 students. 10 week online discussion, reflective journal with embedded analytics; extracted analytics added for weeks 5-10. Guidelines for participation, facilitation, analytics, given out up front. Interviews after course end.

Integration: Connect the purpose of the activity with expectation and how analytics provide indicators. e.g. ‘Try to read as many posts as possible to consider everyone’s ideas …” and guidelines on interpreting the metrics. High student buy in to guidelines/metrics. Hard to distinguish these, because they thought of them together. They interpreted metrics (in their reflective journals) it was in terms of the guidelines.

Diversity: Students found different metrics valuable, multiple pathways. Talked about working on different aspects. Highlighted lack of listening by some of the vociferous speakers, honoured the efforts of others. In the numbers and the reflective journals. Self-critique. Trust of the numbers important, calculation choice became important – range of participation, was exclusive not inclusive – if you’d gone in two days in a row it took a day away. Made them question the numbers.

Agency: Presented as starting points for discussion; class average gives reference points; set personal goals. Students found goal-setting valuable, multiple strategies. Validation, but surprises too, more interesting. Emotional reactions, e.g. clicked on everything but very quick. Hard data contrasting with their perceptions. No major ‘big brother’ issues. Involuntary propensity to target the average.

Reflection: Dual danger of omnipresent analytics – any time means never, distract from engagement. Rhythm for reflection, in-class time for it. It happened. In a single space, they talked about going back and forth. High self-awareness of meeting goals (or not). Did sometimes need support in making change.

Dialogue: Online wiki reflections were private but teacher could view. T made comments in first week as initial support. Started a conversation that continued, was really important. Audience for reflection. Students liked negotiation and contextualisation – personal explanations of choices, strategies, struggles. Instructor really helpful about how to change.

Parity: Turned out not so important. Instructor’s reflection was useful for initial model, but didn’t attend to it later. Students were Ok with having the instructor overseeing.

Future plans – tool development.

Questions

Q How did the analytics affect their behaviour towards the assessment?

Haven’t done that analysis yet. Next step, see did their interactions change, related to the assessment?

Q Was it stated that the software would use that?

We explicitly stated the purpose was learning how to participate in dialogue, analytics to help them improve not for grading. Grading on more standard and overall. Reflection chance to talk about it.

Caroline Haythornethwaite: How do you know the intervention, your implementation has anything to do with what’s going on? The messiness of the class makes it hard to extract the tool effect.

Yes, that’s hard. In first 5 weeks did reflective journal, had interface, but not extractive chart until second half. Hoping we can see there is or isn’t some difference in behaviour.

Emily: This is a lot of dialogue in the classroom. Online not the opportunity for that, how would it transfer?

Want to do this, it’s issue for scalability. There was no f2f dialogue, it was all in the wiki. All asynchronous, turned out to be important. Worked with similar assignment in a different class. Can have larger reflections, more like an assignment. Or peer support. Or reflective dialogue doesn’t happen so often. The audience was important. Maybe could be less frequent. That’s a challenge we’re looking at next.

Q Potential problem – counting number of posts and number of words, give idea that more is better. Especially in a grad class, teaching them to be concise, quickly and clearly making points, that’s a goal.

Two things. One, there’s better measures of dialogue we could integrate. More is always better we were explicit about not being good in the guidelines. The pressure was to be more concise and short. Dialogue was not doing more, but being more directed.

Q You were explicit about it as a pedagogy – don’t make the wrong assumption from these numbers.

Guidelines said here’s the metric, here’s the interpretation, here’s how to use it. For length, need certain amount, but think shorter and invitation to respond.

Understanding Promotions in a case study of student blogging

Bjorn Gunnarsson, Richard Alterman

Different context, same problems. This is in a blogging environment. The blog posts are homework. Everyone can read it. Post it before the due date, all can read and learn from it.

Added feature – can like someone’s homework assignment (=promote it). Different types – badges – e.g. nicely written. (=promotions) Want to determine if high-quality content is the promoted content.

Class on Internet and Society, four books, two posts on each book. Students have own blog, read others (aggregated in reverse chronological order, with the promotions shown), template for blog posts, and feedback. Assignments go in a cycle – do editorial on first book, editorial on second book – can adapt to how it was set up and improve. They use the data in their blog post for their term project. They also need to do it in a set of reflective posts that bring together the ideas from all the books.

Case study – promotions. They require just one click. Different from comments where you put thoughts in to words, heavy cognitive load. Incentive to create these badges, they show up in the view of the blog posts. ‘Like’ as a general overarching one. Do people promote? Yes, they did. Do they act on the promotions from others? Yes – more promotions a post gets, the more reads (monotonic increase of reads with number of promotions). Are the promoted posts the quality ones?

Case study – technology. At Brandeis University. Transition of the technology. Used the format, with viewpoint with comments. Added the social features. Transition from collaborative blogging to social blogging.

Grading – simple grading scheme – 0 not completed, 3 exceeds expectations. TAs graded each post. Questionnaire – six questions on this 0-3 scheme. Less prone to grading errors or miscommunication. Specialist grading view, not influenced by likes/comments. Also peer reviews – part of questionnaire answered by student themselves. We had a lot of students, a lot of work to grade.

Evaluation

From 10 assignments (last ‘was an outlier’), 92 students, 15 filtered out for non-participation (some dropped the class).

They actively promote posts as they read. They promote high quality. Students are reliable – some better than others: some reliably promote poor materials (!), they get better over the semester. Promotions useful as highlighting mechanism, preliminary assessments for grades – grader just verifies community verdict.

Promotion feature used by 90% at least once. 6.77 ae posts promoted. There’s a decline in promotions over the semester – fewer students do promotions as time goes on.

Higher quality posts got more promotions; more likely to get promoted. All types of promotions were useful. [Slightly confusing data presentation of these results] Average number of promotions was higher for higher grades – R^2 0.6, p <0.002. General ‘likes’ were the highest correlated with average grade.

Feedback to students. Grade of promoted posts vs student grade; one student promoted only posts that got the highest grades, but only got an average grade themself. Doesn’t look like there’s much of a correlation here.

Review

These can be useful, for highlighting quality posts.

Questions

Martyn Cooper: How do you handle the timing issue? The timing of the blog post and the promotion by the student.

Once assignment out, they can go and edit it again, and any time you can ‘like’ what’s already in the blogosphere. Spikes of liking are always just after the assignments were due.

Chair: In this social network, sometimes the initial likes have a disproportionate effect, sometimes they’re just random preferential attachment. Did you talk about random variations at the beginning skewing the results.

There was a blank post that got lots of badges. But we didn’t filter that out, and still the data is Ok. We didn’t have to take that in to consideration.

Q:Take any precautions to avoid the bandwagon effect? Advertising and asking students to like them. In social networks, that’s very common.

No, nothing like that. We did not see that happen. It was completely organic. We might have wanted more people to do promotions. Some only used it once during the semester. Did not see that effect.

Naomi Jeffery: Am I right that you judged quality of the post purely by the grade? Or did you have other measures?

Measured by the graders in the grade received. Graders and instructor met and did a few together to get consensus. Same scheme as editorial review. On second term of grading, were in communication and agreement. With course grading scheme it was difficult to get from a 2 to a 3, it was clear what category it fell in to:

Naomi Jeffery: Could it be that the correlation is because students have a good understanding of what’s going to get them a good grade?

Sure, of course. That’s a part of our quality standard. Measure how clearly they state their issue, etc. There’s also an overall grade.

–
This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.