CALRG Conference 2013 (2)

More liveblog notes from CALRG Conference 2013 #calrg2013 – second part of the morning on Tues 11 June 2013.

Patrick McAndrew: Becoming more agile researchers: experiences from researching Open Education Resources

Open society. Private option not available any more! As the OU we’ve been working in open education resources for the last 8 years. But it’s all accelerated recently. Coverage wide – but not to The Sun. Interest in MOOC over time (Google Trends) – exploding up from 2012 – but has now peaked!

It’s a CALRG conference – so need an activity theory triangle. Used to map out activity.

Looked at Research 2.0 (paper with Steven Godwin and Andreia dos Santos, 2009). Web 2.0 characteristics (O’Reilly) mapped on to research approach.

Then OLnet, Evidence Hub, OpenLearn, OER Research Hub. Spotting interesting things, finding the evidence.

Used various tools – Claims Garden (OLnet) – looked at claims, get feedback from people. Produced various visualisations to get messages across. OER Research Hub – exploring hypotheses (have a list). Setting out things agreed with the funder, interested in the policy side – issues we can explore. How do we do that? With OERchery – get collective input.  Two scales – important/not important, provable/not provable – ask to rate on that scale.

Working with range of collaborations in different areas – HE, Schools (K-12), Informal learning, Community Colleges.

Looking for mixture of eureka moments – we know this. Talking to people. Have to balance that with testing the ideas, collective input.

Shows image to inspire – small rodent called Ziggy who’s a bit agile and gets his reward.

Agile Development/ eXtreme Programming. People don’t know what can be done until you’ve done it. Do it very quickly, not spending a lot of time in planning.

OU Camden, FutureLearn developers actually are in the basement. Good exemplar of people following that process. Lot of turnover in staff. Clear on vision, what they’re trying to achieve. Language is about sprints – two-week targets. Have a spring board – sticky notes, classified by idea, not started, started, blocked, finished. Also do a sort of task-size bingo – how big is it. Or T-shirt size – S/M/L/XL, XXXXL. Doing a scrum – stand-up meetings, object passed around, person says 3 things, what did yesterday, what doing today, any issues. Really interesting to observe. E.g. someone saying they couldn’t progress today, so could progress. Burnup, aim at 100 points of work over 2 weeks, draw a graph.

So decided to do this approach for OER Research Hub team. Deliverable -> Spring backlog. Meetings -> Scrum.  Pretended every 20 minutes was a day – amazing how much progress they made. Built a shared, quickly-built collection of what they knew about it.

(Note to self: tell Patrick about Trello if he hasn’t seen it.)

Finding ways to find pointers, directions for the work we do. Being a researcher is changing. It’s becoming a more public job. Martin has good talks on digital scholarship. There’s an element of how we change working practice. Not just an individual rebel, but changing as a group, as an organisation.


Martin: As someone there, it was really interesting doing the evidence dump. Once you start, you think, could use it over here. Could produce a course in two weeks like this. Doesn’t work like this all the time. Longer-term stuff as well. Mix up with sprints.

That came out. Sprints tire you out. Occasional sprint, or whether it’s a permanent change in methodology – which it is for the software world.

Mike: I’ve seen advantages and disadvantages. Advantage, moves quickly, everyone up to speed, as new people come in; very structured, not just random group of anarchists. Difficulty is about user engagement. The problem with FutureLearn is that they’re still developing things, but not trying them out with users, being rather secretive, so not really feeding in from real users yet. So the user engagement isn’t working the way it should. Doing it for research, how do we do it? Users don’t always know what they want, have to inform with deep thinking.

If you look at approach, each sprint should have a product, delivered to the people who need it. FL is treating you and a few others as if we’re the people who need this, but we’re not. Haven’t made these public outputs. We have a blog about it. Need to do the last bit. This will give us something that can feed on collective input. An open sprint, do it as an open process. We have 11 hypotheses, we’ve worked on 1, need to work on others- ask world for input. A divisible process.

Tim: My experience is that in general there are two ways of doing things. Very slowly and carefully, the OU is famous for courses taken 2-3y to do properly. Or you can do it very very fast. I can think of things that people tried to change at the OU that still haven’t changed, similar at Edinburgh, complex network of stakeholders. Once in a while – e.g. in our case we did the MOOCs very fast. The six MOOC teams turned in to one, with a very high level of trust. If don’t have community, trust, it’s immensely difficult to do things at a fast lick – get lots of ‘just a minute, need to consult, has this committee discussed it, what does the pro-vice-chancellor think’. Do things either take a fortnight or two years?

We can’t afford to always live in the two years.

Tim: Most things at OU took 2-3y.

Josie: Intriguing thing is what’s the tipping point – for some things, like CBM, emergence of learning design as a method was the trigger. Not sure about Student Support Teams.

Tim: For us the tipping point was if we want to be first.

A move towards the more rapid approach. Actually the tools have changed. Tools for working in public are definitely there.  You end up with an impossible list of people to check before you do something. Case to do something in the right way, so people can check it while it’s working.

Tim: Trick is to say ‘it’s on the Senate wiki’.

Karen: Trust is really important, that’s when things can move quickly. Links in to something, you had mix of online an physical. Agile in software is very physical.

Almost a rebellion against teams in India, US, UK – more let’s get a team physically together in one place.

Karen: More and more we’re not physically in the same place.

The simple bit is, what did you do yesterday, what will you do today, share with a group of people – not including your line manager (!). (laughter)

Martin Weller: Surviving the Day of the MOOC

This is a Riff a GIF of David Kernohan’s Day of the MOOC poster for the Horror of the MOOCs assignment on #ds106.

#H817Open. Ran open course in Open Education – 7 weeks, informal, OpenLearn. Existing OU students taken out of comfy VLE to mix with informal learners. Had blog aggregator, was the best bit.

Loads of different tools. MailChimp to send out weekly email – this was the killer app. Cloudworks for badges. OpenLearn, Cloudworks, manual blog adding to aggregator.

cMOOCs, xMOOCs – H817Open was somewhere in the middle.

Wasn’t too big – 239 syndicated blogs, 911 posts, 4,689 new visitors. 49 prospectuses requested, 4 registrations on other OU courses.

Survey evaluation, just com ein this morning – completion rates are low for MOOCs, survey rates are low, so this is very small. Satisfaction 78%, MOOC average more like 74%. Peak age 45-54. Big peak of visits in the first week, tailed off over time.

Tech issues with Drupal talking to Moodle – 30s for page loads! Also not clear to enrol. Too much screen real estate used for other stuff, navigation not clear.  “I was trying to make a monkey be a donkey.” Badges in Cloudworks was nice, but required multiple registrations. Had to add the blogs manually, hunting for RSS feeds – some are very well hidden.

Learner issues – didn’t know who to connect with, information overload. What’s the right metaphor? Like a teacher with 50 essays – no, dip in, think of it as a stream you dip in to. Put aside 30 minutes and do something.

What worked – mixing informal and formal learners. Blog aggregator. Weekly email – sending one per week, Downes is daily, that’s too much, adds to pressure; sets the tone for the MOOC. Allowing flexibility, can catch up. Fairly independent, activities don’t build on each other. People liked badges – even for simple things like just give me an answer that’s not trivial. Live sessions good too – not many turned up, but the ones that didn’t liked to know it’s there.

What I’d do differently? The platform – not what OpenLearn does weel. Twitter didn’t work so well, but Google Plus did; so do more work to connect people on Twitter. Advice on reciprocity/engaging. More live sessions. Limit blog options – Blogger, Tumblr, WordPress – because RSS discovery automatable. Update content, but not much, largely the same. More multimedia solutions – many text-based, but the multimedia ones (choose your own tool) went really well.

What I’ve learnt: MOOCs are hard work. MOOCs are scary. MOOCs are fun. I believe in open. There’s a song there. I like learners owning their own solutions, in their own blogs.

Finally .. you don’t get poems like this at the end of most courses.

Key things:

  • Support. If you take the support out of teaching, of course you can do it cheaply. Peer 2 peer, community champions, paid moderators, platform based, encourage self-help, hashtag it.
  • Scale. At <1000 can create community. At >1000, content overwhelming. How do people find each other? How do they cope with overload?
  • Motivation. Get many different motivations – nosy, leisure, people turning up explicitly to moan: free trolls! Key achievements along the way; don’t want to damage people along the way. Highlight vital activities, pathways – don’t have to do everything, allow catchup. Hardline deadlines can work, but dropout. Encouraging messages, assessment and recognition – badges, certificates, etc.
  • Identity. Is the community the identity? DS106 is. Is it the academic? Does it go beyond the MOOC? Explicitly experimental? Don’t be scared of being publicly shamed. Be upfront. How would you craft an opening email?

MOOCs didn’t come from nowhere. Weren’t invented in 2012. Interlinking map of e.g. Open Education, OER, Connectivist MOOCs.

Battle for openness – open is not the same as free. Coursera MOOCs are closed. Announcement that Coursera will be an on-campus elearning provider – blended learning, oh wow. Retreating from open – e.g. FutureLearn T&Cs fuss last week. If claim to be open, people will take you to task. What winds me up: MOOCs as a solution to ‘broken higher education‘. There are narratives excluding higher education from the process, whole Silicon Valley exclusive focus.

Many interesting questions raised by MOOCs – openness, flexibility, smaller achievements, bridging informal and formal learning, technologies, building more automatic support.

MOOCs are our friends so long as we can answer these questions. It’s a mistake to ignore them, but also a mistake to panic. Make it clear that supported learning worthwhile. Understand that this is a crucial time in what it means to be open. We’re at the forefront of that.


Tim: The first third of your talk maps my understanding of where Sian Bayne and colleagues are. They use Coursera but are doing a cMOOC. Were using OERs, no talking heads, particular story. Fabulous one of Sian’s students produced a love letter. Can see the emotion. That leads to an important question – what is the identity? If studying a number of MOOCs, what’s your identity? Haven’t got our heads round that. Qualitative work by Sian. Users have really powerful issues of identity. The Open University is not free, never was. Not open as to people with no money. (Actually I think we are so long as you are in the UK.) But OER is free. VCs putting in 8-figure sums, this is not philanthropy. How does open relate to money? Maybe FutureLearn is a huge act of philanthropy.

John Pettit: That poem, reminded me of tutoring early MAODE module here. 1999. One student, at the end – was a high intensity module, lots of forums. Student left a song about her sadness at leaving. Doesn’t matter whether it’s a MOOC or a more open environment – the innovation, encouraged creativity. I did your MOOC for a few weeks but dropped out because life took over. I’d like a badge for dropping out, I did learn something, it wasn’t a wasted experience even though not picked up in any of the surveys. I entirely agree about the platform, would link that to the metaphor. Thought it was a course where you’re supposed to do stuff.

Eileen Scanlon: Informal learning and science communication

Hope that conversations will go on through the conference. Expected talks would overrun, so planned a shorter talk. Full talk at ICALT2012, on ORO.

Modelling problem-solving in physics, types of CAL that make a difference to science education. Then in a different world. Working to think through how the public might be engaged with science through technology. Few case studies of learning in informal settings. Reflections on issues, especially models of learning in use.

Engagement – participation in science, not the deficit model – ‘doesn’t mean turning everyone into a scientific expert, but enabling them to fulfil an enlightened role in making choices, which affect their environment’.

The oldies are the goodies – informal learning, complex area, hard to clarify – but e.g. Bruner 1966 Towards a theory of instruction. Get student ‘to take part in the process of knowledge-getting. Knowing is a process not a product.’ Also Bruner 1960 The process of education: interest is the best stimulus; aim to arouse interest.

Recent paper (Falk & Needham 2013, Factors contributing to adult knowledge of science and technology, J Res Sci T). Telephone survey, n=1018. People learn a little bit about science from all sorts of activities. Self-report. Informal experiences account for 39% of total variance – the plurality. Interactions between all, so suggests synergistic effects.

Groups that engaged with inquiry-based games at the Exploratorium significantly outperformed control groups in the quality and duration of several inquiry skills.

Different groups/modes of engagement. Indifferent. Concerned. (and two more).

Bevan 2011, four-phase model of interest – how interest developed.

Two brief things. Reporting contested science. Rick and I got some money looking at reporting of contested science. Example of being agile and sideways thinking in research. We were interested in what impact new findings reports would have on general population. One that everyone remembers – finger length and sexuality.

Looked at representations of contested science, exploring media influence. Bruno Latour – ‘ready made science’ vs ‘science in the making’. Stuff that’s already known, vs stuff that’s changing.

Worked with prompt material, small groups, write tabloid articles. Nice data showing the process of how these news products were developed. More interestingly, have the actual artefacts produced. They were quite creative. Creative artefacts can be very powerful, often overlooked, give us a handle on understanding learning.

When asked people where they got ideas – said they hadn’t seen anything on TV or newspapers, but when you had the artefact, you could trace it back.

Incidental learning – MASELTOV. Project looking at recent immigrants to the EU. Have an extremely complex way to map this – an Incidental Learning Framework. Informal learning of this type is massively complex to examine. Look at features of the model – the place, the tasks, the tools, the social support, the learning outcomes, the time.

How to measure informal learning is an unsolved problem, not least because of the complexity. We could work quite hard on thinking more broadly about evaluation activities around MOOCs. What counts as a learning episode? They learned something sometime that we managed to dip in to. When do you have to start measuring? What’s the baseline? What counts as learning? If saw examples in artefacts – e.g. things they’d picked up about modern genetics, but also about tabloid journalism. What counts as success? How good does a service trying to help you have to be? E.g. achieve proximal goal (e.g. catch a train), or more broad ones about increasing integration. Also hard to attribute causality. Our users will have access to many other sources.


Tony: A zillion! (Only allowed one.) Fascinated by MASELTOV. Who decides what people need to learn, how do they make that decision?

From two sides. Has to be the user of the service that decides. Has to be pull rather than a push. Could have a system that sends you everything it knows about train transport when you walk in to a station. But it has to be with the user of the service.

Tony: Massive implications. Who decides which bits of science the public want to learn about?

Interesting privileged position we’re in in universities. Who decides what we should research too? Talk more outside the room.

Patrick: All hooks up with the MOOCs thing. I’ve done three. I’ve ended up studying things because they’re there. Doing Coursera machine learning, with Doug. MOOCs are good at professional development. We’d like them to do something else. Careful not to miss what they’re good at – offering interesting challenges to people who want interesting challenges. But not what the OU does to develop people as learners. MOOCs are getting me back to programming, maths, rather than TV. Cross to social learning is a real challenge. A menu of things you might like to do is what it’s quite good at.

This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.

Author: dougclow

Experienced project leader, data scientist, researcher, analyst, teacher, developer, educational technologist and manager. I particularly enjoy rapidly appraising new-to-me contexts, and mediating between highly technical specialisms and others, from ordinary users to senior management. After 20 years at the OU as an academic, I am now a self-employed consultant, building on my skills and experience in working with people, technology, data science, and artificial intelligence, in the education field and beyond.