Liveblog notes from the afternoon at CALRG’s 40th anniversary, 19 October 2018.

Vision 3 – Teams can successfully teach any number of students at a distance

Patrick McAndrew

(Rebecca Ferguson is sadly unable to attend and present)

Berrill is a challenge for presenting – you can see the people in the room, yawning, and you also know that people will come to you the next day.

The OU is one of the largest universities – in terms of students, in terms of full-time equivalents. And coming up, we are nearly in to the top 20 of students studying full time!

Patrick shows a photo from Rebecca of me (Doug) at a statue showing someone pouring knowledge in to their head and reading a book.

Teaching is when someone acts to help someone learn. “To make someone less inclined to do something” – as in “I’ll teach you to throw rocks at my windows”. Amusing. But here, teaching is causing someone to learn.

What underpins that are a series of beliefs, with a lot of evidence: distance learning works, and anyone can learn. A more open approach to being a university can be effective. Distance education works. That was a challenge for this university. The push to ensuring we have sufficient quality has driven us for many years. One was the formation of an Institute of Educational Technology. Being effective at teaching at a distance, and using technologies. Now synonymous with small devices and computers, but more broadly is how we approach how we teach. Our belief is anyone can learn. We are open to any student studying with us. Have put up work from the early 2000s with a model of pass rates, connects to Bart’s work, predictive models and machine learnings. What can we predict about what the student body does? Jim Peard has done this work, annual update. Three groups of factores – student previous experience, student motivation, module difficulty/match. One key factor is whether they have previous experience of study. As soon as they’ve done one module with us, their previous experience drops down the list of predictors.

Foundations on Supported Open Learning. But also understanding how we design. Derek Rowntree’s work. A lot of work on communication. If you’re not familiar with Diana Laurillard’s conversational framework, get familiar with it.

Also, working at scale. Currently (2011, UNESCO work by John Daniel), there were 165m people in tertiary education, projected to peak at 263m in 2025. Would require more than four major campus universities (30,000 students) opening every week for the next fifteen years.

Support that scales. Several people have come to the OU to find out how we manage, the magic that enables the university to teach at the scale it does. The material design and the support. We have a resilient but not cheapest answer. Building in support, 1:20 or 1:25, if your process can operate at roughly that level, it’s probably going to work. It’s hard to make a resilient system that operates above that ratio. Enables us to give support to our students. A major part of what makes a difference for our students is the support from their ALs. High satisfaction levels around student feedback. The student doesn’t necessarily meet the person who gives the feedback, but we get the highest ratings for the feedback. Bart’s delivered another punchline – what we design makes a difference. The learning design matters.

Success at (moderate) scale. Martin chaired one of the first really large OU courses – 15,000 students studying for credit. T171 You, Your Computer, and the Net. 55% of OU students in England come from disadvantaged backgrounds, only 29% of full-time students do. This was an era when things were able to expand more easily, we could grow supported by the systems around us.

This was also when we started to operate at a truly massive scale – OpenLearn, launched in 2006, hit the 60m visitors mark last week. Open University on YouTube and iTunesU. We have more and more routes to get content through to people. The design is important. We’ve taken away one of our pillars of success – the support – and that means we lose some of the resilience and certainty in what will happen.

I said I’d come back to some of the challenges. There’s the challenge to us: for PT entrants to study in England, there’s been a 61% drop since the introduction of the high cost student loan system. We are no longer as big a university as we were at the end of 2010. The other set of challenges is what happens to our students – from the number who start to register, then who confirm they are starting study, then actually starting, then paying their fee, then keeping going, then completing, then passing. Not everyone makes it to the end point of that journey [something like half by eye from his bar chart].

We are changing our approach. Work on the right start, so students find out in the first three weeks they’re on the wrong course, we haven’t necessarily supported them well. Targeted communication, bridging from one piece of study and the next, and different measures of success – a degree isn’t the right qualification for all our students – a Certificate in HE can be a major achievement.

There’s also being more radical. THe OU was radical in the beginning, and a radical move in 2012 setting up FutureLearn. Are universities the right structure? Changes in support models? Alternative pedagogies? Different attitudes? This is beyond our control.

Pedagogy at scale through MOOCs – led by Mike Sharples. And Rebecca Ferguson has led work looking at what research tells us, within the OU’s own FutureLearn provision. A lot of the lessons are the same: make use of effective distance learning pedagogies; offer well-designed assessment. The largest ever MOOC was the British Council English language course on FutureLearn, with (from memory) 500,000 students on a single presentation.

Rebecca’s also looked at how educators at other univerities experience developing MOOCs. Developing expertise: develop educator teams; identify and share effective learning design. FutureLearn makes the communication element more visible. How do we make this part of the education ecosystem? People still want accreditation. We’re working with FL towards accreditation.

Other pieces of work. Beck Pitt has been working with BizMOOC, to produce a MOOC Book. Shi-Min Chua, how people communicate and how we can support discussion at scale. Francisco Iniesto, how accessible are MOOCs, looking at producer, and learner perspectives. Tina Papathoma looking at the educator perspective.

We are struggling with how to maintain our scale in the current environment. Reaching out to the world, carrying out good research to support it.

Questions

Tim – What’s the claim for superiority of FutureLearn compared to others based on?

Patrick – It is owned by the OU [applause and laughter] It has a different approach to pedagogy. More involving, can see it impacting. It’s the pedagogy, the ownership, and the partners.

Eileen – When we visited Coursera, they were interested in FL’s good design for mobile devices. Because of the timing, it was a year after Coursera, and understood that mobile devices were important. Also, the idea of pedagogy around social learning at all. Nobody else had thought that through. Mike has an interesting slide on the use of the forum equivalent, and the forums not being separate but in the design. Also, Diana says it’s the best one.

Diana – From experience.

Vision 4: Learners engage enthusiastically with science learning

Eileen Scanlon, Thea Herodotou

If you join an interdisciplinary group, your birth discipline can leak out. Long list of collaborators, and past projects. One was Collaborative Learning in Primary Science – originally turned down by a reviewer who couldn’t see the point of learning in groups. Conceptual Change in Science, Alternate Reality Kit, The Talk Factory, Personal Inquiry.

Many theses – from Eileen’s own (Modelling Physics Problem Solving) to Citizen Inquiry in informal settings. We’re digitising our theses, and are going to make that avaiable online.

Some old photos from news in schools, an original Mac SE.

Considering a trajectory for new learners – informal to formal, passive to active, solitary to sharing, learners to teachers. One key step on informal to formal is the Open Science Lab. Originally, a mix of experiments to support formal teaching, but also informal science, an iSpot link. Use of technology to make such experiences available to others is important to us.

The PI project was a bit of bricolage. 2008/9 was just when mobiles were becoming available to help students connect experiences outside the classroom with those inside. Developed an approach to personal inquiry, could be an interest in aspects of yourself, of your environment, or of your community. Personalisation, and inquiry learning needs a lot of scaffolding. Outcome includes a good book, and exhibition at the Royal Society.

With the work for school kids, in the first instantiation of the OpenScience lab, we did some experiments, with Simon Kelley, around moon rocks, to see if the approach worked pedagogically in OU teaching. Just because you’ve worked with school children and technology, doesn’t mean they’re not applicable in OU teaching. We, CALRG, always took a broad approach to which approaches would inspire us and apply to our work within the Open University.

Another experiment, also with Simon Kelley, is some experience with the Virtual Microscope. Thea and Maria Aristeidou (and Mike and Eileen) have done some evaluation. It aimed to explore how students use and engage with virtual microscopy, and what teaching approaches work better for students who study practical science. Contrasted blended learning condition with wholly online one. Mixed methods study, learning analytics, survey, interviews. Blended students were more satisfied with how the virtual microscope is integrated in the module and greater perceived learning improvements (observation skills) than online students. There was a big difference in how it was used – in blended, it was introduced by a tutor and was complementary to a physical one. Online it was the only way to see the images; the learners felt a need for tutor to complement the activities.

Several projects on science learning across settings – Citizen Inquiry. Between citizen science and inquiry learning. New book on Citizen Inquiry, recently published. Active engagement in science, citizens defining their own research agenda. Not just natural and physical sciences, can be social and applied sciences. There are 12 case studies of citizen inquiry.

Started with nQuire-it project, funded by Nominet Trust. Website, and Sense-it application, can capture data using your phone. Whatever data you capture can be uploaded to the nQuire-it platform. A community developed around weather, Maria Aristeidou led. Also worked with the Centre for Policing Research and Learning, to help police understand what the community needs. Latest collaboration, was with the BBC Tomorrow’s World, redesigned the platform. It has open and closed missions, and others. Presented at the Royal Society Summer Science Exhibition. An example – My Life, My Data, #MyTomorrow. Run with the BBC.

Latest work, collaborating with organisations across the world, NSF, Wellcome and ESRC. Three museums – London, SF, and LA, and UCD, OU, and Oxford. LEARN Cit Sci . Museum-led citizen science programme, coastal biodiversity, uploaded to iNaturalist.

The vision – learners engage enthusiastically with science learning. Open learning, inquiry learning, mobile learning, enabling the learner’s transitions between formal and informal settings and contexts can offer benefits for science learners.

Questions

Andrew Ravenscroft – the word ‘disadvantaged’, how do you define that?

Thea – Chidlren or young people who don’t have opportunities ot go to museums, or families not well educated. Museums have activities to go to those communities, to raise their science capital.

Kim Issroff – How science has changed over 40y, and how pedagogical approaches have changed.

Thea – Eileen?

Eileen – A good point, particularly in relation to last points Thea made. Citizen Science as a credible activity. Muki Haklay was tweeting furiously because people were questioning the quality of cit sci data. The notion of being an active science learner has been around for 40y. One of the issues around practical work experience is engagement with practical science became difficult in the secondary curriculum. Noticed as we went round schools, we worked with geography rather than science teachers. They enthusiastically bit our hand off for pilot projects to their whole year groups rather than just 20 or so students. They had got in to science investigation as a way to help young people develop their understanding of geography. They assumed students knew what a hypothesis was. In science teaching, a move away from data collection. There were enquiries, but the activity was managing data that had come from elsewhere. Interesting mismatch cross disciplines about what we assume children will have learned in different contexts. Anyone should understand what a scientific investigation might deliver from them, whether a community group measuring pollution in a creek, or a group of kids deciding they wanted to look at how food rots if you leave it outside a fridge.  Science as even more relevant to daily life. How we include people has changed over the intervening period.

Thea – Research shows people find STEM as not interesting, or boring, or for the gifted ones. What was happening was only a few were involved in science. It wasn’t accessible to the citizens. The idea developed that science was just for a few. We are trying to show there are way to learn science, and give the support to do so.

Tim – You’re too diffident. You can be stronger about the claims you make for nQuire. Look at Zooniverse. Citizen Science and nQuire work is an unambiguous success. The question is how do we get citizen mathematics, or citizen philosophy. The one that has the unambiguous tick is the collective study and engagement in science.

Eileen – We’ve managed to broaden the types of inquiry. Koula has used nQuire to do cultural heritage enquiries. There’s a lot of interest in our work from other museums than just science museums. That’s a strong direction the work might go in.

Diana – The Muki Haklay, it was about data that is used as a research output to demonstrate things about London. There is an issue about how oyu assure the quality, comprehensiveness of the data you collect. You’ve been talking about citizen science as an educational process. But when it’s a process of data gathering, you do have to address data quality, but he didn’t.

Eileen – We do both with iSpot – it’s for learning, and for real science data gathering. The solution is to have seeded experts on your platform to help with identification, then by continuing participation on the platform, through a reputation management platform, develop a more robust way of QAing the observations.

Janice – Before we launched iSpot, we engaged the expert biological recording community. A big job was signing them up, getting their expertise on. That’s the model we’ve used. Get the experts on board. They assist with the verification process. We badge them with the specific expertise they have.

Eileen – In 1975, as a science faculty course manager, we had measurement of SO2 pollution. People filled in a form, we sent them kits where they measured SO2 in their local area. They sent the forms to me, I checked the readings were at least within 100 of where they should be. We had a live BBC programme to report their feedback every year from 1975 onwards. Paper in New Scientist that credits the cohort of OU students.

Thea – With Zooniverse and iNat, partners on LEARN Cit Sci. On iNat, developed a machine learning technique that gives you ideas about the correct answer. Zooniverse, the identification is cross-checked across at least three volunteers.

Diana – That’s a much better answer.

Allison – Do you think the fact that questions, or tasks in science can generally be broken down in to smaller ones, so many people can come together, makes it in some ways simpler to have citizen science rather than citizen philosophy? How do we move forward to bring this together?

Thea – We try to break this down through the design on the site. We structure the steps of the inquiry process. The challenging thing is what kind of questions they can put down – so they’re do-able and can be answered. So young people had ideas about going to space and doing things that couldn’t be done. It needs some scaffolding from experts to narrow them down to link to a conclusion.

–
This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.

The Open University’s Computers and Learning Research Group (CALRG) is 40 years old! To celebrate, we’re having a one-day conference, on Friday 19 October 2018, in the Berrill Lecture Theatre, The Open University, Walton Hall, Milton Keynes. These are my liveblog notes.

Welcome & Introductions

Eileen welcomes everyone and says she’s acting as air hostess, pointing out the exits and so on. She’s had many nice messages. When you look at the collection of old photographs outside, if you want to contribute to the yellow notebook, please do. The goody bags include 10p off coffee!

Mary Kellett – Acting Vice Chancellor, OU

Mary appears in a recorded video. She welcomes everyone. The CALRG has a special place in the history of the OU. In 1978, the OU was still young, and only in its 8th year. From the outset, the potential of the computer to enrich our students’ experience was recognised. [And then there was a technical hitch that made the video stutter and freeze.] The OU is the C20th’s greatest educational experiment, so an ideal testbed for the judicious and imaginative use of computer to help our students. A strong international reputation for international excellence has persisted and flourished. A large cohort of PhD students is crucial, so a special welcome to current and past students who’ve come back today. The group has an enviable record of external funding, well-cited publications, and contributions to the success of the university in research assessment exercises. Their unique position comes from being in a university whose raison d’etre is around using technology in education. Congratulations and have a wonderful day!

[There was some informed discussion in the crowd about the nature of the problem, and the irony of this group being disrupted.]

Hazel Rymer – Pro-Vice-Chancellor, OU

Welcome everyone to this 40th year of CALRG activity! In the room and online. We’re celebrating the story of comimtment, sheer bloody-mindedness, to developing the reputation of the OU for world-leading educational technology. It rests on current and former members who have spread the word nationally and internationally. Welcome particularly our keynotes – Prof Diana Laurillard and Prof Neil Mercer. The group was started a cross-university group of staff, led by IET. First director and founder of the group was Prof Sir Tim O’Shea. Fun stats: over 100 PhDs awarded, at least 10 former students hold professorships, including the OU’s only Regius Professor (and the UK’s only one in education). Excellent results in five research exercises. Not (just) about looking back, but forward. The programme is about visions that underlay the group’s work. Let’s get on with it!

Eileen Scanlon – Regius Professor of Open Education

Thanks for the kind words. Forty seemed quite special, so we were a little more ambitious. We’re looking at work that has persisted over 40 years. We have time for four visions today, but there are a lot more than that. I do love the quote from Voltaire – history is a pack of tricks you play on the dead. We’re playing a trick in trying to retrofit some group visions. You don’t have to agree with us about them, but it’s an attempt to link the past, present and future of the group. We’re hoping it’ll be interactive, and where you’ll disagree and be quite forceful. Tim O’Shea and Ann Jones are the only two from the founding group who are here. Tim, in his interview at the OU, was recruited to work with the Maths faculty, but was told not to do any of the educational computing he was famous for. Within a year we had CALRG. Hazel mentioned bloody-mindedness, it’s a great trait across the group. There’s also the sense of community, of sharing endeavours. Students are the key glue that helps a research group. One common acknowledgement in theses was thanking members of the research group, “past, present and future”.

Four visions, located within the OU.

• Vision 1 – Learning is accessible for everyone. A strong line here. Not just for people with disabilities. Openess, inclusion.
• Vision 2 – Adapting teaching.
• Vision 3 – Learning at scale. The biggest course the OU had was about 15,000 students. So this notion of scale and reach is one we can explore.
• Vision 4 – Science learning. Where Eileen herself started. “Teachers will bite your hand off, Eileen”.

What matters? Many varieties of openness. Transparency and collaboration are key indicators of the health of a group. This has been a cross-university collaboration, led by IET. We make a point of being open, transparent and collaborative.

It’s more than sheer bloody-mindedness. Background to the visions. In 2013, a cross-university group, worked on “Beyond Prototypes” (beyondprototypes.com), an in-depth examination of the processes of innovation in technology-enhanced learning. What makes a difference that lasts? Many case studies, looking at interviews with researchers and industry.

Found three things, key factors: persistent intent working through successive projects and and understanding of the complexity of the infrastructure round Technology Enhanced Learning (the TEL complex) and the process of bricolage. Bricolage is my favourite, it’s something like tinkering, the work you do is influenced by the tools available, it’s of its time – we push the cutting edge – but some of the choices are to do with timeliness in terms of the technology. It sounds a bit theoretical, which I’m not famous for. But in an in-flight magazine I found a definition of bricolage, so thought maybe it’s OK to use.

We’re going to try to produce a book, using persistent intent and bricolage as a frame on which to hang the description of our work.

It’s been exciting, but complex. Demands collaboration. Bricolage sounds like tinkering, creating tools that are no use in the future – but no, one of the key things in this area, even when the product you produce is of its time, the fact that you made your teaching ideas explicit, inspectable and testable – whether RCTs or more qualitatively – that’s what makes the area exciting. Make your ideas concrete, and people can inspect them. There isn’t a distinctive methodological approach in the group, which can be a problem, but so can sticking to one. Studying in teams that are interdisciplinary is part of the challenge.

Finally, a plug for OpenTEL. The CAL group is great, but I have to thank another group, which extends a little further, a priority research area, called openTEL. Has a lot of the features and people of CALRG, but draws even more people in to our work. A link between research and practice – the synergy between TEL research, and the mission and strategic interests of the University. There’s also our OpenAI group, and others. The openTEL group has funded the reception at the end of the day, so thanks to them for that.

Thank you.

Eileen then welcomes Diana. Her contribution to the OU and HE and ed tech policy is huge. We’re delighted she’s back. We used to pack CALRG in to a room and have tea and cake every day.

Keynote – Diana Laurillard

Reflecting on the CALRG vision and its diaspora

Chair of Learning with Digital Technologies, UCL Knowledge Lab.

Thank you for inviting me back. It was such fun working here. Wishing you a happy birthday. 40 years! That was 1978!

An image to reflect on that, and what it was like, and it was this. A British Council visit to Yugoslavia, a conference on ed tech, I was to do a presentation on interactive graphics for computer simulations to teach science. I knew Tim and Eileen’s work. The idea was to take this to Belgrade. The technology hadn’t caught up with our ambitions. I took with me a microcomputer consisting of three enormous boxes. One was the computer, one was the screen, a gigantic TV with a CRT, and another with keyboard and other guff. The plane landed in Belgrade, I watched the luggage trolley with my boxes on top, and I worried it would fall off. Customs were suspicious, even more when I said it was a computer. They got more suspicious and feel on it with knives until someone from the BC explained what was going. You couldn’t show this with an OHP. You had to be bloody-minded to get your point across. Shows how foresighted Tim and Eileen were to setting this up.

I’m not a good digital archaeologist, rooting through my old CDs, it was hard to find stuff to characterise CALRG. The papers for the CALRG conference, I can compare by looking at CALRG papers up to 1995. Two word clouds. 40 years ago, it was the subject areas that dominated – science, music, etc. But today they don’t appear at all. Learning and education dominate, students loom large, and MOOCs replace intelligent systems. They’ve become the focus of my research, although I was skeptical. There were stupid things said about MOOCs, cruel myths. But if we as researchers explore and challenge what we can do, we get something really interesting.

Reflecting on those four visions, and how they are worked on here, illustrated by some things from the UCL Knowledge Lab.

Learning is accessible to everyone.

This makes me think about the UNESCO vision of learning for all. This requires something different to what we’ve been doing. We have to use learning technology to develop high quality HE on the large scale. The sustainable development goals – SDG4 requires 68m teachers by 2030. But technology can help. Teaching skills for digital age – government panic about AI taking people’s jobs, asking what can we do – one of the first thing is to help the teachers who are helping the students, continual change and innovation.

More online models to reach more key audiences? Transnational education, that’s ok for a few hundred thousand students who are well off, but to get to millions it needs online learning. British Council sees that too. TNE demand is currently 4m, but not commensurate with the 200m global demand.

Need to understand relationships between open universities and MOOCs. OU is lamentably undercelebrated within government. Other countries have copied what we’ve done here. Proven for undergraduates. It’s not free, but you get personal tutor support, get tutor discussion, feedback, assessment, and summative assessment. But MOOCs are not suitable for undergraduates – it’s free so no personal tutor support, relies on peer discussion, feedback, assessment. But it is highly suitable for professional development. Teachers value this very highly. They trust their peers, the discussions are vibrant because they know what they’re talking about. You don’t expect and exam and qualification from professional development. Researchers value the pathway to impact on end-users. MOOCs have a role to play.

It’s a two-step cascade model. In the development context, there’s a cascade model – run work for experts and teachers at national level, then a long cascade where there ends up with a pale reflection at the end. But from a MOOC with 10,000 professional participants, if each of those trains 25 local students or adults, that’s 250,000, via blended learning. That’s how we can get to millions.

Teaching is adapted to meet learner’s needs

We developed MOOCs on FutureLearn on blended learning essentials, funding from UFI charitable trust. Running for 2y, 25,000 teachers engage (double that registered). 60% in the UK. The most successful prof dev in the FE sector ever. Has brought in HE teachers too. A way to get to teachers, they get a window on others’ practice. Access to tools and resources, talk to each other – they’re really cooperative and supportive. We learn a lot too. We can take this further e.g. with Padlet, sharing practice. We can explore the real contexts of our learners, which is incredibly important for the research.

Digital competence, technology-enhanced teaching for teachers, a different project. Progressing Technology-Enhanced Teaching – MENTEP. To enhance the self-assessment tool, we created this MOOC, with a form to submit their learning design and how that matched to the competence framework. Again, teachers share what they’re doing, and contribute back. Teachers are isolated in their innovation, but on these platforms they can share. Collaborative knowledge-building, the entire workforce needs to be engage.

Then in research contexts, we have a project looking at refugee settlements, in Lebanon. RELIEF project (Refugees, Education, Learning, IT and Enterprise for the Future). The project us using MOOCs to develop community researchers and teachers professional development. Work with them, what they really need. They said classroom management, so we’re starting with that. This looks like it will work in these challenging refugee camps. The wifi goes down for a few hours a day, but you learn to work around it.

The co-design approach means to engage, co-design, blend, embed it in how they already operate, and work it so it becomes self-sustaining, because we will fade away. That has to keep iterating.

So this is when teachers get support to develop the community knowledge of how best to use computers in learning.

Teams can successfully teach any number of students at a distance

Team of researchers create a MOOC with 10k local professionals, each of them works with 25, those 250,000 people engage – that enables working with them, not dropping stuff on them out of a helicopter.

Some tools to do this – one is the Course Resource Appraisal Modeller (CRAM). Helps people understand what they need to go online. The cost models are different – large initial outlay, but amortised over many years of teaching.

TPD@Scale Coalition – many moved. We hope there is much help for teachers out of this collaboration. Reports often say that teachers or teacher education should do this or that, but rarely much on how they will.

Learners engage enthusiastically with science learning

Yes they do! Not just the teachers bite your hand off, the learners do too. We’ll here more about that through the day.

The CALRG diaspora

It must be huge by now. I want to end with a rather self-aggrandising claim. Two incontrovertible statements, and a conclusion.

There is no clearer force for good in the world than education.

There is no way to provide education for all except through digital technologies.

Therefore, educational technologists are doing the most important job in the world.

Questions

Rose Luckin – Two questions. A slide at the beginning, and near the end. The outside world that we have to cope with. The slide that showed how things had changed, from disciplines to learning. The curriculum doesn’t change, it’s still disciplinary. We’ve changed but it hasn’t. The second is to do with funding. So many initiatives that are really good, do engage in participatory design, still flounder in that transition to scale. Struggle to see how we impact more. How can we do more of that? Second is not related. At what point do we as researchers admit that we’re wrong. It’s hard to publish things that haven’t worked. It’s difficult to accept when we haven’t got things right. Is there a contradiction as researchers?

Diana – that was at least three questions. [laughter] Curriculum – I don’t agree, I think it has agreed. [I was thinking of schools] In university it has changed. In schools politicians get in the way, we get inappropriate things demanded. How we affect those things is part of what we do in co-design, a stakeholder mapping. In change in education that’s always the education ministry. In Lebanon, we start with the guys at the top, and they are all guys, then we work down to the people who do things, we work with all of them. We need them in the MOOCs as well. The co-design works with all these different groups and sometimes mixes them up. When you hear the stories about what it takes to do anything in the camps, the people from the ministry learn something they don’t know. Sustainability is absolutely critical. Our blended learning essentials we started on day 1 working with government and agencies to take it over, wouldn’t cost much to keep it running. They came back with it’s not in alignment with our strategic plan. Unbelievable! What can you do? I spent 3y in the Dept for Education and Skills. No point using government to make viable change. To make things work, had to go back to academe and work with the teachers. They know stuff, they’re committed. That’s the best I can offer.

Ray Ison – On the viability model, I’d challenge you, the reason it’s working is the FL platform provides innovative space because it’s not settled. Why aren’t we as the OU doing what you’re doing ourselves? That’s taking a design turn, absolutely the way we have to do it. Are we not being blinkered by our own adherence to the orthodoxy of named degrees and education meeting prescribed means from others.

Diana – that’s what universities have to be, the market creators of new knowledge, that’s why research is so important. The OU has to take credit for the existence for the OU. How you use it is up to imagination. On co-design, the professionals know stuff we don’t. A perfect design cycle. With younger students it’s more difficult. Different with students than professionals.

Vision 1 – Learning is Accessible for Everyone

Martin Weller & Kate Lister

Kate starts. Four parts – Martin with a video on the Revenge of Open. Then Eileen on remote labs, me on inclusive practice, then all of you engaged. Four themese – making inaccessible.

Martin – Revenge of Open

Starts with a Star Wars crawl. We should embrace the woolliness in the ideas of open access – it enables us.

Different models of open access, illustrated with hats.

He starts with a horned helmet. The traditional model. Open entry, distance education, modular, part time. They’re radical. Ed tech startups seem to emulate it. Like the Vikings, hence the hat, it swept the world.

Next a white curly wig. Another interpretation is boosted by the Internet, Open Educational Researchers. People can do new things with it, as well as accessing it. It’s a legal (hence the wig) as much as anothe process.

Then a yellow builder’s hat. MOOCs. The cost is seen as a barrier, but limitations on rights we didn’t see with the previous. The cost of removing the cost barrier is a loss of support. Constructing barriers.

Then a face mask with respirator. (Chemist) This is mixing everything all together.

Finally, (with no mask) open access takes many different forms. We shouldn’t think of open access as one thing, but as a group of related practices. We should constantly review these, always refine what it means to be educators and an open university.

Eileen – Remote Laboratories and Remote Access

We actually haven’t moved on from thinking about science. Past work – Practical Experimentation by Access to Remote Learning (PEARL) project, Enabling Remote Access, OpenSTEM Lab.

Acknowledgements – small list – many people involved.

How does the introduction of technology change science learning?

There’s a practical problem with practical work in science. Home experiment kits – lasers in the post – or, as the Daily Mail had it, Death Rays Through The Post.

Martyn Cooper, PEARL project, EU Framework V 2000-2003. Providing access at a distance. How to turn some of our introductory science courses in to something people can experience at home or elsewhere. Other partners did manufacturing engineering and electronics. But at the OU, took the physics experiment from our programme – spectroscopy – previously had to build your own at summer school. If you think through how to deliver it at a distance, you improve it for everyone. Helps accessibility and inclusion. From a distance, use a robotic spectroscope set up here in a lab, could do work they would’ve done had they not been unable to attend summer school. This was a complex task.

That is a forerunner of a lot of the work very successfully developed by the Science Faculty, now the STEM Faculty, with a grant from the Wolfson Foundation. History – home experiment kits and broadcasts, residential lab-based classes, on-screen interactive experiments and instruments, livestream experiments, remotely operable analytical instruments.

Another project, led by Trevor Collins, trying to make geology fieldwork. ERA – Enabling remote geology fieldwork by transient wireless networking. Opening practical science for everyone. Persistent intent. Ancient Mountains course (SXR339), came from a course team query about setting up an alternative experience for learners who couldn’t attend. Started in what geologists need to see. Developed idea of the remote activities model. Students brought as close as they could be, but communicating with others. Currently this is working in Access Anglesey. In 2017 had an accessible field trip funded by the NSF, 2014, 3.5d field course on hydrology and meteorology in the field.

Out There and In Here – led by Anne Adams. Cross-university team. EPSRC funded project. Collaborative teams working in a mixed environment, some ‘in here’ working in a command centre, and others out in the field – ‘out there’ – who send and view images. Bringing lots of information together from different sources.

These are examples of the TEL Complex – you can look at what is going to be possible with near-future technology. Persistent intent, bricolage – but also interdisciplinarity. This was like that.

Kate Lister – Evolving inclusive practice

How we’re applying the lessons we’ve learned to our practice.

Another project – EU4All. Who’s heard of this? [most people] Large EU project, 13 partners. Wide stakeholder engagements. Two outputs are a model of professionalism in accessibility, and a learner-centred framework for personalisation f content and service. Three lessons: accessibility requires multi-faceted, multi-stakeholder approach; accessibility is continually evolved – persistent intent is required; understanding user needs, experience and preferences is key.

Three examples where we’re taking thees lessons and applying them.

SeGA – who’s heard of this? [almost everyone] A small team, in LTI, with a goal of embedding accessibility. We connect people to a wider community, multi-stakeholder approach. Has been around for 8y, has revolutionised how accessibility is operationalised at the OU, innovative in the sector.

Second example – the IncSTEM project, led by Trevor Collins. Aim to scale up examples of inclusive practice to the HE sector. 8 OU case studies/mini projects. It’s a huge project. Will take the example of the online practical work, the OpenSTEM labs. The virtual microscope is one of the examples. Can look at an object (e.g. a Martian meteorite), in various orientations, and in the microscope. Working up practical previews – a chance for students to try the tools off-module. A professional demo, then hands on with expert support, time to ID and address accessibility issues, and time to ID and address tech/skills gaps.

Third – understanding students, ‘Our Journey’. Students have diverse backgrounds and goals, but hard to evidence the barriers at scale. Collaborative development. Represent student journeys from the student’s perspective. Shows how various aspects impact on their learning. Can be used online or face to face. Data at scale or guided reflection.

Themes covered: making the inaccessible accessible; making things better for everyone; designing for diversity; breaking down barriers.

The future

Take 3-4 minutes, talk to those around you, pick one of these themes, ID issues you want to see on the research agenda.

[discussion]

Tim – Everyone talked about scalability. That ought to be on the agenda. Eileen’s thing with the rocks, brings to mind many games where people are in virtual environments, or literally running round the streets of Dundee. One of the places to learn from is the big successful multi-user computer games.

Someone – broadening term of accessibility, to include English learners, people from diverse backgrounds, making learning content accessible to them.

Do feel free to communicate more ideas to us!

Vision 2 – Teaching is Adapted to Meet Learner’s Needs

Ann Jones & Bart Rienties

Ann starts. As one of the original members of the CAL group, I thought I’d go back to seek a framework looking at the early days, in a report from the first annual conference in 1981. Then Bart will talk about more recent work.

I’ll talk about three themes in the report – Models of learning, method for studying learning, and institutional research: evaluation.

A photo from 1979, with the CALRG sitting on the grass, in Cambridge, in the garden of APU.

Models of learning

Some were interested in developing production systems – for psychological modelling, computer programs consisting of a set of rules, ways of deciding between different rules, and an interpreter to run the system. A focus on collecting student protocol data, trying to understand what’s going on – but some of us didn’t progress to production systems. There was a strong relationship to supporting student learning, and a range of domains – maths, physics problem solving, and novices learning to program – my [Ann’s] interest.

My thesis work focused on novices learning programming. Has her report on ‘SOLO’. Participants came to the university, worked through the materials, on their own, then recorded talking aloud about what they were doing.

She showed some transcript data from a student talking about how they had gone about a programming task. Some of the more interesting examples are when they have difficulty.

Feeding in to teaching – it fed in to how the design of instructional materials supported understanding. CAL was also used in courses. Tim developed a computer game, used in a mathematical course – developing mathematical thinking (EM235).

Some of our systems were pretty advanced. The CYCLOPs system developed at the OU could capture written input as well.

Institutional research – we evaluated 3 tutorial CAL systems in Science and Social Science courses. (in the early 80s). Some used at summer school. Using questionnaires, observations at summer schools, interviews, automatically recorded usage, and student-completed log books.

Case study – Evaluating tutorial CAL. MCQs answered at home, went to study centre to access a terminal, would get further questions to answer which were more interactive. Use was lower than hoped, and dropped over time. Questionnaires were built into the tutorial. Overall, there were barriers to negotiate. Students had little knowledge of computers – hard to get on system. Other students around, some found it an anxious experience. Tradeoff of perceived benefit and hassle.

Another case study – work metallurgist – Canan Blake et al 1999. Evaluating another tutorial, this on in a game format, interpretations of phase diagrams, used at residential school. It worked well, most students bought a copy. Designed for individual use, but tended to use it in pairs.

Reflections on then and now. The drivers – understanding student behaviour, improving instructional design – are very similar. But in our evaluations we were looking at one part of the course (CAL), because we could change that. Research students were crucial part of group’s work, as now.

Bart – back to the future

Technology portrayal can be positive or negative.

Bart does a quick poll of those in the room – “In comparison to 1979, teaching at the OU is now adapted to the needs of students”. Most are neutral, but a just under half agreed or entirely agreed.

Another one: “With the affordances of analytics and AI, within five years teaching at the OU will be adapted to the needs of students – 36% agree, more neutral, quite a few disagreeing.

We’ll do a pre-post test.

OU Learning Design framework. And how the way we teach has an impact on student satisfaction and retention. Communication is the best predictor for engagement, and for whether they pass the module. Since 2016, have done amazing work on identifying what teachers are actually designing. Quan Nguyen took this further, mapping what the students are actually doing (on the VLE) and how active they are, and how that links to the learning design. You see linkages, and disconnects. Also, how we teach courses at the OU are widely different. Some are peaks and troughs, some are steady all the way. 69% of what students are doing in a week is determined by us, teachers.

Can also see when our students are actually studying. Compare successful to failing students. The vast majority students do not follow the course structure! [In the sense of what activity is done at what time. Some very clear trajectories. Also analysing what students are saying, we can predict emotions – positive, negative, neutral, mixed – just on writing. Our tool is much better than the other tools available.

So, we can look at what effective teachers do, look at what the best paths for students may be, understand what students ‘think’ and feel, and provide personalised feedback what to do next. But are we ready as OU to do this?

Return to the questions. Who would change their answer to the 1979 to now – not much change, but not much change. Perhaps a broader spread. Then will it be in five years?

Final question – what are you hoping that CALRG/technology will bring in 5 years to the OU?

Some suggestions: Design for diversity, replicability, larger inclusion of student-centred design, happy and fulfilled students, robust meaninful analysis of big learning analytics data, get researdh into practice, understanding students’ needs at scale, better evidence bout our treasured theories of learning, personalised, more active learning for time-poor students, research informed new regulations.

Discussion:

Stylianos – I see we have a better way of collecting big data, making meaningful assumptions about students experience of learning. We have gained a lot, but have we lost anything? Ann’s work on close scrutiny, precise social science methods, compared to big data.

Bart – yes. That’s why I like CALRG, we need interdisciplinary researchers. If the learning analytics people were in charge, you wouldn’t want to live in that society, because our view is so limited. You need that triangulation of perspectives. We had 40y of research, but lots of the problems were known but not empirically proven. Now we can show 69% of what students done is determined by us as teachers. We should get our skates on.

Ann – We do still bring in students to try to get a fine-grained view of what they’re doing, interacting with systems, or students with disabilities. We haven’t lost that.

Tim – When Ann and I looked at students using computer system Cicero, we were surprised. BCE – Bad Computer Experiences. e.g. the porter wouldn’t let you in, or you couldn’t get it to work. Another was emotional – feeling others watching you, or the computer would inform your tutor you didn’t really understand. That was helpful for the academic computing service of the time, which explained why usage was dropping off. All of this is sunny uplands. Is there not a proportion of students for whom the modality is adverse. Assumption is this is great for everybody. Is it?

Bart – You’re right. We didn’t show OU Analyse. We couldn’t understand, some students never clicked on anything and still passed. How is that possible, they’re not doing what we expect? But turned out they were in secure environments and couldn’t click. Vast majority of students are not following the course structure, but we focus on it.

Tim – Methodological problem, still around. Students are instrumental. Ask them why did you do something, they’ll answer. If you ask them why did you not do something, they’ll say they were too busy. Won’t say because thought might make a fool of myself, etc. How do you get underneath that to the actual reason?

Ann – We have some students who do not want to engage online. Some are on machines all day, last thing they want is to do that in the evening. A challenge for us.

Andrew Ravenscroft – Following on from Tim. LA, the problem is there may be crucial factors you can’t capture. We measure what we can easily find. Would you capture a moment of insight? Probably not. A cautionary frame round this. A lot of work on LA is less advanced on how you intervene to make things better. Can see students who may fail, too much attention on that and less on what we do to intervene. I have students who are aware of the difficulties, but getting them to improve their performance is really difficult. We need to focus on those interventions.

Bart – Has to be actionable feedback. Giving the feedback to teachers helps them give feedback. Giving feedback to students can help them if – and a big if – they are able and willing. So how can we provide really well designed activities where they’re willing to engage with our feedback. Maybe we should think about how we collect feedback, so we know before the feedback. We can do so much better at the OU than a one size fits all solution. We have the power in the research community to help.

–
This work by Doug Clow is copyright but licenced under a Creative Commons BY Licence.
No further permission needed to reuse or remix (with attribution), but it’s nice to be notified if you do use it.

Liveblog notes from Friday morning at LAK16 [full list of blogging]: Robin Mislevy keynote – A Dispatch from the Psychometric Front.

Continue reading “LAK16 Fri am: Robin Mislevy keynote”

Liveblog notes from Thursday morning at LAK16 [full list of blogging]: Paul Kirschner keynote: Learning Analytics: Utopia or Dystopia.

Continue reading “LAK16 Thu am Paul Kirschner Keynote”

Liveblog notes from Wednesday afternoon at LAK16 [full list of blogging]: Session 1C MOOC discussion analysis.

Alyssa chairing. Says first paper might be really awesome, no bias. [laughter]

Continue reading “LAK16 Weds pm 3C MOOC discussion analysis”

Liveblog notes from Wednesday afternoon at LAK16 [full list of blogging]: Session 2B LA Challenges, accessibility and ethics.

Continue reading “LAK16 Weds pm 2B LA Challenges, accessibility and ethics”

Liveblog notes from Wednesday morning at LAK16 [full list of blogging]: Session 1D Analytics Visualisations and Dashboards. Stephanie Teasley chairing.

Continue reading “LAK16 Weds am Session 1D Analytic Visualisations and Dashboards”

Liveblog notes from Wednesday morning at LAK16 [full list of blogging]: – the opening keynote session.

Fantastic opening as Dragan, Prof Sir Tim O’Shea, and many others are piped in to the auditorium to start the conference.

Continue reading “LAK16 Weds am Mireille Hildebrandt Keynote”