Blog

(Was on slideshare but lost during technology snafu.  Oops.)

What would we do if we were a 2.0 company? Use web analytics to explore student/customer experience to get them to come back.

Helps us answer: What are your students doing online?

T184 Robotics and the meaning of life – 10week presentaiton, two a year, tracked for two years.

How often do they come?  Using webstats: “Most people visited: 15-25 times” over a ten week period.  Breaks down – a lot of people visiting once, but is an artefact – but most ~50% visit ~2-3 times pw.

How many course web pages do they read per visit? “7.66 pages/visit” – again artefact at 1, 2, 3 pp but basically falls off as a tail (exponential? looks more like a polynomial)

How long are they around for? “11:19 minutes average time on site” – artefact of people popping in and out – huge spike 0-10s, but second peak at 601-1800 secs (up to half an hour).

When are they doing it? Mostly daytime, peaks at 12 and mid-evening 7/8pm.

Every day the same? Fridays are bad – big bumps on Sunday and Monday – but that’s when the assessment fell – generally Tuesday seems to be good. (Eh?)

What are they reading?  Mostly the home/landing page. (Duh) But then Calendar, then Search results (to find answers to assessment), then End of Course Assessment, then Assessment landing page, then ‘What is a robot’ which contains several answers, then CMA, then rest of course content.

How long are they reading each page for? Average 2 minutes – can see per page too. Spend a bit more on content pages than other ones – probably artefact from bouncing off home page.

Where are they leaving from? Mostly from ECA – 40% read that then go.  Robotlab is a common one too (27% leave them vs 13% site average) – practical activities described in those, so obvious they read that then go away and use robot (or simulator).  A high rating might flag up that students are unhappy with it.  Outliers and changes are good to watch for.

Does pacing work?  Do they do it like we intend – week 1, 2, 3, 4 etc in order?  Lovely graphs – generally they sort-of do, with nice curves that show that happening, in order, in sort-of the right numbers.

When do students do their assessment? (meaning look at the ECA pages) CMA visits peak around when it happens, then sharp fall.  But some before, and also a little tail afterwards – perhaps search results hitting the CMA page.  ECA – very few look at it at the start! (Blimey, not what I’d have thought.)  Builds up steadily to a huge peak at the due date.

(Wow – students are less strategic than I imagined. At least these students are – as Tony says, would be good to see if this happens with old lag students too.)

Should make it easy for students to get to the assessment – there are extra click involved from the home/landing pages.  Should take a single click – so here’s some user behaviour we can learn from and do sometihng valuable.

What computers do the students use?  (Fun pic of Commodore PET – Tony’s main home machine)  Connection – Most students are on reasonably fast, but 11% on dialup.  “We should encourage dialup to die.”  Screens are getting bigger – so can design screens differently.

Ingrid – can tool tell you what students are printing out?

Tony – could track printable pages, setup not quite right.

Ingrid – evidence that students go to pages and print them out, so how long they spend there is interesting.

Tony – SiteIntelligence – OU signed up for – bigger and better and can segment by PIs and whatever – can track every single page, not just ones with a code on.  Could pull out students who printed out and compare it with other poeoples.

JohnP – Any Data Protection Issues?

Tony – Google can see the URLs, but can’t sniff the content.  Adds to what Google knows about you, could tell you’re a student and add in to their marketing mix.  Google track everything you do.

John – Anything you had to clear?

Tony – Next question!  (laughter)

Steve Godwin – Lots of skimmers, a few detailed learners – does this give you median values rather than just means, because you get very long tails?  Also artefacts for the one-page users – e.g. people putting it up as their home page.

Tony – Google Analytics doesn’t give you raw data, but does some disaggregation.  The OU system does let you do proper stats and segmenting.

Niall Sclater chairing.

The OU Student Experience: How do we measure up across our diverse student population?

Context – Nov 2007 end of course survey.  Very short, QA instrument. KPIs for QI.  60% response rate, like National Student Survey.  15k respondents, 121 courses. Often have issues with ‘neither agree nor disagree’ point – target for followup.  Looked across range of groups, incl lower socio-economic groups (which we get extra funding for, so key for us.  Also low ed quals on entry.

Method – Concentrate on 10 Key Performance Indicators.  Looks at two KPIs where there was a sig diff: 7 of 8 groups differed significantly on: ‘Overall, I was satisfied with the teaching materials provided on this course’ and ‘The workload on this course was higher than I expected’

Younger students were more positive about the online environment.  Steady effect with age – down a few %age pts per 10y age band.  Under 25 64% satified with quality of online interaction, but only 50% of over 56.  (Why only 56 top band?)  Younger students – U25s – less positive about f2f than others.

Workload and scheduling more of an issue for youngest students (actually, correlation with age and keep-up and meet-assignment, but U25s pop back up a little)

Students with no formal quals more positive about the online environment – but again less satisfied with f2f.  ELQ students less positive – so perhaps not a total disaster for us.

Students from lower socio-economic background slightly more positive about the online environment.

Open comments – many nice ones about the online environment, just what you’d hope actually.

Younger students – VLE should be well received, students confident in the environment; difficulty not a problem but keeping up to date is; engaging with employers about this is important too.  No formal quals is a small group but positive; need to make sure get in at the right level.  Lower SEGs – slightly more positive online, struggle with workload and keeping up to date – overlap with no quals.  We have a very diverse student population.

(Wow – Alison on time)

R07 ST? – work with low SEG people – early data he’s getting about online learning – but a surprise that they are finding it.  Issue with the pedagogy of online learning – just giving them computers and broadband isn’t a solution.  Were you surprised they were receptive and positive?

Alison – Offers students opportunity to talk to each other and get support from each other.  Need to dig deeper and see which courses they’re on, why are they having a satisfactory experience and some aren’t.  e.g. training moderators worked well in one place.

s/o interested in prisoners – do you have data on that?

Alison – Don’t ask but can identify them.  Small group in the survey.  Could extract but don’t have it here today.

Pete – Widening participation – looked at financial support or not?  In Scotland, PIs set by funding council – our students are spread across all quintiles of the SIND.  Some very poor students living in very rich areas, even though the areas are very small.  Best predictor of whether you’re hard up is whether you’re on a financial award rather than anything else.

Alison – We have that but haven’t picked it out (yet).  Looked at social deprivation vs FAF for retention modelling, and FAF is a key predictor.  Can provide that info for you – but multivariate analysis too much to do in the time.

s/o – Do you have info on student w/o English as first language?

Alison – Don’t collect data (yet) – but is a question on withdrawal questionnaire pilot at the moment; thinking about it for next years.

s/o – Online tutorials – some students value highly, some quite successful, some run for a long time, hard to get those to be quite successful – info not transferred across the university.  More cross-course information, training ALs properly how to moderate online tutorials.

Alison – Real difference between individual courses – understand which courses are web-enabled, web-focused etc – and share info.  Find out what really works well, and what doesn’t.

Ingrid Nix – Experienced or new online learners?  Your response would be very different if it’s not your first course online.  Would be good to separate in to these groups – see what the range of responses are.  e.g. with older/more experienced learners are better able to formulate discussions and use forums.

Alison – Want to extend Courses Survey

Worried at being ‘introduced’ by Denise Kirkpatrick.

Scholarship in the C21st.  Making connections across all the PVC portfolios.  Building on the literature on scholarship.  Christine Borgman – Scholarship in the Digital Age – techs now a key part.  It’s “inconceivable that practices from 15 years ago are applicable now”.  Must invest more in staff development.

ICT isn’t just part of academic life, but part of life generally.  E Puny in Eur J Ed – new vision of ICT and learning is needed – including having fun.  (Are we having fun? I am – at least some of the time. Nice to have the VC say we should.)

Private sector challenge to universities.  Changes in Govt policy. Univs need to seek other markets.  Our brand not assisted by a proliferation of ‘open universities’ around the world, not with the same quality.

Competitiveness requires distinctiveness – a USP.  OU can claim quality of student exp at scale – upheld by NSS.  Can’t take for granted.  Harness tech to help.  Rep built on that, remains a strategic priority, need to work even harder to maintain.

Cost.  HE more central to the economy, and ‘massified’ – trying to share cost with beneficiaries and contain the costs, not always sensibly (ELQ!).  Tech has added to the cost so far but has the capacity to cut costs too.  Sharing content and even staff and services in much more constructive ways.  In our interests to engage with that sooner rather than later.

Clay Shirky – Here Comes Everybody – wow, the VC is reading the right stuff.

Bill Gates – increased demands for education have strained the system; but education is the cornerstone of economies.

Our mission never more relevant or urgent.  Our best minds must be focused on harnessing the new techs.

What other university would focus on knowledge media and devise new ones, harness them to improve learning and reach new learners? The very stuff of our scholarship.  Covers Boya’s scholarship aspects. [Rhetorical question but I think the answer is: most of them, to a greater or lesser degree.]

Fundamental shifts in HE including in staff resources.  HE undergoing radical shifts. Schuster and Finkelstein book – change is unprecedented – four megatrends.  One – changing nature of what academics do.  Foundations of economy are shifting radically.  Changing conceptions of role of univs.  Walter Perry unimpressed with quality of teaching elsewhere, wanted to do better with the OU.  Staff refocused on student learning, students more demanding.  Teaching and research more distant – some institutions claiming a ‘teaching’ mission, others ‘research’ – but with increasing cost of esp science research, will get worse. (Not sure this is entirely new – polytechnic/univ divide was that.) Unbundling of faculty functions – division of labour – teaching versus prep of materials – more teaching only.

Learning design! Role of staff in the design of the learning experience is important. Design and moderation of the learning experience is the key task of an academic – lot more complex and exciting now that lots of resources are available.  Role of teacher changing, not disappearing.  Can’t delegate this knowledge to one member of the course team.  An academic not engaged can’t appreciate the possibilities.

Unequivocal necessity for the OU to be world leader in four areas of scholarship related to new media.

Policies and practices need to be revisited and are being – Student Support Review, IET review, staff dev, promotion, hiring, induction, research mgt, more. Course offerings and research themes too.

Sum up: Scholarship in this univ in this century has to be irrevocably tied to the technology and knowledge media.  People are proud to be part of the OU mission, and understand that they have to engage with this, but concerned about ways of working and job satisfaction.  Has never met a person at the OU who doesn’t strongly identify with the OU mission.  To be the best in the world in open and distance learning.  Ask “Why not the best?” – drive from original OU.  New answers.  Our mission never more relevant and urgent.

Q&A time

Peter Matrell? Student. –  With all the innovations, are we making every effort to ensure disadvantaged students not comfortable with tech are going to be catered for?  Quality of service, should include costs for students – for students in Europe and outside are increasing horrendously.

VC – Cost is an issue. Tech shouldn’t be an add-on, make it work to reduce cost. Whole range of projects hopefully with that outcome.  E-business streamlining. We are quite an expensive organisation – not that expensive.  Can assure that students not familiar with tech or disadvantaged are protected fiercely by many around the univ, warmly held objective for all sorts of people.  Offer financial aid for hardware, have loosened up the criteria.  Not helpful to students to let them walk away from the technology.

Bob Lambourne, piCETL – Concern about our ability to take ALs with us.  They can be consumers of developments, but can they contribute to creating the innovations, rather than just delivering them.

VC – Staff devt issues are entirely non-trivial.  Not a mistake to have SD as one of ten priorities.  Challenge that many ALs are more than equal to.  Will happen over time with development, induction.

(Technology has collapsed here for getting questions in … so will be bits of paper)

AL from R02, German – Teaching via Lyceum, eTMA, now worried about the time that even keen techs spend on it.  All fully behind it but salaries don’t reflect the time required.  Any way that makes the contribution more financially viable?

VC – We are extremely sensitive to that.  Looking at role of the AL, very large review.  Part of it is exactly that.  Never going to be rich though, sorry.

Darrel Ince, HoComputing – Jeff Besos quote – you innovate in the best of times, you innovate in the worst of times, you should worry weekly that it’ll be closed down.  A lot of innovation – bottom-up, individual islands – don’t see a ton of top-down innovation.  Doesn’t like bottom-up innovation.  How can univ garner bottom-up stuff, start to do more top-down?

VC – It’s one of the distinguishing marks of a successful org – can migrate innovation across whole.  Student Support Review aim to take us on to new level with what’s the baseline – very best practice widespread.  People here are seriously resistant to top-down innovation.  God help the manager who does it.  Have to go with the culture of the organisation and find some balance.  Woe betide the VC who laid down the law esp wrt innovation – there are 1000 people who will tell you you’re entirely wrong and will fight you to the last ditch.  Hard.  Even OpenLearn innovation – most people love – when first mentioned, one person said “Over my dead body”.

Mariann Cantery? – OUSA VP Education – equal proportion of students resistant to technological change.  Plan to maintain less technological course for those who are resistant?

VC – Can’t say it’s in the plan – expensive, and two-tier university, not in students interest.  Migrate students, offer wide palette of possibilities.

Jonathan Fine – LTS and COLMSCT Ting Fellow – Print new forms of social interaction. Technology recent – big changes – Facebook big example.  How will impact university?

VC – It’s not in the future, it’s already happening.  Need to make it more universal.  (Does she know about our lovely OU Facebook apps, with thousands of users?  Possibly.)

Chris Pugh, Arts AL in R05 – Prisoners, have serious access problems.  Is the OU negotiating with HOme Office about this?

Will Swann – At next mtg of LTSS Cttee – considering report on offender learning steering group.  Online access is one of the major issues.  Lot of work in progress.

Lisa Carson, OUSA President – Fresh (?) from OUSA conference.  Resounding message about the initial support, and how daunting it is for students receiving the initial package – “What do I do with this?” – when it’s a brown package, but also when you get on to StudentHome and how it’s presented.  Knowing how to navigate it all is a very definite concern.  Is that being addressed?

VC – Acutely conscious.  Will?

Will Swann – Yes it is. Working on how to make our comms more coherent.  Plan – use student journey framework – let everyone see what everyone is sending to students. (Cool idea.) Across Student Services and Faculties.  Money from his back pocket in to study to pay students to collect everything that we send to them so we understand what we are doing.

Anne Howells, LTS – Using focus groups with students – ‘opening the box experience’ – involving IET and Marketing too – mostly around print, DVDs, but also watched students using their computers in their own environments.

Today and tomorrow is the OU Conference – an almost-annual internal conference that’s grown from an original focus on ‘Tracking Technology for Academic Advantage’ via ‘Curriculum, Teaching and Student Support’ to ‘Making Connections’ more generally.  I’m planning to liveblog notes from it so expect a steady stream of posts.

Annoyingly, the numbering on the parallel sessions is different on the conference programme and the description of the sessions, which makes matching them up a little tricky.  Not impossible, but I predict many people showing up at the wrong place at the wrong time.

First up is the OU Teaching Awards, followed by a keynote from the VC.