OU Conf: Bill Tait – quality framework

Bill Tait – AL in MCT, COLMSCT Associate Teaching Fellow: Towards a Quality Framework for E-learning

Project to make it easy for tutors to write learning objects.  Wrote extensible learning objects and get tutors to apply their own ideas about pedagogy.  But big gap in technological expertise, even among Computing ALs.

Aim: a framework to help practitioners evaluate and design LOs in a more consistent way.  Should apply to e-learning formats and traditional techs.

Generic model: there is a ‘content object’ that is the subject content to be learned, and ‘learning activity’ which is the process in which the content is learned.  “Just like programming objects”  Abstraction – require entry and exit conditions to be met.  Encapsulation means it can have no links with the internal content of other units.  Implementation is how subject content is actually delivered so this is where pedagogy and technology come in. (!)

(Not sure I’m convinced this is a really helpful model of what is going on or should be.)

“Pedagogy is a strategy for implementing learning theory” – learning theories use terminology that may be beyond reach of many practitioners, and implemented using technology which is beyond them too.

So has broken down pedagogy in to Information, Experience, Consolidation, Discussion, Assessment.

Martin LeVoi – ran a project years ago on evaluation of web-based LOs for learning stats.  Fair bit of theory about evaluation, have you looked at those?

Bill Tait – They’ve come and gone.  Recent publications have a lot of advice for helping practitioners.  People are still uncertain about this (?).

OU Conf: Marion Hall – HSC resource bank

Marion Hall, Lecturer in Faculty of Health and Social Care..

Library of resources for use across HSC courses – SORRS project – Shared Online Resources Repository System.

Went live on 16 Jan 2008, HSC resource bank as standard Moodle course website – http://learn.open.ac.uk/site/HSCRB

Can be accessed by: all HSC students, ALs, and OU staff via autoregistration at http://learn.open.ac.uk/course/view.php?id=4443

It’s a meta course so permissions feed in automatically from all HSC courses.  Wanted to use ECM but that’s not available, so the database lives somewhere else as a workaround, so can’t genericise it until it’s available.  Could use it in other Faculties but would need to do your own workaround.

Metadata attached – especially ‘nation’ – England, Wales, Scotland, Northern Ireland – especially for legislation in H&SC contexts.  And keywords from a restricted taxonomy!  Thirty keywords – to help with students putting in mis-spellings, or the synonyms problem.  (Hmm – I think there are cleaner ways of doing this.)

Resources lifted from existing courses, genercised, converted for web.  About 300 resources (pages) at now; another 375 in Phase 2 in Sept 2008, mostly new resources from Jan 2009 from courses in production.

Another component – ‘Milestones’ – significant events in history of H&SC from 1869; each event is an individual resource, course teams can add events (e.g. when doing new course) – can produce search results as a timeline graphic.  (Fun timeline technologies from Tony – e.g. Simile timeline, also BBC have done some nice ones.)

Resources available via programme website; allows for systematic skills development within a named award.

Me – any feedback/evaluation?Marion – Main formal evaluation is to come after stage three. Some small tentative evaluation which gave us some info about study time for these resources.

s/o – What about the generic resources on S Svcs website and getting them to do the extra work>

Marion – Yes, we’ll use those where available, but our students need a lot of handholding compared to the Computing Guide – liaising with Student Services

s/o – Wanted a guide to using Word drawing tool, got the students to produce it, very successful.

OU Conf: Lin Smith – a quality approach

Lin Smith – Regional Manager from East Midlands – “Online e-moderating – a quality approach”

Dean Taylor chairing.

Staff development for OUBS, got OU Teaching Award this morning for whole team, more than were named on award.  Five years programme.  Identified that tutors needed help in becoming good e-moderators on tutor group forums.

ALs did a lot of the work, done online – integrated.

Trying to set up as a Community of Practice.  Courses lasted only two weeks but they became real CoPs, brought in examples from web and own practice.  Of course, was based heavily on Gilly Salmon’s 5 stages but went beyond – got tutors working on knowledge development very quickly.  Also engaging course teams – on d10 and d13, course teams come in, observe, and answer questions – to specific queries about their courses.  Some specific to e-moderating, some general.  Caused a lot of admin work, and is different to e-moderating offered through Janet Macdonald which is available through TutorHome.

Started with Certificate ALs and students, then fed through to more wide audiences (whole BA Business Studies and Law) – based on constructivist ideas of learning.

Principles of excellence – from EQFM – people, processes, results/learn/feedback.  Paid staff to do it.  They did 70% of the exercise and did a PDP afterwards.  85% said they got a lot out of it.

Web monitors (special tutors) – give positive fb that have been changes in behaviour (of tutors) – a pool of tutors who are ‘super tutors’ who can do stuff like Alternative Learning Experiences (online instead of residential schools) and BZX (online versions of ordinary courses).  Tutors demanded a certificate, now ask for it as part of recruitment and selection.

OU Conf: Phil Candy keynote

National Director of Education, Training and Development for NHS Connecting for Health.  “Scholarship and the Rise of Knowledge Work?”

Denise Kirkpatrick welcomes everyone back. Much fewer people here – possibly because the programme is a bit unclear about what’s a keynote and what’s the parallel sessions, or possibly because not half so many ALs here. NHS Connecting for Health is largest civilian ICT project in the world (after Chinese Red Army comms group presumably).  He has four UG degrees, two PG degrees.  (Another ELQ failure then.)  Distinctive, evidence-based approach to adult learning in health and social care.

After an introduction like that, even I’ll be interested to hear what I’m going to say.

Slides will be available … afterwards.  Presentation dense.

Linus’s Law (Peanuts, not Pauling or Torvalds) – “There is no heavier burden than a great potential” – true a lot for the OU.

The Boyer ‘four scholarships’ idea – clearly has a lot of traction in his crowd.

“The work of the academy must be directed toward larger, more human ends” – Boyer summing up his ‘Scholarship of Engagement’ paper (J Pub Srv and Outreach)

Standardish overview of pressures on HE. Scholarship, research, teaching, admin – orthogonal skills.  Commitment to the collegium: Self-governing community of scholars. Boyer – Scholarship REconsidered – good book.

Boyer’s Four Scholarships: Teaching, Discovery (Research), Application (consultancy … or community service), Integration. Teaching and Application is sharing out, Discovery and Integration is drawing in. Theme is making connections between all four parts – it’s artificial, banal – to separate out.  And wrong to assign people to single boxes.

(The fun stuff now – for me? – is Application and Integration.  But the other two are good too.)

Scholarship of Teaching: barriers to it – massification, diversity of students, credentialism, ICT, difficulties in attracting full time faculty, user-pays more demanding ‘clients’, managerialism; good things – outreach to unserved populations, recognition of other forms of learning, new/empowering learning opps, adjunct faculty, true learner-centredness

“Nobody talks about getting release *for* teaching.”

Scholarship of Discovery: globalisation/loss of local relevance, money/getting grants, subversion of the ‘invisible college’ through greater availability of information (?!), shift away from peer-review (Mode 1 knowledge production); global collaboration (e.g. HGP), partnership with industry/interest in research and its applications, career opps for new researches, Mode II knoweldge prductions [GIbbons, Novotny and Limoges 1994]

Scholarship of Application: loss of independence, money-grubbing ‘consultancy’, telling clients what they want to hear, distraction from ‘real’ academic work, worth measured only in terms of commercial goals; real world as a site for practice, career progression in and out of academe, real world problems create ‘natural lab’, opps to create discretionary income streams to do good stuff (!)

Scholarship of Integration: breakdown of disciplinary purity, compromise in language or paradigms, loss of rigour/nobody to hold you to account (Image and Logic book – some physics domain which aren’t theoretical or experimental fields, but sit in intersection between the three, is a trading zone not colonised), difficulty in being judged fairly in one’s specialist field, dilution of scholarly communities; but breakdown of disciplinary purity is a good thing too, truly innovative insights from serendipity, unexpected breakthroughs, enhanced rigour since interdisc, systematic synthesis

Boyer and ‘seasons’ – don’t lock in to one of four as a career trajectory – but an academic is likely to move around and across these during their career.  (As Peter Knight advised me to argue about my profile.  Ho hum.)  Boyer reaffirms trad values in HE – some of them.  John Seely Brown and Paul Duguid – university in the digital age (Xerox Parc) [PDF!}

Knowledge-based occupations: professions, traditional knowledge-intensive jobs – e.g. insurance, publishing; new professions – consulting, info systems; changes in workplaces with more ICT.

Characteristics: never the same, creative, special skills, social distribution of knowledge (Learning in and out of school paper – deconstruct the skills and knowledge required to dock a large naval warship), communication with non-specialists, reflective.  In search of the knowledge worker (Paton, 2005) [PDF] – three axes: theoretical knowledge, contextual knowledge, intellective skills.

So need graduates who can do all this stuff.  And the various aspects require all four scholarships to achieve.  Hooray.  Candy et al 1994 Developing lifelong learners through undergraduate education.

List of what you can tweak in order to produce graduates for the information society.  Teaching is about all four scholarships.

Concept of ‘engaged college’ – H G Wells – universities “floating over the general disorder of mankind like a beautiful sunset over a battlefield” – from a piece called “World Brain: The idea of a permanent world encyclopaedia” – 1937 fascinating article worth another look, seems to be predicting Wikipedia :-).

The quesiton is not just “What does your machine produce” but also “How does your garden grow?” – Pace CR (1971) Thoughts on Evaluation in Higher Education.


Unison Northern Ireland – Individuals needs vs NHS needs?

Candy – inspirational leadership (!) – contented cows produce more milk.

Clare Spencer – AL from Wales – reconceptualise self as a day labourer of the mind.  Skepticism about teaching-only universities – is there a place for teaching-only staff?

Candy – No.  Maybe a period of their career where they are teaching-only, can’t do all four at equal standard all the time, but should test out other options.

SL from OUBS – Profile of knowledge worker versus what our students expect from us.  Geoff Peters course feedback, online Masters expects students to do stuff for themselves “We had to download papers for ourselves, read them … think for ourselves […] Where was the learning?” (Good quote but I missed it all.)

Candy – issue with reframing the language to match.

Josie Taylor – The elephant in the room – don’t disagree, discussion of Boyer is great, should embed it more. But there is a problem with the RAE (and successor).  Good management is good for everyone, especially younger members of community deployed badly and not in a position to recognise it.  Role of very effective and sympathetic management is important – realities of daily life is very tough.

Candy – Luxury of being out of full-time HE.  In NHS – some great managers, manage budgets, chair meetings etc, but still have respect for world of ideas (engage with papers etc) so it can be done.  In Oz – development of shadow admin structure – head of school has a business manager, internationalisation, fundraising, budgeting, HR – take away from academic leaders the things they’re not so good at, but means must be a partnership.

OU Conf: Karen Kear and John Woodthorpe

Piloting VLE Communication tools in a Level 1 course

T175 Networked Living – piloted VLE tools by nine ALs, involved them as investigators. Feb-Oct 2007.  111 students.

Evaluated wikis and blogs – mostly wikis as venue for online tutorials versus FirstClas – alternative offered for a couple of online tutorials (Block 3 and some Block 4).  Also VLE blog versus online learning journal. Online questionnaire to students at end of course.

54 responses = 49% rate.

(Ooh, Twitter has just gone down. Patrick sitting next to me says he’s collecting failures. !)

Demographics – mostly 21-30 and 31-40 males, typical of the course.  5% on dialup, half on slow broadband (<1Mbps).  FirstClass – 48% use client, 11% use web, 41% use both.

Block 3 online tutorials – 52 offered VLE, 44 used it; Block 4 – 41 offtered, 23 used.  Drop off in use.

Enjoyed using it – 73%/56% Block 3/4.  Irritated when others edited work 16%/17%.  Unhappy about editing others’ work 30%/9%.  Prefer firstClass 52%/52%.  VLE tools difficult to use 50%/43%.  Interesting that irritation at others editing you doesn’t drop off but being unhappy about doing it does.

Usability of wiki vs FirstClass – wiki is generally worse, because new? Or other things.

Usability of blog vs FirstClass – blog a bit worse, but not used much, most saying ‘Not sure’.

(Possibly because it’s new)

Qual data – it’s a good idea, usability problems, social discomfort – it was too open (not restricted to student group), but makes it “easy to collaborate jointly on documents which has always been a bit of a logistical nightmare”

Further requirements – students want to know when a new contribution had been added, and who had viewed contributions (FirstClass functionality they didn’t want to lose).

Blog – students interested, but didn’t think suitable for learning journals – because of privacy and access.  Want control of access if it’s a learning journal.

Conclusions “The students enjoyed it”  (fab finding).  Usability problems, half of them preferred FirstClass.  Students concerned about access, privacy, editing others’ work, having others edit their own work.

BUT remember that online tutorials were designed for FirstClass, and the students were already very familiar with FirstClass.

JohnP – How far wikis pull students’ attention away from FirstClass?

Karen – they were using one or the other here.

ChrisP – not surprising, did similar on PROWE project – concern about the etiquette of editing.  Think back to preparing students to use FirstClass, we used to have a whole guide to this sort of stuff – students worried about publishing their thoughts in a permanent state.  Did you provide guidance as part of prep?

Karen – No, it was new to CT and ALs and students.

John – We did have a training day.  Plenty online about wiki etiquette.  Decided not to bring those in but see how they worked without the foundation we give them in the conferences.  In light of experience – give guidance, particular entry doesn’t belong to one person – e.g. ‘this is my page’ – edit it and I’ll edit yours in revenge (!). But better to get them to contribute a phrase to each page, which would’ve established the pattern of permissions, rather than encouraging ownership of a page, which was how the activity was structured.

AL – If I enter a page today, but there had been alterations between yesterday, or can I see what’s in between?

Niall – Yes you can.  VLE tools have been upgraded since then.

Karen – You get History tab and you can see the difference between one thing and the next.  Tutors would have to to see what’s gone on.

ChrisP – tutors can see what’s gone on.

Niall says he didn’t nod off so that’s good.

OU Conf: Tony Hirst on Google Analytics

(Was on slideshare but lost during technology snafu.  Oops.)

What would we do if we were a 2.0 company? Use web analytics to explore student/customer experience to get them to come back.

Helps us answer: What are your students doing online?

T184 Robotics and the meaning of life – 10week presentaiton, two a year, tracked for two years.

How often do they come?  Using webstats: “Most people visited: 15-25 times” over a ten week period.  Breaks down – a lot of people visiting once, but is an artefact – but most ~50% visit ~2-3 times pw.

How many course web pages do they read per visit? “7.66 pages/visit” – again artefact at 1, 2, 3 pp but basically falls off as a tail (exponential? looks more like a polynomial)

How long are they around for? “11:19 minutes average time on site” – artefact of people popping in and out – huge spike 0-10s, but second peak at 601-1800 secs (up to half an hour).

When are they doing it? Mostly daytime, peaks at 12 and mid-evening 7/8pm.

Every day the same? Fridays are bad – big bumps on Sunday and Monday – but that’s when the assessment fell – generally Tuesday seems to be good. (Eh?)

What are they reading?  Mostly the home/landing page. (Duh) But then Calendar, then Search results (to find answers to assessment), then End of Course Assessment, then Assessment landing page, then ‘What is a robot’ which contains several answers, then CMA, then rest of course content.

How long are they reading each page for? Average 2 minutes – can see per page too. Spend a bit more on content pages than other ones – probably artefact from bouncing off home page.

Where are they leaving from? Mostly from ECA – 40% read that then go.  Robotlab is a common one too (27% leave them vs 13% site average) – practical activities described in those, so obvious they read that then go away and use robot (or simulator).  A high rating might flag up that students are unhappy with it.  Outliers and changes are good to watch for.

Does pacing work?  Do they do it like we intend – week 1, 2, 3, 4 etc in order?  Lovely graphs – generally they sort-of do, with nice curves that show that happening, in order, in sort-of the right numbers.

When do students do their assessment? (meaning look at the ECA pages) CMA visits peak around when it happens, then sharp fall.  But some before, and also a little tail afterwards – perhaps search results hitting the CMA page.  ECA – very few look at it at the start! (Blimey, not what I’d have thought.)  Builds up steadily to a huge peak at the due date.

(Wow – students are less strategic than I imagined. At least these students are – as Tony says, would be good to see if this happens with old lag students too.)

Should make it easy for students to get to the assessment – there are extra click involved from the home/landing pages.  Should take a single click – so here’s some user behaviour we can learn from and do sometihng valuable.

What computers do the students use?  (Fun pic of Commodore PET – Tony’s main home machine)  Connection – Most students are on reasonably fast, but 11% on dialup.  “We should encourage dialup to die.”  Screens are getting bigger – so can design screens differently.

Ingrid – can tool tell you what students are printing out?

Tony – could track printable pages, setup not quite right.

Ingrid – evidence that students go to pages and print them out, so how long they spend there is interesting.

Tony – SiteIntelligence – OU signed up for – bigger and better and can segment by PIs and whatever – can track every single page, not just ones with a code on.  Could pull out students who printed out and compare it with other poeoples.

JohnP – Any Data Protection Issues?

Tony – Google can see the URLs, but can’t sniff the content.  Adds to what Google knows about you, could tell you’re a student and add in to their marketing mix.  Google track everything you do.

John – Anything you had to clear?

Tony – Next question!  (laughter)

Steve Godwin – Lots of skimmers, a few detailed learners – does this give you median values rather than just means, because you get very long tails?  Also artefacts for the one-page users – e.g. people putting it up as their home page.

Tony – Google Analytics doesn’t give you raw data, but does some disaggregation.  The OU system does let you do proper stats and segmenting.