Week 1: What difference do learning designers make?

Hi all,

I totally agree that evaluation is critical - and I think organsiations (or parts of them) are at last beginning to realise that that's a bit different from doing a simple 'satisfaction' survey. I do hope so! So even tho 'most of the paperwork gets binned' , at least Bron is developing processes. I feel much the same abou the start of the design process - much of that paperwork is also unpopular and 'gets binned' but I have proved that it is is valauable later on.

From my perspective as a free-lance learning designer - I rarely would still be involved in a project at evaluation stage. That raises a whole lot of other issues - is there someone else in the organisation who would carry the flag for doing an evaluation? Do they already have some systems in place? I do work with one private company, creating courses on a very small scale - and the tutor and I have a free rein.... so there we can do some (informal) testing and evaluation before the course runs. Although few courses have run properly yet, the results are encouraging, especially as the senior management were very doubtful that this form of teaching would work. Because I remain involved, we can adjust courses according to the feedback we get, and I can keep an overall view to make sure we are delivering consistent messages.

In those very narrow terms (ie my involvement with that small private company) I'd answer Lyn's questions as follows:

What is it that matters to our managers?
In this case, the courses have to make money. We have had plenty of leeway so far, and the design and development costs vastly outweigh the returns. They have yet to do some effective marketing, but have a vast potential pool of clients. But accuracy is also essential (and I mean really really picky!) The constant reviewing by what seems like everyone in the organisation adds mucho hours to the job.

What matters to me?
It matters to me that my client is happy - fortunately my goals are the same as theirs.

How do we stay sane, but find some rigour?
In this case, the rigour is not an issue - my main problem is finding ways to meet their needs within a crappy LMS - and convincing them that other things are possible if we re-work them another way. I' measure my success in tiny ways - such as an evaluation coming back saying "I learned heaps! and I liked the environment, even tho I was sure i wouldn't"

What should we be measuring?
Hummm, not so sure on this one. I do think completion and retention  are valid measures, but capturing the qualitative comments is also important. There is a risk, tho, with the qualitative stuff - that teachers/the organisation have a reflex response to individual comments: "X said this, so we have to change it". In larger organisations the measures are different - can ppl staircase to higher level courses?, for example.

Lyn says" I would like to think that the impact of design improvements influence some identified change in either student or teacher behaviour which in turn helps or improves learning"

I totally agree: one of the most satisfying moments this year was at Moodle Moot: hearing a tutor talking with excitement about her online course;  repeating many of the things that I had tried to impress upon her at the beginning of last year, and her enthusiasm in going forward and finding her own way. i don't very often get to see the end of that change - and it's always possible that the result is the opposite - they remain unconvinced, or revise courses in ways that are singulalry inneffective.

That's enuff from me for now. Scuse all typos -- Write Group would never let me get away with it!

k8

Paul said: "Or maybe they are using other measures...?"

The most comon managerial measure around  here is student retention and success - however they decide to measure that, usually figures from the sms! - so if more students stay with the course than before and actually complete, then something made a difference.  It could well be the ID that goes into the re-development (usually) as the teacher works with a different group than their own department to discuss an approach and produce the teaching materials etc etc. 

We did a project here with some courses that had really poor retention historically. A team worked on introducing some elearning aspects (though the classes were a face to face ones) and evaluated the outcomes.  The students were certainly more motivated, more engaged and more stayed and completed the units.  So for the R&S measure, we improved things.  But was it the ID or was it the fact that the teacher was also more engaged, or was it the Hawthorne effect - we asked the students for their reactions of course...

But if the "powers that be"were only interested in the R&S, then did it matter what made the difference?  We'll claim the Iearning design did of course :-)

Oriel