Week 1: What difference do learning designers make?
Last updated by: John Delany on 2 November 2008 - 7:25pm
Hi everyone
Let’s get underway with our first discussion topic.
As I understand it, the daily notification function is not working yet, so that’s a bit of a nuisance. Any feedback on the way the forum is working can be emailed to John (John@inova.co.nz) or placed on the ‘General Comments’ discussion.
------------
Since my FLLinNZ project in 2004-5 I have worked in a variety of roles and had 2 years away from tertiary education. As I come back to get steeped in the education sector again, I find myself wondering if I ever managed to answer a question I asked as part of my FLLinNZ project.
"As Learning Designers how do we know that what we do is making a difference? How do we evaluate this?"
Do you have any thoughts from your experience??
Regards
Lyn Williams
- If you're member of this group, log-in to post a reply
Summing up..
What is some of the evidence?
From outside an organisation
Hi all,
I totally agree that evaluation is critical - and I think organsiations (or parts of them) are at last beginning to realise that that's a bit different from doing a simple 'satisfaction' survey. I do hope so! So even tho 'most of the paperwork gets binned' , at least Bron is developing processes. I feel much the same abou the start of the design process - much of that paperwork is also unpopular and 'gets binned' but I have proved that it is is valauable later on.
From my perspective as a free-lance learning designer - I rarely would still be involved in a project at evaluation stage. That raises a whole lot of other issues - is there someone else in the organisation who would carry the flag for doing an evaluation? Do they already have some systems in place? I do work with one private company, creating courses on a very small scale - and the tutor and I have a free rein.... so there we can do some (informal) testing and evaluation before the course runs. Although few courses have run properly yet, the results are encouraging, especially as the senior management were very doubtful that this form of teaching would work. Because I remain involved, we can adjust courses according to the feedback we get, and I can keep an overall view to make sure we are delivering consistent messages.
In those very narrow terms (ie my involvement with that small private company) I'd answer Lyn's questions as follows:
What is it that matters to our managers?
In this case, the courses have to make money. We have had plenty of leeway so far, and the design and development costs vastly outweigh the returns. They have yet to do some effective marketing, but have a vast potential pool of clients. But accuracy is also essential (and I mean really really picky!) The constant reviewing by what seems like everyone in the organisation adds mucho hours to the job.
What matters to me?
It matters to me that my client is happy - fortunately my goals are the same as theirs.
How do we stay sane, but find some rigour?
In this case, the rigour is not an issue - my main problem is finding ways to meet their needs within a crappy LMS - and convincing them that other things are possible if we re-work them another way. I' measure my success in tiny ways - such as an evaluation coming back saying "I learned heaps! and I liked the environment, even tho I was sure i wouldn't"
What should we be measuring?
Hummm, not so sure on this one. I do think completion and retention are valid measures, but capturing the qualitative comments is also important. There is a risk, tho, with the qualitative stuff - that teachers/the organisation have a reflex response to individual comments: "X said this, so we have to change it". In larger organisations the measures are different - can ppl staircase to higher level courses?, for example.
Lyn says" I would like to think that the impact of design improvements influence some identified change in either student or teacher behaviour which in turn helps or improves learning"
I totally agree: one of the most satisfying moments this year was at Moodle Moot: hearing a tutor talking with excitement about her online course; repeating many of the things that I had tried to impress upon her at the beginning of last year, and her enthusiasm in going forward and finding her own way. i don't very often get to see the end of that change - and it's always possible that the result is the opposite - they remain unconvinced, or revise courses in ways that are singulalry inneffective.
That's enuff from me for now. Scuse all typos -- Write Group would never let me get away with it!
k8
evaluating - more questions
Hi everyone
Other measures
Paul said: "Or maybe they are using other measures...?"
The most comon managerial measure around here is student retention and success - however they decide to measure that, usually figures from the sms! - so if more students stay with the course than before and actually complete, then something made a difference. It could well be the ID that goes into the re-development (usually) as the teacher works with a different group than their own department to discuss an approach and produce the teaching materials etc etc.
We did a project here with some courses that had really poor retention historically. A team worked on introducing some elearning aspects (though the classes were a face to face ones) and evaluated the outcomes. The students were certainly more motivated, more engaged and more stayed and completed the units. So for the R&S measure, we improved things. But was it the ID or was it the fact that the teacher was also more engaged, or was it the Hawthorne effect - we asked the students for their reactions of course...
But if the "powers that be"were only interested in the R&S, then did it matter what made the difference? We'll claim the Iearning design did of course :-)
Oriel
Bron said: 'the lack of
Bron said: 'the lack of interest from managers to acknowledge the need for them let alone read them. Basically they did not really care.'
It seems the message there is that it doesn't really matter to managers whether or not you are making a difference! Or maybe they are using other measures...?
Paul
making a difference evaluating
Great questions to start us off Lyn. Sometimes it is a long process finding out if we have made a difference - a bit like evolution and we know how long that takes! However most of us haven't got as long as a dinousaur - so we need to know reasonably quickly if what we are doing is working. It is very important to me that I estimate straight away that I am meeting the needs of the target audience - clients, staff, students etc. As I am an educational developer, I will refer to the target audience as staff and students rather than clients.
On a day to day basis, we soon know if our suggestions and ideas are "too out there" - the staff member will not follow through and will not come back! Success, however, can be a process which manifests over several months.
Online learning flexible development projects: In 2007, I worked with several staff to help them develop course materials and teaching strategies for several online learning flexible development projects. Some wanted to use Blackboard as the platform and others were keen to try a wiki approach or a combination.
Each time I met with the staff member or project group, we re-evaluated our course of action. I also built in formative evaluation processes as we developed materials and once they were piloted - needs analysis, review and usability - getting feedback from peers, experts and students. staff also conducted summative evaluations later on. Because I had 12 projects on the go over the year, it was really hard to finish off the formal evaluations properly, and formal evaluation is usually the are which gets shelved if there are time constraints. Yes I gathered the data and made design changes on the basis of what experts, peers and students were saying, however, most of the evaluations did not end up as proper reports.
So in any design project adequate time to do proper evaluation does need to be built in and adhered to properly. of course, the other thing which reduced the need for formal evaluation reports was the lack of interest from managers to acknowledge the need for them let alone read them. Basically they did not really care. so when time is short some of the paperwork gets binned. That is the reality. But is this a good idea? I think not. Someone sometime will ask for evidence of success.
Evaluation is a process I am pretty pedantic about - probably cos I teach the subject in the Grad Cert in applied eLearning. There are some materials in the Resources section of the evaluation course.
In the eCDF online information literacy (OIL) project (http://oil.otago.ac.nz) evaluation was a pivotal part of the project. all the material relating to the processes we used are on the evaluation section of the website. Interestingly, our first evaluation process was to get feedback on the design prototypes - we created a storyboard of a prototype design on big sheets of butcher paper and asked colleagues for their feedback. It was a very useful exercise and unusual to use good old paper storyboarding when everything else was multimedia.
In my opinion, if we are going to know if we are making a difference a lot of intensive support for staff is needed and adequate time to build a close rapport with them and a mutual trust - also a clear project plan, regular meetings and constant re-evaluation of the process as well as formal evaluation processes. I am interested to find out how others evaluate "success" and hear about their experiences.
Bron