Is an exam a valid and reliable assessment method in an online course ?

Jane Dillon's picture

Hi LDNet Community !

Thanks John for the opportunity to kick off a discussion on a topic that is of increasing interest here at the Nelson Marlborough Institute of Technology (NMIT).  

This issue came into focus (in my work as a Flexible Learning Advisor) in the development of an online programme to run concurrently with an established campus programme. Should online students be assessed in the same way with an exam when the nature of their learning environment and activities might suggest this was less appropriate ?

Look forward to your views !

Cheers

Jane Dillon

 

Parent group: 
Forum:
John Delany's picture

useful resource

Hi Jane

Discussion seems to have dried up somewhat - maybe everyone is busy marking! But thanks Jane for stimulating some interesting ideas here.

I came across this site today from AUTC, well worth exploring in terms of tertiary assessment in general, and some useful material on online assessment in particular: useful in that the resources are well set out, downloadable for printing, and the powerpoints can be downloaded and adapted. See what you think.

http://www.cshe.unimelb.edu.au/assessinglearning/index.html

 

Cheers

John

Jane Dillon's picture

Some ironies

Hi David

Many thanks for your reply. There can be ironies in all of this. I have been reflecting on Gilly Salmon's comment that "we need to move towards 'aligning' our assessment with our online teaching aproaches" (Salmon 2003). Elsewhere she mentions a fundamental irony in students studying online only to have to appear in the exam room with pen and paper ...... As you point out when this is required  there are ways of making it happen - complex as the arrangements might be !

Cheers

Jane

David Whyte's picture

Out of my depth

Good questions Jane. Sorry I am showing my age :) and I have not participated in any online assesment. Although I am being encouraged to put my multichoice questions onto Moodle here at UoW. I am going to leave that adventure till next year as there has been many other changes in the course that have kept me busy. 

I have had a very small example of none traditional exam situation. Last year in one of my courses there was a writer for a student, and for both the tests and the exams she sat them at same time, but in another part of the university. I am unsure if they had a writer for the test/exam or just extra time.

In the test situation I emailed or sent via internal mail (can't remember) to a person in student services who looked after these special case tests etc. It also struck me as been ironic that there is all this procedure and controls etc around exams and handling of questions and scripts, and there is this exam going in internal mail in a envelope with holes in it, and the exam script comes back the same way.

I would suspect that most tertiarys have the ability to handle test/exams in the way mentioned above. So if your students could easily travel to a tertiary then they could sit the exam there. However I doubt this would happen due to the politics of this kind of thing across tertiaries......

David

Jane Dillon's picture

Related questions ......

 

Hi David, Mark, Nick and Oriel
Many thanks for your interest in this discussion and for sharing your wisdom !
As you point out David there are really two topics in my initial posting.
Mark provides some sound advice to the issue of whether students in concurrent programmes – one f2f and one online – should both be assessed in exams by saying:
“I would be inclined to:
1.    Challenge the rationale for a final exam.
2.    Outline a process for facilitating final exams, which might involve distance-proctoring rather than on-campus conditions, and
3.    Depending on the outcomes of 1. and 2., either work to change the assessment or else implement the process. “       Thanks Mark !
Now to the actual question of whether an exam is a valid and reliable assessment method in an online course ? David sees value in “hard /thinking exams that were open book “ and considers an exam could be seen as an authentic assessment tool if a real world scenario requires an answer under pressure. As you say a quick diagnosis may be called in some circumstances namely medicine !  However,  in other circumstances we might well ask ourselves the question which Oriel has posed – do we need exam at all - are there other more important skills to be learned ?
(Nick I believe that in online learning also assessment does need to be aligned with the specific learning outcomes and student learning dimensions. My concerns are much more related to validity. )
A couple of questions:
1.    Have you experienced an exam as an online learner yourself and did this feel like a valid experience in that subject context ?
 
2.    In terms of reliability have you been involved in making arrangements for online students to undertake exams ( on or off line) and if off line have you followed the same or a different process to Mark?
 
 “ weeks before the exam, the student is required to find an exam supervisor (we do set conditions on who is suitable). We assess the relationship between the student and their proposed supervisor and authorise the supervisor to, well, supervise the student under exam conditions that we set. This requires a high level of trust, and clear instructions for supervisors. The supervisor returns the student script and provides a signature confirming that exam conditions were in place.”
 
Look forward to your thoughts !
 
Cheers
 
 
Jane
 
 
 
 
kellyo's picture

Do we need exams at all any mo...?

This is a good topic for debate.  As Nick says, there may well be a place for exams in general, if the SLOs warrant it - ie there is a  certain body of knowledge that needs to be learned - vocabulary, process whatever - and the actual student needs to demonstrate that under pressure.  So we have to have a process that is authentic as well - and identify the student taking the exam - another reason they maybe are not the best option at a distance, and even face to face sometimes.

But, increasingly exams in general are less use - the skills students need more now are where to find the information, how to evaluate and synthesise it into something else that they can apply - and information readily available doubles every day, and half of it is wrong!  Only we don't know which half, whether its in the teacher's notes or on the web either!

Another problem I have with exams - and this is personal experience - is that they don't necessarily test a student's knowledge of the subject......sometimes all they test is your knowledge of Greek and Latin derived vocabulary, if the exam questions are written in exam speak.  7th form history exam - I knew all there was to know about Henry 7th - complete bastard, tyrant, etc.  Exam question "How despotic was Henry 7th?"  Hand up, what does despotic mean ?  Can't tell you, it's an exam.  I didn't have enough confidence to write "If despotic means tyrant then this is my answer..." so I chose another question I was less prepared for and didn't get a chance to show what I knew. (David Corson did a thesis on this, if you can't understand the words in the question, how can you answer it?).  Hopefully exam questions these days are a little more fair to students????  But do we need them at all??

Oriel

Nick's picture

My 2 cents!


Hi Jane

I believe that assessment has to be tightly aligned with the specific learning outcome(s) (SLO) and the students learning dimension i.e. the initial acquisition > independence> mastery continuum.

Once the SLOs and the students learning dimension have been identified assessments maybe designed and rolled out utilising appropriate supportive technologies.

So in response to your question I say yes. An exam CAN be a valid and reliable assessment method in an online course. However as with anything this is context specific with the contexts being slo's, learning dimensions and subject domain.

Do you think online learning changes the notion of assessment needing to be aligned with slo’s and student learning dimensions or are your concerns related to validity?

Cheers, Nick

Mark Nichols's picture

The Laidlaw experience

Hi Jane,

At Laidlaw College we are preparing a course on the ancient Greek language. We are using online tests in a low stakes (that is, small percentage of final grade) way whereby students are tested on their memorisation of key vocabulary and noun/verb paradigms in timed tests that they can take again and again, based ona rich pool of randomsied questions.This works for us in that:

  • Memorisation is appropriate for the nature of the course.
  • Having timed tests puts the pressure on for students to respond to questions quickly.
  • The grade gives an incentive for memorisation, yet is not so high as to permit a student to pass the entire course.

A final exam will also be a part of the assessment, under proctored conditions. The system is this: weeks before the exam, the student is required to find an exam supervisor (we do set conditions on who is suitable). We assess the relationship between the student and their proposed supervisor and authorise the supervisor to, well, supervise the student under exam conditions that we set. This requires a high level of trust, and clear instructions for supervisors. The supervisor returns the stduent script and provides a signature confirming that exam conditions were in place.

The key terms in your own question are 'should' and 'appropriate'. There are times when exams and tests make good sense, others when they do not. High-stakes exams can still be done, even at a distance. Administratively It is a lot of hassle, but the assessment type suits many of our early-level courses where memorised knowledge in the student is a defensible objective.

In your situation I would be inclined to:

  1. Challenge the rationale for a final exam.
  2. Outline a process for facilitating final exams, which might involve distance-proctoring rather than on-campus conditions, and
  3. Depending on the outcomes of 1. and 2., either work to change the assessment or else implement the process.

I wouldn't be too quick to dismiss the value of exams in e-learning. There are ways and means, and it does provide a verifiable outcome where you know that the student has engaged with course concepts.

Hope this helps!

David Whyte's picture

A very thorny issue

Jane

There would appear to be two distinct topics, (1) should the on-line assesment be same as the on campus course and (2) is an exam a valid assesment tool. I will limit my comments here to the 2nd point. So people know were I come from - after my Masters I spend approx 10 years in industry before becoming a contract lecturer -teaching on the business of science topics.

As a Sever to extreme dyslexic that was diagnosed while at tertiary (UoW) I found the concept of rote learning to regurgitate in an exam an abhorrent concept  ( and no surprise I was not that good at it). And after a decade in industry I experienced nothing that would validate exams as a measure or knowing - In fact I came to realizes that some of the best academic students made some of the worst researchers.. Because research is about the unknown and they struggled with the fact there was no "right" answer - only steps on the journey.

However in saying that, I think assesment is important as it motivates people to learn ! I found the Physics department has the best idea (and this became my major). they set hard/thinking exam's that were open book. This meant that the you had to know the materiel and could apply it, but didn't need to memorize fact to regurgitate them.

Also in some feilds eg medical where large volumes of data floating around in the mind is important. When one goes to a doctor / medical person and ask for a diagnoses, you wouldn't want them taking 1/2 and hour to figure out you had a sore throat caused by an infection....... so exams might be valid.

 

 

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.