Innovative practice with elearning - case study 3

The third case study from the JISC Tangible Benefits publication was

The University of Nottingham Case Study: Moving from Optical Mark Recognition (OMR) to Computer Based Assessment (CBA) for summative exams in medicine

The Nottingham Medical School used optical mark recognition (ORM) for scoring exams, and wished to move to computer-based assessment (CBA) instead. There were two key drivers for this, namely the time pressures of marking (with OMR each exam paper must be scanned, so marking time increases in line with class size) and desire to incorporate images (e.g. microscope and x-ray slides) into questions, which was difficult to do with sufficiently accurate reproduction on paper. There was also the attraction of being able to use interactive question types such as drag and drop labelling and image hotspots.

The case study appears to relate to 2003/4 when the Medical School began developing the TouchStone online assessment system and populating it with the question bank. At this time there was no institutional steer regarding computer-based assessment, and very few departments were using online assessment for high-stakes summative assessment.

The problems anticipated with moving from an OMR to a CBA approach were actually administrative, rather than technical or attitude-related. There was a difficulty finding computer labs large enough to examine the whole cohort simultaneously, and there were concerns about plagiarism that might be afforded by online exams (i.e. seeing other student’s screens and accessing forbidden materials during the exam).

This case study focused a lot on getting the electronic workflow process to support online exams right. This was clearly a large project with many stakeholders involved, including academics, administrators, subject matter experts, external examiners, disability experts and IT support. There was an awareness that by computerising all aspects of the workflow, quality could be enhanced by ensuring backups, collaborative working if any stakeholder was unavailable (within appropriate permissions-based access) and transparency of work.

Tangible benefits identified include:

• electronic access permits all stakeholders to access and work on their part of the assessment process. Also electronic storage of questions, past papers, student profiles and exam results can be analyses statistically.
• marking time was greatly reduced - has positive impacts on scheduling subsequent activities such as moderation, follow up exams, etc
• ability to include multimedia questions types, in particular image labelling and hotspots. Labelling and hotspot questions are similar to multiple choice with a diagram, but reduce cognitive load. Hotspot questions also reduce the probably of guessing correctly
• online assessment makes it simple to accommodate some disabilities such as dyslexia and other visual problems
• students can change their answer as many times as they wish before submitting - previously multiple marks on an OMR form could cause scanner errors

My thoughts:

• This appears to be a good case study of process improvement through applying technology - there have been cost and time savings as a result
• Technology also afforded better assessment - interpretation of various medical images is a key skill, but could not be adequately assessed by the previous method
• Involvement of all key stakeholders from the beginning of the process was again identified as necessary for success - this is particularly important while there are staff who are IT-averse or shy.
• The case study pointed out the inherent risks of online assessment, in particular the reliance on servers, client computers, network and power all being necessary for an exam to run. While this does appear to be adding risk to the exam process, I suspect that as daily life makes ever greater expectations of availability of these systems, prevention and mitigation of these risks becomes easier.