Subjectivity is the curse on examinations for UK doctors -
get rid of it
There is currently a controversy raging in UK healthcare
about doctors. For many years it was known and was passively accepted that if
you were of Black or Minority Ethnic origins more so if you were an
International Medical Graduate (IMG - a doctor with a primary medical
qualification outside the UK or European Union) facing a Royal College
examination you would have a much lesser chance of passing the examination. If
you were of BME or IMG origins and were of a generous persuasion you would call
this sub-conscious bias but most called this racism, whether it was racism or
not. If you were representing the establishment you put out phrases that are
superficial gibberish, such as 'reasons are multi-factorial and complex' but
certainly not racism.
The issue came to a head with the MRCGP examination where in
the new version of the examination the differences between white and non-white
candidates were so gross that you would notice it even if you were colour blind.
The medical post graduate examinations conducted by the
Royal Colleges are essentially about medical knowledge both theoretical and
applied. Given that these are knowledge tests, why did the results show racial
differences? We will not discuss racial supremist reasoning here. Many of us will remember the days before the
MCQs - the essay answers were often a demonstration of your wizardry in medical
English. Apparently even in the MCQ based knowledge tests we can use linguistic
jugglery so that a non-native English speaker comes out as having poor medical
knowledge - we are not discussing that further here.
The curse of subjectivity
Applied knowledge in medicine is tested in vivas, OSCEs,
with patients and simulated patients. Here the marking is done by examiners,
that is where subjectivity comes in despite current best efforts, subjectivity
is ruining careers.
The rest of this blog post is about subjectivity (the
collection of the perceptions, experiences, expectations, personal or cultural
understanding, and beliefs specific to a person - Wikipedia)
The sad paradox is knowledge especially in medicine is
objective but part of the testing process of this knowledge is subjective. The
tension that results from an objective topic tested subjectively is where the
fundamental flaw lies. Where subjectivity exists, there bias exists and hence unethicality
at the best and fraud at the worst exists.
Subjective assessments must not have a place in career make
or break decisions such as exit examinations or in any arena where career
progress or ability to practice the chosen profession can be stopped.
Subjective assessments do have a place and can be used for progressing in
learning and development - some of which are known as formative assessments.
Must not be used for stop-go decisions where only objective assessments should
be used.
Reducing or Eliminating subjectivity
Examiners in vivas, OSCEs, patient encounters, interview and
other areas currently suffering due to subjectivity, are generally given
questions - they should also be given answers and as long as the candidates
answers fit in with the recognised accepted answers the candidate passes, when
the answers fit in with recognised unacceptable answers the candidate fails and
where the answers fit in with recognised borderline, a published formula for
accepted number of borderline for a pass or fail should be defined (no, this is
not the 'borderline method' that is used in standard setting).
This may beg the question whether vivas are needed at all -
verbal communication is essential in all walks of life and especially so in
healthcare; a candidate should be able to answer effectively and accurately
under stressful verbal conditions and hence vivas are needed but the
subjectivity of the vivas must be eliminated.
Subjectivity cannot be sometimes avoided but when forced to
use it the answers should be 'force fit' in a pre-defined uniform manner and
the candidates be assessed against that uniform force fit. The candidate does
not have to know what the defined force-fit answer is but all candidates would
be marked against the same answer.
Lets look at an example: Let us assume that in a scenario where
there is a certain level of oxygen desaturation which does not impact on life
or limb but where a candidate has to act - say an peripheral oxygen saturation
that has fallen from 98 to 89 but where the patient is otherwise very stable.
The candidate has to make preparations for an adverse eventuality but there was
no need to act immediately. Let us also assume that currently this is
subjective and hence an assessor would mark someone and this would be variable
(depending on the other skills of the candidate). Let us try a force-fit answer
for this scenario - the examiner would be given a set of answers and would give
the candidate a mark for each correct answer, for instance, a) the patient if
conscious was asked if she was okay within two seconds 1 mark
b) the pulse oximeter probe was checked and re-applied within
4 seconds
c) capnograph reading checked within 6 seconds
d) the oxygen flow and any gas mix ups were checked within 8
seconds
d) airway tube position checked within 10 seconds
e) airway change kit and reversal drugs asked to be brought
in and kept ready with 12 seconds
etc. You get the picture.
These answers may not be based on evidence because there is no
evidence to base it on. However, for the purposes of the assessments the
answers are defined on the basis of agreement between examiners and are used
uniformly with all candidates. Then the chances of the examiner being
influenced by mastery of the language, social status of an accent, the image
projected by clothes, the false confidence provided by a charming smile or
colour of the skin would be less.
Subjective experts are simply socially acceptable
influential frauds providing a certain voyeuristic celebrity value when they are
reviewing wines, films or restaurants. Techniques similar to those have no
place in medical examinations. It is of course a completely different story
that the British are not able to trust the training provided to their young
doctors for somewhere between a minimum five years (in the case of general
practice) or an approximate minimum of twelve years in the case of surgeons that
makes an 'exit' exam essential to cross check knowledge (which is then pretty
badly due to the subjective components). In the USA exit exams are not
mandatory, they are voluntary, the Americans obviously have a great degree of
confidence in their trainers, trainees and training system. The British system
needs reform and a commitment to eliminate subjectivity when the stakes are
high could be core to whether the UK will ever have a equitable outcome in
examination results.
© HEMADRI
This blog has argued for reducing or eliminating
subjectivity from re-validation http://successinhealthcare.blogspot.co.uk/2012/11/revalidation.html
We have discussed differential results in surgical Royal
College examinations http://successinhealthcare.blogspot.co.uk/2012/12/exit-exam.html
