Yo BAPIO - What is that Moral Victory thing?

This article was first published in Sushruta Vol 7 Issue 1 by BAPIO

It is now very well known that BAPIO filed for a judicial review of the MRCGP examination, especially with regards to the CSA component; well there is no point beating about the bush, BAPIO lost the case.

That means that the MRCGP was ruled to be a fair examination, by indirect inference it may be assumed that other examinations were also likely to be fair. BAPIO members, BME doctors, IMGs can be reassured that things are rosy and live in joy. I was just about to do that when I found that Prof Rajan Madhok, Chairman, BAPIO tweeted ‘JR judge says: moral success but not legal victory! So our laws go against our morals? Crazy’ It is true. The judge said BAPIO had a moral victory.

So what is that moral victory thing? It is just a judge being polite?

Pause. Reflect.

To understand this, we have to go back to 7 June 1893. One Mr MK Gandhi who had a first-class ticket and was travelling in a first-class rail compartment was thrown off the train. He had a legal right to be on that train but he still lost his seat in the train; Gandhiji had a moral victory. The rest was history and what a history it was.

The judicial review has set off a number of changes which we are beginning to hear about. The GMC is now considering seriously introducing a common licensing examination for UK graduates and IMGs (similar the concept of USMLE). The GMC is introducing English language competency tests for EU doctors (where there is cause for concern). The time allowed for the AKT MCQ examination of the MRCGP is being increased. There could be changes to the way CSA is conducted and assessed. There are numerous other changes and many Royal Colleges and medical educational establishments are engaging with BAPIO and its partners.

The RCGP and GMC activities considered in the Judicial Review were ruled legal. Yet they and other institutions are making changes that further cause of equality. BAPIO contends that these changes would not have happened at this juncture and at this pace, without the Judicial Review? Are we beginning to understand the concept of a legal loss and a moral victory?

By the way, Gandhiji protested and was allowed to travel the next day by first class. In the continuation of the same journey he was beaten by a driver, banned from hotels and subjected to other forms of abuse.

BAPIO should be under no illusion that things are or soon will be rosy. The path is strewn with thorns and BAPIO should be prepared for its skin to be pricked in this journey. What does BAPIO want? BAPIO wants, what you have always wanted. A level playing field, no bias, high standards, fair assessment and equal opportunity to progress.

Here are some suggestions on the specifics that BAPIO should be asking for

1) Real patients rather than role players

2) Increased number of BME/IMG examiners

3) Two examiners on each station

4) Video recording of the session.

5) Improved training of the candidates.

6) Improved training of the trainers and holding to account of trainers with poor record of success of their trainees.

7) Feedback and mentoring for those who fail

8) Removal of hawk examiners/trainers (especially those who have negative impact on BME/IMG doctors)

9) Removal of dove examiners/trainers (especially those who have a negative impact on BME/IMG doctors)

10) Testing and continued monitoring of sub-conscious bias in examiners/trainers.

11) Examiners with extreme bias not to be selected, examiners with non-extreme bias to be provided training followed by monitoring.

12) Pass-fail threshold and other standard setting (such as ARCP/RITA progress) should be tested for impact on various populations with protected characteristics and where there is no evidence of impact on patient outcomes the thresholds should be adjusted to reduce any possible negative impact on doctors with protected characteristics.

13) Objective assessments/examinations for summative, pass-fail, high-stakes situations/examinations/assessments (with any subjective assessments reserved for formative processes)

There are many more ideas that will benefit the system.

If BAPIO decides to ask for these and more, you can be assured BAPIO will be vilified and denounced. The hope is, after the abuse is done, the changes would happen, even if they were slow.

A couple of thousand of years before Gandhi, we hear of one Jesus Christ, who lost a legal case and was crucified; he seemed to have won the moral case quite convincingly. Time will tell, but BAPIO’s moral victory may turn out to be a very strong force for change.

M HEMADRI

 

Subjectivity is the curse on examinations for UK doctors - get rid of it

Subjectivity is the curse on examinations for UK doctors - get rid of it

There is currently a controversy raging in UK healthcare about doctors. For many years it was known and was passively accepted that if you were of Black or Minority Ethnic origins more so if you were an International Medical Graduate (IMG - a doctor with a primary medical qualification outside the UK or European Union) facing a Royal College examination you would have a much lesser chance of passing the examination. If you were of BME or IMG origins and were of a generous persuasion you would call this sub-conscious bias but most called this racism, whether it was racism or not. If you were representing the establishment you put out phrases that are superficial gibberish, such as 'reasons are multi-factorial and complex' but certainly not racism.

The issue came to a head with the MRCGP examination where in the new version of the examination the differences between white and non-white candidates were so gross that you would notice it even if you were colour blind.

The medical post graduate examinations conducted by the Royal Colleges are essentially about medical knowledge both theoretical and applied. Given that these are knowledge tests, why did the results show racial differences? We will not discuss racial supremist reasoning here.  Many of us will remember the days before the MCQs - the essay answers were often a demonstration of your wizardry in medical English. Apparently even in the MCQ based knowledge tests we can use linguistic jugglery so that a non-native English speaker comes out as having poor medical knowledge - we are not discussing that further here.

The curse of subjectivity

Applied knowledge in medicine is tested in vivas, OSCEs, with patients and simulated patients. Here the marking is done by examiners, that is where subjectivity comes in despite current best efforts, subjectivity is ruining careers.

The rest of this blog post is about subjectivity (the collection of the perceptions, experiences, expectations, personal or cultural understanding, and beliefs specific to a person - Wikipedia)

The sad paradox is knowledge especially in medicine is objective but part of the testing process of this knowledge is subjective. The tension that results from an objective topic tested subjectively is where the fundamental flaw lies. Where subjectivity exists, there bias exists and hence unethicality at the best and fraud at the worst exists.

Subjective assessments must not have a place in career make or break decisions such as exit examinations or in any arena where career progress or ability to practice the chosen profession can be stopped. Subjective assessments do have a place and can be used for progressing in learning and development - some of which are known as formative assessments. Must not be used for stop-go decisions where only objective assessments should be used.

Reducing or Eliminating subjectivity

Examiners in vivas, OSCEs, patient encounters, interview and other areas currently suffering due to subjectivity, are generally given questions - they should also be given answers and as long as the candidates answers fit in with the recognised accepted answers the candidate passes, when the answers fit in with recognised unacceptable answers the candidate fails and where the answers fit in with recognised borderline, a published formula for accepted number of borderline for a pass or fail should be defined (no, this is not the 'borderline method' that is used in standard setting).

This may beg the question whether vivas are needed at all - verbal communication is essential in all walks of life and especially so in healthcare; a candidate should be able to answer effectively and accurately under stressful verbal conditions and hence vivas are needed but the subjectivity of the vivas must be eliminated.

Subjectivity cannot be sometimes avoided but when forced to use it the answers should be 'force fit' in a pre-defined uniform manner and the candidates be assessed against that uniform force fit. The candidate does not have to know what the defined force-fit answer is but all candidates would be marked against the same answer.  

Lets look at an example: Let us assume that in a scenario where there is a certain level of oxygen desaturation which does not impact on life or limb but where a candidate has to act - say an peripheral oxygen saturation that has fallen from 98 to 89 but where the patient is otherwise very stable. The candidate has to make preparations for an adverse eventuality but there was no need to act immediately. Let us also assume that currently this is subjective and hence an assessor would mark someone and this would be variable (depending on the other skills of the candidate). Let us try a force-fit answer for this scenario - the examiner would be given a set of answers and would give the candidate a mark for each correct answer, for instance, a) the patient if conscious was asked if she was okay within two seconds 1 mark

b) the pulse oximeter probe was checked and re-applied within 4 seconds

c) capnograph reading checked within 6 seconds

d) the oxygen flow and any gas mix ups were checked within 8 seconds

d) airway tube position checked within 10 seconds

e) airway change kit and reversal drugs asked to be brought in and kept ready with 12 seconds

etc. You get the picture.

These answers may not be based on evidence because there is no evidence to base it on. However, for the purposes of the assessments the answers are defined on the basis of agreement between examiners and are used uniformly with all candidates. Then the chances of the examiner being influenced by mastery of the language, social status of an accent, the image projected by clothes, the false confidence provided by a charming smile or colour of the skin would be less.

Subjective experts are simply socially acceptable influential frauds providing a certain voyeuristic celebrity value when they are reviewing wines, films or restaurants. Techniques similar to those have no place in medical examinations. It is of course a completely different story that the British are not able to trust the training provided to their young doctors for somewhere between a minimum five years (in the case of general practice) or an approximate minimum of twelve years in the case of surgeons that makes an 'exit' exam essential to cross check knowledge (which is then pretty badly due to the subjective components). In the USA exit exams are not mandatory, they are voluntary, the Americans obviously have a great degree of confidence in their trainers, trainees and training system. The British system needs reform and a commitment to eliminate subjectivity when the stakes are high could be core to whether the UK will ever have a equitable outcome in examination results.

 

© HEMADRI

 

Follow me on twitter @HemadriTweets

This blog has argued for reducing or eliminating subjectivity from re-validation http://successinhealthcare.blogspot.co.uk/2012/11/revalidation.html

We have discussed differential results in surgical Royal College examinations http://successinhealthcare.blogspot.co.uk/2012/12/exit-exam.html