Subjectivity is the curse on examinations for UK doctors - get rid of it

Subjectivity is the curse on examinations for UK doctors - get rid of it

There is currently a controversy raging in UK healthcare about doctors. For many years it was known and was passively accepted that if you were of Black or Minority Ethnic origins more so if you were an International Medical Graduate (IMG - a doctor with a primary medical qualification outside the UK or European Union) facing a Royal College examination you would have a much lesser chance of passing the examination. If you were of BME or IMG origins and were of a generous persuasion you would call this sub-conscious bias but most called this racism, whether it was racism or not. If you were representing the establishment you put out phrases that are superficial gibberish, such as 'reasons are multi-factorial and complex' but certainly not racism.

The issue came to a head with the MRCGP examination where in the new version of the examination the differences between white and non-white candidates were so gross that you would notice it even if you were colour blind.

The medical post graduate examinations conducted by the Royal Colleges are essentially about medical knowledge both theoretical and applied. Given that these are knowledge tests, why did the results show racial differences? We will not discuss racial supremist reasoning here.  Many of us will remember the days before the MCQs - the essay answers were often a demonstration of your wizardry in medical English. Apparently even in the MCQ based knowledge tests we can use linguistic jugglery so that a non-native English speaker comes out as having poor medical knowledge - we are not discussing that further here.

The curse of subjectivity

Applied knowledge in medicine is tested in vivas, OSCEs, with patients and simulated patients. Here the marking is done by examiners, that is where subjectivity comes in despite current best efforts, subjectivity is ruining careers.

The rest of this blog post is about subjectivity (the collection of the perceptions, experiences, expectations, personal or cultural understanding, and beliefs specific to a person - Wikipedia)

The sad paradox is knowledge especially in medicine is objective but part of the testing process of this knowledge is subjective. The tension that results from an objective topic tested subjectively is where the fundamental flaw lies. Where subjectivity exists, there bias exists and hence unethicality at the best and fraud at the worst exists.

Subjective assessments must not have a place in career make or break decisions such as exit examinations or in any arena where career progress or ability to practice the chosen profession can be stopped. Subjective assessments do have a place and can be used for progressing in learning and development - some of which are known as formative assessments. Must not be used for stop-go decisions where only objective assessments should be used.

Reducing or Eliminating subjectivity

Examiners in vivas, OSCEs, patient encounters, interview and other areas currently suffering due to subjectivity, are generally given questions - they should also be given answers and as long as the candidates answers fit in with the recognised accepted answers the candidate passes, when the answers fit in with recognised unacceptable answers the candidate fails and where the answers fit in with recognised borderline, a published formula for accepted number of borderline for a pass or fail should be defined (no, this is not the 'borderline method' that is used in standard setting).

This may beg the question whether vivas are needed at all - verbal communication is essential in all walks of life and especially so in healthcare; a candidate should be able to answer effectively and accurately under stressful verbal conditions and hence vivas are needed but the subjectivity of the vivas must be eliminated.

Subjectivity cannot be sometimes avoided but when forced to use it the answers should be 'force fit' in a pre-defined uniform manner and the candidates be assessed against that uniform force fit. The candidate does not have to know what the defined force-fit answer is but all candidates would be marked against the same answer.  

Lets look at an example: Let us assume that in a scenario where there is a certain level of oxygen desaturation which does not impact on life or limb but where a candidate has to act - say an peripheral oxygen saturation that has fallen from 98 to 89 but where the patient is otherwise very stable. The candidate has to make preparations for an adverse eventuality but there was no need to act immediately. Let us also assume that currently this is subjective and hence an assessor would mark someone and this would be variable (depending on the other skills of the candidate). Let us try a force-fit answer for this scenario - the examiner would be given a set of answers and would give the candidate a mark for each correct answer, for instance, a) the patient if conscious was asked if she was okay within two seconds 1 mark

b) the pulse oximeter probe was checked and re-applied within 4 seconds

c) capnograph reading checked within 6 seconds

d) the oxygen flow and any gas mix ups were checked within 8 seconds

d) airway tube position checked within 10 seconds

e) airway change kit and reversal drugs asked to be brought in and kept ready with 12 seconds

etc. You get the picture.

These answers may not be based on evidence because there is no evidence to base it on. However, for the purposes of the assessments the answers are defined on the basis of agreement between examiners and are used uniformly with all candidates. Then the chances of the examiner being influenced by mastery of the language, social status of an accent, the image projected by clothes, the false confidence provided by a charming smile or colour of the skin would be less.

Subjective experts are simply socially acceptable influential frauds providing a certain voyeuristic celebrity value when they are reviewing wines, films or restaurants. Techniques similar to those have no place in medical examinations. It is of course a completely different story that the British are not able to trust the training provided to their young doctors for somewhere between a minimum five years (in the case of general practice) or an approximate minimum of twelve years in the case of surgeons that makes an 'exit' exam essential to cross check knowledge (which is then pretty badly due to the subjective components). In the USA exit exams are not mandatory, they are voluntary, the Americans obviously have a great degree of confidence in their trainers, trainees and training system. The British system needs reform and a commitment to eliminate subjectivity when the stakes are high could be core to whether the UK will ever have a equitable outcome in examination results.

 

© HEMADRI

 

Follow me on twitter @HemadriTweets

This blog has argued for reducing or eliminating subjectivity from re-validation http://successinhealthcare.blogspot.co.uk/2012/11/revalidation.html

We have discussed differential results in surgical Royal College examinations http://successinhealthcare.blogspot.co.uk/2012/12/exit-exam.html

Revalidation - GMC must make it objective as soon as possible

The shortest overview of revalidation

GMC is commencing the process of revalidation for doctors in December 2012. The revalidation demands that we have evidence of 

1) Continuing professional development

2) Quality Improvement Activity

3) Significant events

4) Feedback from colleagues

5) Feedback from patients

6) Review of complaints and compliments

These six will populate the annual appraisal which apart from its main domains include the personal development plan, probity and health

Based on the above, the responsible officer will make a 'judgement' on whether the doctor can be recommended for revalidation. The GMC will then make a decision on whether the doctor has been successfully revalidated.

There is plenty of guidance on GMC website : http://www.gmc-uk.org/doctors/revalidation.asp

Concerns about the background for revalidation

While the issue of periodic quality assurance of licensed doctors has been discussed for a long time, the common view is that the current revalidation efforts commenced after the Bristol enquiry and gathered momentum after the Shipman enquiry. Bristol was an outlier, there was no trend that many hospitals or many cardiac surgery units were having unacceptably bad outcomes. Shipman was an outlier, there was no trend that many doctors were behaving or beginning to behave in a Shipman like manner. Outliers need to be analysed properly so that outliers can be stabilised to a performance level compatible with other performers within the general system. Quality principles would suggest that outliers should not trigger a process change for the whole system. Process change for a system could be triggered by an unacceptable trend (there are other reasons to change the process as well, but outlier is generally not one of them). To create a process change on the basis of outliers is thought to result in unnecessary expense and wasted effort.

This does not mean that we cannot learn from outliers, undoubtedly there are extraordinarily important lessons to be learned from Bristol and Shipman.

Linking the background to current revalidation method

Bristol is about performance and Shipman is about behaviour. We can safely assume that this is what the GMC seeks to assure. Quality assurance needs to be demonstrated in an objectively measureable manner.

Revalidation criteria - Not Objective

The six areas of evidence that the GMC asks for seem to be mostly subjective.

Continuous professional development (CPD) is generally accepted as a reflection of time spent on courses and conferences or other learning opportunities. It is certainly not a measure of the knowledge or skills gained or updated though that might happen. Some professional bodies have not defined the time needed to be spent on CPD. Hence while CPD is often measured and entered as a number it is a measure of time spent rather than a number to show the knowledge or skills gained. It could therefore be argued that CPD is either subjective or fit for purpose for revalidation if the intention was to validate or assure knowledge and/or skills.

Quality Improvement Activity: within this areas a range of activity is included. Activity is neither outcome nor achievement. Therefore activity is again time spent rather than gains (or losses) measured. An important point in quality improvement activity is that people fail more often than they succeed, that is the nature of quality improvement. Doctors, hospitals and the GMC should be comfortable with that. This could be potentially be an objective criteria but currently it could probably be considered unsure.

Learning from significant events and review of complaints and compliments are about self-reflection and reflective writing. It is obviously subjective. Feedback from colleagues and feedback from patients though done through validated tools by external or independent service providers is essentially the conversion of subjectivity into a scale to be able to measure.

Are subjective criteria relevant?

Absolutely yes. But only when looked along with objective criteria. Any form of quality assurance process must include subjectivity. The current criteria for revalidation seems mostly subjective and hence the concerns.

Why are objective criteria important?

We are talking about doctors who are essentially already very highly qualified and doing an extremely complex job under phenomenally varying conditions. We are taking about professionals on whom we have already spent somewhere between half-a-million to a million pounds before they are employed to do their role. Revalidation is about making a decision about their careers which could potentially be halted. To make such major decisions on mostly subjective criteria would not make sense. Further, there are planet loads of data already gathered and analysed and hence objective criteria are possibly already available if we wanted to use them.

Next is the issue of who may potentially be adversely affected to a higher degree than most. When subjective criteria are used there is a risk that often the weak, the easy targets and usual suspects may be affected. This has been seen in a few exam situations where certain sections of candidates pass the objective knowledge and skill components but fail the subjective elements of vivas, communication, simulation etc. There is a fear that it is possible that IMGs and BMEs (and SAS) doctors  would be affected by the level of subjectivity involved in revalidation.

There are good reasons behind these fears which relate to the culture and history of healthcare institutions and the culture and mind-set of BME/IMG doctors which is not explored here.

Increasing objectivity

Testing knowledge has traditionally been done by examinations. Americans revalidate their doctors on the basis of an objective examination of knowledge. This while reducing bias increases the validity of assurance of knowledge. Skills assessment could quite relevantly be based on performance data. Speaking from a hospital doctor perspective, this should be quite easy to do with some minimal tweaking on how data is gathered. Operational performance data is either the best or as good as any other indicator of a doctors skill.

Increasing objectivity still would not resolve the underlying issue of a process change for all doctors based on outliers and not trends. The GMC also needs to resolve other issues. Is revalidation a quality assurance process or a quality improvement process? Because the theory and the tools for assurance are different from improvement

Revalidation is important. It is likely that as it stands the revalidation process is heavily subjective. Given the importance of healthcare of the nation it would be advisable to quickly move to mainly objective criteria. We are where we are, let us make it better and fit for purpose.

©M HEMADRI 

Follow me on twitter @HemadriTweets