Recruitment of newly qualified doctors – a work in progress

Exiting UK medical students are currently allocated to their foundation training (FT) posts based on the combination of an Educational Performance Measure (EPM) encompassing scores from medical school performance in addition to additional degrees and publications as well as results from a Situational Judgement test (SJT), a new written test aiming to assess the professional attributes of a newly qualified doctor (namely commitment to professionalism, coping with pressure, effective communication, patient focus and working effectively as part of a team). Both the EPM and SJT are scored equally at 50 points for each (The Foundation Programme, 2016).


Limited supervisor sample
Only 46 % of supervisors reporting on the low scoring group responded, compared to 37 % of supervisors reporting on the high scoring group. In addition, perceived accuracy of performance ratings by supervisors did not correspond directly with length of supervision; supervisors with the shortest supervision reported overall high levels of perceived accuracy. Supervisor questionnaires focused on five domains: commitment to professionalism, coping with pressure, patient focus, problem solving and decision making and working effectively as part of a team. The Likert scale ranged from 1 = needed significant development to 6 = clear area of strength. Means for the high scoring application groups ranged from 4.8 to 5.2 and for the low scoring 4.4 to 4.8. Both groups in effect scored above average. It is unlikely for newly starting doctors to gain a perfect 6 and this would be true for most working doctors. Doctors are of course rarely perfect practitioners. Further analysis highlights that the majority of supervisors were educational or foundation programme directors (FPD) as opposed to actual clinical supervisors (CS) (Educational Supervisors (ES) 187, FPD 169 and CS 25). Interestingly the report concludes that some low SJT scorers did go on to achieve high performance ratings (Patterson F et al, 2015).

Educational Performance Measure vs the SJT
The EPM was noted to be a stronger predictor of performance for the higher scoring SJT group. When analysing the effect of additional EPM points, the report noted improvement to the extent to which EPM scores differentiated between applicants and that this was more pronounced for the higher scoring SJT group. The report however discouraged against any increase in the weighting of the EPM against the SJT at present. For the 2013 FP applicant cohort a weakly positive correlation was noted between EPM and SJT scores n=8,127 (rs .30). Correlations with the EPM in 2014 was noted at r=.30 and 2015 r=.34. (Patterson F et al, 2015).

Relationship of ARCP and SJT outcomes
ARCP outcomes were grouped into two domains: 'satisfactory' completion of F1 and 'unsatisfactory' comprising inadequate progress additional training time required, released from training programme and incomplete evidence presented. This in itself is problematic as aspects of an unsatisfactory outcome encompassing these three domains does not necessarily imply a poor performer and a multitude of reasons may account for an unsatisfactory outcome which are not directly linked to 'on the job' working habits. Further analysis showed no significant differences between SJT or EPM scores for those who achieved satisfactory ARCP outcomes compared to those who received unsatisfactory outcomes. The report notes that there is a need to determine a true association between ARCP outcome and low scoring SJTs. (Patterson F et al, 2015).

Remedial action and the SJT
For those that received remedial action the report noted lower scores on the EPM, SJT and total application score. However the number analysed was small at n=16/ 391. For doctors who had received remedial action, analysis of receipt of lower SJT scores showed a small r of 0.19. (Patterson F et al, 2015).

SJT scores and F1 performance
The report concluded that for those who achieved lower SJT scores, the SJT showed a moderate relationship with F1 performance; a moderate Pearson correlation of 0.54. Interestingly for higher SJT scorers there was a small performance relationship with the SJT with a poor Pearson correlation of 0.20. The report concluded that the SJT could only be useful for screening out extremely low scorers as opposed to differentiating between highest performers. It is likely however that low scores would be by and large minimal and the real purpose of the SJT Sharma N MedEdPublish https://doi.org/10. 15694/mep.2016.000138 Page | 3 should be to ensure a fair and accurate differentiation between all candidates when competition for jobs is at stake. Furthermore, the report highlights that it is not known how those who received middle SJT scores perform in practice. Surely this is worth knowing as the majority of doctors are likely to fall into this group. (Patterson F et al, 2015).

Students' concerns
In 2013, a total of 7770 applicants were asked to provide their reactions to the SJT. Only 52.5 % thought that the content seemed relevant to what they thought the role of a foundation doctor should be (vs 57.1 % in 2014 and 56.5 % in 2015). 38.6 % agreed or strongly agreed that the content of the SJT appeared to be fair to the foundation programme (vs 40.4 % in 2014 and 40.6 % in 2015), with 25.4 % applicants agreeing or strongly agreeing that the results of the SJT could help selectors to differentiate between weaker and stronger applicants (vs 26.1 % in 2014 and 26.3 % in 2015). (Patterson F et al, 2013(Patterson F et al, , 2014(Patterson F et al, , 2015a.

SJT and demographics
From 2013 to 2015 group differences analysis revealed significant differences between performance in the test based on age, gender and ethnicity. Females scored higher than males and white applicants outperformed BME candidates (Patterson F et al, 2013(Patterson F et al, , 2014(Patterson F et al, , 2015a.

Further evidence needed
The SJT AMEE guide concluded that in fact research is needed to determine the link between SJTs and performance in independent practice. And that SJTs have different predictive validity at different stages during medical education. (Patterson F et al, 2016).

Notes On Contributors
Dr Neel Sharma graduated from the University of Manchester and is currently based at the Albert Einstein College of Medicine, New York, USA