Dimensionless squared jerk: An objective differential to assess experienced and novice probe movement in obstetric ultrasound

Abstract Objective Widely accepted, validated and objective measures of ultrasound competency have not been established for clinical practice. Outcomes of training curricula are often based on arbitrary thresholds, such as the number of clinical cases completed. We aimed to define metrics against which competency could be measured. Method We undertook a prospective, observational study of obstetric sonographers at a UK University Teaching Hospital. Participants were either experienced in fetal ultrasound (n = 10, >200 ultrasound examinations) or novice operators (n = 10, <25 ultrasound examinations). We recorded probe motion data during the performance of biometry on a commercially available mid‐trimester phantom. Results We report that Dimensionless squared jerk, an assessment of deliberate hand movements, independent of movement duration, extent, spurious peaks and dimension differed significantly different between groups, 19.26 (SD 3.02) for experienced and 22.08 (SD 1.05, p = 0.01) for novice operators, respectively. Experienced operator performance, was associated with a shorter time to task completion of 176.46 s (SD 47.31) compared to 666.94 s (SD 490.36, p = 0.0004) for novice operators. Probe travel was also shorter for experienced operators 521.23 mm (SD 27.41) versus 2234.82 mm (SD 188.50, p = 0.007) when compared to novice operators. Conclusion Our results represent progress toward an objective assessment of technical skill in obstetric ultrasound. Repeating this methodology in a clinical environment may develop insight into the generalisability of these findings into ultrasound education.


| INTRODUCTION
Ultrasound is a dynamic, real-time imaging modality that is widely used in clinical obstetrics for screening of congenital anomalies, to monitor fetal growth and to assess fetal well-being. The quality of an ultrasound examination is known to be operator dependent and to have high inter-operator variability. 1 This directly affects the information available to the clinician. Consequently, training and competence assessment are of great importance to ensure effective, reproducible and safe clinical practice. Varying standards for training and assessment of skills have been suggested by bodies such as the American Institute of Ultrasound in Medicine (AIUM) 2 and The International Society of Ultrasound in Obstetrics and Gynaecology (ISUOG). 3 National colleges have introduced varying requirements. The French National College of Gynaecologists and Obstetricians (CNGOF) require practitioners to pass a national examination to obtain certification. On the other hand in Sweden and Italy there is neither a formal curriculum in obstetric ultrasound, nor a minimum number of scans to complete prior to independent practice. 4 Similar challenges for trainees have been reported in other medical 5 and surgical 6 specialities, where reduced training hours have led to concerns 7 that current specialists-in-training will have had fewer training hours than their predecessors. 8 Unlike first trimester ultrasound, 9 a widely accepted threshold for competency and service audit has not been established for fetal biometry. Where defined, outcomes of training curricula are often based on arbitrary thresholds, such as the number of clinical cases completed rather than objective measures of competence. 10 These thresholds assume competence based on the expectation that experience equates to expertise. This assumption has been challenged by Tolsgaard et al, 11 which found that that some experienced ultrasound operators did not display expert-like behaviours, despite daily use of obstetric ultrasound in their clinical practice. The authors hypothesised that inadequate basic training may be a root cause, as the operators did not have the correct foundation to benefit from later clinical training. The authors further hypothesised that the expected improvement in performance was not seen because sustained deliberate practice rarely occurs in clinical practice.
The acquisition of complex skills and psychomotor tasks follows a learning curve. Ultrasound examinations, much like in minimally invasive surgery (MIS) hand-eye coordination, require the operator to interpret a dynamic image produced by the three-dimensional (3D) position and motion of the ultrasound probe by means of a twodimensional (2D) visual display. The challenge of delivering high quality training leading to safe, reproducible practice has been considered extensively in minimally-invasive surgery. 12,13 Obstetric ultrasound requires a detailed understanding of the capabilities and limitations of ultrasound equipment, as well as knowledge of normal maternal and fetal anatomy, as well as an appreciation of abnormal findings. The experienced operator will be able to combine this knowledge with the technical ability to manipulate transducer and display settings to optimise the image, leading to higher objective image quality. It is accepted that performance improves with training and experience, 7,8,14 but this observation has not been objectively measured or classified.
Dimensionless squared jerk has been proposed as an objective parameter to discriminate between expert and novice operators in endoscopic surgery. 15 Dimensionless squared jerk is a measure of deliberate hand movements and is a derivative of acceleration with respect to time and distance while remaining independent of spurious peaks and dimension within the movement. 16 Dimensionless squared jerk is normalised with respect to time duration, so that only the shape of the trajectory -and not its dimension/extension -contributes to jerk. Dimensionless squared jerk quantifies common deviations from smooth, coordinated movement and it has been accepted as an objective parameter to quantify hand motion in different disciplines, such as parkinsonism, kinetics, and optometry. 17,18 To our knowledge, no study quantifying hand motion using Dimensionless squared jerk during simulated ultrasound has been performed. Inspired by comparable training challenges in endoscopic surgery, this study observed operator behaviour by tracking the ultrasound probe during an ultrasound scan and by comparing the images obtained. We hypothesise that dimensionless squared jerk could be used to differentiate experienced from novice operators. We compared the results between experienced and novice ultrasound operators with the aim of establishing specific, measurable and reproducible performance differences between experienced and novice operators.

| METHODS
We undertook a prospective, observational study of medical practitioners who were accredited specialists or specialty trainees in Obstetrics & Gynaecology at University College London Hospital NHS Foundation Trust, London, UK (UCLH). Participants were empirically divided between either experienced in fetal ultrasound (n = 10, >200 fetal ultrasound examinations, "experienced") or novice operators What is already known about this topic?
• There is little global consensus on how to train, assess and evaluate skill in obstetric ultrasound.
• Curricula, where present, are often based on clinical volume, rather than objective outcomes of competence.
• The acquisition of complex psychomotor tasks follows a learning curve, but they have not been objectively measured or classified.
• DSJ has been proposed as an objective parameter in endoscopic surgery where similar issues are faced.
What does this study add?
• Inspired by comparable training challenges in endoscopic surgery, this study observed operator behaviour by tracking the ultrasound probe during an ultrasound scan.
• We included experienced and novice operators.
• We describe measurable and reproducible performance differences between the groups.
• When time and DSJ are considered together, the experienced group had less unwanted and more purposeful movements.
• Our results show that quantifiable performance changes can be defined using objective measures.
(n = 10, <25 ultrasound examinations, "novice"). These numbers were selected as The European Board and College of Obstetrics and Gynaecology (EBCOG) guideline on obstetric training recommends that trainees complete a logbook as part of their training which contains 200 obstetric ultrasound examinations. 19 The study was undertaken at the Obstetric Ultrasound Unit at UCLH. Clinical experience varied from Foundation Year 2 Doctors to Consultants in Fetal Medicine.
The study was exempt from review by the NHS Research Ethics Committee as it was not performed on patients. Informed consent was obtained from each participant.
Each participant was asked to obtain standard 2D fetal measurements using a clinical GE Voluson E8 ultrasound machine  Figure 1(B).
The required views were: • The trans-ventricular plane, with bi-parietal diameter (BPD) measurement.
• The trans-abdominal plane, on which the Anterior Posterior Abdominal Diameter (APAD), the Transverse Abdominal Diameter (TAD) and the Abdominal Circumference (AC) can be measured.
• A view of the femur and a measurement of its length (FL).
These views were chosen as these are the views are necessary to calculate fetal weight, in accordance with Hadlock B formula and are in accordance with guidance on the estimation of fetal weight from ISUOG. 20 The Hadlock B formula was used to calculate the estimated fetal weight as it is widely used to estimate fetal weight and it has a lower error rate than other formulas. 21 Prior to commencing the scan each participant was given written instructions (supplement 1), including images of the required planes, Figure 1, and up to 5 min to familiarise themselves with the operation of the ultrasound scanner. Both novice and expert operators were allowed unlimited time to complete the task. Participants were permitted to refer to the instructions at any time during the task, if they wished to.
The position of the probe was tracked using the Aurora electromagnetic tracking system (NDI Inc., Ontario, Canada). The experimental set-up is shown in Figure 2. To track the probe a 6 degree-of-freedom (DOF) sensor was attached to the ultrasound probe by means of a custom 3-D printed holder, as shown in Figure 1  Dimensionless squared jerk (DSJ) 16,23 was calculated for each novice and expert participant, using the formula described by Hogan and Sternad.
where: A is movement amplitude or extent, t1 = initial time; t2 = final time; t2 − t1 = completion time; D = path length and D = t 2 − t 1 is duration.

| RESULTS
Experienced operator performance was associated with a shorter time to task completion and probe travel. On average a novice operator took almost four times longer to perform the task. The mean experi- When evaluating the images used to estimate fetal weight in accordance with the scoring system detailed in the methodology, we found that experienced operators achieved a higher quality image score (mean score 11.8, SD 1.87) than novice operators (10.2 SD 1.46, p = 0.04) ( Table 1). We found that more experienced operators achieved a higher score for the submitted images, but this did not cor-   we reported in our literature review, 24 it is worth noting that some clinicians will never achieve expert levels of performance despite daily use of ultrasound. We remind the reader that each ellipse contains results within an 95% confidence interval and, as such, outliers are to be expected. Our aim at this stage is not to try to move the outliers

| Strengths and Limitations
This study used clinical ultrasound systems, a commercially available probe tracking system and the participants were reflective of the skills mix in many ultrasound departments.

| CONCLUSION
DSJ may represent a metric by which training progress could be assessed. Alone, a single metric is unlikely to represent experienced skill, but could be used as a waypoint for trainees, signifying the transition from simulated learning to supervised practice. Integrating ultrasound simulation into training curricula and promoting self-directed learning would give trainees the opportunity to contribute to the clinical service while learning a complex skill. In using DSJ, we propose a metric which gives trainees, and their trainers an objective measure of their progress over time which could be used in addition to other metrics that may, for example, be based on image data.
DSJ, alone cannot be assumed to equate to clinical competence, but objectively measures one of several technical and non-technical behaviours required to perform obstetric ultrasound to a high standard. DSJ supports the idea that the process of performing an ultrasound scan is highly discriminatory between the novice and experienced operator. Further work is required to correlate this metric with tools traditionally used to evaluate training and standardised quality control gradings for images produced by the operator. These might include OSATs, as used by the RCOG and the ultrasoundspecific assessment OSAUS assessment, or the quality control framework adopted by the Fetal Medicine Foundation.
Our future work in this area will be to repeat this experiment in a clinical environment, adequately powered to show difference. Repeating the study in a clinical setting will allow us to understand if skills assessed in a simulated environment can be translated to clinical reality and how robust skills are when presented with the variability inherent in obstetric scanning owing to maternal habitus, stage of pregnancy, fetal presentation and position.