pISSN 2005-291X   eISSN 2005-4408
Open Access, Peer-reviewed
    Clin Orthop Surg. 2021 Dec;13(4):513-519. English.
    Published online Nov 15, 2021.
    https://doi.org/10.4055/cios20259
    Copyright © 2021 by The Korean Orthopaedic Association
    Original Article

    Detection of Rotator Cuff Tears by Ultrasound: How Many Scans Do Novices Need to Be Competent?

    Dong Min Kim, MD,# Jae-Seong Seo, MD,* In-Ho Jeon, MD, Changho Cho, MD and Kyoung Hwan Koh, MD
      • Department of Orthopaedic Surgery, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea.
      • *Department of Orthopaedic Surgery, Seoul Medical Center, Seoul, Korea.
    • Correspondence to: Kyoung Hwan Koh, MD. Department of Orthopaedic Surgery, Asan Medical Center, University of Ulsan College of Medicine, 88 Olympic-ro 43-gil, Songpa-gu, Seoul 05505, Korea. Tel: +82-2-3010-3530, Fax: +82-2-488-7877, Email: osdoc.koh@gmail.com

      #Current affiliation: Department of Orthopaedic Surgery, Sarang Plus Hospital, Seoul, Korea
    Received October 21, 2020; Revised January 02, 2021; Accepted February 02, 2021.

    This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

    Abstract

    Background

    Ultrasound is commonly used for evaluating rotator cuff tears. However, little training in ultrasound imaging is provided during orthopedic residents’ training period. Therefore, we performed this study to determine how many ultrasound scans are required for orthopedic residents to be competent and self-confident in the diagnosis of supraspinatus tendon tears and to investigate whether senior residents outperformed junior residents.

    Methods

    We studied two third-year residents who had no previous experience of shoulder ultrasound and evaluated their ability to detect rotator cuff pathologies. Their learning curves were plotted using a cumulative summation analysis with a 20% acceptable failure rate compared to arthroscopic findings. Downward, upward, and horizontal cumulative summation trends indicated incompetence, exceptional competence, and competence, respectively. The diagnostic accuracy of third-year residents was compared with that of second-year residents and the number of cases required to gain self-confidence was evaluated.

    Results

    Cumulative summation analysis showed that after 26–28 scans, residents achieved the competence to correctly diagnose supraspinatus tears: an upward trend was observed from the beginning for full-thickness tears and a downward trend was observed for partial-thickness tears. Sensitivity and specificity were 0.95 and 0.79, respectively, for third-year residents and 0.91 and 0.58, respectively, for second-year residents. Residents reported self-confidence after 30 ultrasound scans for the detection of rotator cuff tears.

    Conclusions

    The number of scans that novices needed to be competent for detecting rotator cuff tears was approximately 30 cases, and the diagnostic accuracy of third-year residents was significantly higher than that of second-year residents.

    Keywords
    Rotator cuff injuries; Shoulder; Ultrasonography; Diagnosis; Learning curve

    Ultrasound (US) is commonly used for evaluating shoulder injuries and has been utilized as a reliable and accurate imaging modality for identifying rotator cuff tears.1, 2, 3, 4) When US is performed by experienced radiologists, US and magnetic resonance imaging (MRI) results are comparable regarding the identification and quantification of rotator cuff pathologies.1, 2, 5, 6, 7) Furthermore, advances in US technology have led to the more widespread use of this imaging method, even among general physicians. More recently, US technology has produced more affordable, portable, and higher-resolution scans. Although more general physicians now utilize this modality to evaluate pathologies and administer injections, these physicians lack specific training in US imaging. It has resulted in some debate surrounding the training and level of competence in general physicians’ use of US.

    The American Institute of Ultrasound in Medicine and the European Federation of Societies for Ultrasound in Medicine and Biology announced training guidelines for the musculoskeletal US for general physicians.8, 9) These guidelines state that a minimum of 150 to 300 scans must be read under the supervision of a radiologist to obtain proficiency. However, a few studies have reported that orthopedic surgeons can evaluate rotator cuff tears using US approximately as well as musculoskeletal radiologists.10, 11, 12) Unfortunately, more knowledge is needed to determine the required amount of background knowledge to achieve US proficiency. No previous study has evaluated the learning curve for the interpretation of US images; such knowledge is particularly required by orthopedic surgeons and orthopedic residents because the specialists’ background knowledge may significantly shorten the training period required to reach competency for the diagnosis of rotator cuff pathologies using US imaging. Furthermore, orthopedic surgeons efficiently learn US because they can combine image, history taking, physical examination, and arthroscopic findings. However, little training in US imaging is provided during orthopedic residents’ four-year training period.

    Therefore, we performed this study to determine how many US scans are required for orthopedic residents to be competent and self-confident in the diagnosis of supraspinatus tendon tears and to investigate whether senior residents outperformed junior residents. We hypothesized that senior residents would need a smaller number of US scans to be competent and self-confident than junior residents in the diagnosis of supraspinatus tendon tears.

    METHODS

    All procedures performed in studies involving human participants were in accordance with the ethical standards of the Institutional and National Research Committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. The study was approved by the Institutional Review Board of Seoul Medical Center (IRB No. SMC 2013-017). Informed consent was obtained from all patients and residents before the start of this study.

    Patients

    From July 2013 to February 2014, 53 consecutive patients underwent arthroscopic subacromial decompression with or without rotator cuff repair. Those patients with a history of prior fracture around the affected shoulder, revision surgery, accompanying calcific tendinitis, or infection were excluded from this study. After the additional exclusion of 3 patients with incomplete data and 3 who refused to participate, 47 patients were analyzed. The included patients were 18 men and 29 women with a mean age of 59 years (range, 41–79 years).

    Methods for US Examination

    Two third-year and 2 second-year residents without previous experience in shoulder US participated in this study. All residents completed a hands-on workshop provided by the Korean Orthopaedic Ultrasound Society. Additionally, the in-hospital training of US scans was performed. In March 2013, all trainees were taught shoulder anatomy through a shoulder anatomy seminar. Subsequently, they attended US lectures conducted by a shoulder surgeon (KHK). Supervised scanning of 5 normal shoulders was also provided by the shoulder surgeon (KHK). Each resident’s ability to detect rotator cuff pathologies was evaluated. A high standard of patient care was ensured by the principal investigator (KHK), who had more than 10 years of experience in shoulder US, examining all participants at an outpatient clinic. The residents performed US scans (HD 11 XE; Philips, Korea) one day before arthroscopic surgery without any prior knowledge of the patient condition and without performing any physical examination.

    US Assessment

    We reviewed the literature on US for rotator cuff tears and developed a case report form to standardize US examination results and reports of pathologies in the supraspinatus tendon.13) The following items were recorded in this report after each scan: (1) discontinuity, (2) focal thinning or loss of convexity in the bursal surface, (3) cartilage interface sign, (4) cortical irregularity, and (5) intra-articular or bursal fluid collection. A final diagnosis of the pathology of the supraspinatus tendon was recorded as a normal, partial tear, or full-thickness tear at the end of the examination. Additionally, the residents’ level of self-confidence regarding diagnostic accuracy was documented.

    The diagnostic accuracy of US imaging with the arthroscopic diagnosis used as the reference standard and each scan’s time-to-completion were calculated for the 2 third-year residents. Learning curves were plotted using cumulative summation (CUSUM) analysis. The second part of this study involved evaluating the differences between residents in different years of training. Senior residents have more background knowledge about the anatomy of the shoulder and rotator cuff tears, so we hypothesized that senior residents would outperform junior residents at diagnosing rotator cuff pathologies. The data for 2 second-year residents were available for 40 US scans, and the number of cases required to achieve self-confidence was evaluated for these 2 second-year residents, and diagnostic accuracy was compared between the second- and third-year residents.

    Statistical Analysis

    Learning curves were plotted using CUSUM analysis14, 15) with a 20% acceptable failure rate compared to the arthroscopic findings. We set 20% as the failure rate by referring to a similar study on spinal US.14) If a resident successfully performed US, 0.2 points were awarded; if a resident failed to perform US successfully, (1–0.2) = 0.8 points were deducted. The CUSUM curve was plotted with the CUSUM value representing continuously accumulated scores on the Y-axis and the number of consecutive attempts on the X-axis. Downward, upward, and horizontal CUSUM trends indicated a lack of competence, exceptional competence, and competence, respectively. Continuous variables were compared by Student t-test, and categorical data were compared by chi-square test and Fisher’s exact test. Statistical analyses were performed with IBM SPSS ver. 22.0 (IBM Corp., Armonk, NY, USA) with a significance level of 0.05 for all analyses.

    RESULTS

    The mean US scan time-to-completion was 14 minutes (range, 9–23 minutes; standard deviation, 3 minutes). The diagnostic accuracy of supraspinatus tears is shown in Tables 1 and 2 for the second- and third-year residents, respectively. Diagnostic parameters (sensitivity, specificity, and positive and negative predictive values) for supraspinatus tear are summarized in Table 3. The third-year residents’ diagnostic parameters were higher than those of the second-year residents. A CUSUM plot revealed that 26–28 scans were required to achieve competence in detecting supraspinatus tears (Fig. 1). When we divided the supraspinatus tears into partial tears and full-thickness tears, the CUSUM graph showed an upward trend from the beginning of the examinations in the diagnosis of full-thickness tears and a downward trend in the diagnosis of partial-thickness tears (Fig. 2).

    Fig. 1
    Cumulative summation graph showing that 26–28 scans are needed to achieve competency in detecting supraspinatus tears.

    Fig. 2
    (A) Cumulative summation graph of full-thickness tears showing an upward trend from the beginning of examinations. (B) Cumulative summation graph of partial-thickness tears, revealing a downward trend until the last examination.

    Table 1
    Diagnostic Accuracy of Examiners 1 and 2 (2 Third-Year Residents)

    Table 2
    Diagnostic Accuracy of Examiners 3 and 4 (2 Second-Year Residents)

    Table 3
    Diagnostic Parameters of Both Third-Year and Second-Year Residents

    Following the completion of all 47 consecutive US scans, the 2 third-year residents reported that it took approximately 30 cases to gain self-confidence in detecting rotator cuff tears. Although Fig. 3 shows that roughly 20 cases were sufficient to have self-confidence, those 2 third-year residents answered that it took 30 cases, probably because of the subjective nature of self-report. The number 30 was smaller than the self-reported number for one of the second-year residents, who was uncertain about his abilities until he had read 30 consecutive scans; the other second-year resident reported self-confidence after reading 30 scans. The self-confidence level was documented at the end of each examination, and the resultant graphs show that third-year residents were confident about their diagnosis after approximately 20 scans, whereas second-year residents reported self-confidence after approximately 30 scans (Fig. 3). The overall diagnostic accuracy of second-year residents was lower than that of third-year residents.

    Fig. 3
    (A) Two second-year residents exhibited self-confidence after about 30 scans. (B) Two third-year residents showed self-confidence after a smaller number of scans. Upper dots indicate an ultrasound scan in which the examiner exhibited self-confidence about the ultrasound diagnosis for rotator cuff pathology; lower dots indicate a lack of confidence.

    DISCUSSION

    Our study reveals that the learning curve for orthopedic residents who are novices in US imaging is very steep: only about 30 scans were necessary for the second- and third-year residents to achieve competence. In particular, orthopedic residents were remarkably competent in the detection of full-thickness supraspinatus tears from the beginning. These results differ from the guidelines published by the American Institute of Ultrasound in Medicine and the European Federation of Societies for Ultrasound in Medicine and Biology, which state that 150–300 scans are necessary for general physicians who are not radiologists.8, 9)

    We also reported a difference in diagnostic accuracy and self-confidence between second- and third-year residents. Orthopedic residents in this study who had more background knowledge and were familiar with surgical anatomy seemed to have comparable or shorter learning curves than those previously reported for general physicians for US imaging. Similarly, the third-year residents with greater knowledge and experience in orthopedic disease showed better diagnostic accuracy and earlier self-confidence in their diagnoses than the second-year residents.

    Improvements in US equipment technology have led to the more widespread use of US in almost every medical and surgical department, including general surgery, gynecology, anesthesiology, urology, and internal medicine. Although US has traditionally been considered an examination that must be performed by a radiologist or orthopedic surgeon, general physicians have started using this imaging method, particularly in cases of the musculoskeletal disease. Studies on the amount of experience required to gain US competency are rare.16, 17, 18, 19) Two studies reported results from the training of inexperienced non-orthopedic trainees.20, 21) One recent study by Alavekios et al.22) evaluated US imaging competency by two inexperienced orthopedic surgeons and reported that the majority of learning to detect full-thickness supraspinatus tears occurs within the first 50 scans; this study also reported that peaks in sensitivity and specificity were achieved at around 30 scans. Similarly, we report that 2 third-year residents required approximately 30 scans to obtain competency; these results imply that orthopedic surgeons (even third-year trainees) who have more background knowledge and are familiar with surgical anatomy have shortened learning curves compared to non-orthopedic specialists. Thus, US imaging training would be a valuable addition to the training period for orthopedic surgeons.

    In contrast, there appears to be a longer learning curve for novices to reach competency at detecting partial-thickness rotator cuff tears using US. Previous studies have reported that US shows comparable diagnostic accuracy to MRI for both partial- and full-thickness rotator cuff tears.23) In this study, inexperienced orthopedic residents performing US examinations showed a downward trend in CUSUM analysis although there were few scanned partial-thickness rotator cuff tears. Residents who were involved in this study showed a trend of mistaking partial-thickness rotator cuff tears for normal or full-thickness rotator cuff tears, which is similar to the results of a study by Murphy et al.24) The mistaken identification of a partial-thickness tear as either normal or as a full-thickness tear led to decreased diagnostic accuracy in US examination among novice examiners in our study. During the evaluation of shoulders using US, indirect signs such as cartilage interface signs or the cortical irregularity of greater tuberosity provide clues for the detection of rotator cuff tears.25) Particularly in partial-thickness tears, detecting these signs might require more experience and a longer learning curve than the detection of full-thickness tears. Although many cases of full-thickness tears with persistent symptoms necessitate surgical treatment, the determination of surgical indication in a partial-thickness tear depends on the tear’s degree and clinical manifestations, and thus requires more clinical experience. A sufficient number of US scans should be done for the proper detection of a partial-thickness rotator cuff tear using US imaging.

    All residents involved in this study had completed a hands-on workshop on US imaging and had in-hospital training with US with supervision and feedback from the principal investigator. Although orthopedic residents have more background knowledge of surgical anatomy and clinical presentation, training sessions focused on US imaging should be included before starting US imaging practice. Furthermore, standardized training in US imaging should be included in orthopedic residents’ training regimen.

    One of the limitations of our study is that true blind evaluation was impossible. Because the residents knew that all the shoulders they examined were scheduled for arthroscopic surgery, the residents might have tried to find any pathologic findings in the US scans, which could lead to a high false-positive rate. However, they did not know the detailed clinical presentations and diagnoses and did not have the ability to perform a clinical examination that could have helped them in determining which structure was damaged. These facts are rather the strengths of this study. A second limitation is that the number of normal rotator cuff cases was small due to the study design. We thought it to be most accurate to evaluate the diagnostic accuracy of US through the arthroscopic operation. It was indeed impossible to decide whether to perform the operation according to the sonographic finding by the resident. Therefore, we had to carry out this study with patients who had already decided to undergo surgery, which led to a small number of normal rotator cuff patients. Although there was a small number of cases of the normal rotator cuff, the residents showed a satisfactory level of diagnostic accuracy. A third limitation is that there may be differences in the learning curve between examiners. A fourth limitation is that we could not determine the number of scans required to develop competence in detecting partial tears because of the small number of patients. Additionally, we could not obtain a CUSUM plot for those second-year residents because CUSUM analysis requires consecutive datasets. Indeed, it was difficult to perform four US scans with the same patient. Also, we could not derive intraclass coefficient because of the missing patients in the second-year resident group. Finally, our definition of self-confidence was somewhat arbitrary and not validated.

    The number of scans that novices needed to be competent for detecting rotator cuff tears was approximately 30 cases, and the diagnostic accuracy of third-year residents was significantly higher than that of second-year residents.

    Notes

    CONFLICT OF INTEREST:No potential conflict of interest relevant to this article was reported.

    References

      1. Teefey SA, Hasan SA, Middleton WD, Patel M, Wright RW, Yamaguchi K. Ultrasonography of the rotator cuff: a comparison of ultrasonographic and arthroscopic findings in one hundred consecutive cases. J Bone Joint Surg Am 2000;82(4):498–504.
      1. Ottenheijm RP, Jansen MJ, Staal JB, et al. Accuracy of diagnostic ultrasound in patients with suspected subacromial disorders: a systematic review and meta-analysis. Arch Phys Med Rehabil 2010;91(10):1616–1625.
      1. Park I, Lee HJ, Kim SE, et al. Evaluation of the effusion within biceps long head tendon sheath using ultrasonography. Clin Orthop Surg 2015;7(3):351–358.
      1. Kim MS, Kim IW, Lee S, Shin SJ. Diagnosis and treatment of calcific tendinitis of the shoulder. Clin Shoulder Elb 2020;23(4):210–216.
      1. Teefey SA, Rubin DA, Middleton WD, Hildebolt CF, Leibold RA, Yamaguchi K. Detection and quantification of rotator cuff tears: comparison of ultrasonographic, magnetic resonance imaging, and arthroscopic findings in seventy-one consecutive cases. J Bone Joint Surg Am 2004;86(4):708–716.
      1. de Jesus JO, Parker L, Frangos AJ, Nazarian LN. Accuracy of MRI, MR arthrography, and ultrasound in the diagnosis of rotator cuff tears: a meta-analysis. AJR Am J Roentgenol 2009;192(6):1701–1707.
      1. Fotiadou AN, Vlychou M, Papadopoulos P, Karataglis DS, Palladas P, Fezoulidis IV. Ultrasonography of symptomatic rotator cuff tears compared with MR imaging and surgery. Eur J Radiol 2008;68(1):174–179.
      1. American College of Radiology (ACR); Society for Pediatric Radiology (SPR); Society of Radiologists in Ultrasound (SRU). AIUM practice guideline for the performance of a musculoskeletal ultrasound examination. J Ultrasound Med 2012;31(9):1473–1488.
      1. Terslev L, Hammer HB, Torp-Pedersen S, et al. EFSUMB minimum training requirements for rheumatologists performing musculoskeletal ultrasound. Ultraschall Med 2013;34(5):475–477.
      1. Jeyam M, Funk L, Harris J. Are shoulder surgeons any good at diagnosing rotator cuff tears using ultrasound?: a comparative analysis of surgeon vs radiologist. Int J Shoulder Surg 2008;2(1):4–6.
      1. Al-Shawi A, Badge R, Bunker T. The detection of full thickness rotator cuff tears using ultrasound. J Bone Joint Surg Br 2008;90(7):889–892.
      1. Iossifidis A, Ibrahim EF, Petrou C. Ultrasound for the detection of full-thickness rotator cuff tears: the learning curve for an orthopaedic surgeon using a novel training method. Shoulder Elbow 2015;7(3):158–162.
      1. Daenen B, Houben G, Bauduin E, Lu KV, Meulemans JL. Ultrasound of the shoulder. JBR-BTR 2007;90(5):325–337.
      1. Deacon AJ, Melhuishi NS, Terblanche NC. CUSUM method for construction of trainee spinal ultrasound learning curves following standardised teaching. Anaesth Intensive Care 2014;42(4):480–486.
      1. Starkie T, Drake EJ. Assessment of procedural skills training and performance in anesthesia using cumulative sum analysis (cusum). Can J Anaesth 2013;60(12):1228–1239.
      1. Ohrndorf S, Naumann L, Grundey J, et al. Is musculoskeletal ultrasonography an operator-dependent method or a fast and reliably teachable diagnostic tool? Interreader agreements of three ultrasonographers with different training levels. Int J Rheumatol 2010;2010:164518
      1. Ok JH, Kim YS, Kim JM, Yoo TW. Learning curve of office-based ultrasonography for rotator cuff tendons tears. Knee Surg Sports Traumatol Arthrosc 2013;21(7):1593–1597.
      1. Ziegler DW. The use of in-office, orthopaedist-performed ultrasound of the shoulder to evaluate and manage rotator cuff disorders. J Shoulder Elbow Surg 2004;13(3):291–297.
      1. Moosmayer S, Smith HJ. Diagnostic ultrasound of the shoulder: a method for experts only? Results from an orthopedic surgeon with relative inexpensive compared to operative findings. Acta Orthop 2005;76(4):503–508.
      1. Dessieux T, Estebe JP, Bloc S, Mercadal L, Ecoffey C. Evaluation of the learning curve of residents in localizing a phantom target with ultrasonography. Ann Fr Anesth Reanim 2008;27(10):797–801.
      1. Sites BD, Gallagher JD, Cravero J, Lundberg J, Blike G. The learning curve associated with a simulated ultrasound-guided interventional task by inexperienced anesthesia residents. Reg Anesth Pain Med 2004;29(6):544–548.
      1. Alavekios DA, Dionysian E, Sodl J, Contreras R, Cho Y, Yian EH. Longitudinal analysis of effects of operator experience on accuracy for ultrasound detection of supraspinatus tears. J Shoulder Elbow Surg 2013;22(3):375–380.
      1. Dinnes J, Loveman E, McIntyre L, Waugh N. The effectiveness of diagnostic tests for the assessment of shoulder pain due to soft tissue disorders: a systematic review. Health Technol Assess 2003;7(29):iii, 1–166.
      1. Murphy RJ, Daines MT, Carr AJ, Rees JL. An independent learning method for orthopaedic surgeons performing shoulder ultrasound to identify full-thickness tears of the rotator cuff. J Bone Joint Surg Am 2013;95(3):266–272.
      1. Jacobson JA, Lancaster S, Prasad A, van Holsbeeck MT, Craig JG, Kolowich P. Full-thickness and partial-thickness supraspinatus tendon tears: value of US signs in diagnosis. Radiology 2004;230(1):234–242.
    Cited by
    Crossref 3
    Google Scholar 
    Web of Science 2
    MeSH Terms
    Diagnosis
    Learning Curve
    Mental Competency
    Pathology
    Respect
    Rotator Cuff Injuries*
    Rotator Cuff*
    Sensitivity and Specificity
    Shoulder
    Tears
    Tendons
    Ultrasonography*
    Metrics
    Share
    Links to
    Figures
    slide
    slide
    slide

    1 / 3

    Tables
    slide
    slide
    slide

    1 / 3

    ORCID IDs
    PERMALINK