Skip to main content
SpringerLink
Log in

Inter-reader reliability of contrast-enhanced ultrasound Liver Imaging Reporting and Data System: a meta-analysis

  • Hepatobiliary
  • Published:
Abdominal Radiology Aims and scope Submit manuscript

Abstract

Purpose

To systematically determine the inter-reader reliability of the contrast-enhanced ultrasound (CEUS) Liver Imaging Reporting and Data System (LI-RADS), with emphasis on its major features for hepatocellular carcinoma (HCC) and LR-M (LI-RADS category M) features for non-HCC malignancy.

Methods

MEDLINE, EMBASE, and Cochrane databases were searched from January 2016 to March 2021 to identify original articles reporting the inter-reader reliability of CEUS LI-RADS. Meta-analytic pooled kappa values (κ) were calculated for major features [nonrim arterial-phase hyperenhancement (APHE), mild and late washout], LR-M features (rim APHE, early washout), and LI-RADS categorization using the DerSimonian-Laird random-effects model. Meta-regression analysis was performed to explore any causes of study heterogeneity.

Results

Twelve studies with a total of 2862 lesions were included. The meta-analytic pooled κ of nonrim APHE, mild and late washout, rim APHE, early washout, and LI-RADS categorization were 0.73 [95% confidence interval (CI), 0.67 − 0.79], 0.69 (95% CI, 0.54–0.84), 0.54 (95% CI, 0.37–0.71), 0.62 (95% CI, 0.45–0.79), and 0.75 (95% CI, 0.64–0.87), respectively. Compared with the major features, LR-M features had a lower meta-analytic pooled κ. Substantial study heterogeneity was noted in the LI-RADS categorization, and lesion size (p = 0.03) and the homogeneity in reader experience (p = 0.03) were significantly associated with study heterogeneity.

Conclusions

CEUS LI-RADS showed substantial inter-reader reliability for major features and LI-RADS categorization, but relatively lower reliability was found for LR-M features. In our opinion, the definitions of imaging features require further refinement to improve the inter-reader reliability of CEUS LI-RADS.

Graphic abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Abbreviations

HCC:

Hepatocellular carcinoma

LI-RADS:

Liver Imaging Reporting and Data System

CEUS:

Contrast-enhanced ultrasound

CT:

Computed tomography

MRI:

Magnetic resonance imaging

APHE:

Arterial-phase hyperenhancement

κ :

Kappa value

CI:

Confidence interval

References

  1. Marrero JA, Kulik LM, Sirlin CB, et al. (2018) Diagnosis, staging, and management of hepatocellular carcinoma: 2018 practice guidance by the American Association for the Study of Liver Diseases. Hepatology 68:723-750. https://doi.org/10.1002/hep.29913.

    Article  PubMed  Google Scholar 

  2. Mitchell DG, Bruix J, Sherman M, Sirlin CB (2015) LI-RADS (Liver Imaging Reporting and Data System): summary, discussion, and consensus of the LI-RADS Management Working Group and future directions. Hepatology 61:1056-1065. https://doi.org/10.1002/hep.27304.

    Article  PubMed  Google Scholar 

  3. Wilson SR, Lyshchik A, Piscaglia F, et al. (2018) CEUS LI-RADS: algorithm, implementation, and key differences from CT/MRI. Abdom Radiol (NY) 43:127-142. https://doi.org/10.1007/s00261-017-1250-0.

    Article  Google Scholar 

  4. American College of Radiology (2021) CEUS LI-RADS v2017 core. https://www.acr.org/-/media/ACR/Files/RADS/LI-RADS/CEUS-LI-RADS-2017-Core.pdf. Accessed 31 March 2021.

  5. Shin J, Lee S, Bae H, et al. (2020) Contrast-enhanced ultrasound Liver Imaging Reporting and Data System for diagnosing hepatocellular carcinoma: a meta-analysis. Liver Int 40:2345-2352. https://doi.org/10.1111/liv.14617.

    Article  PubMed  Google Scholar 

  6. Son JH, Choi SH, Kim SY, et al. (2020) Accuracy of contrast-enhanced ultrasound Liver Imaging Reporting and Data System: a systematic review and meta-analysis. Hepatol Int 14:1104-1113. https://doi.org/10.1007/s12072-020-10102-5.

    Article  PubMed  Google Scholar 

  7. Li J, Chen M, Wang ZJ, et al. (2020) Interobserver agreement for contrast-enhanced ultrasound of Liver Imaging Reporting and Data System: a systematic review and meta-analysis. World J Clin Cases 8:5589-5602. https://doi.org/10.12998/wjcc.v8.i22.5589.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Li J, Ling W, Chen S, et al. (2019) The interreader agreement and validation of contrast-enhanced ultrasound Liver Imaging Reporting and Data System. Eur J Radiol 120:108685. https://doi.org/10.1016/j.ejrad.2019.108685.

    Article  PubMed  Google Scholar 

  9. Terzi E, Iavarone M, Pompili M, et al. (2018) Contrast ultrasound LI-RADS LR-5 identifies hepatocellular carcinoma in cirrhosis in a multicenter restropective study of 1,006 nodules. J Hepatol 68:485-492. https://doi.org/10.1016/j.jhep.2017.11.007.

    Article  PubMed  Google Scholar 

  10. Kang HJ, Lee JM, Yoon JH, Han JK (2021) Role of contrast-enhanced ultrasound as a second-line diagnostic modality in noninvasive diagnostic algorithms for hepatocellular carcinoma. Korean J Radiol 22:354-365. https://doi.org/10.3348/kjr.2020.0973.

    Article  PubMed  Google Scholar 

  11. Kang HJ, Lee JM, Yoon JH, et al. (2020) Contrast-enhanced US with sulfur hexafluoride and perfluorobutane for the diagnosis of hepatocellular carcinoma in individuals with high risk. Radiology 297:108-116. https://doi.org/10.1148/radiol.2020200115.

    Article  PubMed  Google Scholar 

  12. Li W, Li L, Zhuang BW, et al. (2021) Inter-reader agreement of CEUS LI-RADS among radiologists with different levels of experience. Eur Radiol. https://doi.org/10.1007/s00330-021-07777-1.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Makoyeva A, Kim TK, Jang HJ, et al. (2020) Use of CEUS LI-RADS for the accurate diagnosis of nodules in patients at risk for hepatocellular carcinoma: a validation study. Radiol Imaging Cancer 2:e190014. https://doi.org/10.1148/rycan.2020190014.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Zhou H, Zhang C, Du L, et al. (2020) Contrast-enhanced ultrasound Liver Imaging Reporting and Data System in diagnosing hepatocellular carcinoma: diagnostic performance and interobserver agreement. Ultraschall Med. https://doi.org/10.1055/a-1168-6321.

    Article  PubMed  Google Scholar 

  15. Stroup DF, Berlin JA, Morton SC, et al. (2000) Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 283:2008-2012. https://doi.org/10.1001/jama.283.15.2008.

    Article  CAS  PubMed  Google Scholar 

  16. Liberati A, Altman DG, Tetzlaff J, et al. (2009) The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 339:b2700. https://doi.org/10.1136/bmj.b2700.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Moher D, Liberati A, Tetzlaff J, et al. (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339:b2535. https://doi.org/10.1136/bmj.b2535.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Kottner J, Audige L, Brorson S, et al. (2011) Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. J Clin Epidemiol 64:96-106. https://doi.org/10.1016/j.jclinepi.2010.03.002.

    Article  PubMed  Google Scholar 

  19. IntHout J, Ioannidis JP, Borm GF (2014) The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Med Res Methodol 14:25. https://doi.org/10.1186/1471-2288-14-25.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159-174.

    Article  CAS  PubMed  Google Scholar 

  21. Higgins JP, Thompson SG, Deeks JJ, Altman DG (2003) Measuring inconsistency in meta-analyses. BMJ 327:557-560. https://doi.org/10.1136/bmj.327.7414.557.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Ling W, Wang M, Ma X, et al. (2018) The preliminary application of Liver Imaging Reporting and Data System (LI-RADS) with contrast-enhanced ultrasound (CEUS) on small hepatic nodules (</= 2cm). J Cancer 9:2946-2952. https://doi.org/10.7150/jca.25539.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Schellhaas B, Pfeifer L, Kielisch C, et al. (2018) Interobserver agreement for contrast-enhanced ultrasound (CEUS)-based standardized algorithms for the diagnosis of hepatocellular carcinoma in high-risk patients. Ultraschall Med 39:667-674. https://doi.org/10.1055/a-0612-7887.

    Article  PubMed  Google Scholar 

  24. Tan Z, Teoh WC, Wong KM, et al. (2020) Analysis of comparative performance of CEUS and CECT/MR LI-RADS classification: Can CEUS dichotomize LI-RADS indeterminate lesions on CT or MRI? Clin Imaging 62:63-68. https://doi.org/10.1016/j.clinimag.2020.02.002.

    Article  PubMed  Google Scholar 

  25. Wang JY, Feng SY, Xu JW, et al. (2020) Usefulness of the contrast-enhanced ultrasound Liver Imaging Reporting and Data System in diagnosing focal liver lesions by inexperienced radiologists. J Ultrasound Med 39:1537-1546. https://doi.org/10.1002/jum.15242.

    Article  PubMed  Google Scholar 

  26. Wang JY, Feng SY, Yi AJ, et al. (2020) Comparison of contrast-enhanced ultrasound versus contrast-enhanced magnetic resonance imaging for the diagnosis of focal liver lesions using the Liver Imaging Reporting and Data System. Ultrasound Med Biol 46:1216-1223. https://doi.org/10.1016/j.ultrasmedbio.2020.01.023.

    Article  PubMed  Google Scholar 

  27. Kang JH, Choi SH, Lee JS, et al. (2020) Interreader agreement of Liver Imaging Reporting and Data System on MRI: a systematic review and meta-analysis. J Magn Reson Imaging 52:795-804. https://doi.org/10.1002/jmri.27065.

    Article  PubMed  Google Scholar 

  28. Kang JH, Choi SH, Lee JS, et al. (2021) Inter-reader reliability of CT Liver Imaging Reporting and Data System according to imaging analysis methodology: a systematic review and meta-analysis. Eur Radiol. https://doi.org/10.1007/s00330-021-07815-y.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Schellhaas B, Hammon M, Strobel D, et al. (2018) Interobserver and intermodality agreement of standardized algorithms for non-invasive diagnosis of hepatocellular carcinoma in high-risk patients: CEUS-LI-RADS versus MRI-LI-RADS. Eur Radiol 28:4254-4264. https://doi.org/10.1007/s00330-018-5379-1.

    Article  PubMed  Google Scholar 

  30. Jang HJ, Kim TK, Burns PN, Wilson SR (2007) Enhancement patterns of hepatocellular carcinoma at contrast-enhanced US: comparison with histologic differentiation. Radiology 244:898-906. https://doi.org/10.1148/radiol.2443061520.

    Article  PubMed  Google Scholar 

  31. Grimm LJ, Anderson AL, Baker JA, et al. (2015) Interobserver variability between breast imagers using the fifth edition of the BI-RADS MRI lexicon. AJR Am J Roentgenol 204:1120–1124. https://doi.org/10.2214/ajr.14.13047.

    Article  PubMed  Google Scholar 

  32. Greer MD, Shih JH, Lay N, et al. (2019) Interreader variability of Prostate Imaging Reporting and Data System version 2 in detecting and assessing prostate cancer lesions at prostate MRI. AJR Am J Roentgenol 212:1197-1205. https://doi.org/10.2214/ajr.18.20536.

    Article  Google Scholar 

  33. Park KJ, Choi SH, Lee JS, et al. (2020) Interreader agreement with Prostate Imaging Reporting and Data System version 2 for prostate cancer detection: a systematic review and meta-analysis. J Urol 204:661-670. https://doi.org/10.1097/ju.0000000000001200.

    Article  PubMed  Google Scholar 

  34. Bartolotta TV, Taibbi A, Midiri M, Lagalla R (2019) Contrast-enhanced ultrasound of hepatocellular carcinoma: where do we stand? Ultrasonography 38:200-214. https://doi.org/10.14366/usg.18060.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Stevens WR, Gulino SP, Batts KP, et al. (1996) Mosaic pattern of hepatocellular carcinoma: histologic basis for a characteristic CT appearance. J Comput Assist Tomogr 20:337-342. https://doi.org/10.1097/00004728-199605000-00001.

    Article  CAS  PubMed  Google Scholar 

Download references

Funding

The authors state that this work has not received any funding.

Author information

Authors and Affiliations

  1. Department of Radiology, Hanyang University Guri Hospital, Hanyang University College of Medicine, Guri-si, Republic of Korea

    Ji Hun Kang

  2. Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, 88 Olympic-Ro 43-Gil, Songpa-Gu, Seoul, 05505, Republic of Korea

    Sang Hyun Choi, Dong Wook Kim & Jong Keon Jang

  3. Department of Clinical Epidemiology and Biostatistics, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Republic of Korea

    Ji Sung Lee

  4. Clinical Research Center, Asan Medical Center, Asan Institute for Life Sciences, University of Ulsan College of Medicine, Seoul, Korea

    Ji Sung Lee

Authors
  1. Ji Hun Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sang Hyun Choi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ji Sung Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dong Wook Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jong Keon Jang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sang Hyun Choi.

Ethics declarations

Conflict of interest

All authors declare that they have no conflict of interest.

Ethical approval

Institutional review board approval and written informed consent is not required because this study is a meta-analysis based on previously published studies.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 113 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kang, J.H., Choi, S.H., Lee, J.S. et al. Inter-reader reliability of contrast-enhanced ultrasound Liver Imaging Reporting and Data System: a meta-analysis. Abdom Radiol 46, 4671–4681 (2021). https://doi.org/10.1007/s00261-021-03169-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00261-021-03169-7

Keywords

Search

Navigation