Skip to main content

Advertisement

SpringerLink
Log in

CACTUS: A Digital Tool for Quality Assurance, Education and Evaluation in Surgical Pathology

  • Original Article
  • Published:
Journal of Medical and Biological Engineering Aims and scope Submit manuscript

Abstract

Purpose

To examine the factors that affect inter/intra-observer variability for breast cancer grading on histopathology images, to compare the level of agreement between histopathologists for assessment of virtual slides, and to introduce a new application, CACTUS, for the quality assurance of the cancer grading programme.

Methods

A new web-based tool for image archiving, annotation, distribution and evaluation (CACTUS) was developed. Four pathologists of varying practice experience in surgical pathology took part in three web-based image circulations using CACTUS. All pathologists evaluated the same 50 images, of which ten images were repeated to assess consistency. Reproducibility was evaluated by intra-observer (intraclass correlation coefficient) and inter-observer (kappa statistics) concordance rates and additional analyses.

Results

All pathologists found the interface of the tool and presentations of the images quick to learn and easy to use. The pathologists were asked to evaluate all of the images using a menu of questions meant to determine their assessment of the Nottingham grade of a series of \(\times\)40 images of breast carcinoma cases. For each image, they were asked to give their overall grade impression based on the \(\times\)40 single image. They were then directed to separate menus of questions to assess the principal components of Nottingham grading: mitotic rate, nuclear morphology and glandular morphology. Their consistency in re-evaluating the set of ten images was excellent agreement (> 0.90). While the overall inter-agreement for three sets is moderate (0.50–0.75), the overall intra-agreement between initial grades and mean value of mitosis, tubular and nuclear scores is excellent, even though opinions were based on single, \(\times\)40 images. The correlation between initial grade and four criteria/features (mitotic rate, nuclear pleomorphism, tubule formation and the mean value of these three criteria) has been calculated to show which feature is important among others or contributes to the actual score more. Tubular and nuclear scores are highly correlated with initial grades. Outliers among observers are identified by finding their distances from each other and computing the clusters.

Conclusion

This easy to use, easily modifiable, web-based image archiving, distribution and assessment tool can be useful for evaluating agreement amongst pathologists and disseminating anatomical pathology images for teaching. Moreover, it can be used against a gold standard for the purpose of diagnosis or quality assurance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Sinard, J. H. (2006). Pathology Informatics and The Future of Medicine. Practical Pathology Informatics: Demstifying Informatics for the Practicing Anatomic Pathologist (pp. 355–380). New York: Springer.

    Google Scholar 

  2. Leong, F. J., Leong, A. S., et al. (2004). Digital photography in anatomical pathology. Journal of Postgraduate Medicine, 50(1), 62.

    CAS  PubMed  Google Scholar 

  3. Farahani, N., & Pantanowitz, L. (2015). Overview of telepathology. Surgical Pathology Clinics, 8(2), 223–231.

    Article  PubMed  Google Scholar 

  4. Griffin, J., & Treanor, D. (2017). Digital pathology in clinical use: where are we now and what is holding us back? Histopathology, 70(1), 134–145.

    Article  PubMed  Google Scholar 

  5. Furness, P. N. (1997). The use of digital images in pathology. The Journal of Pathology: A Journal of the Pathological Society of Great Britain and Ireland, 183(3), 253–263.

    Article  CAS  Google Scholar 

  6. Al-Janabi, S., Huisman, A., & Van Diest, P. J. (2012). Digital pathology: current status and future perspectives. Histopathology, 61(1), 1–9.

    Article  PubMed  Google Scholar 

  7. Irshad, H., Montaser-Kouhsari, L., Waltz, G., Bucur, O., Nowak, J. A., Dong, F., Knoblauch, N. W., & Beck, A. H. (2014). Crowdsourcing image annotation for nucleus detection and segmentation in computational pathology: Evaluating experts, automated methods, and the crowd. In Pacific symposium on biocomputing Co-chairs, pp. 294–305. World Scientific.

  8. Park, S. L., Pantanowitz, L., Sharma, G., & Parwani, A. V. (2012). Anatomic pathology laboratory information systems: A review. Advances in Anatomic Pathology, 19(2), 81–96.

    Article  PubMed  Google Scholar 

  9. Dee, F. R. (2009). Virtual microscopy in pathology education. Human Pathology, 40(8), 1112–1121.

    Article  PubMed  Google Scholar 

  10. Wright, A. I., Clarke, E. L., Dunn, C. M., Williams, B. J., Treanor, D. E., & Brettle, D. S. (2020). A point-of-use quality assurance tool for digital pathology remote working. Journal of Pathology Informatics, 11, 17.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Aksac, A., Ozyer, T., Demetrick, D. J., & Alhajj, R. (2020). Cactus: cancer image annotating, calibrating, testing, understanding and sharing in breast cancer histopathology. BMC Research Notes, 13(1), 1–7.

    Article  Google Scholar 

  12. Jahn, S. W., Plass, M., & Moinfar, F. (2020). Digital pathology: Advantages, limitations and emerging perspectives. Journal of Clinical Medicine, 9(11), 3697.

    Article  PubMed Central  Google Scholar 

  13. Hamilton, P. W., Wang, Y., & McCullough, S. J. (2012). Virtual microscopy and digital pathology in training and education. Apmis, 120(4), 305–315.

    Article  PubMed  Google Scholar 

  14. Elston, C. W., & Ellis, I. O. (1991). Pathological prognostic factors in breast cancer. i. the value of histological grade in breast cancer: experience from a large study with long-term follow-up. Histopathology, 19(5), 403–410.

    Article  CAS  PubMed  Google Scholar 

  15. Bloom, H. J. G., & Richardson, W. W. (1957). Histological grading and prognosis in breast cancer: a study of 1409 cases of which 359 have been followed for 15 years. British Journal of Cancer, 11(3), 359.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  16. Robbins, P., Pinder, S., De Klerk, N., Dawkins, H., Harvey, J., Sterrett, G., Ellis, I., & Elston, C. (1995). Histological grading of breast carcinomas: A study of interobserver agreement. Human Pathology, 26(8), 873–879.

    Article  CAS  PubMed  Google Scholar 

  17. Frierson, H. F., Wolber, R. A., Berean, K. W., Franquemont, D. W., Gaffey, M. J., Boyd, J. C., & Wilbur, D. C. (1995). Interobserver reproducibility of the nottingham modification of the bloom and richardson histologic grading scheme for infiltrating ductal carcinoma. American Journal of Clinical Pathology, 103(2), 195–198.

    Article  PubMed  Google Scholar 

  18. Longacre, T. A., Ennis, M., Quenneville, L. A., Bane, A. L., Bleiweiss, Ira J., Carter, B. A., Catelano, E., Hendrickson, M. R., Hibshoosh, H., Layfield, L. J., et al. (2006). Interobserver agreement and reproducibility in classification of invasive breast carcinoma: An nci breast cancer family registry study. Modern Pathology, 19(2), 195.

    Article  PubMed  Google Scholar 

  19. Boiesen, P., Bendahl, P.-O., Anagnostaki, L., Domanski, H., Holm, E., Idvall, I., Johansson, S., Ljungberg, O., & Ringberg, A. (2000). Histologic grading in breast cancer: Reproducibility between seven pathologic departments. Acta Oncologica, 39(1), 41–45.

    Article  CAS  PubMed  Google Scholar 

  20. Gomes, D. S., Porto, S. S., Balabram, D., & Gobbi, H. (2014). Inter-observer variability between general pathologists and a specialist in breast pathology in the diagnosis of lobular neoplasia, columnar cell lesions, atypical ductal hyperplasia and ductal carcinoma in situ of the breast. Diagnostic Pathology, 9(1), 121.

    Article  PubMed  PubMed Central  Google Scholar 

  21. van Riel, S. J., Sánchez, C. I., Bankier, A. A., Naidich, D. P., Verschakelen, J., Scholten, E. T., de Jong, P. A., Jacobs, C., van Rikxoort, E., Peters-Bax, L., et al. (2015). Observer variability for classification of pulmonary nodules on low-dose ct images and its effect on nodule management. Radiology, 277(3), 863–871.

    Article  PubMed  Google Scholar 

  22. Bueno-de Mesquita, J. M., Nuyten, D. S. A., Wesseling, J., van Tinteren, H., Linn, S. C., & van De Vijver, M. J. (2010). The impact of inter-observer variation in pathological assessment of node-negative breast cancer on clinical risk assessment and patient selection for adjuvant systemic treatment. Annals of Oncology, 21(1), 40–47.

    Article  CAS  PubMed  Google Scholar 

  23. Gilchrist, K. W., Kalish, L., Gould, V. E., Hirschl, S., Imbriglia, J. E., Levy, W. M., Patchefsky, A. S., Penner, D. W., Pickren, J., Roth, J. A., et al. (1985). Interobserver reproducibility of histopathological features in stage ii breast cancer. Breast Cancer Research and Treatment, 5(1), 3–10.

    Article  CAS  PubMed  Google Scholar 

  24. Aksac, A., Demetrick, D. J., Ozyer, T., & Alhajj, R. (2019). Brecahad: A dataset for breast cancer histopathological annotation and diagnosis. BMC Research Notes, 12(1), 82.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Kayser, K., Kayser, G., Radziszowski, D., & Oehmann, A. (1999). From telepathology to virtual pathology institution: the new world of digital pathology. Romanian Journal of Morphology and Embryology, 45, 3–9.

    PubMed  Google Scholar 

  26. Szymas, J. (2000). Teleeducation and telepathology for open and distance education. Analytical Cellular Pathology, 21(3–4), 183–191.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  27. Preston, K., Jr. (1986). High-resolution image analysis. Journal of Histochemistry & Cytochemistry, 34(1), 67–74.

    Article  Google Scholar 

  28. Jondet, M., Agoli-Agbo, R., & Dehennin, L. (2010). Automatic measurement of epithelium differentiation and classification of cervical intraneoplasia by computerized image analysis. Diagnostic Pathology, 5(1), 7.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Kamenetsky, I., Rangayyan, R. M., & Benediktsson, H. (2010). Analysis of the glomerular basement membrane in images of renal biopsies using the split-and-merge method: a pilot study. Journal of Digital Imaging, 23(4), 463–474.

    Article  PubMed  Google Scholar 

  30. Loukas, C., Kostopoulos, S., Tanoglidi, A., Glotsos, D., Sfikas, C., & Cavouras, D. (2013). Breast cancer characterization based on image classification of tissue sections visualized under low magnification. Computational and Mathematical Methods in Medicine. https://doi.org/10.1155/2013/829461

    Article  PubMed  PubMed Central  Google Scholar 

  31. Yoshida, H., Shimazu, T., Kiyuna, T., Marugame, A., Yamashita, Y., Cosatto, E., Taniguchi, H., Sekine, S., & Ochiai, A. (2018). Automated histological classification of whole-slide images of gastric biopsy specimens. Gastric Cancer, 21(2), 249–257.

    Article  PubMed  Google Scholar 

  32. Fung, K.-M., & Tihan, T. (2009). Internet and world wide web-based tools for neuropathology practice and education. Brain Pathology, 19(2), 323–331.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Jones, K. N., Kreisle, R., Geiss, R. W., Holliman, J. H., Lill, P. H., & Anderson, P. G. (2002). Group for research in pathology education online resources to facilitate pathology instruction. Archives of Pathology & Laboratory Medicine, 126(3), 346–350.

    Article  Google Scholar 

  34. Chen, Y.-K., Hsue, S-S., Lin, D.-C., Wang, W-C., Chen, J.-Y., Lin, C.-C., & Lin, L.-M. (2008). An application of virtual microscopy in the teaching of an oral and maxillofacial pathology laboratory course. Oral Surgery, Oral Medicine, Oral Pathology, Oral Radiology, and Endodontology, 105(3), 342–347.

    Article  PubMed  Google Scholar 

  35. Johnston, D. J., Costello, S. P., Dervan, P. A., & O’Shea, D. G. (2005). Development and preliminary evaluation of the vps replaysuite: a virtual double-headed microscope for pathology. BMC Medical Informatics and Decision Making, 5(1), 10.

  36. Bruch, L. A., De Young, B. R., Kreiter, C. D., Haugen, T. H., Leaven, T. C., & Dee, F. R. (2009). Competency assessment of residents in surgical pathology using virtual microscopy. Human Pathology, 40(8), 1122–1128.

    Article  PubMed  Google Scholar 

  37. Dessauvagie, B. F., Lee, A. H. S., Meehan, K., Nijhawan, A., Tan, P. H, Thomas, J., Tie, B., Treanor, D., Umar, S., Hanby, A. M., et al. (2018). Interobserver variation in the diagnosis of fibroepithelial lesions of the breast: a multicentre audit by digital pathology. Journal of Clinical Pathology, 71(8), 672–679.

    Article  PubMed  Google Scholar 

  38. Linder, E., Lundin, M., Thors, C., Lebbad, M., Winiecka-Krusnell, J., Helin, H., Leiva, B., Isola, J., & Lundin, J. (2008). Web-based virtual microscopy for parasitology: a novel tool for education and quality assurance. NLoS neglected Tropical Diseases, 2(10), 10371

  39. Kldiashvili, E, & Schrader, T. (2011). Reproducibility of telecytology diagnosis of cervical smears in a quality assurance program: the georgian experience. Telemedicine and e-Health, 17(7), 565–568.

    Article  PubMed  Google Scholar 

  40. Lee, E. S., Kim, I. S., Choi, J. S., Yeom, B. W., Kim, H. K., Han, J. H., Lee, M. S., & Leong, A. S Y. (2003). Accuracy and reproducibility of telecytology diagnosis of cervical smears: a tool for quality assurance programs. American journal of Clinical Pathology, 119(3), 356–360.

    Article  PubMed  Google Scholar 

  41. Cristiani, P., Costa, S., Schincaglia, P., Garutti, P., Bianchi, P. S. D., Naldoni, C., Sideri, M., & Bucchi, L. (2014). An online quality assurance program for colposcopy in a population-based cervical screening setting in italy: Results on colposcopic impression. Journal of Lower Genital Tract Disease, 18(4), 309–313.

    Article  PubMed  Google Scholar 

  42. Theissig, F., Kunze, K. D., Haroske, G., & Meyer, W. (1990). Histological grading of breast cancer: Interobserver, reproducibility and prognostic significance. Pathology-Research and Practice, 186(6), 732–736.

    Article  CAS  PubMed  Google Scholar 

  43. Malon, C., Brachtel, E., Cosatto, E., Graf, H. P., Kurata, A., Kuroda, M., Meyer, J. S., Saito, A., Shulin, W., & Yagi, Y. (2012). Mitotic figure recognition: Agreement among pathologists and computerized detector. Analytical Cellular Pathology, 35(2), 97–100.

    Article  Google Scholar 

  44. Gandomkar, Z., Brennan, P. C., & Mello-Thoms, C. (2019). Computer-assisted nuclear atypia scoring of breast cancer: A preliminary study. Journal of Digital Imaging, 32(5), 702–712.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Rakha, E. A., Aleskandarani, M., Toss, M. S., Green, A. R., Ball, G., Ellis, I. O., & Dalton, L. W. (2018). Breast cancer histologic grading using digital microscopy: Concordance and outcome association. Journal of Clinical Pathology, 71(8), 680–686.

    Article  PubMed  Google Scholar 

  46. Dalton, L. W., Page, D. L., & Dupont, W. D. (1994). Histologic grading of breast carcinoma. A reproducibility study. Cancer, 73(11), 2765–2770.

    Article  CAS  PubMed  Google Scholar 

  47. Aksac, A., Özyer, T., & Alhajj, R. (2019). Cutesc: cutting edge spatial clustering technique based on proximity graphs. Pattern Recognition, 96, 106948.

    Article  Google Scholar 

  48. Aksac, A., Ozyer, T., & Alhajj, R. (2020). Data on cut-edge for spatial clustering based on proximity graphs. Data in Brief, 28, 104899.

    Article  PubMed  Google Scholar 

  49. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.

    Article  Google Scholar 

  50. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378.

    Article  Google Scholar 

  51. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420.

    Article  CAS  PubMed  Google Scholar 

  52. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.

    Article  CAS  PubMed  Google Scholar 

  53. Altman, D. G. (1990). Practical Statistics for Medical Research. London: CRC Press.

    Book  Google Scholar 

  54. Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163.

    Article  PubMed  PubMed Central  Google Scholar 

  55. Oommen, B. J., & Loke Loke, R. K S. (1997). Pattern recognition of strings with substitutions, insertions, deletions and generalized transpositions. Pattern Recognition, 30(5), 789–800.

    Article  Google Scholar 

  56. Comaniciu, D., & Meer, P. (2002). Mean shift: A robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(5), 603–619.

    Article  Google Scholar 

  57. R Core Team. (2018). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/.

  58. Contesso, G., Mouriesse, H., Friedman, S., Genin, J., Sarrazin, D., & Rouesse, J. (1987). The importance of histologic grade in long-term prognosis of breast cancer: A study of 1,010 patients, uniformly treated at the institut gustave-roussy. Journal of Clinical Oncology, 5(9), 1378–1386.

    Article  CAS  PubMed  Google Scholar 

  59. Henson, D. E., Ries, L., Freedman, L. S., & Carriaga, M. (1991). Relationship among outcome, stage of disease, and histologic grade for 22,616 cases of breast cancer. the basis for a prognostic index. Cancer, 68(10), 2142–2149.

    Article  CAS  PubMed  Google Scholar 

  60. Elmore, J. G., Barnhill, R. L., Elder, D. E., Longton, G. M., Pepe, M. S., Reisch, L. M., et al. (2017). Pathologists’ diagnosis of invasive melanoma and melanocytic proliferations: Observer accuracy and reproducibility study. BMJ, 357, 2813.

    Article  Google Scholar 

  61. Geller, B. M., Nelson, H. D., Carney, P. A., Weaver, D. L., Onega, T., Allison, K. H., Frederick, P. D., Tosteson, A. N A., & Elmore, J. G. (2014). Second opinion in breast pathology: Policy, practice and perception. Journal of Clinical Pathology, 67(11), 955–960.

    Article  PubMed  Google Scholar 

  62. Swapp, R. E., Aubry, M. C., Salomão, D. R., & Cheville, J. C. (2013). Outside case review of surgical pathology for referred patients: The impact on patient care. Archives of Pathology & Laboratory Medicine, 137(2), 233–240.

    Article  Google Scholar 

  63. Wright, K. C., Harnden, P., Moss, S., Berney, D. M., & Melia, J. (2011). A practical application of analysing weighted kappa for panels of experts and eqa schemes in pathology. Journal of Clinical Pathology, 64, 257–260.

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Calgary, Calgary, AB, Canada

    Alper Aksac, Jon Rokne & Reda Alhajj

  2. Department of Pathology & Laboratory Medicine, University of Calgary and Calgary Laboratory Services, Calgary, AB, T2L 2K8, Canada

    Douglas J. Demetrick, Adrian Box, Lisa DiFrancesco & Parham Minoo

  3. TOBB University of Economics and Technology, Ankara, Turkey

    Tansel Ozyer

Authors
  1. Alper Aksac
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Douglas J. Demetrick
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Adrian Box
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lisa DiFrancesco
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Parham Minoo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Tansel Ozyer
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jon Rokne
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Reda Alhajj
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alper Aksac.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (DOCX 481 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aksac, A., Demetrick, D.J., Box, A. et al. CACTUS: A Digital Tool for Quality Assurance, Education and Evaluation in Surgical Pathology. J. Med. Biol. Eng. 41, 470–481 (2021). https://doi.org/10.1007/s40846-021-00643-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40846-021-00643-x

Keywords

Search

Navigation