Skip to main content
SpringerLink
Log in

A teaching intervention in a contouring dummy run improved target volume delineation in locally advanced non-small cell lung cancer

Reducing the interobserver variability in multicentre clinical studies

Verbesserung der Zielvolumendefinition durch einen Dummy Run mit interventionellem Konturierungstraining beim lokal fortgeschrittenen nicht-kleinzelligen Lungenkarzinom

Reduktion der Interobserver-Variabilität in klinischen Multizenterstudien

  • Original Article
  • Published:
Strahlentherapie und Onkologie Aims and scope Submit manuscript

Abstract

Introduction

Interobserver variability in the definition of target volumes (TVs) is a well-known confounding factor in (multicentre) clinical studies employing radiotherapy. Therefore, detailed contouring guidelines are provided in the prospective randomised multicentre PET-Plan (NCT00697333) clinical trial protocol. This trial compares strictly FDG-PET-based TV delineation with conventional TV delineation in patients with locally advanced non-small cell lung cancer (NSCLC). Despite detailed contouring guidelines, their interpretation by different radiation oncologists can vary considerably, leading to undesirable discrepancies in TV delineation. Considering this, as part of the PET-Plan study quality assurance (QA), a contouring dummy run (DR) consisting of two phases was performed to analyse the interobserver variability before and after teaching.

Materials and methods

In the first phase of the DR (DR1), radiation oncologists from 14 study centres were asked to delineate TVs as defined by the study protocol (gross TV, GTV; and two clinical TVs, CTV-A and CTV-B) in a test patient. A teaching session was held at a study group meeting, including a discussion of the results focussing on discordances in comparison to the per-protocol solution. Subsequently, the second phase of the DR (DR2) was performed in order to evaluate the impact of teaching.

Results

Teaching after DR1 resulted in a reduction of absolute TVs in DR2, as well as in better concordance of TVs. The Overall Kappa(κ) indices increased from 0.63 to 0.71 (GTV), 0.60 to 0.65 (CTV-A) and from 0.59 to 0.63 (CTV-B), demonstrating improvements in overall interobserver agreement.

Conclusion

Contouring DRs and study group meetings as part of QA in multicentre clinical trials help to identify misinterpretations of per-protocol TV delineation. Teaching the correct interpretation of protocol contouring guidelines leads to a reduction in interobserver variability and to more consistent contouring, which should consequently improve the validity of the overall study results.

Zusammenfassung

Einleitung

Die Interobserver-Variabilität bei der Definition der Zielvolumina („target volumes“, TVs) ist ein anerkanntes Problem in strahlentherapeutischen (multizentrischen) klinischen Studien. In der prospektiven randomisierten multizentrischen PET-Plan-Studie (NCT00697333) wird eine strikt FDG-PET-basierte mit einer konventionellen TV-Definition bei Patienten mit einem lokal fortgeschrittenen nicht-kleinzelligem Lungenkarzinom (NSCLC) verglichen. Im Studienprotokoll ist eine detaillierte Konturierungsanleitung implementiert. Trotz genauer Vorgaben zur TV-Definition kann deren Interpretation durch verschiedene Strahlentherapeuten sehr differieren und zu unerwünschten Diskrepanzen bei den TVs führen. Unter diesem Gesichtspunkt wurde als ein Aspekt der Qualitätssicherungsmaßnahmen der PET-Plan-Studie ein Konturierungs-„Dummy-Run“ in zwei Abschnitten durchgeführt. Analysiert wurde die Interobserver-Variabilität vor und nach einem Konturierungstraining.

Material und Methoden

Im ersten Teil des „Dummy Run“ (DR1) wurden Strahlentherapeuten aus 14 Studienzentren gebeten, eine protokollgemäße Definition der TVs (GTV, CTVA, CTV-B) an einem Beispielpatienten durchzuführen. Bei einem Studientreffen wurden die Ergebnisse mit Fokus auf Diskrepanzen im Vergleich zur protokollgemäßen Konturierung diskutiert. Mit dem Ziel, den Einfluss dieses Konturierungstrainings zu evaluieren, wurde der zweite Teil des „Dummy Run“ (DR2) durchgeführt.

Ergebnisse

Das Training der protokollgemäßen Konturierung nach dem DR1 führte zu einer Reduktion der absoluten Volumina sowie zu einer besseren Übereinstimmung der TVs im DR2. Der „Overall“-Kappa erhöhte sich von 0,63 auf 0,71 (GTV), von 0,60 auf 0,65 (CTV-A) und von 0,59 auf 0,63 (CTV-B).

Schlussfolgerung

Durch Qualitätssicherungsmaßnahmen wie „Dummy Runs“ und Studientreffen können Fehlinterpretationen bei der protokollgemäßen TV-Definition in multizentrischen klinischen Studien in der Strahlenheilkunde identifiziert und durch entsprechendes Training korrigiert werden. Dies sollte sich positiv auf die Validität der Studienergebnisse auswirken.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. WHO (2014) World Health Organization. World cancer factsheet January 2014: world cancer burden (2012). Accessed 6 Nov. 2014

  2. Auperin A, Le Pechoux C, Rolland E, Curran WJ, Furuse K, Fournel P, Belderbos J, Clamon G, Ulutin HC, Paulus R, Yamanaka T, Bozonnat MC, Uitterhoeve A, Wang X, Stewart L, Arriagada R, Burdett S, Pignon JP (2010) Meta-analysis of concomitant versus sequential radiochemotherapy in locally advanced non-small-cell lung cancer. J Clin Oncol 28:2181–2190. doi:10.1200/JCO.2009.26.2543

    Article  CAS  PubMed  Google Scholar 

  3. Goeckenjan G, Sitter H, Thomas M, Branscheid D, Flentje M, Griesinger F, Niederle N, Stuschke M, Blum T, Deppermann KM, Ficker JH, Freitag L, Lubbe AS, Reinhold T, Spath-Schwalbe E, Ukena D, Wickert M, Wolf M, Andreas S, Auberger T, Baum RP, Baysal B, Beuth J, Bickeboller H, Bocking A, Bohle RM, Bruske I, Burghuber O, Dickgreber N, Diederich S, Dienemann H, Eberhardt W, Eggeling S, Fink T, Fischer B, Franke M, Friedel G, Gauler T, Gutz S, Hautmann H, Hellmann A, Hellwig D, Herth F, Heussel CP, Hilbe W, Hoffmeyer F, Horneber M, Huber RM, Hubner J, Kauczor HU, Kirchbacher K, Kirsten D, Kraus T, Lang SM, Martens U, Mohn-Staudner A, Muller KM, Muller-Nordhorn J, Nowak D, Ochmann U, Passlick B, Petersen I, Pirker R, Pokrajac B, Reck M, Riha S, Rube C, Schmittel A, Schonfeld N, Schutte W, Serke M, Stamatis G, Steingraber M, Steins M, Stoelben E, Swoboda L, Teschler H, Tessen HW, Weber M, Werner A, Wichmann HE, Irlinger Wimmer E, Witt C, Worth H, German Respiratory S, German Cancer S (2011) Prevention, diagnosis, therapy, and follow-up of lung cancer: interdisciplinary guideline of the German Respiratory Society and the German Cancer Society. Pneumologie 65:39–59. doi:10.1055/s-0030-1255961

    Article  CAS  PubMed  Google Scholar 

  4. Fleckenstein J, Hellwig D, Kremp S, Grgic A, Groschel A, Kirsch CM, Nestle U, Rube C (2011) F-18-FDG-PET confined radiotherapy of locally advanced NSCLC with concomitant chemotherapy: results of the PET-PLAN pilot trial. International journal of radiation oncology, biology, physics. 81:e283–e289. doi:10.1016/j.ijrobp.2011.01.020

  5. van Baardwijk A, Wanders S, Boersma L, Borger J, Ollers M, Dingemans AM, Bootsma G, Geraedts W, Pitz C, Lunde R, Lambin P, De Ruysscher D (2010) Mature results of an individualized radiation dose prescription study based on normal tissue constraints in stages I to III non-small-cell lung cancer. J Clin Oncol 28:1380–1386. doi:10.1200/JCO.2009.24.7221

    Article  PubMed  Google Scholar 

  6. Hoffmann AL, Troost EG, Huizenga H, Kaanders JH, Bussink J (2012) Individualized dose prescription for hypofractionation in advanced non-small-cell lung cancer radiotherapy: an in silico trial. International journal of radiation oncology, biology, physics. Int J Radiat Oncol Biol Phys 83:1596–1602. doi:10.1016/j.ijrobp.2011.10.032

    Article  PubMed  Google Scholar 

  7. van Elmpt W, De Ruysscher D, van der Salm A, Lakeman A, van der Stoep J, Emans D, Damen E, Ollers M, Sonke JJ, Belderbos J (2012) The PET-boost randomised phase II dose-escalation trial in non-small cell lung cancer. Radiother Oncol 104:67–71. doi:10.1016/j.radonc.2012.03.005

    Article  PubMed  Google Scholar 

  8. Nestle U, Kremp S, Grosu AL (2006) Practical integration of [18F]-FDG-PET and PET-CT in the planning of radiotherapy for non-small cell lung cancer (NSCLC): the technical basis, ICRU-target volumes, problems, perspectives. Radiother Oncol 81:209–225. doi:10.1016/j.radonc.2006.09.011

    Article  CAS  PubMed  Google Scholar 

  9. MacManus M, Nestle U, Rosenzweig KE, Carrio I, Messa C, Belohlavek O, Danna M, Inoue T, Deniaud-Alexandre E, Schipani S, Watanabe N, Dondi M, Jeremic B (2009) Use of PET and PET/CT for radiation therapy planning: IAEA expert report 2006–2007. Radiother Oncol 91:85–94. doi:10.1016/j.radonc.2008.11.008

    Article  PubMed  Google Scholar 

  10. Belderbos JS, Heemsbergen WD, De Jaeger K, Baas P, Lebesque JV (2006) Final results of a Phase I/II dose escalation trial in non-small-cell lung cancer using three-dimensional conformal radiotherapy. Int J Radiat Oncol Biol Phys 66:126–134. doi:10.1016/j.ijrobp.2006.04.034

    Article  PubMed  Google Scholar 

  11. De Ruysscher D, Wanders S, van Haren E, Hochstenbag M, Geeraedts W, Utama I, Simons J, Dohmen J, Rhami A, Buell U, Thimister P, Snoep G, Boersma L, Verschueren T, van Baardwijk A, Minken A, Bentzen SM, Lambin P (2005) Selective mediastinal node irradiation based on FDG-PET scan data in patients with non-small-cell lung cancer: a prospective clinical study. Int J Radiat Oncol Biol Phys 62:988–994. doi:10.1016/j.ijrobp.2004.12.019

    Article  PubMed  Google Scholar 

  12. Caldwell CB, Mah K, Ung YC, Danjoux CE, Balogh JM, Ganguli SN, Ehrlich LE (2001) Observer variation in contouring gross tumor volume in patients with poorly defined non-small-cell lung tumors on CT: the impact of 18FDG-hybrid PET fusion. Int J Radiat Oncol Biol Phys 51:923–931

    Article  CAS  PubMed  Google Scholar 

  13. Nestle U, Weber W, Hentschel M, Grosu AL (2009) Biological imaging in radiation therapy: role of positron emission tomography. Phys Med Biol 54:R1–R25. doi:10.1088/0031-9155/54/1/R01

    Article  PubMed  Google Scholar 

  14. Steenbakkers RJ, Duppen JC, Fitton I, Deurloo KE, Zijp LJ, Comans EF, Uitterhoeve AL, Rodrigus PT, Kramer GW, Bussink J, De Jaeger K, Belderbos JS, Nowak PJ, van Herk M, Rasch CR (2006) Reduction of observer variation using matched CT-PET for lung cancer delineation: a three-dimensional analysis. Int J Radiat Oncol Biol Phys 64:435–448. doi:10.1016/j.ijrobp.2005.06.034

    Article  PubMed  Google Scholar 

  15. Nestle U, Mix M, Weber W, Grosu AL (2011) Klinische Studien zum Einsatz der PET in der Bestrahlungsplanung in Deutschland: Ein Update. Nuklearmediziner 34:130–132. doi:10.1055/s-0031-1280790

    Article  Google Scholar 

  16. Schimek-Jasch T, Küsters A, Hoffmans H, Nestle U (2014) Optimierung der Strahlentherapieplanung bei lokal fortgeschrittenen NSCLC durch 18F-FDG PET-CT: Stand der PET-Plan-Studie. Nuklearmediziner 37:175–180. doi:10.1055/s-0034-1376972

    Article  Google Scholar 

  17. Djarv E, Nyman J, Baumann P, Ekberg L, Hoyer M, Lax I, Lewensohn R, Levin N, Lund JA, Morhed E, Ericsson SR, Traberg A, Wittgren L, Johansson KA (2006) Dummy run for a phase II study of stereotactic body radiotherapy of T1-T2 N0M0 medical inoperable non-small cell lung cancer. Acta Oncol 45:973–977. doi:10.1080/02841860600919241

    Article  PubMed  Google Scholar 

  18. Foppiano F, Fiorino C, Frezza G, Greco C, Valdagni R, Radiotherapy ANWGoP (2003) The impact of contouring uncertainty on rectal 3D dose-volume data: results of a dummy run in a multicenter trial (AIROPROS01-02). Int J Radiat Oncol Biol Phys 57:573–579

    Article  PubMed  Google Scholar 

  19. Giraud P, Elles S, Helfre S, De Rycke Y, Servois V, Carette MF, Alzieu C, Bondiau PY, Dubray B, Touboul E, Housset M, Rosenwald JC, Cosset JM (2002) Conformal radiotherapy for lung cancer: different delineation of the gross tumor volume (GTV) by radiologists and radiation oncologists. Radiother Oncol 62:27–36

    Article  PubMed  Google Scholar 

  20. Matzinger O, Poortmans P, Giraud JY, Maingon P, Budiharto T, van den Bergh AC, Davis JB, Musat E, Ataman F, Huyskens DP, Gulyban A, Bolla M, Group ERO (2009) Quality assurance in the 22991 EORTC ROG trial in localized prostate cancer: dummy run and individual case review. Radiother Oncol 90:285–290. doi:10.1016/j.radonc.2008.10.022

    Article  PubMed  Google Scholar 

  21. Poortmans PM, Venselaar JL, Struikmans H, Hurkmans CW, Davis JB, Huyskens D, van Tienhoven G, Vlaun V, Lagendijk JJ, Mijnheer BJ, De Winter KA, Van der Hulst MH, Van den Bogaert WF (2001) The potential impact of treatment variations on the results of radiotherapy of the internal mammary lymph node chain: a quality-assurance report on the dummy run of EORTC Phase III randomized trial 22922/10925 in Stage I-III breast cancer(1). Int J Radiat Oncol Biol Phys 49:1399–1408

    Article  CAS  PubMed  Google Scholar 

  22. Spoelstra FO, Senan S, Le Pechoux C, Ishikura S, Casas F, Ball D, Price A, De Ruysscher D, van Sornsen de Koste JR, Lung Adjuvant Radiotherapy Trial Investigators G (2010) Variations in target volume definition for postoperative radiotherapy in stage III non-small-cell lung cancer: analysis of an international contouring study. Int J Radiat Oncol Biol Phys 76:1106–1113. doi:10.1016/j.ijrobp.2009.02.072

  23. Senan S, van Sornsen de Koste J, Samson M, Tankink H, Jansen P, Nowak PJ, Krol AD, Schmitz P, Lagerwaard FJ (1999) Evaluation of a target contouring protocol for 3D conformal radiotherapy in non-small cell lung cancer. Radiother Oncol 53:247–255

    Article  CAS  PubMed  Google Scholar 

  24. Senan S, Le Pechoux C, Spoelstra FO, Ishikura S, Casas F, Ball D, Price A, De Ruysscher D, van Sörnsen de Koste JR (2007) A need to standardize post-operative radiotherapy (PORT) fields used for non-small cell lung cancer (NSCLC): analysis of an international dummy-run study: A1–02. J Thorac Oncol 2:S308–S309

    Article  Google Scholar 

  25. Valley JF, Bernier J, Tercier PA, Fogliata-Cozzi A, Rosset A, Garavaglia G, Mirimanoff RO (1998) Quality assurance of the EORTC radiotherapy trial 22931 for head and neck carcinomas: the dummy run. Radiother Oncol 47:37–44

    Article  CAS  PubMed  Google Scholar 

  26. Van de Steene J, Linthout N, de Mey J, Vinh-Hung V, Claassens C, Noppen M, Bel A, Storme G (2002) Definition of gross tumor volume in lung cancer: inter-observer variability. Radiother Oncol 62:37–49

    Article  Google Scholar 

  27. Vorwerk H, Beckmann G, Bremer M, Degen M, Dietl B, Fietkau R, Gsanger T, Hermann RM, Alfred Herrmann MK, Holler U, van Kampen M, Korber W, Maier B, Martin T, Metz M, Richter R, Siekmeyer B, Steder M, Wagner D, Hess CF, Weiss E, Christiansen H (2009) The delineation of target volumes for radiotherapy of lung cancer patients. Radiother Oncol 91:455–460. doi:10.1016/j.radonc.2009.03.014

    Article  PubMed  Google Scholar 

  28. Schaefer A, Nestle U, Kremp S, Hellwig D, Grgic A, Buchholz HG, Mischke W, Gromoll C, Dennert P, Plotkin M, Senftleben S, Thorwarth D, Tosch M, Wahl A, Wengenmair H, Rube C, Kirsch CM (2012) Multi-centre calibration of an adaptive thresholding method for PET-based delineation of tumour volumes in radiotherapy planning of lung cancer. Nuklearmedizin 51:101–110. doi:10.3413/Nukmed-0452-11-12

    Article  CAS  PubMed  Google Scholar 

  29. Schaefer A, Kremp S, Hellwig D, Rube C, Kirsch CM, Nestle U (2008) A contrast-oriented algorithm for FDG-PET-based delineation of tumour volumes for the radiotherapy of lung cancer: derivation from phantom measurements and validation in patient data. Eur J Nucl Med Mol Imaging 35:1989–1999. doi:10.1007/s00259-008-0875-1

    Article  PubMed  Google Scholar 

  30. Chapet O, Kong FM, Quint LE, Chang AC, Ten Haken RK, Eisbruch A, Hayman JA (2005) CT-based definition of thoracic lymph node stations: an atlas from the University of Michigan. Int J Radiat Oncol Biol Phys 63:170–178. doi:10.1016/j.ijrobp.2004.12.060

    Article  PubMed  Google Scholar 

  31. Giraud P, De Rycke Y, Lavole A, Milleron B, Cosset JM, Rosenzweig KE (2006) Probability of mediastinal involvement in non-small-cell lung cancer: a statistical definition of the clinical target volume for 3-dimensional conformal radiotherapy? Int J Radiat Oncol Biol Phys 64:127–135. doi:10.1016/j.ijrobp.2005.06.043

    Article  PubMed  Google Scholar 

  32. Rucker G, Schimek-Jasch T, Nestle U (2012) Measuring inter-observer agreement in contour delineation of medical imaging in a dummy run using Fleissʼ kappa. Methods Inf Med 51:489–494. doi:10.3414/ME12-01-0005

    Article  CAS  PubMed  Google Scholar 

  33. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159–174

    Article  CAS  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Radiation Oncology, University Medical Center Freiburg, Robert-Koch-Str. 3, 79106, Freiburg, Germany

    Tanja Schimek-Jasch M.D., Vesna Prokic Ph.D., Christian Doll M.D., Anca-Ligia Grosu M.D., Ph.D. & Ursula Nestle M.D., Ph.D.

  2. Department of Radiation Oncology (MAASTRO), GROW School for Oncology and Developmental Biology, Maastricht University Medical Centre, Maastricht, The Netherlands

    Esther G. C. Troost M.D., Ph.D.

  3. Institute for Medical Biometry and Statistics, Centre for Medical Biometry and Medical Informatics, University Medical Center Freiburg, Freiburg, Germany

    Gerta Rücker Ph.D.

  4. Department of Radiation Oncology, Ortenau-Klinikum Offenburg-Gengenbach, Gengenbach, Germany

    Viola Duncker-Rohr M.D.

  5. Department of Nuclear Medicine, University Medical Center Freiburg, Freiburg, Germany

    Michael Mix Ph.D.

  6. German Cancer Research Center (DKFZ), Heidelberg, Germany and German Cancer Consortium (DKTK) partner site: Freiburg, Heidelberg, Germany

    Tanja Schimek-Jasch M.D., Vesna Prokic Ph.D., Melanie Avlar, Michael Mix Ph.D., Christian Doll M.D., Anca-Ligia Grosu M.D., Ph.D. & Ursula Nestle M.D., Ph.D.

Authors
  1. Tanja Schimek-Jasch M.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Esther G. C. Troost M.D., Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gerta Rücker Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Vesna Prokic Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Melanie Avlar
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Viola Duncker-Rohr M.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Michael Mix Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Christian Doll M.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Anca-Ligia Grosu M.D., Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Ursula Nestle M.D., Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tanja Schimek-Jasch M.D..

Ethics declarations

Conflict of interest

T. Schimek-Jasch, E.G.C. Troost, G. Rücker, V. Prokic, M. Avlar, V. Duncker-Rohr, M. Mix, C. Doll, A.-L. Grosu and U. Nestle state that there are no conflicts of interest.

This study was conducted with the approval of the Ethics Committee of the Albert-Ludwigs-University Freiburg.

All studies on humans described in the present manuscript were carried out with the approval of the responsible ethics committee and in accordance with national law and the Helsinki Declaration of 1975 (in its current, revised form). Informed consent was obtained from all patients included in studies.

Consent was obtained from all patients identifiable from images or other information within the manuscript. In the case of underage patients, consent was obtained from a parent or legal guardian.

Additional information

On behalf of the PET-Plan study group

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Schimek-Jasch, T., Troost, E., Rücker, G. et al. A teaching intervention in a contouring dummy run improved target volume delineation in locally advanced non-small cell lung cancer. Strahlenther Onkol 191, 525–533 (2015). https://doi.org/10.1007/s00066-015-0812-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00066-015-0812-8

Keywords

Schlüsselwörter

Search

Navigation