A retrospective validation study of three models to estimate the probability of malignancy in patients with small pulmonary nodules from a tertiary oncology follow-up centre

Highlights

• The BTS Pulmonary Nodule Guidelines 2015 recommend the use of composite prediction models in clinical practice.

• This study shows that 3 the recommended models have a lower predictive value than published in the original papers.

• This study identifies the limitations of these models, especially in patients with sub-centimeter nodules.

Aim

To estimate the probability of malignancy in small pulmonary nodules (PNs) based on clinical and radiological characteristics in a non-screening population that includes patients with a prior history of malignancy using three validated models.

Materials and methods

Retrospective data on clinical and radiological characteristics was collected from the medical records of 702 patients (379 men, 323 women; range 19–94 years) with PNs ≤12 mm in diameter at a single centre. The final diagnosis was compared to the probability of malignancy calculated by one of three models (Mayo, VA, and McWilliams). Model accuracy was assessed by receiver operating characteristics (ROC). The models were calibrated by comparing predicted and observed rates of malignancy.

Results

The area under the ROC curve (AUC) was highest for the McWilliams model (0.82; 95% confidence interval [CI]: 0.78–0.91) and lowest for the Mayo model (0.58; 95% CI: 0.55–0.59). The VA model had an AUC of (0.62; 95% CI: 0.47–0.64). Performance of the models was significantly lower than that in the published literature.

Conclusions

The accuracy of the three models is lower in a non-screening population with a high prevalence of prior malignancy compared to the papers that describe their development. To the authors' knowledge, this is the largest study to validate predictive models for PNs in a non-screening clinically referred patient population, and has potential implications for the implementation of predictive models.

Introduction

The management of patients with small pulmonary nodules (PNs) is clinically challenging. With the widespread availability of multidetector row computed tomography (CT) and the significant increase in the number of scans performed,¹ this scenario is likely to increase in clinical practice, and clear evidence is required to guide management strategies.

Data from published case series in Europe,2, 3 Asia,4, 5 and America6, 7, 8 report the prevalence of PNs from 2% to 24% (mean 13%) with a mean lung cancer prevalence of 1.5%. The prevalence of PNs in a screening population is higher (mean 33%),⁹ but management of nodules in this scenario is predetermined for the majority of screening programmes.

The effective management of PNs requires knowledge of the “pre-test” probability of malignancy in each individual patient, prior to conducting further investigation.¹⁰ The British Thoracic Society Guidelines for the management of PNs recommends the use of prediction models when evaluating these patients.

Twenty-eight studies 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33 in the literature have evaluated the clinical and radiological characteristics of PNs in relation to the probability of malignancy. Four studies have developed composite prediction models that can be used in clinical practice and are easily accessible.

Swensen et al.¹¹ (Mayo model) and Gould et al.¹² (VA model) identified independent clinical and radiological predictors of malignancy and used this to create and validate models with an AUC of 0.8. The Mayo model was validated with the additional parameter of positron-emission tomography (PET) avidity by Herder et al.,¹³ increasing the AUC to 0.92 (the Herder model). McWilliams et al.¹⁴ developed a model based on a lung cancer screening population. The AUC for this model was 0.94 and specifically 0.91 for nodules ≤10mm.

A number of case series have attempted to address the likelihood of malignancy in patients with known extra-thoracic cancer34, 35; however due to their heterogeneity, and small patient numbers, these studies do not provide sufficient data to distinguish benign from malignant nodules or a metastasis from a primary lung cancer.

The aim of this study was to validate three existing composite prediction models in a non-screening clinically referred patient population with small, predominantly subcentimetre PNs.