Assessing validity of a depression screening instrument in the absence of a gold standard

Abstract

Purpose

We evaluated the extent to which use of a hypothesized imperfect gold standard, the Composite International Diagnostic Interview (CIDI), biases the estimates of diagnostic accuracy of the Patient Health Questionnaire-9 (PHQ-9). We also evaluate how statistical correction can be used to address this bias.

Methods

The study was conducted among 926 adults where structured interviews were conducted to collect information about participants' current major depressive disorder using PHQ-9 and CIDI instruments. First, we evaluated the relative psychometric properties of PHQ-9 using CIDI as a gold standard. Next, we used a Bayesian latent class model to correct for the bias.

Results

In comparison with CIDI, the relative sensitivity and specificity of the PHQ-9 for detecting major depressive disorder at a cut point of 10 or more were 53.1% (95% confidence interval: 45.4%–60.8%) and 77.5% (95% confidence interval, 74.5%–80.5%), respectively. Using a Bayesian latent class model to correct for the bias arising from the use of an imperfect gold standard increased the sensitivity and specificity of PHQ-9 to 79.8% (95% Bayesian credible interval, 64.9%–90.8%) and 79.1% (95% Bayesian credible interval, 74.7%–83.7%), respectively.

Conclusions

Our results provided evidence that assessing diagnostic validity of mental health screening instrument, where application of a gold standard might not be available, can be accomplished by using appropriate statistical methods.

Introduction

Screening for mental health problems in clinical- and community-based settings is an important component in overall public health disease prevention and health promotion strategies. The success of screening procedures, however, is largely dependent on the accuracy of the diagnostic procedure, used as the gold standard or criterion standard, to evaluate the screening instruments [1], [2], [3]. Psychometric properties such as sensitivity and specificity are common measures used to evaluate the quality of a screening test. These psychometric properties are unbiased if the screening test results are compared with a gold standard measure [2], [3]. We recognize that there is no perfect gold standard and we use the term to refer to the best available method used to determine the presence or absence of the condition or disease of interest. However, verification of the true status using the gold standard may be impossible to obtain because of cost and human resources. In some instances, the gold standard may be invasive, impractical to obtain, or unethical to conduct. For example, gold standard diagnosis of Alzheimer's disease cannot be ascertained until a patient dies and an autopsy is performed. In epidemiologic studies, where a comparison with a gold standard is not possible, validation studies often compare screening instruments with instruments that are imperfect but more precise than the screening instrument. The key assumption is that the measurement error for the imperfect reference or gold standard is unlikely to be correlated with the screening instrument [4]. If an imperfect standard is used as if it were a gold standard, the estimated accuracy of the tests would be biased due to misclassification [3], [5]. Zhou et al. [3] call this type of bias “imperfect gold standard bias.” A number of authors have proposed model-based estimates or estimates that make use of prior information to reduce or correct this imperfect gold standard bias without retesting subjects [1].

The Patient Health Questionnaire-9 (PHQ-9) is a very brief, easy to administer, and interpret depression screening instrument [6]. Because of its brevity, the PHQ-9 is widely used as a depression screening instrument in primary care settings among racially and ethnically diverse populations. Additionally, it has been reported to be a valuable tool for the detection and management of depression [6]. Use of the PHQ-9 instrument in new clinical and research settings requires evaluating the validity of the instrument in comparison with a diagnostic “gold standard.” The Schedules for Clinical Assessment in Neuropsychiatry (SCAN), a semistructured clinical interview widely considered as gold standard, is used to assess and diagnose psychiatric disorders including depression among adults [7]. The instrument offers flexibility for clinicians to phrase questions about particular symptoms taking into account local context. However, it requires that clinicians make their clinical decisions following the definitions and criteria provided in the Diagnostic and Statistical Manual (DSM) [8], [9]. Although the SCAN has been reported to have excellent accuracy in depression diagnosis, it is time demanding, expensive, and requires a trained clinician, thus limiting its use in resource-limited clinical settings [10].

To address these limitations, alternative diagnostic tools for the measurement of depression have been developed. One of these tools is the Composite International Diagnostic Interview (CIDI). The CIDI is a fully structured lay-administered diagnostic interview that can be used to diagnose major depressive disorder (MDD) according to DSM criteria, although it is regarded as a less optimal gold standard compared with the SCAN [11], [12]. In an effort to use a less burdensome assessment, some investigators have used the CIDI as the criterion standard to evaluate the diagnostic accuracy of the PHQ-9 [13], [14]. Use of CIDI as gold standard (an imperfect but more precise instrument than PHQ-9) is expected to bias the validity of the PHQ-9. However, no study has systematically evaluated the impact of using the CIDI as a gold standard. Therefore, we conducted this study to assess the extent to which the use of what we hypothesized to be an imperfect gold standard [3] biases the estimates of diagnostic accuracy of a screening instrument (the PHQ-9). We further sought to demonstrate how statistical methods can be used to correct the bias and improve psychometric properties of the PHQ-9.