Accuracy of the Depression, Anxiety and Stress Scale (DASS-21) for Screening on Comorbid Internalizing Disorders among Youth in Substance Use Disorder Treatment

Introduction: Major depression, anxiety disorders, and post-traumatic stress disorder (PTSD) are among the most prevalent comorbid mental disorders in youth addiction treatment. Hence, screening for these internalizing disorders should be part of the standard routine at intake in substance use disorder treatment. We investigated the usefulness of the Depression, Anxiety and Stress Scale (DASS-21) as a screener for this purpose. Methods: A nationally representative sample of 421 treatment-seeking youths aged 16–22 years with a primary cannabis, alcohol, cocaine, or amphetamine use disorder participated in the study. At intake, we administered the DASS-21 and the Mini International Neuropsychiatric Interview (MINI; Sheehan et al., 1998) based on the Diagnostic and Statistical Manual of Mental Disorders, fifth edition (DSM-5), as “gold standard,” as part of a broader baseline assessment of the Youth in transition study [Moska et al. BMC Psychiatry. 2021;21(1):1–11]. Results: At comparable sensitivity (0.80–0.84), based on the optimal cut-off value, specificity was higher for the DASS-21 total score detecting any DSM-5 internalizing disorder (0.62) than for the DASS subscales specifically detecting depression, anxiety, or PTSD (0.44, 0.49, and 0.51, respectively). Receiver operating characteristic curve analyses showed an area under the curve (AUC) value of 0.80 for the DASS total score to detect any internalizing disorder (“good discrimination”), compared with AUC values of 0.70–0.75 of the DASS depression and anxiety subscales to detect DSM-5 depression, anxiety, and PTSD (“fair discrimination”). The optimal DASS total score cut-off value of ≥44 for detecting any internalizing disorder resulted in 0.81 sensitivity, 0.62 specificity, 0.80 positive predictive value, and 0.64 negative predictive value. Conclusion: Given the high prevalence of comorbid internalizing disorders in youth addiction care, the need to address these comorbid disorders in treatment, and the favorable accuracy of the DASS to detect these disorders, we recommend to implement the DASS-21 as routine screener in youth addiction treatment in the Netherlands.


Introduction
Many mental health disorders originate in adolescence and early adulthood.More than one-third of adolescents in the general population meet the criteria of a past-year mental health disorder, and in about half of these cases, the disorder is associated with severe impairment [1,2].Past-year prevalence is highest for any anxiety disorder (25%), followed by any behavior disorder (16%), mood disorder (10%), and substance use disorder (SUD) with or without dependence (8%).Compared with the other adolescent mental disorders, SUDs have a relatively late age of onset (median approximately 15 years), and show a steep increase in incidence following this median age.For example, in the USA, "lifetime" prevalence of SUD in adolescents showed a sixfold increase from 3.7% at age 13-14 to 22.3% at age 17-18 [1,3].The few available general population studies in adolescents with SUD suggest highup to 60%rates of comorbid mental disorders in this subgroup (e.g., [4]), and more recent data indicate that these comorbid mental disorders oftenin 2 out of 3 casesprecede the onset of SUD in adolescents [5].Not surprisingly, rates of comorbid mental disorders found among adolescents who seek treatment for SUD are even higher.In a review of 10 studies of adolescents in SUD treatment, Couwenbergh et al. [6] found prevalence rates of any comorbid mental disorder ranging from 61 to 88%.
These findings indicate that systematic screening of cooccurring mental disorders is warranted in all youths in SUD treatment, and should be followed by a comprehensive diagnostic assessment of the comorbid disorder(s) in case of a positive screen.However, current best clinical practice in Dutch youth addiction care is that the clinician conducts an unstructured diagnostic evaluation with the patient if (s)he suspects a comorbid non-SUD mental disorder, but systematic screening and use of a structured diagnostic interview are by no means routine in most patients or in most addiction care organizations.Consequently, comorbid mental disorders often remain undetected and, hence, untreated.Notably, the scientific literature indicates that treatment of a comorbid mental disorder does have a favorable effect on the comorbid disorder but generally does not improve the substance use outcomes of SUD treatment.Hence, treatment of both disorders is recommended, and is likely to improve the overall prognosis of patients with concurrent SUD and other mental health disorders in SUD treatment [7,8].
The main objective of screening is to efficiently distinguish between individuals who likely have or do not have a comorbid disorder, without going through the full, often time-consuming, diagnostic process in all individuals.Given this objective, a screening test should be brief and easy to administer by nonspecialists in the relevant field.Screening is useful only when the prevalence of the disorder in the population in question is sufficiently high and when the disorder concerns a treatable condition.
In the context of youth substance abuse treatment, internalizing disorders, including major depression and anxiety disorders, are among the most prevalent comorbid mental disorders [9,10].Hence, these disorders should be a primary target for screening.One of the briefest and most commonly used screening tests for depression and anxiety disorders in adults is the 21-item version of the Depression, Anxiety and Stress Scale (DASS-21) [11].In a systematic review of 48 studies on the measurement properties of the DASS-21 inmostly nonclinicaladult populations, Lee et al. (2019) found moderate-to high-quality evidence for high internal consistency reliability for the DASS subscales and total scale.Although the original authors proposed a threefactor solution of the instrument [11], Lee et al. [12] conducted a systematic review of studies on the measurement properties of the DASS-21 and found highquality evidence for both a common factorrepresenting "general distress" or "negative emotional state"and three specific factors, i.e., depression, anxiety, and stress [12].Concerning adult populations, only two psychometric studies of the DASS-21 have been conducted among patients in substance abuse treatment [13,14].Both (small-scale) studies investigated the criterion validity of the instrument in adults, and found fair to good area under the curve (AUC) values of the DASS total score for identifying depression [14], and post-traumatic stress disorder (PTSD) [13], respectively.
Concerning youths, both Moore et al. [15] and Evans et al. [16] investigated the structural validity of the DASS-21 in a community population study among adolescents.Consistent with the findings of Lee et al. [12] in adults, they found strong support for the presence of both three specific factors representing the domains of depression, anxiety, and stress in adolescents, and of a general, overarching factor of general distress which seems to be represented by the DASS-21 total score.To date, no studies have been conducted on the usefulness and psychometric properties of the DASS-21 among youth in substance abuse treatment.Against this background, the present study aims to investigate the usefulness of the DASS-21 as a screening instrument to identify comorbid past-year major depressive disorder, anxiety disorder, and PTSD in youths who enter substance abuse treatment.In addition to the performance validity of the DASS-21 subscales, we investigated the accuracy of the DASS-21 total scoreas a measure of general distressto identify any past-year internalizing disorder, i.e., depression, anxiety disorder, and/or PTSD.

Participants and Setting
Data for the present study were collected by trained research assistants, who were independent from treatment staff, as part of a broad baseline assessment of a nationally representative prospective cohort study into the long-term course of addiction, mental health problems, and social functioning in youth entering addiction treatment in the Netherlandsthe Youth in transition study (YIT) [17].Four hundred and twenty-four youths who entered addiction treatment at ten treatment organizations were recruited to participate in the study.Inclusion criteria were as follows: age 16-22 years; primary problem substance concerned cannabis, alcohol, cocaine, or amphetamines; and willingness to participate in the study (oral and written informed consent).Exclusion criteria were as follows: referral for diagnostic evaluation only; and insufficient comprehension of basic Dutch language.Three participants had a missing value pertaining to the classification of one of the three internalizing disorders, and were excluded from the analysis.Hence, data were analyzed of 421 participants (Fig. 1).Recruitment of study participants took place between January 2019 and August 2021.The YIT study was funded by The Netherlands Organization for Health Research and Development (No. 60-63600-98-317) and was approved by the Medical Ethical Board of the Leiden University Medical Center (No. NL65903.058.18;file number P18.175).

Instruments
The DASS-21 is a self-report measure designed to assess the severity of the core symptoms of depression, anxiety, and stress, and is derived from the original 42-item questionnaire developed by Lovibond and Lovibond [11].Each subscale contains seven items that are rated on a 4-point severity/frequency scale ranging from 0 ("did not apply to me at all") to 3 ("applied to me very much or most of the time").Scores for each subscale and the total score of the DASS-21 are derived by summing the scores of the relevant item-set, which are subsequently doubled to correspond with the scoring range of the original 42-item version.Hence, DASS-21 subscale scores range from 0 to 42, and the total score ranges from 0 to 126.The usual time frame of the DASS refers to the past week.

Screening for Internalizing Disorders in Youth Addiction Treatment
We used a time frame of past 30 days (prior to treatment entry) to harmonize the time period covered by the DASS with that of other instruments administered at baseline.In addition, we administered the DASS in an interview format, to enable questions for clarification, if needed.
In earlier research, the subscales of the Dutch version of the DASS-21 showed good (Cronbach's alpha 0.8-0.9) to excellent (alpha >0.9) internal consistency in both a nonclinical and clinical sample [18].In the present study sample, internal consistency amounted to alpha = 0.89 for depression, 0.78 for anxiety, 0.83 for stress, and 0.92 for the total score.
To investigate the usefulness of the DASS-21 as a screening instrument for identifying comorbid internalizing disorders among youth who apply for addiction treatment, we administered the Mini International Neuropsychiatric Interview (MINI) [19] based on the Diagnostic and Statistical Manual of Mental Disorders, fifth edition (DSM-5) in its Dutch translated version (MINI 7.0.2,2019) as "gold standard."From the disorders covered by the MINI, we used the sections of major depressive disorder, panic disorder, agoraphobia, social phobia, generalized anxiety disorder (GAD), and PTSD for the current study.In addition to screening for each of these disorders separately, we pooled each of these disorders to screen for the presence of "any internalizing disorder" (i.e., any of these classifications).The DASS-21 and MINI were administered during the same interview session at treatment intake, which took place shortly prior to the start of addiction treatment.
Concerning the structure of the MINI, each diagnostic section starts with one to three screening questions pertaining to the key diagnostic criteria of the disorder.If the patient answers negatively to these key questions, the disorder is coded as absent and the interview proceeds with the next diagnostic section [19].In case of a positive answer, additional symptom questions are asked, closely following the diagnostic criteria and structure of the DSM-5.As a rule, and particularly relevant for the current study, symptoms were excluded if they were clearly induced by the physiological effect of a psychoactive substance, i.e., if the symptoms occurred only during or shortly following intoxication or withdrawal.
In the MINI, priority is given to the identification of "current" disorders to keep the instrument short [19].However, the time frame in the DSM-5 version of the MINI for "current disorder" differs substantially across the disorders, withfor the internalizing disordersa range of (at least) "past 2 weeks" for depression; "past month" for panic disorder and PTSD; and "past 6 months" for agoraphobia, social anxiety, and GAD.In the context of our prospective YIT cohort study, with its longitudinal assessments at treatment entry (baseline) and 2 and 4 years after treatment entry, we deemed it most relevant to use a standard time frame of "past-year," as is most common in epidemiological studies.For this reason, and to better accommodate the different time frames included in the MINI diagnoses, patients in our sample received a past-year diagnosis for a specific disorder in the MINI if they met the minimum required duration of that disorder anytime during the previous year.
Demographic background, substance use, and SUD data were derived from the baseline YIT questionnaire and the Measurements in the Addictions for Triage and Evaluation -Youth version (MATE-Y) [20].For an extensive description of the YIT instruments and study procedures, the reader is referred to the protocol paper of the YIT [17].

Data Analysis
Differences in mean scores on the DASS-21 subscales and total score between participants with and without DSM-5 depression, anxiety, PTSD, and any internalizing disorder were tested by independent t tests.Although the Likert-type items in the DASS-21 are in essence ordinal, quite a number of experts in the measurement field have argued that, while a single Likert item may be ordinal, the sum or the average of several of these Likert items (i.e., the Likert scale) represents an approximate continuous variable, which can well be analyzed with parametric tests, like the independent t test [21][22][23][24].We determined the sensitivitythe proportion of disordered individuals with a positive test result ("true-positive rate"; see Table 1); the specificitythe proportion of non-disordered individuals with a negative test result ("truenegative rate"); the positive predictive value (PPV)the probability that an individual with a positive test actually has the disorder; and the negative predictive value (NPV)the probability that an individual with a negative test actually does not have the disorderof the DASS-21 depression and anxiety subscales and total score.Criterion diagnoses were DSM-5 past-year major depressive disorder, anxiety disorders (i.e., panic disorder; agoraphobia; social phobia; GAD), PTSD, and any internalizing disorder.
In addition, we conducted receiver operating characteristic (ROC) curve analyses, in which sensitivity is plotted against 1specificity (i.e., the true-positive vs. the false-positive rate) for each cut-off value of the DASS subscales and total score, to determine the overall classification accuracy in terms of AUC.AUC values can range between 0.50, indicating chance prediction (represented in the ROC space by a 45°diagonal) and 1.00, signifying a perfect test.As a rule of thumb, and derived from Hosmer et al. [25], AUC values of 0.5-0.69represent poor discrimination; 0.70-0.79fair discrimination; 0.80-0.89good discrimination; and ≥0.90 excellent discrimination.
We prioritized high sensitivity by opting for a relatively low threshold value, determined by the lowest DASS cut-off at which sensitivity reached a value of 0.80 or higher.Opting for higher sensitivity (less false negatives) generally comes at the expense of lower specificity (more false positives), but we deemed this acceptable because we wanted to minimize the number of false negatives (i.e., participants with the actual disorder not identified as positives by the screener).Data were analyzed with the IBM Statistical Package for the Social Science (SPSS), version 27.0.

Results
The baseline characteristics of the participants at treatment entry are summarized in Table 2. Most study participants were male (68.4%) and defined their cultural identity as Dutch-western (89.3%).On average, participants were 20 years old.The majority of youth entered treatment for a primary problem with cannabis (62.2%), followed by alcohol (18.3%), and stimulants (cocaine and amphetamine pooled; 19.5%).Based on the MINI, nearly half of the participants received a past-year DSM-5 classification of depression, one-third were classified with an anxiety disorder, and one-quarter were classified with PTSD.Nearly two-thirds met the criteria of one or more internalizing disorders.

Screening for Internalizing Disorders in Youth Addiction
Treatment not reported).Mean scores on the DASS-21 were significantly higher for those with the disorder than for those without the disorder on all DASS-21 subscales and on the total score.
Table 1 shows the main accuracy parameters of the DASS-21 and their calculation, using the identification of any internalizing disorder as an example, with an optimal DASS total score cut-off value of ≥44 (see below).Among youths with the target disorder present, 81% (222/274) consisted of true positives (sensitivity) on the DASS and 19% (52/274) were false negatives.Similarly, among youths with the target disorder absent, 62% (91/147) were true negatives (specificity) on the DASS and 38% (56/147) were false positives.Concerning the PPV and NPV of the DASS for detecting any internalizing disorder, 80% (222/ 278) of youths with a positive test consisted of true positives (PPV).Likewise, 64% (91/143) of those with a negative test were true negatives (NPV).
Figure 2 displays the ROC curve in which the truepositive rate is plotted against the false-positive rate for each cut-off of the DASS total score, again using identification of any internalizing disorder as example.Overall classification accuracy in terms of AUC was 0.80 ("good discrimination").Separating girls (N = 132) and boys (N = 289) in the sample, again using the optimal DASS total score value of ≥44 for detecting internalizing disorder resulted in 0.86 sensitivity for girls versus 0.78 for boys, and specificity amounted to 0.53 for girls and 0.64 for boys.The AUC value was 0.79 for both girls and boys.Hence, no relevant differences in these values were found between girls and boys and the total sample.
Table 3 shows the main accuracy parameters of the DASS for specifically identifying depression, anxiety disorders, and PTSD using the depression and anxiety subscales (upper panels) and for detecting any internalizing disorder using the total score (lower panel).At comparable sensitivity (0.80-0.84) based on the optimal cut-off value (defined as the lowest DASS cut-off at which sensitivity reached a value of 0.80 or higher), specificity was higher for the total score detecting any internalizing disorder (0.62) than for the subscales detecting depression, anxiety, or PTSD (0.44-0.51).This was, consequently, also the case for the AUC, with a value of 0.80 versus 0.70-0.75,respectively.Supplemental analyses indicated that AUC values of the specific subscales were not superior (0.70-0.75; "fair discrimination") to those found when the total score was used for specifically detecting depression, anxiety, and PTSD (0.72-0.78; "fair discrimination"; not shown in Table 2).As shown in the lower panel, opting for higher sensitivity came at the expense of lower specificity and vice versa.Lastly, the PPV was higher and the NPV was lower when screening for any internalizing disorder using the DASS total score (lower panel: 0.80 and 0.64, respectively) compared with the values for each DASS subscale separately (upper panels: 0.34-0.61and 0.73-0.87,respectively).

Discussion
We evaluated the usefulness of the DASS-21 as a screener for past-year DSM-5 depression, anxiety disorder, or PTSD and as a screener for the presence of any internalizing disorder in a nationally representative sample of youths aged 16-22 years old with a primary cannabis, alcohol, cocaine, or amphetamine use disorder entering substance abuse treatment in the Netherlands.The optimal DASS total score cut-off value of ≥44 for detecting any internalizing disorder resulted in 0.81 sensitivity, 0.62 specificity, 0.80 PPV, and 0.64 NPV and the AUC value amounted to 0.80 ("good discrimination").This AUC value was higher compared to the values when using the DASS subscales for detecting depression, anxiety, or PTSD ("fair discrimination").Hence, the DASS subscales seem to have no added value for identifying separate comorbid internalizing disorders among youths in SUD treatment, and it seems most appropriate to use the DASS total score as broadspectrum screener for the decision of whether or not subsequent diagnostic evaluation of internalizing disorders is warranted in this population.Such application of the DASS-21 as broad-spectrum screener is in line with emerging evidence for the use of the total score as a valid measure of overall psychological distress in adolescents [16].Moreover, broad screening seems preferable because internalizing disorders often show considerable overlap among themselves [26].
In our study, high sensitivity (≥0.80) was accompanied by relatively low specificity.In all (non-perfect) screening tests, there is a tradeoff between sensitivity and specificity, in that higher sensitivity nearly always comes at the expense of lower specificity, andceteris paribushigher specificity comes at the expense of lower sensitivity.Both test values are not inherently high or low (or different from each other), but their value is highly dependent upon the choices made by the researcher or clinician.
We intentionally opted for a relatively low cut-off value on the DASS-21 total score to achieve sufficiently high sensitivity.This decision was made because we considered it crucial to minimize the number of false negativesi.e., the number of patients who actually had the disorder but were missed by the screener.Alternatively, if we would have prioritized higher specificity, by increasing the cut-off value on the DASS-21, this would not only have resulted in less false positives (i.e., the number of patients with a positive screening test who did not have the actual disorder), but also in a higher number of false negativesthe scenario that we wanted to avoid.Please see the bottom rows in Table 3, which illustrate the tradeoff between higher/lower sensitivity and specificity.
When screening in clinical practice, actual disease status is often unknown, and sensitivity and specificity are less useful measures of test validity [27].In practice, the most prominent question for the clinician pertains to the probability of the disease being present or absent, given the outcome of screening.We found a sufficiently high PPV, with four in five true positive screens, accompanied by a relatively low NPV, with two in three true negatives, when screening on any internalizing disorder.However, an important caveat with regard to these findings is that the predictive value of a test is highly dependent on (among others) the prevalence of the disorder in the sample, with higher prevalence "inflating" the PPV, and lower prevalence inflating the NPV.Hence, the higher PPV found when screening on any internalizing disorder and higher NPV when screening on each disorder separately may at least partly be explained by this association.
Strengths of the study are that (1) this is the first study which investigates the usefulness of the DASS-21 as a screening instrument for youths in substance abuse treatment, (2) the results are based on a representative, well-defined, sample of youths seeking treatment for the most prevalent SUDs in the Netherlands (covering nearly three quarters of total youth addiction treatment demand), and (3) we used a structured DSM-5 diagnostic interview in which participants received a negative classification of a disorder if the disorder symptoms only occurred during intoxication or withdrawali.e., when it concerned a substanceinduced disorder (e.g., substance-induced depression) only.Study limitations include, first, that we assessed the accuracy of the DASS-21 shortly after intakea time at which acute effects of alcohol and drugs may affect DASS-21 scores, and not at a second time point, Screening for Internalizing Disorders in Youth Addiction Treatment e.g., after discontinuation or substantial reduction of substance usewhen acute alcohol and drug effects are likely diminished [14], and, second, that we did not compare the performance of the DASS in youth with that of other broad-spectrum screeners of general distress, like the Outcome Questionnaire (OQ-45) [28] or Brief Symptom Inventory (BSI) [29].Nevertheless, despite possible acute drug effects during intake, the DASS performed quite well at this time point, in particular given that symptoms that were solely druginduced resulted in a negative classification in our structured diagnostic interview.Third, we used a time frame of "past month" for the DASS-21 instead of the usual "past week" to better accommodate the different time frames of "current disorder" between the internalizing disorders in the MINI.In addition, we considered a disorder in the MINI as "present" in the past year if a patient met the minimum required duration of that disorder -2 weeks for depression, 1 month for panic disorder and PTSD, and 6 months for agoraphobia, social anxiety, and GADanytime during the previous year.Given the strong positive association in earlier research between illness severity and comorbidity of mental disorders with treatment seeking [30][31][32], it is likely that people seek treatment for addiction and concurrent mental health problems at a time when they are experiencing the most acute and severe problems.Hence, for most patients in our sample, their "past-year disorder" likely pertains to the weeks or months directly preceding treatment entry, rather than to the first months of the previous year.

Conclusion
Given the high prevalence of comorbid internalizing disorders in youth addiction care, the need to address these comorbid disorders in treatment, and the favorable accuracy of the DASS to detect these disorders, we recommend to implement the DASS-21 as routine screener in youth addiction treatment in the Netherlands.

Fig. 1 .
Fig. 1.Flowchart of the Youth in transition study.

Table 3 .
Accuracy of DASS-21 subscales and total score for detecting DSM-5 depression, anxiety, PTSD, and any internalizing disorder predictive value; NPV, negative predictive value; AUC, area under the curve.