Figures
Abstract
Background
Symptoms of depression and anxiety are highly prevalent in adolescence and they are the cause of considerable suffering. Even so, adolescents are not inclined to seek professional help for emotional problems. Internet-based preventive interventions have been suggested as a feasible method of providing appropriate care to adolescents with internalizing symptoms. The objective of this study was to evaluate the effects of preventive Internet-based (guided) self-help problem-solving therapy (PST) for adolescents reporting mild to moderate symptoms of depression and/or anxiety as compared to a waiting list control group (WL).
Methodology/Principal Findings
A total of 45 participants were randomized to the 2 conditions. PST consisted of 5 weekly lessons. Participants were supported by e-mail. Self-report measures of depression and anxiety were filled in at baseline and after 3 weeks, 5 weeks, and 4 months. Of the 45 participants, 28 (62.2%) completed questionnaires after 3 weeks, 28 (62.2%) after 5 weeks, and 27 (60%) after 4 months. Hierarchical linear modeling analyses revealed overall improvement over time for both groups on depressive and anxiety symptoms. However, no significant group x time interactions were found. No differences were found between completers and non-completers.
Conclusions/Significance
Results show that depressive and anxiety symptoms declined in both groups. No support was found, however, for the assumption that Internet-based PST was efficacious in reducing depression and anxiety in comparison to the waiting list control group. This finding could represent lack of power.
Citation: Hoek W, Schuurmans J, Koot HM, Cuijpers P (2012) Effects of Internet-Based Guided Self-Help Problem-Solving Therapy for Adolescents with Depression and Anxiety: A Randomized Controlled Trial. PLoS ONE 7(8): e43485. https://doi.org/10.1371/journal.pone.0043485
Editor: James G. Scott, The University of Queensland, Australia
Received: March 26, 2012; Accepted: July 20, 2012; Published: August 31, 2012
Copyright: © Hoek et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This study is funded by ZonMw – the Netherlands organisation for Health Research and Development, grant number 120610006. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Despite the high prevalence and impact of depression and anxiety in adolescence [1], [2], many young people do not seek professional help for emotional problems [3]. Internet-based interventions, however, may enable treatment consideration and participation. Adolescents are particularly comfortable interacting within the computer environment [4], and Internet-based interventions may reduce stigma, maximize autonomy, save travelling time, provide low cost access, reduce waiting-lists, abolish the need to schedule appointments with a therapist, and reduce objections like lack of willingness to talk to a stranger about personal problems [5], [6].
Studies on adolescent Internet-based interventions for depression or anxiety are scarce (see [4], [7], [8]) and the effectiveness of merely two programs was examined in randomized controlled trials (RCTs) [9], [10], [11]. These concern a five session cognitive behavior therapy (CBT) based intervention program (MoodGYM) in preventing and reducing the symptoms of depression and anxiety in an adolescent school-based population, and a six session guided CBT group course (Master Your Mood) for adolescents aged 16 to 25 years with depressive symptoms. Previous controlled pilot trials of the MoodGYM intervention in two private single-sex schools demonstrated benefits on self-reported depression for adolescent boys who completed more modules [9] and for girls at follow-up [10]. A previous study with a more diverse sample reported decreases in anxiety symptoms for the total sample and decreases in depression for males only [7]. The online group course Master Your Mood was effective in reducing depressive and anxiety symptoms and in increasing mastery in young people [11].
As far as we know, there is no study which evaluates Internet-based problem-solving therapy (PST) for adolescents. PST is aimed at determining the major goals in life, investing energy only in those problems that are related to what matters and learning to accept those situations that cannot be changed, with problem-solving skills as the core element of this approach [12], [13]. Face-to-face problem-solving therapy has been found to be effective in a variety of problem areas [12], [14], and given high comorbidity and shared risk factors of depression and anxiety [15], the provision of an intervention that can concurrently decrease depressive and anxiety symptoms in adolescents would be appealing. A PST-based intervention that could be applied through the Internet was developed and studied in an adult population [13]. The intervention was successful in reducing adults' symptoms of depression, anxiety, and work related stress, and was consequently adapted for adolescents. The aim of the current study is to evaluate the effects of preventive Internet-based PST for adolescents reporting mild to moderate symptoms of depression and/or anxiety as compared to a waiting list control group.
Methods
The protocol for this trial and supporting CONSORT checklist are available as supporting information; see Checklist S1 and Protocol S1.
Participants
Participants were recruited through press releases, banners and advertisements on the Internet, advertisements in magazines, referral by school-doctors, through brochures and posters in schools, and through information to parents who are treated in mental health care institutions for depression and anxiety. Recruitment took place from January 2009 to May 2011. After application through a website, applicants received a brochure about this study and an informed consent form by email. If adolescents were younger than 18 years of age, parents received a brochure and an informed consent form by regular mail after application by the adolescent. Adolescents aged 17 years or younger needed parental consent for study participation. After receiving informed consent, adolescents received the baseline questionnaire by email.
To be included, applicants had to report mild to moderate depressive and/or anxiety symptoms (there was no lower limit for entering the study) and have sufficient knowledge of the Dutch language, access to the Internet, and an email address. At first, adolescents aged 12 to 18 years with depressive and/or anxiety symptoms were found eligible for this study. It was difficult to include adolescents in the study. Inasmuch as we had strong indications that parental consent requirements often withheld adolescents from participating [16] and because older adolescents showed interest in our intervention, the upper age limit was altered during the study and adolescents aged 19 to 21 years were also included from April 2010. Adolescents were excluded from the study if they were younger than 18 years of age and had no parental permission, were receiving other psychological treatment for mental health problems, had severe depressive symptoms (defined as a score above 40 on the Centre for Epidemiologic Studies Depression scale; CES-D [17]), had severe anxiety symptoms (indicated by a score above 14 on the anxiety subscale of the Hospital Anxiety and Depression Scale; HADS-A [18]), and/or had prominent suicide ideation (indicated by a score above 1 on the suicide item of the Beck Depression Inventory-II; BDI-II [19]). Respondents who were excluded due to the presence of severe symptoms received a telephone call in which they were advised to consult their general practitioner. If younger than 18 years, their parents were also informed by telephone. The study was approved by the Medical Ethics Committee of the VU University Medical Center, Amsterdam, the Netherlands. For a detailed description of the study and intervention we refer to the study protocol [20].
Intervention
The PST intervention was web-based and participants were provided with a username and password to access the website. Every week an automated email was sent to participants to explain the contents and exercises for the coming week. Mental health care professionals and the authors (WH, JS) offered feedback on the completed exercises. This feedback was not therapeutic but was directed at mastering the proposed problem-solving strategies. PST consisted of three steps. First, respondents described what really mattered to them. Second, they wrote down their current worries and problems, and they divided these problems into three categories: (a) unimportant problems, (b) solvable problems, (c) problems which cannot be solved. For each of these three types of problems a different strategy was proposed to solve the problems or to learn to cope with the unimportant and unsolvable ones. This step is the most important element of PST. For the solvable problems, we proposed the following six-step procedure: (1) write a clear definition of the problem, (2) generate multiple solutions to the problem, (3) select the best solution, (4) work out a systematic plan for this solution, (5) carry out the solution, and (6) evaluate as to whether the solution has resolved the problem. During the third and last step, the respondents made a plan for the future in which they described how they would try to accomplish those things that matter most to them. The course took 5 weeks and consisted of one lesson a week.
Design
This study is a randomized controlled trial with two groups: an Internet-based guided self-help intervention (PST) and a waiting list control group (WL). The study was designed to compare the effects of the intervention with the waiting list. Based on a power of 0.80 and an alpha of 0.05, we needed 63 patients in each condition in order to be able to demonstrate moderate effects of d = 0.40. Therefore, the total sample size was set at 126. 45 participants were included in total (see Fig. 1). Participants on the waiting list got access to a website with general information on depression and anxiety, and could complete the intervention 4 months after baseline. Questionnaires were administered on the Internet and were sent after application (baseline) and 3 weeks, 5 weeks, and 4 months later.
Randomization
Randomization took place at an individual level after the baseline measurement. We used block randomization, with each block containing 10, 12, 14, or 16 allocations. An independent researcher made the allocation schedule with a computerized random number generator. Immediately after randomization, adolescents were informed about the randomization outcome by email. If younger than 18 years of age, the subject's parents also received this email. We randomized 22 participants to the PST intervention and 23 participants to the waiting list condition.
Measures
Structured diagnostic interview.
Depression and anxiety subscales of the National Institute of Mental Health Diagnostic Interview Schedule for Children Version IV (NIMH DISC-IV) were conducted by telephone before randomization. The Diagnostic Interview Schedule for Children is a reliable and valid structured diagnostic interview designed for lay interviewers, which includes algorithms to diagnose DSM-IV disorders in children and adolescents [21]. Participants completed a telephone version of the generalized anxiety disorder, social phobia, panic, agoraphobia, major depression, and dysthymia modules of the NIMH DISC-IV. The NIMH DISC-IV assesses the presence of diagnoses occurring within both the past 12 months and the past 4 weeks. The interview was not a requisite of the study, 33 participants decided to participate. Telephone versions of structured psychiatric interviews in both adults [22] and youth [23] have been found to have a high correlation with in-person interviews.
Depressive symptoms.
Depressive symptoms were assessed with the Centre for Epidemiologic Studies Depression scale (CES-D) [17]. The CES-D consists of 20 items for which subjects rate the frequency of symptoms during the past week with scores ranging from 0 (rarely or none of the time present [less than 1 day]) to 3 (most or all of the time present [5–7 days]), with a total score ranging between 0 and 60. A cut-off score of ≥16 can be used to identify adolescents with mild or greater levels of depression. The validity of the CES-D has been tested in different populations, including studies with adolescents [24]–[26]. In the current study, Cronbach's alpha ranged from 0.89 to 0.93.
Anxiety symptoms.
To measure the prevalence of anxiety symptoms, the anxiety subscale of the Hospital Anxiety and Depression Scale (HADS-A) [18] was used. The HADS-A consists of seven items rated on a 4-point scale ranging from 0 (not at all) to 3 (a great deal of the time), with 3 indicating higher symptom frequency. A total score ranges from 0 to 21 and score ranges are classified as normal (0–7), mild (8–10), moderate (11–14) and severe (15–21). The HADS shows good homogeneity and reliability, with Cronbach's alpha ranging from 0.81 to 0.84 in different normal and clinical Dutch samples [27], and has been found valid and adequate for use with adolescents [28]. Cronbach's alpha in this study varied from 0.76 to 0.90.
Client satisfaction.
The eight-item Client Satisfaction Questionnaire (CSQ-8) [29] assesses patient satisfaction. The CSQ-8 items can be scored on a scale from 1 (low satisfaction) to 4 (high satisfaction), with labels specific to particular items. The total CSQ score was obtained by averaging the items answered. Good construct validity and internal consistency have been previously reported [29]. Cronbach's alpha was 0.86 in this study.
In conformity with grade rating scales in Dutch education, participants were also asked to give an overall satisfaction rating on a scale from 1 (very bad) to 10 (excellent), followed by an open question on how this rating could have been improved.
Statistical analyses
Missing values.
All analyses were performed on the intention-to-treat sample. Baseline data were available for all participants. The Linear Mixed Modeling (LMM) procedure was used for all analyses to estimate missing values. LMM includes incomplete cases in the analysis and employs restricted maximum likelihood estimation to calculate parameter estimates. LMM assumes that missing data are missing at random.
Effects
Treatment differences.
LMM, using the variance components covariance structure, was used to investigate treatment differences. Time was treated as a continuous variable, condition was treated as a fixed effect, and the intercept was included as a random effect.
Recovery.
For the CES-D total score, the standard cut-off score of 16 was used as an indication of recovery. A HADS score lower than 8 [30] was used as an indication of recovery. This was calculated only for those participants who did meet these criteria at baseline. The differences in recovery rate between the intervention and control group were also calculated with binary logistic regression.
Completion status.
LMM was used to investigate differences in development of depressive symptoms between treatment completers, non-completers, and WL. Time was treated as a continuous variable. The intercept was included as a random effect. Completers were defined as respondents who completed all lessons. All analyses were conducted using SPSS for Windows version 17.0.
Results
Participants
Of the 378 adolescents who were potentially interested in participating, 312 adolescents did not give informed consent. From 66 adolescents we received baseline questionnaires and written informed consent. Of these, 17 did score above the cut-off on either the CES-D, HADS, or BDI suicide item, 1 already received different treatment, and 3 withdrew without specifying a reason. The remaining 45 participants were randomized. Table 1 presents sample characteristics. Participants were mainly female (75.6%, n = 34). The mean age was 16.07 years (SD = 2.31). Almost all participants were of Dutch origin (91.1%, n = 41). The mean score of all participants on the CES-D at baseline was 25.02 (SD = 9.06, range = 5–40), on the HADS 8.84 (SD = 3.60, range = 2–14). Both averages are above cut-off scores for mild levels of depression and anxiety. In the 12 months before our baseline measurement, 45% of participants encountered any mood or anxiety disorder, with social phobia being the most common (42%). There were no statistically significant differences between the groups at baseline with respect to demographics, symptoms or diagnostic status (Table 1 and Table 2).
Attrition
Attrition rates for the full sample were 37.8% (n = 17) at the 3-week assessment, 37.8% (n = 17) at 5 weeks, and 40% (n = 18) at 4 months. Reasons for the high level of attrition were unknown. There were no significant differences in attrition rates between groups. Participants who completed post measures were younger than participants who didn't (15.46 and 16.89 years respectively, t(43) = 2.137, P = 0.038).
Effects
Treatment differences.
Table 2 reports the means and standard deviations for depression and anxiety for the intervention and waiting list control group over 4 months. For depression, a significant overall improvement over time was found for the complete sample on the CES-D, Β = −2.31, SE = 0.86, P = 0.009. Neither significant group (Β = −0.10, SE = 3.58, P = 0.978) nor group x time interaction (Β = 0.54, SE = 1.14, P = 0.637) effects were found. Regarding anxiety, no significant time (Β = −0.73, SE = 0.39, P = 0.062), group (Β = −0.06, SE = 1.57, P = 0.968) or group x time interaction (Β = 0.25, SE = 0.51, P = 0.621) effects were found.
Recovery.
36 participants suffered clinical levels of depression according to the CES-D at baseline (80% of total sample). 37.5% of participants who completed post-measures had recovered at 5 weeks and 25% had recovered at 4 months across both groups. Regarding anxiety, 31 participants suffered clinical levels at baseline (68.9%). After 5 weeks, 60% had recovered while 52.4% had recovered after 4 months. Recovery effects were not significantly different between intervention and waiting list groups.
Treatment completers versus non-completers.
Many participants failed to complete the whole course. Of those participants assigned to the intervention condition, 10 (45%) participants completed three or more lessons and 6 (27%) completed all five. All but 1 participant completed the first lesson. Completers were younger than non-completers (14.83 years versus 16.94 years), t (20) = −2.20, P = 0.04. Regarding symptom measures at baseline, no differences were found between completers and non-completers.
We investigated differences in development of depression and anxiety scores between treatment completers, non-completers, and waiting list condition. No differences were found in improvement of depressive and anxiety symptoms between completers and non-completers. Moreover, we examined differences in development of depression and anxiety between participants who completed three or more lessons, participants who completed less than three lessons, and participants on the waiting list. No differences were found in improvement of depression and anxiety.
Client satisfaction.
Table 3 provides an overview of the CSQ items and the total scale score that were used to estimate participants' satisfaction with the intervention. Given the 1–4 metric for the CSQ scale, participants expressed moderate satisfaction (total score M = 2.51, SD = 0.55). The mean overall satisfaction grade on a 1–10 scale was 6.45 (SD = 1.28) and 85% of participants reported grades of 6 or above. Fourteen participants provided suggestions on how to improve the intervention which have been abstracted into five main suggestions. Suggestions were: better communication and guidance, more feedback from supervisor and receiving emails when having received feedback (adolescents now had to be logged onto the website in order to see whether they got a response on their assignments) (n = 5), more clarity on what to do and why, a clearer website (n = 3), more exercises and more elaborative on exercises, more exercises in negative thinking (n = 3), more time to work on exercises, extension of course length (n = 2), more originality in self-help program (n = 1).
Discussion
This is the first RCT examining the effects of preventive Internet-based problem solving therapy (PST) for adolescents with depressive and anxiety symptoms. Internet-based PST was compared to a waiting list control group and intervention effects were investigated over 4 months. Results show that depressive and anxiety symptoms declined in both groups. However, no support was found for the effects of Internet-based PST in reducing depression and anxiety in comparison to the waiting list control group, but the small sample size and high drop-out rate precluded meaningful interpretation of this finding.
Previous research into the problem-solving treatment with an adult population has found that the program obtains significant effects [13], [31]. Several explanations may be considered for not finding any additional effect of PST above usual care for our adolescent population. One reason for the lack of effects in this study is that our intervention was not implemented precisely in the way it was intended due to website problems. That is, the website content was not presented in an optimally format which could have caused difficulties for adolescents in understanding and working through the intervention. Moreover, the intervention may have lacked sufficient guidance and email support for some participants because of technical difficulties in the email support module. Six participants have encountered delayed feedback. Both suppositions are supported by satisfaction reports and may have impacted completion status. The level of completers in our study (27%) was relatively low compared to adult studies into web-based self-help [31]–[33]. Differences in depression and anxiety outcomes between completers and non-completers were not found, however.
Another reason for the non-significant findings here may be reduced appropriateness of our intervention for this age group. The self-help and individual format of our intervention requires extensive self-discipline and motivation, and this may be too informal in nature for treatment of adolescents. Also, depression impacts on individuals' motivation and their ability to concentrate in a self-directed format [10]. It is also possible that PST may not be attractive enough for adolescents, or the intervention may have been of too high difficulty or of insufficient duration for problem solving skills to become internalized to produce lasting effects.
A different possible reason for the lack of effects may be participants' severity of symptom levels at baseline. To illustrate, most adolescents reported clinically significant levels of depression or anxiety and several had encountered a mood or anxiety disorder in the past 12 months. Possibly, the PST intervention is not suitable for adolescents with more severe symptoms.
Finally, we should note that participants in the waiting list control condition may have attended other treatments during the study, possibly affecting our results. Two out of 11 waiting-list participants received other treatment. Although it is not clear to what extent this may have affected the results, our results are vulnerable to many influences considering our small sample size.
The present study has a number of significant strengths. The study is one of only a small number of randomized controlled trials to date into web-based interventions for adolescents with depression and anxiety. The intervention being examined required minimal training of therapists and was relatively cheap to administer, thereby increasing the probability that the intervention would be sustained and distributed within mental health care services. Moreover, reliable and valid measures of depressive and anxiety symptoms were used, recruitment was widespread, and diagnostic status at baseline was assessed. However, limitations should be noted. First, the sample size was very small and questions could be raised about the representativeness of the sample and the generalizability of our findings to the wider population. Specifically, immigrants were fairly underrepresented among participants, and we predominantly excluded early and middle adolescents who were reluctant to request parental consent. It is likely that participants differed in other important ways from those who did not sign up for the study. Second, in common with many other online interventions [34], we were faced with a high attrition rate, possibly due to adolescents losing interest in participating in the study. Another possible limitation is the use of a wait-list control group in that it is unclear whether the prospect of receiving an intervention within a couple of months could have influenced results. A final limitation is that demand characteristics, such as receiving a telephone call when reporting possible severe depressive or anxiety symptoms, may have affected participant responses to the questionnaires.
Overall, the results of this study do not provide support for the use and implementation of an Internet-based problem-solving therapy for adolescents with depressive and anxiety symptoms. Several explanations for this lack of effect may be provided. In general, this study may have had inadequate power to demonstrate effects and further research is needed to establish whether web-based PST will be effective in adolescents if adapted or aimed at a specific subgroup of adolescents. More research is required to ascertain what intervention content, amount, timing, and duration of delivery is most effective and for which group of adolescents [35], as well as the potential of Internet-based interventions for adolescents with internalizing symptoms.
Acknowledgments
We would like to thank Prezens/GGZinGeest for supervising participants in the PST intervention.
Author Contributions
Conceived and designed the experiments: WH JS HMK PC. Performed the experiments: WH. Analyzed the data: WH. Wrote the paper: WH JS HMK PC.
References
- 1. Cole DA, Peeke LG, Martin JM, Truglio R, Seroczynski AD (1998) A longitudinal look at the relation between depression and anxiety in children and adolescents. J Consult Clin Psychol 66: 451–460.
- 2. Hankin BL, Abramson LY, Moffitt TE, Silva PA, McGee R, et al. (1998) Development of depression from preadolescence to young adulthood: Emerging gender differences in a 10-year longitudinal study. J Abnorm Psychol 107: 128–140.
- 3. Tillett J (2005) Adolescents and informed consent: Ethical and legal issues. J Perinat Neonatal Nurs 19: 112–121.
- 4. Abeles P, Verduyn C, Robinson A, Smith P, Yule W, et al. (2009) Computerized CBT for adolescent depression (“Stressbusters”) and its initial evaluation through an extended case series. Behav Cogn Psychother 37: 151–165.
- 5. Cuijpers P (1997) Bibliotherapy in unipolar depression: A meta-analysis. J Behav Ther Exp Psychiatry 28: 139–147.
- 6. Christensen H, Griffiths KM (2002) The prevention of depression using the Internet. Med J Aus 177: S122–S125.
- 7. Calear AL, Christensen H, Mackinnon A, Griffiths KM, O'Kearney R (2009) The YouthMood Project: A cluster randomized controlled trial of an online cognitive behavioral program with adolescents. J Consult Clin Psychol 77: 1021–1032.
- 8. Cunningham MJ, Wuthrich VM, Rapee RM, Lyneham HJ, Schniering CA, et al. (2009) The Cool Teens CD-ROM for anxiety disorders in adolescents: A pilot case series. Eur Child Adolesc Psychiatry 18: 125–129.
- 9. O'Kearney R, Gibson M, Christensen H, Griffiths KM (2006) Effects of a cognitive-behavioural Internet program on depression, vulnerability to depression and stigma in adolescent males: A school-based controlled trial. Cogn Behav Ther 35: 43–54.
- 10. O'Kearney R, Kang K, Christensen H, Griffiths K (2009) A controlled trial of a school-based Internet program for reducing depressive symptoms in adolescent girls. Depress Anxiety 26: 65–72.
- 11. Van der Zanden R, Kramer J, Gerrits R, Cuijpers P (2012) Effectiveness of an online group course for depression in adolescents and young adults: A randomized trial. J Med Internet Res 14: e86.
- 12. Cuijpers P, van Straten A, Warmerdam L (2007) Problem solving therapies for depression: A meta-analysis. Eur Psychiatry 22: 9–15.
- 13. Van Straten A, Cuijpers P, Smits N (2008) Effectiveness of a web-based self-help intervention for symptoms of depression, anxiety, and stress: Randomized controlled trial. J Med Internet Res 10: e7.
- 14. Malouff JM, Thorsteinsson EB, Schutte NS (2007) The efficacy of problem solving therapy in reducing mental and physical health problems: A meta-analysis. Clin Psychol Rev 27: 46–57.
- 15. Van Lang NDJ, Ferdinand RF, Ormel J, Verhulst FC (2006) Latent class analysis of anxiety and depressive symptoms of the Youth Self-Report in a general population sample of young adolescents. Behav Res Ther 44: 849–860.
- 16. Hoek W, Aarts F, Schuurmans J, Cuijpers P (2012) Who are we missing? Non-participation in an Internet intervention trial for depression and anxiety in adolescents. Eur Child Adolesc Psychiatry in press.
- 17. Radloff LS (1977) The CES-D scale: A self-report depression scale for research in the general population. Appl Psych Meas 3: 385–401.
- 18. Zigmond AS, Snaith RP (1983) The Hospital Anxiety and Depression Scale. Acta Psychiatr Scand 67: 361–370.
- 19.
Beck AT, Steer RA, Brown GK (1996) Manual for the Beck Depression Inventory-II. San Antonio, TX: Psychological Corporation.
- 20. Hoek W, Schuurmans J, Koot HM, Cuijpers P (2009) Prevention of depression and anxiety in adolescents: A randomized controlled trial testing the efficacy and mechanisms of Internet-based self-help problem-solving therapy. Trials 10: 93.
- 21. Shaffer D, Fisher P, Lucas CP, Dulcan MK, Schwab-Stone ME (2000) NIMH Diagnostic Interview Schedule for Children Version IV (NIMH DISC-IV): Description, differences from previous versions, and reliability of some common diagnoses. J Am Acad Child Adolesc Psychiatry 39: 28–38.
- 22. Wells KB, Burnam MA, Leake B, Robins LN (1988) Agreement between face-to-face and telephone-administered version of the depression section of the NIMH diagnostic interview schedule. J Psychiatr Res 22: 207–220.
- 23. Wasserman GA, McReynolds LS, Lucas CP, Fisher P, Santos L (2002) The voice DISC-IV with incarcerated male youths: Prevalence of disorder. J Am Acad Child Adolesc Psychiatry 41: 314–321.
- 24. Sawyer MG, Pfeiffer S, Spence SH (2009) Life events, coping and depressive symptoms among young adolescents: A one-year prospective study. J Affect Disord 117: 48–54.
- 25. Garrison CZ, Addy CL, Jackson KL, McKeown RE, Waller JL (1991) The CES-D as a screen for depression and other psychiatric disorders in adolescents. J Am Acad Child Adolesc Psychiatry 30: 636–641.
- 26. Cuijpers P, Boluijt P, van Straten A (2008) Screening of depression in adolescents through the Internet: Sensitivity and specificity of two screening questionnaires. Eur Child Adolesc Psychiatry 17: 32–38.
- 27. Spinhoven P, Ormel J, Sloekers PPA, Kempen G, Speckens AEM, et al. (1997) A validation study of the Hospital Anxiety and Depression Scale (HADS) in different groups of Dutch subjects. Psychol Med 27: 363–370.
- 28. White D, Leach C, Sims R, Atkinson M, Cottrell D (1999) Validation of the Hospital Anxiety and Depression Scale for use with adolescents. Br J Psychiatry 175: 452–454.
- 29. Nguyen TD, Attkisson CC, Stegner BL (1983) Assessment of patient satisfaction: Development and refinement of a service evaluation questionnaire. Eval Program Plann 6: 299–313.
- 30. Olsson I, Mykletun A, Dahl AA (2005) The Hospital Anxiety and Depression Rating Scale: A cross-sectional study of psychometrics and case finding abilities in general practice. BMC Psychiatry 5: 46.
- 31. Warmerdam L, van Straten A, Twisk J, Riper H, Cuijpers P (2008) Internet-based treatment for adults with depressive symptoms: Randomized controlled trial. J Med Internet Res 10: e44.
- 32. Christensen H, Griffiths KM, Jorm AF (2004) Delivering interventions for depression by using the internet: Randomised controlled trial. Br Med J 328: 265–269.
- 33. Andersson G, Bergstrom J, Hollandare F, Carlbring P, Kaldo V, et al. (2005) Internet-based self-help for depression: Randomised controlled trial. Br J Psychiatry 187: 456–461.
- 34. Eysenbach G (2005) The law of attrition. J Med Internet Res 7: e11.
- 35. Spence SH, Holmes JM, March S, Lipp OV (2006) The feasibility and outcome of clinic plus Internet delivery of cognitive-behavior therapy for childhood anxiety. J Consult Clin Psychol 74: 614–621.