Development and Validation of the Cognitive Behavioral Therapy Skills Scale Among Japanese College Students

Background: There are many different skill components used in cognitive-behavioral therapy (CBT). However, there is currently no comprehensive way of measuring these skills in patients. Assessing pretreatment CBT skills will contribute to prediction of treatment responses in the context of CBT for depression. Methods: We developed the CBT Skills Scale from ve pre-existing instruments measuring major CBT components: self-monitoring, behavioral activation, cognitive restructuring, assertiveness training, and problem-solving. University students (N = 847) who participated in a fully factorial randomized controlled trial of smartphone CBT were assessed with the CBT Skills Scale, the Patient Health Questionnaire-9 (PHQ-9), the Generalized Anxiety Disorder 7 (GAD-7), and the short form of the Japanese Big Five Scale. Structural validity was estimated with exploratory factor analysis (EFA) and conrmatory factor analysis (CFA), and internal consistency evaluated with Cronbach’s α coecients. Construct validity was evaluated with the correlations between each factor of the CBT Skills Scale, the PHQ-9, the GAD-7, and the Big Five Scale. Results: The EFA supported a ve-factor solution based on the original instruments assessing each CBT skill component. The CFA showed sucient goodness-of-t indices for the ve-factor structure. The Cronbach’s α of each factor was 0.75-0.81. Each CBT skills factor was specically correlated to the PHQ-9, GAD-7, and the Big Five Scale. Conclusions: The CBT Skills Scale has a stable structural validity and internal consistency with a ve-factor solution and appropriate content validity concerning the relationship with depression, anxiety, and personality. between CBT skills possessed by patients beforehand and the specic interventions to be selected for optimal treatment. Therefore, this study aims to develop a psychometrically sound, comprehensive, yet brief measure of ve main CBT skills based on the existing questionnaires developed specically for each skill.


Background
Packaged cognitive-behavioral therapy (CBT) is an effective treatment for depression and its acute phase response rate has previously been estimated at 44% [1], which is as effective as second-generation antidepressant medication. However, more individualized approaches are needed to continually improve upon response rates. CBT is a multi-component treatment that employs skills focused on cognition and behavior. Flexible treatment that selects customized individual skill components help to build optimal treatment. The components provided in a typical CBT package for depression include self-monitoring (SM), behavioral activation (BA), and cognitive restructuring (CR) [2]. In some cases, assertiveness training (AT) is added to those packages [2]. For some packages, BA and problem-solving therapy (PS) are implemented alone [3], [4]. These ve components are the major intervention techniques of CBT for depression.
Selecting components, or combinations of components, is done in accordance with patient characteristics and relies on clinical judgement. However, the empirical base consists of little evidence guiding the component selection.
There are two possible directions for optimizing CBT on an empirical basis. The rst direction is to derive skills, or combination of skills, that are effective for the entire population of depressed patients. Such optimal combination is yet to be discovered [5][6] [7]. Towards this solution, several research groups are currently conducting fully factorial randomized controlled trials of CBT optimization [8] [9].
Second, individualized or precision CBT interventions tailored to individual characteristics are expected to improve the remission rates [10] [11]. This approach examines the interaction between individual patient characteristics and each CBT component. Among the patient characteristics needed for individualized interventions, the patient's a nity of each CBT skill can be a clinically meaningful predictor of outcomes or treatment effect modi ers. For example, the CBT skills a patient has before or after treatment predicts relapse [12] [13]. This nding suggests that a strengths-based approach of drawing on patients' pretreatment CBT skills may improve outcomes.
Thus, assessing a patient's CBT skills before the intervention may help to predict outcomes and to select appropriate intervention components. Scales measuring each of the patient's CBT skills, such as such as SM [14], BA [15], CR [16], AT [17], and PS [18], have been developed separately and their psychometric properties have been validated. However, each of these scales contain 8 to 36 items and completing them may take up to 20-30 minutes. It is impossible to administer all of them, in addition to other important questionnaires, in the routine clinical care. We need a concise and comprehensive measure of multicomponent skills to reveal the complex interactions between CBT skills possessed by patients beforehand and the speci c interventions to be selected for optimal treatment. Therefore, this study aims to develop a psychometrically sound, comprehensive, yet brief measure of ve main CBT skills based on the existing questionnaires developed speci cally for each skill.

Participants
Participants were 847 junior college, undergraduate, and graduate students enrolled in a college or university in Kyoto or Nagoya, Japan, and participated in the Healthy Campus Trial (HCT) between September 2018 and May 2020. The HCT is a fully factorial randomized controlled trial to optimize smartphone cognitive behavioral therapy developed for mental health promotion and depression prevention in healthy college students. Details of the clinical trial are described in the protocol paper [9].

Cognitive and behavioral skills
We used the following established questionnaires to integrate the ve constructs of cognitive or behavioral skills. We have obtained written permission from the developers of all the original scales to use parts of their items in the CBT Skills Scale.
Self-monitoring (SM): From the Cognitive Behavioral Self-Monitoring 17-item scale originally developed by Tsuchida et al [14], we used the Cognitive Monitoring subscale for measuring SM skills. This scale contains questions such as, "I don't bother to think about how my actions relate to my feelings" and "Sometimes I don't understand how my actions and feelings are related to each other." Each item was rated on a four-point Likert scale from 0 = very untrue of me to 3 = very true of me, with a total score between 0 and 15. Cronbach's α reported in a previous study of Japanese university students was 0.77, indicating su cient internal consistency [14].
Cognitive restructuring (CR): The six highest loading items from the Competencies of the Cognitive Therapy Scale developed by Strunk et al [16] were used for measuring CR skills. The questions include, "When I became distressed because of a negative thought or feeling, I come up with a speci c plan of action for what I could do to deal with it," and "When something upset me, I paid attention to what I was thinking so I could have a more balanced view." Items are rated from 0 = not very true about me, to 3 = very true about me, with a total score between 0 and 18. We con rmed the semantic equivalence of the Japanese and English versions by translation and back translation.
Behavioral activation (BA): We used the ve-item Behavioral Activation Subscale of Behavioral Activation for Depression Scale -Short Form (BADS-SF) developed by Manos et al [15] for measuring BA skills. BADS-SF was translated to Japanese from the English version and validated in the Japanese population [19]. The scale includes items such as, "I'm satis ed with the amount and type of things I've done." Items are rated from 0 = not very true in my mind to 3 = very true in my mind. Cronbach's α reported in a previous study of Japanese university students was 0.71, indicating su cient internal consistency [19].
Assertiveness training (AT): We used the seven-item Self-Assertion subscale of the Adult Social Skills Scale developed by Aikawa et al [17] for measuring AT skills. The scale includes items such as, "I complain clearly when I'm made uncomfortable." Items are rated between 0 = not very true about oneself to 3 = very true about oneself. Cronbach's α reported in a previous study of Japanese university students was 0.73, indicating su cient internal consistency [17].
Problem-solving (PS): The six highest loading items of the Approach Avoidance Style subscale of the Problem-Solving Inventory [18] were used for measuring PS skills. A Japanese version of the inventory was translated and back translated. The scale includes items such as, "When making a decision, I weigh the consequences of each alternative and compare them against each other." Items are rated from 0 = very untrue of me to 3 = very true of me, with a total score ranging from 9 to 18.

Mood and personality
Depression: The Patient Health Questionnaire-9 (PHQ-9) [20] was used for measuring depression. The PHQ-9 uses 9 items of diagnostic criteria for major depressive episode (MDE) in the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV), and is rated from 0 = not at all to 3 = nearly every day. The Japanese version of the PHQ-9 has been shown to be valid and reliable in primary care patients [21] and the university students [22].
Anxiety: The Generalized Anxiety Disorder-7 (GAD-7) [23] was used for measuring anxiety. The GAD-7 is an established measure including 7 items of anxiety, worry, and hypersensitiveness. The scale is rated from 0 = not at all to 3 = nearly every day. The Japanese version of the PHQ-9 has been shown to be valid and reliable in primary care patients with anxiety disorder or major depressive disorder [24].
Personality: For investigating the correlation between the CBT skills and personality traits, we used the short form of the Big Five Scale of Personality Traits (Big Five Scale), which is well established in reliability and validity, and is frequently used in Japan [25] [26]. The Big Five Scale is rated from 0 = untrue for me to 4 = true for me on each of the ve personality trait factors: Neuroticism, Extraversion, Openness, Conscientiousness, and Agreeableness.

Data collection
Data were collected as an assessment before participants were enrolled in the HCT and assigned to each intervention group, and all participants completed all questionnaires on the trial registration website and the smartphone CBT app on their smartphones. Of these, those participants who gave consent for the use of the data were used for the current analysis. The HCT is being conducted with the approval of the Ethics Committee of Kyoto University School of Medicine.
We analyzed the cross-sectional data of baseline instruments of the trial in this investigation.

Statistical analysis
Data analysis was conducted with SPSS 25 (IBM Corp., Armonk, NY, USA) for investigating construct validity and internal consistency. Amos 22 (IBM SPSS Statistics, Chicago, IL, USA) was used for con rmatory factor analysis.

Structural validity
We rst conducted an exploratory factor analysis (EFA) using maximum likelihood and Promax rotation to identify the factor structure of the ve CBT skill scales collectively. Next, a con rmatory factor analysis (CFA) was performed based on the factor loadings of each factor to con rm the goodness-of-t for the factor structure. Chi-squared (CMIN), goodness-of-t index (GFI), adjusted goodness-of-t index (AGFI), comparative t index (CFI), and root mean square error of approximation (RMSEA) were calculated. The criterion for a good model t was set to CMIN/df ≤ 2, GFI ≥ 0.95, AGFI ≥ 0.90, CFI ≥ 0.97, and RMSEA ≤ 0.05. An acceptable model t was set to CMIN/df ≤ 3, GFI ≥ 0.90, AGFI ≥ 0.85, CFI ≥ 0.95, and RMSEA ≤ 0.08 [27].

Reliability
Cronbach's α coe cients were calculated to test the internal consistency reliability of each of the ve CBT skills scales. The α value considered to be of su cient internal consistency ranges from 0.7 to 0.8. [28] Construct validity Pearson's correlation coe cients were calculated to analyze the correlation between the ve CBT skill scales with the Big Five Scale, PHQ9, and GAD7 with symptoms of depression, anxiety, and speci c personality traits.

Structural validity
On the CBT Skills Scale, ve factors were extracted according to the EFA ( Table 1)

Construct validity
The correlation coe cients for each CBT skill with the ve-factor personality, depression, and anxiety were calculated for construct validity and are shown in Table 2. Depression symptoms were negatively correlated with BA and SM and less strongly so with CR and AT. Anxiety symptoms were negatively correlated with SM and less strongly with BA, CR and AT.
With regard to the Big Five personality Traits, (1) Extraversion was weakly and positively correlated with AT and BA, (2) Conscientiousness was weakly positively correlated with BA, CR, and PS, (3) Openness was weakly and positively correlated with AT, BA, CR, and PS, and (4) Agreeableness was weakly and positively correlated with CT and PS, and (5) neuroticism was weakly negatively correlated with AT, BA, and SM.

Discussion
The purpose of this study was to develop a brief and comprehensive measure of the ve major CBT skills and to establish the validity and reliability of a measure of said skills. To investigate this, a factor structure was validated by EFA and CFA. Then, internal consistency was veri ed by calculating the Cronbach's α. Moreover, construct validity was tested by investigating the correlation between each CBT skill and the potential outcomes of depression, anxiety, and personality traits.
The present ndings contribute to the literature by developing a useful instrument that is necessary to determine the yet to be discovered optimal CBT components and combinations [5], [6], [7]. The results of the EFA showed that SM, BA, CR, AT, and PS could indeed be classi ed as distinct factors representing separate CBT skills. Additionally, the CFA con rmed that the factor structure of the CBT Skills Scale is appropriate. Furthermore, Cronbach's α for each skill showed acceptable internal consistency. Each of the ve CBT skills (SM, BA, CR, AT, and PS) could be considered separately to measure speci c skills in each intervention factor. These ndings will allow researchers to validate the matching of skills with each intervention in clinical trials using multiple components of CBT. Such studies would then contribute to the optimization of interventions in clinical practice according to patient characteristics.
The present ndings advance existing research in individualized treatment [10] [11] by demonstrating that individual CBT skills can be differentially associated with varied potential mental health outcomes. We identi ed signi cant and differential associations between each CBT skill and potential outcomes of depression, anxiety, and personality traits. In the present ndings, we found that the higher the SM, BA, and AT, the lower the depression symptoms. Additionally, we found that the higher the SM, the lower the anxiety. In the treatment of both depression and anxiety disorders, SM-based assessments are one of a key component of CBT and SM itself promoted symptom reduction by objectively observing the problem related emotions, cognitions, and behaviors [29]. Because BA and AT are activities that directly antagonize depressive behaviors [30], it appears that a particularly relevant association with depression was found in this cross-sectional investigation.
Concerning each skill and the personality traits measured by the Big Five Scale, speci c correlations were found between each trait and each skill. Higher openness tended to be associated with higher levels of all skills except SM, which suggests that being open to new experiences makes it easier to activate a wide variety of CBT skills. Higher extroversion was associated with higher skills in BA and AT, which suggests that these skills are more likely to be honed by relationships with others. Higher conscientiousness was associated with higher levels of BA, CR, and PS, which are skills commonly assigned as homework of continuous habits in CBT. This may mean that those low in conscientiousness or low in pretreatment BA, CR and PS, may struggle in treatment adherence. Those who were more agreeable tended to have higher CR and PS skills, which may indicate that they are better at the demonstrating empathy. On the other hand, neuroticism tended to have lower skills in SM, BA, and AT, which may indicate that they may have more skills to develop as targets for CBT. These ndings are consistent with the results of a meta-analysis that examined the relationship between resilience, which addressed in each component of CBT, and the Big Five personality traits [31]. Neuroticism was negatively correlated with resilience and other personality traits were positively correlated with resilience.
The optimization of CBT components and combination their combinations is important to reduce depressive symptoms and prevent major depressive episodes. The CBT Skills Scale developed in this study will enable researcher and potentially clinicians to predict the effect of each intervention element or combination of interventions and to explore effect modi ers. It has the potential to help quantitatively re ne and individualize CBT and provide optimal interventions.

Limitations
The participants of the current study were college students below the depression threshold, which may differ from those with clinical depression. However, for those with higher levels of depression, the CBT skills and outcome associations may be even stronger. Future studies should be conducted in a variety of patient groups with differing characteristics and symptom severity. Second, the present study only looked at the association between CBT skills and crosssectional depression and anxiety, and the association with their longitudinal changes is not clear. The association will be investigated after the trial is completed.

Conclusions
The CBT Skills Scale showed stable structural validity, su cient internal consistency, and su cient reliability. Importantly, we demonstrated that there is a vefactor model of CBT skills aligned with the ve most commonly used CBT intervention components. The Scale also demonstrated an interpretable relationship with depression, anxiety, and the speci c personality traits. The CBT Skills Scale could be used as a potential predictor and effect modi er in studying the optimization of CBT interventions. The CBT Skills Scale may then help with clinical decision-making by conceptualizing patient strengths associated with component selection in relation to differential outcomes.

Consent for publication
Not applicable.

Availability of data and materials
Data sharing is not applicable as the scope of data use is limited to informed consent to participants.