Validation of a Self-Assessment Scale for Therapists’ Perception of Cognitive Behavioural Therapy Skills in China: A Mixed Method

Cognitive behavioural therapy (CBT) is a first-line psychotherapy in Western countries, yet there have been limited studies examined the required skills based on perception of CBT therapists in the context of Western countries, and there has been no such research in China. Currently, there is no exist of a selfassessment scale to evaluate the required competence perceived by CBT therapists in China. This study used both qualitative and quantitative method to develop a self-assessment tool by CBT therapist to provide their perception about required skills in the treatment of mental disorders in China. Qualitative research used semistructured behavioural event interviews to identify the essential competences perceived by CBT therapists in China. Based on the qualitative research results, quantitative research was further used to develop a self-rated instrument with 52 questions. Three-hundred therapists who were conducting CBT therapy for clients with mental disorders were invited to participate in the study, 278 of which answered the survey. Among 25 therapists in the qualitative research, three key themes relating to the success of treatment using CBT were extracted from the interview. These themes including generic psychotherapy competencies, common CBT theory, mastering basic CBT techniques are essential skills in the effective application of CBT treatment. Another two themes on challenges and difficulties in the application of CBT include formulating clients’ problems and plans and using CBT techniques properly. Among 52 questions, 44 questions were analysed, which identified four CBT competence related factors: 1. generic psychotherapy competencies; 2. mastering CBT theory and skills; 3. using CBT techniques properly; 4. formulating clients’ problems and plans. The reliability using Cronbach’s α and half-reliability coefficient analysis showed that the therapist competence self-assessment scale had a high level of reliability of 0.94. The content validity, convergent validity and discriminant validity were high, and the structure of the instrument was reasonable. The self-assessment scale may provide a reliable and valid tool to self-assess the required competence and skills perceived by CBT therapists in China, and provide useful information for CBT training curriculum and program in China.


Introduction
There is substantial evidence indicating that cognitive behavioural therapy (CBT) is an effective therapy in the treatment of depression, anxiety [1], personality disorder [2], and obsessive compulsive disorder [3]. In the area of mental disorders treatment, there is compelling evidence suggesting the benefits 2 Qualitative Research

Participants
Thirty-seven therapists who mainly used CBT to treat clients in their daily work were invited by email to participate in this study. Once they agreed to participate in the study, they were asked to provide consent and subsequently to fill in a questionnaire to provide information to assess whether they met the inclusion criteria. They need to meet the following selection criteria: (1) psychology or psychiatry degree; (2) attended CBT junior training level more than 100 hours; (3) At least three years treatment experience with four hours treatment per week and 10 cases; (4) Have attended supervision for at least three cases and had been supervised for over 50 hours.
Ethics approval for the present study was obtained through the Human Research Ethics Committee of Beijing Anding Hospital, Capital Medical University. The participants were informed that their involvement was completely voluntary, that the information they provided would be kept confidential, and that they could withdraw from the interview at any time if they wished to do so.

Critical Cases Provided by Interviewees
In the interview, all the therapists were asked to narrate three successful cases and three failed cases that they had completed, and describe the therapeutic process involved in the mentioned cases. They can describe cases which they dealt with frequently in practice. Success was defined as "at least 30% symptom reduction at the end of therapy in subjective impression". Failure was defined as "less than 30% symptom reduction at the end of therapy in subjective impression". Tab. 2 presents the diagnosis type of critical cases provided by all therapists. Note: PTSD = post-traumatic stress disorder.

Behavioural Event Interview
All participants were asked to complete a semi-structured interview (See Appendix 1) comprised of behavioural event interviews (BEI). BEIs are open-ended and semi-structured interviews originally developed by McClelland [11], a professor of psychology at Harvard University. The BEI technique is a valid content assessment method that obtains a sample of individuals' actual behaviours as they perform a given task. In the classic BEI format, the manner that therapists deal with cases during the therapeutic process and the way they act when they encounter obstacles reflect their competence as a therapist.
During the interview process, interviewees were free to discuss their feelings, thoughts and selfreflections. This method gradually reveals a specific pattern of competence. The BEI included asking the interviewees to describe events in which they felt effective in their job, and events in which they felt ineffective. The interviewees described their behaviours, thoughts and feelings in detail when discussing a number of critical incidents related to job competence (the questions we asked the interviewees was attached as appendix). Then these events were systematically coded for various traits, attitudes, knowledge, skills and ability. The open interview format of the BEI gives the researcher multiple opportunities to discover and code new indicators of competence. In this study, each interviewee was allowed to casually narrate six cases involving three successes and three failures. Questions were asked after the participants described each successful and failed case, to investigate the key facilitator and barrier of treatment, participant's feelings about the "success and failure" of the treatment, their aspirations and insights, and anything the participant believed he or she could have done to improve the treatment.
All the interviews were finished by one PhD student in clinical psychology. She received one interview training and carried out two experimental interviews prior to the formal interview. The interviews were audio-recorded using a Sony Audio Recorder ICD-TX650. The recordings were transcribed, coded and analysed later.

Thematic Analysis
We used grounded theory analysis to guide our data analysis [11,12]. The grounded theory method has been used by many previous psychology researchers to help develop theories and explore subjective experience [13,14]. Data were analysed via an inductive process and coded in qualitative analysis software package (Leximancer). Such theories can be developed via a systematic interview as opposed to developing and testing hypotheses.
The interviews were transcribed by the first author and a member of the research team using the Transcription software of IFLYTEK Co., Ltd., (https://www.iflyrec.com/). The Accuracy and truthfulness are over 95%. The data were then checked again by two senior Master degree students. Leximancer version 4.5 [15] was used to extract themes using thematic analysis method. Leximancer is an advanced computer program that conducts content analysis using a machine learning technique. The main CBT related concepts in a text and how they relate to each other were analysed. A thematic analysis and a relational analysis of the interview data for qualitative study was then conducted to extract themes based on these relational concepts relating to CBT competencies and skills. The concepts and themes are extracted based on word frequency counts co-occurrence counts of concepts present in the transcripts of the narrative interviews.

Qualitative Results
The interviewees' characteristics are presented in Tab. 1. All participants were psychiatrists in general hospitals or mental health hospitals, teachers in university psychology departments or full-time members of a university psychological counselling centre. There were 11 interviewees were junior CBT therapists with less than three years treatment experience, and 14 interviewees were senior CBT therapists with more than 5 fives year treatment experience. As shown in Tab. 2, the cases provided by the CBT therapists mainly included neurosis disorders and mood and anxiety disorders. All the borderline personality disorders cases had failed, and the theraHpists believed that CBT was not effective for the treatment of personality disorders. Three of the therapists only provided four cases due to their busy workload. All cases were diagnosed by psychiatrists prior to the beginning of CBT, according to the criteria in the ICD-10 (International Statistical Classification of Diseases and Related Health Problems, Version 10) [16].
Three main themes were inductively derived from qualitative analysis of open-ended responses. The themes had a significant degree of overlap in terms of respondents' perceptions of competence and skills. The derived themes are as follows.
Theme 1: Generic psychotherapy competencies Generic psychotherapy competencies refer to basic knowledge of psychological treatment, building treatment alliance, positive regarding, keeping empathy and complying with ethic. It is essential to engage in psychotherapy and reflects the professionalism and universality of psychological treatment.
A therapist stated that 'therapists need to have empathy to clients, and have flexibility in dealing with different clients' treatment needs'. Another therapist highlighted that basic psychotherapy competencies will help with treatment, stating 'how to achieve listening and how to empathise and ask questions, how can we really show respect and concern for others'. Finally, another respondent stated that the reasoning, logical thinking, drawing conclusion, establishing relationship, and expression ability, all need to be improved'.
Theme 2: Common CBT theory Conceptual mapping showed that understanding common CBT theory was the dominant concept (118 ranked counts) and therapists recognised that understanding CBT was core for their professional training, providing treatment to clients and solving treatment problems. Many of the therapists, in particular, male doctors, medical degree students and master degree students stated that understanding different CBT models was the main skill on which they should focus, and that it was part of their professional training. They stated that understanding behaviour was involved in role plays stimulating patient interaction. A therapist stated that clinicians use CBT theory to understand their clients from different angles and different contexts.
Theme 3: Mastering basic CBT techniques In addition to CBT theory, basic skills were discussed. A participant identified that specific forms of competence and skills were required for effective treatment, stating that 'specific skills, attributes will be necessary to work in a fast-paced environment with lots of clients-therapists contact'. Consultants with medical degree strongly emphasized the importance to develop professional skills in consultation process, and develop experience in using CBT techniques.
Several CBT techniques were identified as important skills. These included (1) Case formulation. Case formulation is a hypothesis of the causes, precipitants and maintaining factors of a person's psychological, interpersonal and behavioural problems. The case formulation helps the therapist to organise information about a patient, particularly when that information contains contradictions or inconsistencies in behaviour, emotion and thought. There are two types of case formulations: longitudinal and horizontal formulation. As stated by one participant: "As a therapist, we sort out clients' whole situation, including the analysis of his cross-section symptoms. Specially, behaviour function analysis includes micro-analysis and macro-analysis in a case. Then we discuss it with the clients and amend it in this way". Case conceptualisation is a guide for intervention. It is most useful when viewed as a dynamic, iterative process that invites frequent revisiting of hypotheses as new client data becomes available. (2) Cognitive reconstruction. Cognitive reconstruction was identified as an important process of correcting irrational cognition by using certain cognitive techniques that eventually improve emotion and behaviour in the patient. On participant stated "Through a horizontal formulation, I found out her automatic thoughts. The next step was to use the narrow down technique to find her intermediate beliefs. I found her core beliefs finally. And then I used some cognitive techniques, such as pros and cons analysis, pie chart analysis and so on, in order to help her to correct the irrational core beliefs and adjust her distorted cognition". (3) Behaviour modification. Behaviour modification was stated as a group of techniques that improve emotions and adjust cognition through changing behaviours. This often includes relaxation, exposure and desensitisation. "Therapists design some behavioural experiments with clients to change his irrational cognition. They sometimes design some daily activity sheets to improve his mood". (4) Arrange and review homework. All the CBT therapists we interviewed (25 therapists) stated that they assign appropriate homework for their clients at the end of each treatment session. Before the subsequent session, therapists reviewed the homework and attempt to resolve any problems that emerge. "In the first three sessions, psychological education for anxiety or depression was completed, and the cognitive model and cognitive distortion were introduced. Homework was assigned to record his thoughts in daily life." It was accepted by all participants that homework can strengthen the effect of treatment and the efficacy of techniques the clients may learn during CBT. In addition, to some extent, the completion of homework can reflect the compliance of the clients to the treatment, and the stability of the therapeutic alliance. (5) Summary and feedback. Some CBT therapists (11/25) require their clients to make various reports during the treatment course. This involves spending 5-10 minutes to summarize the contents of their discussion and guiding the patient in giving feedback on their treatment. "We found some problems in the process of treatment through reflection or summary. We paid more attention to it in another case in the future". (6) Prevention of recurrence. The patient's condition may fluctuate after a course of treatment is completed. Many therapists reported that they used coping card to respond to the recurrence. "After the treatment, the patient's condition may be repeated. I will provide the solution to the problems that the patient cannot handle after the treatment is completed in the last two sessions. The most commonly used ones are the coping cards".

Themes on challenges and difficulties in application of CBT
Another two themes were extracted from the negative comments about the application of CBT skills in the treatment when therapists encountered difficulties and failures.
Theme 1: Formulating clients' problems and plans Understanding clients is crucial to achieve the treatment success. "It needs time to understand clients' thoughts and emotions". Therapists try their best to focus on symptoms or their behaviours. "It is difficult to really help patients if we do not understand them or to understand them from our own point of view". In some circumstances, the implementation of CBT technique was not meeting the clients' expectations as most clients expected the therapists could solve their problems immediately. So "the failure is clients are not satisfied with the treatment outcome or they have very high expectations about the treatment". It is worth mentioning that a good relationship is the foundation of understanding clients' problems and establishing plans. Theme 2: Using CBT techniques properly Using CBT techniques properly is a kind of meta-competences, which is generally considered a comprehensive set of skills that facilitate adaptation and flexibility in the therapist. For example, "For some cases, we did not use the CBT technique flexibly to adapt to the changes of clients' personality and thinking". This is an advanced ability, yet qualified therapists should at least grasp the appropriate "rhythm" of treatment; they must apply therapeutic techniques in an appropriate manner and deftly overcome any obstacles that may arise during treatment.

Design of Questionnaire
Based on the five themes of the qualitative interview results, 52 questions were designed. The questions were rated according to a 5-point Likert scale ranging from 'not at all' to 'absolutely matched'. 1. Generic psychotherapy competences. This measure has the following domains: (1) alliance building. (

Participants
The subjects of this study were all therapists who were participating in a CBT training courses in China, including all junior CBT therapists from Psychiatric hospitals, psychological therapists, general hospital psychologists, and community hospital mental health workers. The therapists could be recruited to the study if they (1) had received at least 30 credit hours for all courses for initial training in cognitive activities; (2) had a college degree or above, or major in psychiatry, nursing (psychiatry) or psychology; (3) participated in case supervision more than 30 hours (including group supervision and personal supervision); (4) had a secondary or tertiary psychological consultant certificate. A total of 300 questionnaires were distributed; 278 valid questionnaires were received, eventually yielding a high response rate of 92.7%.
All participants provided consent to participate in the study; all participants filled in the paper-based survey and returned the survey to the chief investigator of the study. The study obtained approval through the hospital research ethics committee's assessment. Based on the competence model of cognitive behavioural therapists in China, a self-assessment scale on competence characteristics of Chinese cognitive behavioural therapists was preliminarily prepared, and 52 items were selected and modified to be suitable to clients in the Chinese context (see Supplemental Tab. 1).
The structure of the scale was explored by exploratory factor analysis, and the reliability of the scale was tested. Then, confirmatory factor analysis (CFA) was used to verify the construct validity of the therapist competence self-assessment scale following three steps: •Step 1. The CBT therapist competence self-assessment scale was designed according to the Chinese competence model of cognitive behavioural therapists. There were 52 original items. According to the suggestions of an expert review panel and project group members, 52 items were included in the assessment measure. A five-level rating was used, with 1 = not at all, 2 = mostly not matched, 3 = not sure, 4 = mostly matched, and 5 = absolutely matched.
•Step 2. The data of 134 questionnaires were randomly selected from the total 278 questionnaires. Exploratory factor analysis was conducted to identify the dimensions of the therapist scale and confirm the final scale items. Items with correlation coefficients for the total scores less than 0.5, critical values that were not significant and t-values less than 3 were excluded.
•Step 3. The data of 144 questionnaires were analysed using CFA to assess the scale structure, and whether the newly identified structure is optimal to assess therapists' competence and provides a good model fit.

Statistical Analysis
SPSS 25.0 was used for exploratory factor analysis, t-tests and correlation analysis, and Amos 7.0 version was used for CFA.
CFA was used to examine the model fit for the three-, four-and five-factor structures. The structure of the scale derived in the exploratory factor analysis was examined for model fit by using the comparative fit index (CFI), tucker fit index (TFI) and root mean squared error of approximation (RMSEA). The cut of score for CFI, TFI is around 0.90, and RMSEA less than 0.08 are considered as excellent model fit [18]. The factor structure of this scale was tested using the Amos restrictive factor analysis approach with maximum likelihood estimations. The goodness-of-fit indicators are summarized and the model derived in this study was compared among the three-factor, four-factor and five-factor structures of the scale.

Participant Demographics
The mean age for the 278 participants was 36.97 years (SD = 8.77); 71 (25.6%) were male, and 206 (74.4%) were female. Regarding educational background, 140 (51.3%) therapists had a bachelor's degree or below, while 133 (48.7%) had a master's or doctorate. There were 115 (44.7%) therapists working in a hospital setting, while 142 (55.3%) worked in clinics or the community. There were 133 (55.2%) therapists with CBT treatment experience of less than 1 year, and 108 (44.8%) therapists with CBT treatment experience of more than 1 year.

Exploratory Factor Analysis
Principal component analysis was used to extract the dimensions of the instrument. According to the results of the analysis, items with a factor loading score of less than 0.4 were removed from the project, and items where the average score of a single item tended to an extreme value were excluded because of a lack of discrimination. Exploratory factor analysis was performed on the remaining items by using SPSS 25.0. The Kaiser-Meyer-Olkin (KMO) value was 0.91 and the approximated chi-square value of the Barrlett's sphere test was 8970.531, the Barrlett's sphere test df was 2850, and the p-value was less than 0.001, indicating that there were many common factors suitable for exploratory factor analysis. indicating that there were common factors among the matrix's correlation matrix to be identified and the exploratory factor analysis was useful to identify these common factors.
In the factor analysis process, the principal component analysis method was adopted, and varimax rotation was used. The extracted eigenvalues were greater than 1, and the number of extractions of each factor was not limited. According to the factor analysis results, the items with factor loading less than 0.4 were removed until the cumulative rate of variation became stable.
Tab. 3 presents the results of exploratory factor analysis of the instrument. In total, 44 items and four dimensions were extracted from the exploratory factor analysis. Seventeen items (17,22,23,26, 30, 31, 32, 33, 34, 36, 37, 39, 40, 41, 44, 45 and 50) were loaded into ability to master CBT theory and skills; For example, questions were related to CBT theory and skills such as recording automatic thinking, using cognitive triangle to do horizontal case analysis; help clients to perform cognitive reconstruction; master theory of CBT for depression and anxiety. 15 items (1, 2, 6, 13, 15, 16, 20, 21, 25, 27, 46, 47, 48, 49 and 51) were loaded into generic psychotherapy competencies. For example, establish a good relationship with the clients; accept and embrace the issue of clients; communicate with the clients naturally. Seven items (3,5,9,12,14,18 and 38) were loaded into using CBT techniques properly. For example, questions were related to issues of flexibility of treatment, speed up or slow down the treatment pace according to the need of clients. Finally, five items (4, 10, 19, 29 and 42) were loaded formulating clients' problems and plans. For example, questions were related to identify the type of cognitive distortions of clients, identify the automatic thoughts of clients, form a treatment plan for CBT based on the issues of clients. Eight items (7,8,11,24,28,35,43 and 52) out of 52 questions with low factor loadings of less than 0.40 were deleted from the final analysis. The four dimensions explained 55.58% of the total variance, with each factor representing 38.61%, 8.41%, 4.81% and 3.75%, respectively.

Reliability (Internal Consistency)
The reliability level for the total scale was at a high level, with a total Cronbach's α of 0.94 for the tatal scale, and 0.94, and 0.92, 0.86 and 0.87, respectively, for the four factors (see Tab. 4).

Confirmatory Factor Analysis
The four-factor model showed the good fit to the data, Chi square is 1.845, CFI = 0.844, TLI = 0.845, RMSEA = 0.065, compared with the three-factor and five-factor structures. The four-factor structure was found to be a good model fit in terms of chi-square, CFI and RMSEA results, compared with the other two structures (see Tab. 5 and Fig. 1). Compared with 3 factors and 5 factors model, 4 factors model has the best model fit as it has the least level of error rate by RMSEA, smaller chi square, biggest fit indices as they were indicated by CFI and TLI, and smaller differences as indicated by AIC and BIC values.

Discriminant Validity of Therapist Competence Test
Tab. 6 shows that there were significant differences in non-technical components of CBT between male and female participants. There were significant differences in generic psychotherapy competencies and using CBT techniques properly between age categories. Basic competencies of CBT were significantly different between occupations. Ability to use CBT common theories and techniques, general ability of psychotherapy and total scores were significantly different between positions. There were significant differences in the four factors and total scores between participants with different years of working experience. There were also significant differences in each factor and total scores between treatment times per week (see Tab. 6).

Discussion
The Qualitative study identified the CBT components including common CBT theory, generic psychotherapy competencies, basic CBT techniques, using CBT techniques properly, and formulating clients' problems and plans are important for the training of CBT therapists. The Quantitative research confirms the structure of competence model derived from the qualitative study and exhibited an excellent level of convergent, construct and content validity and reliability of the competence survey with the four-factor structure. The competence instrument showed a high level of discriminant validity to differentiate therapist competence according to age, gender, working experience and profession. Valid instruments with proven reliability, capable of consistent measurements, are becoming increasingly important for CBT therapists who seek to understand therapists' perceptions about CBT competence.
In addition to testing the reliability and validity of this instrument, we gained insight into CBT therapists' perceptions of CBT competence for their current career. Current skills were loaded into four factors: generic psychotherapy competencies, mastering CBT theory and skills, using CBT techniques properly, formulating clients' problems and plans. The same to the requirements of the UK CBT framework, and Europe's and America's professional training programs [8][9][10] are the first three themes. The difference is that the last one becomes an important theme for CBT therapists in China.
To date, therapist competence has no uniform scale, and because of this, studies examining the relation between competence and treatment outcomes have yielded mixed findings [19][20][21][22]. The cognitive behavioural competence self-assessment scale developed in this study aims to allow Chinese therapists to gain an accurate evaluation of their competence, to understand the current lack of expertise and to specify directions for future career planning.
The largest proportion of variance is explained by the first dimension; therefore, the ability to have CBT theory and skills is most important to therapists. This is consistent with the UK CBT competence framework in component 2, basic cognitive and behavioural therapy, and component 3, specific cognitive and behavioural therapy technique-the core technical interventions employed in most forms of CBT. Qualitative study also showed that common CBT theory and CBT technique are core components of their professional training.
All the therapists emphasized the importance of general psychotherapy skills through training, supervision and treatment of cases to be better prepared to conduct CBT in clients. This is consistent with the content of 'Generic Competences-Used in all Psychological Therapies' in the UK CBT competence framework [8]. Qualitative results also reflect the important of basic training, which can help therapists acquire basic and important skills relating to listening, asking questions, showing respect to others, reasoning, logical thinking and drawing conclusions.
Using CBT techniques properly is an important aspect of the structure of our scale. Flexibility reflects therapists' level of techniques and clients may prefer natural, flexible treatment manners. Similarly, "use therapeutic techniques in an appropriate manner" is also mentioned in Pilling's and Miller's model [8]. To every therapist, it is a challenge in the application of CBT.
What is different from other competence framework, formulating clients' problems and plans become an independent part of the competences structure. Maybe it is because the development of cognitive behavioral therapy in China is still insufficient. This means the ability of formulation will be overemphasized in China, which will be much helpful to lay a good foundation for the development of CBT at this stage.

Limitations
CBT is still at the developing stage in China. There are a limited number of therapists practising CBT in China. Therefore, the sample size of therapists engaged in this study is small. In this study, 278 questionnaires were collected through meetings, training and networking. The 278 questionnaires were randomly divided into two parts, one was used for exploratory factor analysis and the other for CFA to cross-validate the structure of the scale. This might have further limited the power of the statistical analysis as it has been suggested in the previous study [23]. The next stage of the study is to increase the number of therapists and cross-validate the measurements to confirm the reliability, validity and structure of the instrument. The effect of psychotherapy is related to their cultural background, however, how CBT should be adapted in Chinese context and what skills are required for the therapists are not researched in this study. Another limitation is the instrument is self-reported in nature that may have not provided objective results on the required CBT skills. Further limitation is that no comparison to formally assessed competence.
Future work might compare self-rated results to ratings of competence by other experts [24].
In conclusion, the therapist competence self-assessment identified the structure of the final model with four dimensions: mastering CBT theory and skills, generic psychotherapy competencies, using CBT techniques properly, formulating clients' problems and plans. The reliability using Cronbach's α and halfreliability coefficient analysis showed that the therapist competence self-assessment scale had a high level of reliability. The content validity, convergent validity and discriminant validity were high, and the structure of the instrument was reasonable. The therapist competence self-assessment scale may provide a reliable and valid tool to self-assess the competence characteristics of cognitive behavioural therapists in China.
Disclosure Statement: None of the authors has any conflicting interest to declare.