The preliminary development and psychometric properties of the Psychotherapy Side Effects Scale

Abstract Background Side effects in psychotherapy are common and have a negative impact on patients or clients. However, effective evaluation tools are still lacking and have not been fully studied. The present study aims to develop a scale with good reliability and validity to measure the side effects of psychotherapy. Methods The 25 items in the Psychotherapy Side Effects Scale (PSES) were condensed and distributed to 420 subjects online to test its psychometric properties. Results The internal consistency of the PSES was satisfactory to excellent (Cronbach's ɑ coefficient was .95, and the Guttman split‐half coefficient was 0.88). A statistically significant negative correlation between the satisfaction score and the total score of the PSES was shown (r = −0.51, p < .001). The PSES could effectively discriminate between two groups with and without side effects (F = 250.95, p < .001) and was able to predict the occurrence of side effects in psychotherapy with an area under curve of 0.932 and a 95% confidence interval of 0.900–0.964 (p < .001). A cutoff was set at 36 points in total PSES score, from which the maximum Youden's index (= 0.72) could be obtained. The positive rate of the PSES was 24% (101/420). Conclusion The PSES showed good internal consistency, content validity, concurrent validity, discriminant validity and predictive validity in evaluating and identifying side effects in psychotherapy. More advanced reliability testing methods and structural validity testing for PESE need to be practiced in the future to better serve clinical practice.

dardized instruments may help to decrease potential under reporting and allow for a more accurate assessment of potential harm. Regrettably, the existing tools for assessing the side effects of psychotherapy have not reached a consensus on the most important domains, and the psychometric properties of these instruments were not good (Herzog et al., 2019).
In addition, the accurate measurement to psychotherapy side effects can reduce the therapist bias effect (Waller & Turner, 2016), which may lead the therapist to abandon effective treatments that are considered to be harmful for patients. Identifying and managing the side effects in psychotherapy is a valuable indicator for an ethical therapist, which can significantly improve the effectiveness of treatment Scott & Young, 2016;Whipple et al., 2003). In conclusion, it is necessary to develop a set of high-quality psychological scales to measure side effects in psychotherapy.
Our team took the lead in carrying out qualitative research on the side effects of psychotherapy in China. In our previous research results (Qiang et al., 2020), four kinds of first-order themes and 14 kinds of second-order themes applying to psychotherapy side effects were extracted, which provided a good basis for developing a new scale for evaluating side effects. This study posed the following research hypotheses: (1) some experiences reported by patients or clients could be used to evaluate the side effects of psychotherapy; (2) these items taken from the experiences mentioned above could be integrated to form a scale with good reliability and validity to measure the side effects of psychotherapy; and (3) the higher the satisfaction of psychotherapy, the lighter the side effects experience should be.

Scale development
In our previous qualitative research (Qiang et al., 2020), the themes applying to psychotherapy side effects were extracted clearly (Table 1).
These 14 types of side effects belong to four dimensions of "pressure from the therapeutic relationship," "emergence of new symptoms," "problem unsolved," and "pressure outside the treatment context." Among them, the "special affection in the therapeutic relationship" refers to the love of the client to the therapist, and the "maladaptive behavior" refers to these behaviors that have a negative impact on oneself or others, such as overeating and self harm. In addition, the "symptom persistence" refers to those symptoms persist in psychotherapy and are not reduced or alleviated.
Based on these themes, the Psychotherapy Side Effects Scale (PSES) was preliminarily developed. First, an item pool was built according to the second-order themes, with each secondary theme being expanded into 1-3 questions to assess the patient's experience. Twenty-eight entries were carefully generated using a description regarding psychometric habits and cultural background. The developers were major in the theoretical orientation of systemic family therapy. Second, three experts in the field of psychotherapy were invited to review the content of the item pool to improve content validity. Three entries were combined according to expert recommendations. The theoretical schools of the three experts were psychodynamics, cognitive behavioral therapy, and systemic family therapy. Finally, the first version of the 25-item TA B L E 1 Themes applying to side effects in psychotherapy (Qiang et al., 2020)  Subjects rated the negative effects of each item according to their own experience as "not at all," "mild," "moderate," "severe," and "extremely severe." Participants were asked to respond on a five-point Likert scale (1 = not at all, 2 = mild 3 = moderate, 4 = severe, 5 = extremely severe), and the instrument contained no reverse-scored items. The sum of the scores of each item constituted the total score of the PSES, which was used to evaluate the overall severity of side effects.

Side effects experience
Meanwhile, the following question was asked separately after the PSES to determine whether patients experienced side effects: "Through the overall presentation of these items, do you think that psychotherapy has caused side effects for you?". "Yes" meant that there were side effects experience, and "no" meant that there were no side effects experience. To date, there is no standardized tool for the diagnosis of side effects in psychotherapy. The measurement was used to test the predictive validity of the scale. In other words, predictive validity measures the ability of the PSES to predict the real side effects experience of the subject.

Satisfaction evaluation
Because no accepted scales of side effects in psychotherapy have been developed, patient satisfaction with psychotherapy was measured to test the concurrent validity of the PSES. Participants rated satisfaction on a scale of 1-10 according to their own experience in psychotherapy in response to the following question: "What is your satisfaction with the psychotherapy you are currently receiving?". The higher the score was, the higher satisfaction was. If satisfaction was rated as 1, it meant that the patient was very dissatisfied, and if satisfaction was rated as 10, it meant that the patient was very satisfied.

Data collection
The instrument was edited via the internet using an interface for administering surveys (Wen Juan Xing; www.wjx.cn). Demographic and psychotherapy characteristics for the sample were also released alongside the PSES. The detailed information collected is shown in

Participants
Participants were enrolled through the online PSES published on the official WeChat account from February 7, 2021 to July 1, 2021.
The inclusion criteria were that the participants (1) had received at least one session of psychotherapy in the last month, (2) had received psychotherapy from a therapist who had a licensed qualification to practice psychotherapy issued by the government, (3) were 18−65 years old, and (4) gave informed consent. Exclusion criteria included participants being (1) in acute phase of serious mental disorders (e.g., schizophrenia, bipolar disorder, etc.) or physical illness (e.g., pneumonia, heart disease, etc.); (2) ethical faults, such as the therapist's malpractice, non-therapeutic dual relationship, etc. and; (3) disagreements with the public release of the research data.

Statistical analysis
All data were integrated into one dataset. The statistical work was performed using IBM SPSS Statistics, version 22.0. Descriptive statistics were used to calculate the participants' demographic and psychotherapy characteristics and the item scores. The internal consistency reliability was tested by Cronbach's alpha (= ɑ) coefficient and Guttman's split-half test. The Pearson test was used to measure the correlation between the item score and the total score of PSES and between the satisfaction score and the total PSES score. Furthermore, itemtotal score correlations < 0.30 were considered inappropriate for the purpose of the scale. The continuous variables among groups were compared by analysis of variance. A receiver operating characteristic (ROC) curve was generated to test the predictive validity with specificity and sensitivity. Youden's index was introduced to evaluate the ability of the scale to discriminate between clients with and without side effects. The larger the index, the stronger the authenticity of discrimination. The formula for Youden's index was as follows: (specificity + sensitivity − 1). Multi-level factor analysis was performed using R 4.1.3 (Revelle, 2021). It was used to calculate McDonald's omega, which estimates general and total factor saturation (Zinbarg et al., 2005). ω t estimated the total reliability of the scale, whereas ω h and ɑ calculated the internal consistency. In this study, bilateral tests were performed, and the difference was considered statistically significant when p < .05.

Participants
A total sample of 456 participants tried to participate in the survey, and 420 participants ultimately met the inclusion and exclusion criteria. Table 2 shows the demographic and psychotherapy characteristics of the sample.

TA B L E 2
Demographic and psychotherapy characteristics of the sample (n = 420)

Variables Mean (SD) Frequency
Age (

Reliability
For all items in the scale, Cronbach's ɑ coefficient was .95, ω t was .97, and ω h was .8, which indicated great reliability. Only when Item 17 was deleted did Cronbach's ɑ coefficient slightly increase to .96 (Table 3).
The Guttman split-half coefficient was 0.88.

Content validity
The total PSES score ranged from 25 to 121, with an average of 33.15 ± 12.49 and a median of 28. The Kolmogorov-Smirnov value was 0.26, and p < .001, which indicates that the total score presented a normal distribution. The 95% percentile of the total score was 58.95, and the 25% percentile was 25. The histogram of the total score frequency is shown in Figure 1.
The correlation coefficient between items and the total score was between 0.41 and 0.84, in which the highest correlation coefficient F I G U R E 2 Receiver operating characteristic (ROC) curve for the prediction ability of Psychotherapy Side Effects Scale (PSES) for side effects was between Item 7 and the total score (r = 0.84), and the lowest was between Item 17 and the total score (r = 0.41). No correlation coefficient was < 0.30 (Table 3).

Concurrent validity
The correlation test between the satisfaction score and the total PSES score was used to assess concurrent validity by the Pearson test. The satisfaction score ranged from 1 to 10, with an average of 7.32 ± 1.99.
The correlation coefficient between the satisfaction score and the total PSES score was −0.51 with p < .001, which showed a statistically significant negative correlation.

Discriminant validity
Of 420 subjects, 39 responded to the following question with "Yes," and 381 responded with "No": "Through the overall presentation of these items, do you think that psychotherapy has caused side effects for you?". There was a significant difference in the total score of the scale between the two groups with and without side effects (p < .001) (Table 4).

Predictive validity
ROC curve analysis was performed with the total score of the PSES as the test variable and with (= 1) or without (= 0) side effects as the state variable. The results of the ROC curve demonstrated that the AUC was 0.932 with a 95% confidence interval (CI) of 0.900-0.964 (p < .001) ( Figure 2). Next, the total score was tested by ROC curve analysis to determine which cutoff for the total score was the most conducive TA B L E 3 Cronbach's α coefficient and correlation coefficient between items and total score (n = 420) to distinguishing whether there were side effects. Table 5

DISCUSSION
The current study evaluated a new self-report instrument for assessing the side effects of psychotherapy, the PSES. Items were generated from the themes of psychotherapy side effects discovered in previous research (Qiang et al., 2020) by our team and by using consensus among researchers and experiences of patients undergoing TA B L E 5 Classification accuracy of the total score of Psychotherapy Side Effects Scale (PSES) for cases with versus without side effects (n = 420) The SEPS has been applied to patients with obsessive-compulsive disorder, and its Cronbach's ɑ coefficient was .83 (Moritz et al., 2015).
In the initial attempt to use the "Psychotherapy Side Effects Questionnaire (PSEQ)," the Cronbach's ɑ coefficient was only .74 (Yao et al., 2020 (Coen, 1996). A literature review indicated that approximately 33%−73.3% of patients reported that they felt sexually attracted to therapists; 5.2%−6.5% admitted that sexual attraction often occurred (Sonne & Jochai, 2014). Interestingly, the proportion of patients expressing sexual attraction directly to the therapist is very small (1%-2%), which is far lower than that of patients' reports of attraction. Not surprisingly, when difficult topics perceived by patients were discussed or disclosed in therapy, treatment could continue and their alliance with the therapist could be strengthened. The reason for the low reliability of item 17 in the PSES may be the imprecision of the description, as subjects were not sure whether a feeling of love for the therapist would lead to bad results. In conclusion, we believe that the expression of the intention of item 17 should be clearer, but the item should be retained.
An effective psychotherapy side effect evaluation tool should be able to accurately distinguish the core characteristics of the target (Herzog et al., 2019). The core feature of the side effects in psychotherapy is inconsistency with the treatment goal and the infliction of burden or harm (Linden & Schermuly-Haupt, 2014;Parry et al., 2016), which may significantly reduce patient satisfaction or the effects of psychotherapy (Whipple et al., 2003). However, there were different views concerning the side effects of psychotherapy depending on whether they were independent of the goals of therapy or whether they were unavoidable or even necessary to achieve a therapeutic effect (Linden & Schermuly-Haupt, 2014). In any case, such side effects result in an unsatisfactory experience in psychotherapy. In the current study, there was a significant negative correlation between the results of the PSES and satisfaction, which was consistent with clinical observation and theoretical hypothesis.
To date, no measurement tools concerning psychotherapy side effects have been tested for predictive validity (Herzog et al., 2019).
The most likely reason for this lack is that there are no standardized tools or gold standards for identifying side effects in psychotherapy. In our further analysis, the results indicated that the PSES could discriminate and predict side effects well based on the subjective experience of participants. The therapist is the "producer" of psychotherapy and is therefore responsible for all side effects, which may result in a perceptional bias toward positive rather than negative effects (Hatfield et al., 2010). Therefore, the actual side effects should be determined by clients rather than therapists. Meanwhile, it should be noted that the PPV of the PSES was high, while the NPV was low (0.347). Our current findings suggest that although the PSES, a self-report scale, could measure the side effects experienced by patients in psychotherapy, it may be more suitable for the screening stage or preliminary assessment.
According to a National Audit of Psychological Therapies (NAPT) launched in England and Wales, 5.2% of patients reported long-lasting negative effects by psychological treatment .
The incidence of side effects in psychotherapy was 31.1% (115/370) in a sample of Chinese patients from an online survey (Yao et al., 2020). Among the young patients who had received psychotherapy, the incidence of side effects was about 41% (Lorenz, 2021). In a recent study, the negative effects of psychotherapy were assessed by the INEP