Patients.

The CBT for insomnia dataset consisted of two combined datasets. The first trial [14] consisted of 104 adult patients with insomnia disorder and suffering from (subclinical) symptoms of depression, randomized to a 5-session guided online treatment ‘i-Sleep’ including sleep diary monitoring (n = 52) or control group (n = 52; sleep diary monitoring control group without intervention). The second trial [15] consisted of a total of 134 adult patients with insomnia disorder recruited through general practices who were randomized to the same 5-session guided online i-Sleep treatment (n = 69) or control group (n = 65; care-as-usual control group without further interference). As both trials offered the same intervention, and were assumed to induce similar processes of change, we combined data from both trials to increase sample size; we refer to the combined dataset as the ‘CBT for insomnia dataset’. Assessments were performed online, before randomization (baseline) and 8 [15] or 9 weeks [14] after the intervention or waiting period (post-assessment). For more information regarding in- and exclusion criteria and recruitment methods see [14, 15] respectively. The combined dataset consisted of 121 participants in the intervention group, of whom 91 participants (75%) fulfilled the criteria of ‘completer’ (followed 4 out of 5 sessions; as this was considered to be an adequate dose of CBT as no new information is offered in session 5), and 77 of those 91 participants completed post-assessment. The control group consisted of 117 participants, of whom 2 participants (2%) were excluded because they reported to have received an alternative psychological treatment, and a total of 86 participants completed post-assessment.

Intervention.

The online intervention i-Sleep consisted of five sessions of CBT for insomnia [20, 21]. The five sessions focused on (1) sleep hygiene and lifestyle, (2) stimulus control and sleep restriction therapy (in which the time in bed is restricted to the average time slept in the last week (with a 5-h minimum), with the aim of increasing the time in bed spent asleep and decreasing time in bed spent awake, (3) relaxation, (4) cognitive therapy addressing dysfunctional thoughts about sleep, and (5) relapse prevention [20]. Online guidance entailed feedback on exercises, discussing sleep restriction therapy/bedtimes based on the diary patients had to keep, and motivating participants to persevere the treatment. Patients in the control condition were offered i-Sleep after completing post-assessment [14] or 6 months after inclusion [15].

Primary outcome.

Insomnia severity was measured using the insomnia severity index (ISI; [22]). The ISI consists of seven items that assess the nature, severity and the impact of insomnia symptoms (see S1 Table in S1 Appendix). Items are scored on a 5-point Likert scale, with higher scores indicating more severe insomnia symptoms. The ISI has good internal consistency (Cronbach’s α = 0.78) and is sensitive to changes in perceived sleep difficulties over time [23].

Predictors.

As possible predictors of response shift effects we included: the trial type (insomnia only trial, versus insomnia with comorbid depression trial), age, anxiety (as measured with the Anxiety subscale of the Hospital Anxiety and Depression Scale (HADS-A; seven items on a four-point Likert scale, scores ranging 0–21, higher scores indicating higher levels of anxiety; [24]), and sleep-efficiency (percentage of sleep of the total time in bed; calculated from the online patients’ sleep-diary records).

Patients.

A total of 170 cancer survivors with a need for psychological support were randomly assigned to one of the three study arms: meaning-centred group psychotherapy for cancer survivors to improve personal meaning (MCGP-CS; n = 57), supportive group psychotherapy (SGP; n = 56), or control group (n = 57; care-as-usual control group without group intervention). As our aim was to investigate the process of change induced by the primary psychological treatment as compared to the control group, we only included data from patients in the MCGP-CS and control groups. Assessments were performed before randomization (baseline) and 9 weeks after the intervention or waiting period (post-assessment); questionnaires were filled out online or via pen and paper based on patients preference. In the treatment group, a total of 45 participants (79%) fulfilled the criteria of ‘completers’ (followed at least 4 out of 8 sessions; as completion of at least 4 sessions was considered to be sufficient to transfer the essential elements of the MCGP-CS treatment), and all participants completed post-assessment. Participants in the control group did not receive alternative treatments, but 13 participants did not complete post-assessment, thus a total of 44 participants had complete data. For more information regarding the methods of this trial see [16, 18].

Intervention.

The main purpose of MCGP-CS is to sustain or enhance a sense of meaning or purpose in a patient’s life, in order to cope better with the consequences of cancer. Theoretically, enhanced meaning is considered to be the catalyst of positive psychological outcomes. MCGP-CS is a manualized eight-week intervention that makes use of didactics such as a personal workbook called ‘Life lessons portfolio’ containing homework-exercises, group discussions and experiential exercises that focus around themes related to meaning and cancer survivorship. The sessions lasted two hours each and were held weekly. The participants used the Life lessons portfolio to complete homework assignments every week. MCGP-CS was led by a psychotherapist with considerable experience in treating patients with cancer. Each session addressed a theme related to the concepts and sources of meaning, for example ‘meaning before and after cancer’, or ‘attitudinal sources of meaning: encountering life’s limitations’.

Primary outcome.

Personal meaning was assessed with the personal meaning profile-Dutch version (PMP-DV) [25]. The PMP-DV consists of 39 items scored on a 7-point Likert scale, that yield the following five subscales: relation with God/higher order (RG; 8 items), dedication to life (DL; 11 items), fairness of life (FL; 8 items), goal orientation (GO: 6 items), and relations with others (RO; 6 items) (see S4 Table in S1 Appendix for an overview of the items per subscale). The subscale scores are computed as mean item scores, with higher scores indicating more personal meaning.

Predictors.

Age and religiousness (yes/no) were included as possible predictors for the detected response shift.

Patients.

A total of 255 adult persons with diabetes with elevated depressive symptoms were randomly assigned to an internet-based CBT treatment (n = 130) or a control group (n = 125; waitlist control group). Assessments were performed online, before randomization (baseline) and 8 weeks after the intervention or waiting period (post-assessment). A total of 62 participants (48%) in the treatment group fulfilled the criteria for ‘completers’ (completed at least 5 out of 8 sessions; because all core concepts were explained in sessions 1 to 5), of whom 54 participants completed post-assessment. From the control group a total of 117 participants did not follow other psychological treatment, and 92 participants also completed post-assessment. For more information regarding the methods of this trial see [17].

Intervention.

Participants individually went through eight consecutive online lessons that provided written and spoken information and videos of diabetes patients with depression explaining how they learned from the course. Coaches (certified health psychologists) provided feedback on homework assignments. Feedback was to a large degree standardized and consisted of a concise, constructive reply on the applied CBT techniques, and meant to help patients understand and apply the CBT skills in daily practice. Participants allocated to the waiting list control group were offered the Web-based intervention if they still had elevated depressive symptoms 12 weeks after randomization.

Primary outcome.

Symptoms of depression were assessed with the centre for epidemiological studies depression scale (CES-D), a widely used 20-item self-report instrument. Respondents are asked to indicate the frequency with which they experienced depressive symptoms in the preceding week. The total score ranges from 0 to 60, with higher scores indicating more severe depression and scores of >16 representing a clinically significant level of depressive symptoms. Previous research [26, 27] has shown that the items can be grouped into four distinct aspects of depression, namely: depressed affect (DA; 7 items), well-being (WB; 4 items), somatic symptoms (SS; 7 items), and interpersonal problems (IP; 2 items) (see S5 Table in S1 Appendix for an overview of the items per subscale). For analyses in the current paper, we use these four subscale scores (calculated as mean item scores), where higher scores indicate higher levels of depressive symptoms.

Predictors.

As possible predictor of detected response shift effects we included diabetes distress at baseline as measured with the Dutch version of the problem areas in diabetes (PAID) scale, a widely used 20-item self-report questionnaire [28]. Items pertain to negative emotions related to living with diabetes, rated on a 5-point Likert scale, ranging from 0 (“not a problem”) to 4 (“a serious problem”). Sum scores are converted to a 0–100 scale, with higher scores indicating higher distress.

Objective 1: The occurrence of response shift.

In the first step of the SEM procedure, one has to establish an interpretable and well-fitting “measurement model” in which the relationships between the observed scores and the underlying latent variables are specified. For all three primary outcome variables we used a one-factor model, where the underlying target construct (i.e. insomnia severity, personal meaning, and depression) is measured by the observed item- or subscale-scores from the associated questionnaire, both at baseline and follow-up assessment, for both treatment and control groups separately (see Fig 1). The model-fit of the measurement model was evaluated using the chi-square test of exact fit, and the root mean square error of approximation (RMSEA) as a measure of approximate fit. A non-significant chi-square test indicates exact model fit. An RMSEA value above 0.10 indicates ‘poor’ fit, below 0.08 indicates ‘reasonable’ fit and one below 0.05 ‘close’ fit [30]. To inspect possible indications of model misfit we used modification indices and standardized residuals [31]. The power to detect a meaningful level of model misspecification of the measurement model was calculated based on the RMSEA [32]. The power to reject a measurement model with poor fit (i.e. RMSEA > 0.10) in favour of close fit (i.e. RMSEA < 0.05) for the three selected datasets are 1.0, 0.80 and 0.82 respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. An example of the measurement model used for response shift investigation.

Notes: The squares represent observed variables, i.e., questionnaire scores (X), measured at both baseline and follow-up assessment. The solid single-headed arrows at the bottom represent the residual factors of each observed variable. The dotted double-headed arrow represents the relations between the residual factors, where the residual factors of the same observed variable are allowed to correlate over time. The circles represent the target construct that the observed variables aim to measure (e.g., insomnia severity, depression, or personal meaning; both at baseline and follow-up assessment). Each arrow from a circle to an observed variable represents a factor loading. The double-headed arrows between the circles represent the correlations between the target construct over time.

https://doi.org/10.1371/journal.pone.0252035.g001

The second step of the SEM procedure entails the specification of a model where all measurement parameters associated with response shift (i.e., factor loadings and intercepts) are constrained to be equal across baseline and follow-up assessments. The model fit of the resulting “no response shift model” can be compared to the model fit of the measurement model, using the difference in chi-square test statistics. Significant deterioration in model fit indicates the mere presence of overall response shift (any type of response shift). We also continued our investigation of response shift when the omnibus test for response shift was not significant, in order not to miss small but meaningful effects. To illustrate, the power to detect a difference in intercept of 0.5 (corresponding to a medium Cohen’s d effect-size) and a difference in factor loading of 0.3 (corresponding to a medium Cohen’s r effect-size) is only 0.55, 0.37, and 0.46 for the omnibus test in the three datasets respectively, whereas the power to detect such effects increases to 0.94, 0.74 and 0.77 when these effects would be tested one-by-one (see Appendix for details on power calculations). We note, however, that power calculations in the context of SEM are complicated and return to this issue in the Discussion.

The third step of the SEM procedure is used to investigate which variable is affected by which type of response shift. A change in the pattern of factor loadings (i.e., when a factor loading becomes zero) is taken as evidence of reconceptualization, as this indicates that the associated observed variable is no longer relevant to the measure of the target construct. A change in the value of factor loadings is taken as evidence of reprioritization, where a lower or higher value of the factor loading at follow-up assessment indicates that the associated observed variable has become less or more important to the measure of the target construct respectively. Finally, a change in the intercepts is taken as evidence of recalibration, where lower or higher values of the intercept at follow-up indicate that the same level of the target construct is associated with lower or higher values of the associated observed variable respectively. We used an iterative procedure to indicate specific response shift, where we tested all possible model modifications (i.e., all types of response shift) in each iteration using the difference in chi-square test statistics. A significant chi-square difference test indicates that inclusion of the specific response shift effect significantly improves model fit. The response shift effect associated with the model modification that showed the largest (significant) improvement in model fit was included in the model. When different model modifications showed equivalent improvement in model fit, we used substantive considerations to choose between effects, in order to arrive at a sensible model. When response shift was evidenced, it was investigated whether response shift was present to the same extent in treatment versus control groups. The final model, that includes all possible indications of response shift, is referred to as the “response shift model”.

Objective 2: Impact of response shift on interpretation of treatment effects.

The impact of response shift on the interpretation of change can be evaluated at two levels. First, the impact of response shift on the estimated change in the variable for which response shift was detected can be calculated using a decomposition of change [33]. The decomposition entails that observed change is decomposed into change due to change in the underlying target construct and change due to response shift. Second, in the final model from step 3, the change in means of the underlying latent factors is indicative of change in the target construct. Comparison of the estimated change in the target construct from the final model with the estimated change in the no response shift model gives an indication of the impact of response shift on the overall treatment effect. For both types of impact of response shift we calculated SEM-based effect-size indices of change, where values of 0.3, 0.5 and 0.8 are indicative of ‘small’, ‘moderate’, and ‘large’ effects respectively [31].

Objective 3: The predictive role of clinical and background variables.

In the fifth step of the SEM procedure, we extended the model to include possible predictors for detected response shift. The prediction model includes direct effects of the predictive variable on the item(s) or scale(s) that showed response shift. When an effect of a predictive variable is significant, this indicates that the predictor is associated with the detected response shift. Standardized effects of predictive variables can be interpreted as correlation coefficients, where values of 0.1, 0.3 and 0.5 are indicative of ‘small’, ‘moderate’, and ‘large’ effects respectively [31].

All statistical analyses were performed using the freely available software package Lavaan [34] in R [35]. Syntaxes of all statistical analyses reported in this paper are available as online supplementary material.

Objective 1.

The multi-group longitudinal measurement model (see S1 Fig in S1 Appendix) with an added residual covariance between item 5 and item 6 showed the best fit (see Table 4). As expected, the no response shift model showed a significant deterioration in model fit as compared to the measurement model (Δχ² (24) = 45.30, p = .005), indicating the overall presence of response shift. The equality restriction on the intercept of item 2 (“Difficulty staying asleep”) was found not to be tenable (Δχ² (2) = 17.26, p < .001), indicating the presence of recalibration response shift. As expected, this recalibration response shift was only significant in the CBT group (Δχ² (1) = 14.98, p < .001) and not in the control group (Δχ² (1) = 2.28, p = .131). Inspection of parameter estimates showed that the intercept “Difficulty staying asleep” was lower at follow-up as compared to baseline (see S1 Fig in S1 Appendix). In terms of response shift, this indicates that patients in the treatment group may have changed their interpretation of the response scale on this item such that it became easier to score lower on this item after treatment as compared to before treatment, given the same insomnia severity.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Overall goodness of fit of the models in steps 1–3 of the SEM-approach for investigation of response shift (objective 1) for each of the three datasets.

https://doi.org/10.1371/journal.pone.0252035.t004

Objective 2.

The detected response shift had a moderate impact on the change in item 2, where the observed reduction in this item score was larger than what would have been expected if the item response scale was equivalent as compared to baseline (response shift effect, Cohen’s d = -0.75; see also S1 Table in S1 Appendix). The decrease in insomnia severity in the CBT group was somewhat smaller when taking into account response shift (d = -1.75) as compared to when response shift was not taken into account (d = -1.92). Hence, other than expected, taking into account response shift only slightly reduced the overall treatment effect.

Objective 3.

Anxiety and sleep-efficiency showed small but significant associations with the detected recalibration response shift (r = -.18, p = .047; r = -.22, p = .009), indicating that more anxiety and better sleep-efficiency are associated with more recalibration response shift in item 2. That is, patients who score higher on anxiety and sleep-efficiency at baseline tend to more easily indicate less problems with difficulty staying asleep at follow-up. There were no significant effects of age or the type of trial (i.e., the insomnia trial versus the insomnia and depression trial).

Objective 1.

The multi-group longitudinal measurement model (see S2 Fig in S1 Appendix) showed good fit (see Table 4). As expected, the no response shift model showed a significant deterioration in model fit as compared to the measurement model (Δχ² (16) = 35.05, p = .004), indicating the overall presence of response shift. The equality restriction on the factor loading of the subscale RG (“relations to God/higher order”) was found not to be tenable (Δχ² (2) = 13.17, p = .001), indicating the presence of reprioritization response shift. Unexpectedly, this reprioritization response shift was significant in both the MCGP-CS group (Δχ² (1) = 5.73, p = .017) as well as in the control group (Δχ² (1) = 7.54, p = .006). However, inspection of parameter estimates showed that the effect was in opposite direction. The factor loading of the subscale RG was higher after treatment in the MCGP-CS group as compared to baseline, whereas the value of the same factor loading in the control group decreased over time (see S2 Fig in S1 Appendix). In terms of response shift, this indicates that for patients in the treatment group the subscale RG became more important to the measurement of personal meaning after treatment, whereas the same subscale became less important to patients in the control group.

Objective 2.

The detected response scale had a very small impact on the estimated change in the subscale RG. The increase in this subscale was larger than what would be expected if the item was equally important as compared to baseline (response shift effect, Cohen’s d = 0.05; see also S2 Table in S1 Appendix), whereas the decrease in this subscale was smaller than expected due to a decrease in importance of the subscale in the control group (Cohen’s d = 0.06). The increase in personal meaning in the MCGP-CS group was of comparable size with (d = 0.52) and without taking response shift into account (d = 0.53). Also, the decrease in personal meaning in the control group was only marginally different with (d = -0.50) and without (d = -0.52) taking into account response shift. Thus, other than expected, the detected response shift did not impact the treatment effect.

Objective 3.

In both the treatment- and control groups there was a small but significant association of religiousness with the detected reprioritization response shift in the subscale RG (MCGP-CS group: r = .17, p = .007; control group: r = .19, p = .026;). These results indicate that being religious before the start of treatment is associated with more reprioritization response shift in the treatment group (i.e., relation with god/higher order becomes more important, and this is especially true for religious people), and with less reprioritization response shift in the control group (i.e., relation with god/higher order becomes less important, but this is especially true for non-religious people). There were no significant effects of age.

Objective 1.

The multi-group longitudinal measurement model (see S3 Fig in S1 Appendix) showed good fit (see Table 4). The no response shift model did not show a significant deterioration in model fit as compared to the measurement model (Δχ² (12) = 13.71, p = .319). Other than expected, this indicates there is no overall presence of response shift. To prevent missing small but meaningful effects, we continued the investigation for possible specific indications of response shift. The equality restriction on the factor loading of the subscale DA (“Depressive Affect”) was found not to be tenable (Δχ² (2) = 8.63, p = .013), indicating the presence of reprioritization response shift. Unexpectedly, this reprioritization response shift was significant in both the CBT group (Δχ² (1) = 4.45, p = .035), and the control group (Δχ² (1) = 4.18, p = .041). Inspection of parameter estimates showed that the factor loading of DA became smaller at follow-up, indicating that the subscale depressive affect became less important to the measurement of depression in both groups (see S3 Fig in S1 Appendix).

Objective 2.

The detected response shift had a moderate to large impact on the change in the subscale DA, where the estimated change was smaller than what would be expected if the subscale was equally important as compared to baseline (response shift effect, Cohen’s d = 0.46 (control group) and d = 0.96 (CBT group); see also S3 Table in S1 Appendix). The decrease in depression in the CBT group was only slightly larger with (d = -1.58) as compared to without taking response shift into account (d = -1.51). Also, the decrease in depression in the control group was slightly larger with (d = -0.80) as compared to without (d = -0.77) taking into account response shift. Thus, other than expected, the detected response shift did only marginally impact the treatment effect.

Objective 3.

We found no significant association between diabetes distress and the detected reprioritization response shift of the subscale DA, indicating that diabetes distress at baseline is not related to a change in the importance of the DA subscale.