Staging and profiling for schizophrenia spectrum disorders: Inter-rater reliability after a short training course

OBJECTIVE
Clinical staging and profiling have been proposed as a new approach in order to refine the diagnostic assessment of schizophrenia spectrum disorders. However, only limited evidence is available for the inter-rater reliability of the clinical staging and profiling model. The aim of the present study was therefore to determine the inter-rater reliability of the clinical staging and profiling model for schizophrenia spectrum disorders, and to investigate whether a short course can improve inter-rater reliability.


METHODS
Consecutively recruited inpatients with schizophrenia spectrum disorders were included between January 2015 and January 2016 (study 1), and between March 2018 and October 2018 (study 2). By contrast with the assessors in study 1, all the assessors in study 2 were trained in clinical staging and profiling. We used the clinical staging model proposed by McGorry and identified profile characteristics. Inter-rater reliability was measured using the Intraclass Correlation Coefficient (ICC).


RESULTS
The ICC score for clinical staging in study 1 was moderate (0.578). It improved considerably in study 2 (0.757). In general, the ICC scores for the profile characteristics in studies 1 and 2 ranged from poor to sufficient (0.123-0.781).


CONCLUSION
This study demonstrated that inter-rater reliability in clinical staging was sufficient after training. However, inter-rater reliability for clinical profile characteristics was highly variable. The general implementation of the clinical staging model for schizophrenia spectrum disorders is therefore feasible but clinical profile characteristics should be used with caution.


Introduction
Schizophrenia spectrum disorders are heterogeneous and complex mental disorders (Marwaha and Johnson, 2004) (Kooyman et al., 2007). This heterogeneity is seen at the symptom level, and in a large range of social consequences, and aetiological or prognostic factors (Owen et al., 2016) (Kahn et al., 2015). Recurrence rates of psychosis following the first episode are up to 90% after discontinuation of antipsychotic drugs, compared to 27% who remained on medication (Leucht et al., 2012;Emsley et al., 2013). Nevertheless, 13.5% of patients diagnosed with schizophrenia achieve complete recovery and do not require any further treatment (Jaaskelainen et al., 2013).
The principles of clinical staging and profiling are applied to map this heterogeneity and refine the diagnosis of schizophrenia spectrum disorders (Wigman et al., 2013). Clinical staging provides information about disease progression, recurrence and remission status (Godin et al., 2019). The objective is to relate diagnosis more precisely to treatment options and therefore to help prevent progression towards the advanced stages of disease (McGorry et al., 2010).
Staging could deliver a range of benefits in comparison with our current diagnostic system. For instance, it provides a different perspective on the course of illness, and information about where patients are situated on the continuum of psychosis extending across different classifications (McGorry et al., 2010;McGorry, 2007;McGorry and Mei, 2018). Furthermore, staging provides a dynamic framework for targeted intervention and the prevention of over-or under-treatment with, for example, antipsychotic medication. Indeed, ultra-high-risk patients benefit substantially from staged treatment. However, further evaluation of the prognostic and therapeutic validity of this potential clinical A second way of describing the heterogeneity of schizophrenia spectrum disorders is clinical profiling with profile characteristics. Clinical profile characteristics are state and trait characteristics that clinicians can easily identify and apply in daily practice. They are used to reflect the large inter-individual variation in a particular diagnostic class and to personalise the diagnosis (Wigman et al., 2013) and they are established on the basis of a large body of systematic reviews and studies investigating prognostic factors in schizophrenia spectrum disorders. For instance, factors such as compliance, social support from relatives, or psychiatric or somatic comorbidity have a significant impact on the course of disease (Sprah et al., 2017) (Rabinovitch et al., 2009).
Profiling may be useful since it may provide an overview of specific problem areas that require swift attention from practitioners. Furthermore, the profile characteristics used in profiling are also targets for psychiatric or social interventions such as learning social skills, health education or medication adherence interventions (Abdel Aziz et al., 2016;Dodell-Feder et al., 2015). Seen from this perspective, the clinical profile characteristics for which specific interventions are currently available point to the multi-dimensionality of need for care in patients with psychotic disorders in terms of both actual presence and life-time prevalence (Wigman et al., 2013).
However, the instruments used in studies are not always easily applicable in clinical practice: they can be time-consuming or difficult to apply in acute situations. Combining both staging and clinical profiling could result in a comprehensive framework for personalising diagnosis and targeting treatment with the ultimate goal of improving outcomes in schizophrenia spectrum disorders (Wigman et al., 2013).
Until now, evidence concerning the psychometric properties of the clinical staging and profiling model has been scarce. One study has demonstrated acceptable inter-rater reliability (κ 0.71) for the staging model in adolescents with subclinical psychotic symptoms and the authors reported evidence that the staging model is useful in practice for the selection of early interventions (Hickie et al., 2013). However, we are not aware of any studies that discuss the inter-rater reliability of clinical staging or profiling in acute or chronic episodes of the disease. It is therefore uncertain whether staging and profiling can be reliably applied in the regular clinical diagnosis and treatment of schizophrenia spectrum disorders. Despite the fact that stage-dependent treatment is not yet available for schizophrenia, the evaluation of reliability is important to ensure valid ratings and provide an accurate model for further research. We therefore consider the assessment of inter-rater reliability to be essential before broad application in clinical practice can be advised.
Our aim in the present study is therefore to determine the inter-rater reliability of both clinical staging and profiling for schizophrenia spectrum disorders. In addition, we look at whether a short training course could improve the inter-rater reliability of the clinical staging and profiling model.

Study design
The design of the present study was cross-sectional. The study is divided into two sub-studies: study 1 and study 2. The current study was part of a larger research project looking at clinical staging and profiling in schizophrenia spectrum disorders. The data from study 1 were used in an earlier study to examine the construct validity of the clinical staging model (Berendsen et al., 2018).

Samples
We included acute patients admitted consecutively to the acute ward of a general psychiatric hospital in Amsterdam, the Netherlands, between May 2015 and January 2016 (study 1), and between March 2018 and November 2018 (study 2). Our psychiatric hospital caters to a catchment area in Amsterdam. Admissions are primarily for acute hazardous situations arising from a psychiatric illness. Approximately 80% of the admissions are involuntary. The acute wards of the hospital have capacity for 90 inpatients. The medical staff consists of four psychiatrists and ten residents. Inclusion criteria for the present study were (1) age over 18 and < 65 years, (2) fulfilment of the DSM-5 criteria for a schizophrenia spectrum disorder.

Study 1
The inter-rater reliability of the clinical staging and profiling model was determined by comparing the assessment of the physician who had actually seen the patient with the assessment of an independent researcher (S.B.) based on a review of all the available medical records before admission. Inter-rater reliability here therefore means agreement between a physician and an independent researcher. This procedure was adopted on the basis of earlier studies of inter-rater reliability for medical diagnosis in different patient samples (Varmdal et al., 2015) (Chung et al., 2010) (Kang et al., 2013).
The physician, often a resident, who had actually seen the patient made the primary assessment of the clinical stages and profile characteristics directly after the admission of the patient to the acute ward. In study 1, this physician was briefly informed by the psychiatrists present at the daily general staff meeting about staging and profiling. Accordingly, the physician was informed that different stages can be distinguished in the course of psychosis, and that profile characteristics are important patient features that may affect outcome of the disease. After this brief introduction, the physicians observed other physicians applying the staging and profiling model. The physicians did not receive any specific training in clinical staging and profiling.
The procedure was as follows: the patient was presented by the physician who had seen the patient in the daily staff meeting with all the psychiatrists and residents in the department. This presentation included the evaluation of the patient's current psychiatric status. The physician was able to draw on all the available information from the medical records covering the psychiatric and somatic history, social status and other relevant information. During this meeting the clinical stage and profile characteristics were proposed by the physician and discussed with the staff until consensus was reached. This staging and profiling by the physician was compared with an independent assessment performed by one researcher (S.B.) with clinical experience in staging and profiling. The researcher was blind to the first assessment of staging and profiling by the physician. This assessment was based on a review of all the information available in the medical records at the time of admission. The medical record included the diagnostic interview at the time of admission and medical or legal documents drafted before admission.

Study 2
Before starting the second study, we organised a short and practical training course for all the physicians concerned. Two interactive presentations of clinical staging and profiling lasting approximately thirty minutes were given to small groups of two or three physicians. The presentation included general information about staging and profiling, and the physicians were given the opportunity to raise questions. In addition, we demonstrated how to determine the clinical stage and clinical profile characteristics, and discussed several illustrative vignettes. After practising with the vignettes, we gave the physicians direct individual feedback about their performance. Moreover, we emphasised the importance of completing the diagnostic interview in the records given the fact that the independent researcher (S.B.) would be using these documents for the second assessment of the clinical stages and profile characteristics. In addition, the independent researcher was present once every two weeks at the clinical staff meeting where clinical stages and profile characteristics were determined in order to provide feedback to the physicians during the inclusion period of study 2. Patients discussed during staff meetings attended by the independent researcher (S.B.) were not included in the present study. With the exception of the training course and the presence of the independent researcher once every two weeks at the staff meeting, the procedure for determining inter-rater reliability was the same as in study 1. Table 1 shows the clinical staging model. It is based on the criteria proposed by McGorry et al. 2007. Patients assigned to stages, 0, 1A and 1B (increased risk of psychosis or mild non-psychotic symptoms) were not admitted to our acute ward. Furthermore, we interpreted stage 3B as recurrent psychosis with one or more episodes and symptomatic remission between episodes since there is ample evidence that symptomatic remission after multiple episodes is possible (Albus, 2012;Wiersma et al., 1998).

Application of clinical stages and profile characteristics
The clinical profile characteristics we used were patient features selected on the basis of a review of the literature on prognostic factors in schizophrenia: we performed a thorough PUBMED search for reviews and prospective studies with the following terms: schizophrenia, schizoaffective disorder, schizophrenia spectrum disorders, clinical predictors, relapse, non-adherence and clinical decline (Rabinovitch et al., 2009) (Weiden, 2004). On the basis of this review, we identified four important domains: psycho-social factors, comorbidity, judicial aspects and psychiatric condition. We then determined the profile characteristics associated with these domains that were significantly linked in the scientific literature with an increase in psychotic symptoms, recurrence or an elevated risk of psychiatric re-hospitalisation. Not all profile characteristics from the scientific literature were applicable in an acute psychiatric ward and so we presented the profile characteristics to the research group 'Severe Psychotic Disorders of the Academic Medical Centre of the University of Amsterdam and Arkin', which consists of clinical psychiatrists and psychologists who provided input based on their clinical experience. By combining scientific evidence with the input from these psychiatrists and psychologists, we aimed to develop a list of clinically relevant profile characteristics, applicable in daily practice. The final set of relevant profile characteristics for our study was adopted after discussion by the experts present. The profile characteristics adopted for the psycho-social factors theme were: living situation, work and daily activities, support from close relatives; for the comorbidity theme: psychiatric comorbidity, addiction, personality features, physical health problems; for judicial aspects: judicial history, aggressive behaviour; for psychiatric condition: trauma, number of episodes last year, premorbid functioning, disease insight, compliance. All the profile characteristics were rated using a three-point severity scale -(0) minimal or no problems, (1) moderate problems, (2) severe problems -and scored using the standard classification form shown in Table 1.

Statistical analysis
Mann-Whitney U or Chi-Square tests were used to test for differences in diagnosis, gender and age between studies 1 and 2. We decided to use the Intraclass Correlation Coefficient (ICC, 2-way, mixed-effects model with absolute agreement) with a 95% confidence interval to rate agreement between the physicians and the researcher because our primary end point was an ordinal variable. Another advantage of the ICC by comparison with, for instance, Cohen's kappa, is that the ICC circumvents the Cohen's kappa paradox (Zec et al., 2017). Furthermore, recent literature has proposed stricter cut-off values for the interpretation of ICC values to minimise measurement errors and ensure that results are more reliable. We therefore adopted the following ICC cutoff values: ICC < 0.5 reflects poor agreement, ICC 0.5-0.75 reflects moderate agreement and ICC > 0.75 reflects good agreement (Koo and Li, 2016). We used Student t-tests to determine whether ICC scores differed significantly between studies 1 and 2. We did not correct for multiple testing.

Demographic and clinical characteristics
A total of 214 of all the patients admitted (N = 943) to the acute ward during the study period met the inclusion criteria. The main reasons for exclusion were age (above 65 or under 18), no DSM-5 diagnosis of schizophrenia spectrum disorder or no assessment of the clinical stage or profile characteristics at admission. The patients discussed during the staff meeting attended by the independent researcher (S.B.) were also excluded (N = 32). Table 2 lists the demographic and clinical characteristics of participants included in the two studies. Study 1 included 114 participants and study 2 included 100 participants. No significant differences in gender, age or diagnosis were found between the studies. As shown in Table 2, we included patients with schizophrenia spectrum disorders. The DSM-5 diagnoses for the included patients were: schizophrenia (51.40%), unspecified schizophrenia spectrum disorder (29.91%), schizoaffective disorder (13.55%), otherwise specified schizophrenia spectrum disorder (2.34%), psychotic disorder induced by substances (0.93%), and delusional disorder (0.93%), postpartum psychosis (0.47%) and brief psychotic disorder (0.47%). Table 3 also shows the  distribution of participants across clinical stages.

3.2.
Inter-rater reliability study 1 Table 4 and Fig. 1 lists the inter-rater reliability scores from study 1. The ICC score for the clinical stages was moderate (0.578). ICC scores were high (0.753-0.781) for the profile characteristics addiction and personality disorders. ICC scores were moderate (0.575-0.654) for the profile characteristics judicial history, trauma, living situation and somatic histry. ICC scores were poor (0.123-0.491) for other profile characteristics. Table 4 and Fig. 1 lists the inter-rater reliability scores from study 2. The ICC score for the clinical stages was in study 2 significantly higher (0.757) than the ICC clinical stage scores in study 1 (0.578). Other ICC scores did not significantly improve. The ICC scores were moderate (0.536-0.720) for the profile characteristics judicial and somatic history, aggression, living situation, addiction and personality disorders. ICC scores were poor (0.258-0.484) for other profile characteristics.

Discussion
The aim of the present study was to determine the inter-rater reliability of clinical staging and profiling in schizophrenia spectrum disorders in day-to-day clinical practice. We also looked at whether a short training course could improve reliability scores. Generally, reliability is found to increase after training procedures. However, this could not be taken for granted because an earlier study found no improvement in IRR after training of assessors (Demitrack et al., 1998). Previous studies have found the assessment of earlier clinical stages with mild subclinical psychotic symptoms to be sufficiently reliable (Hickie et al., 2013).
Our results demonstrated that observed inter-rater reliability for clinical staging in day-to-day practice could easily be improved to a significantly higher level by a relatively short training course for clinicians. Our study also demonstrated that assessment in more advanced stages of schizophrenia spectrum disorders is sufficiently reliable. All in all, we conclude that, after a short training course, the reliable implementation of the clinical staging model is achievable in practice.
The staging model may help practitioners, patients and their families in regular care since it provides a framework that allows patients to move between stages depending on their clinical status and to receive individualized treatment, with the benefits such treatment entails. Another potential benefit of this framework is the possibility to inform patients and their relatives concerning predictive factors.
On the other hand, the reliability of clinical profiling based on the assessment of profile characteristics proved to be highly variable, even after training. Inter-rater reliability was sufficient for the profile characteristics personality disorders and addiction. It may be considered remarkable that the presence and severity of such a complex but clinically relevant diagnosis as personality disorder can be made quite reliably in an acute setting. Nevertheless, this finding is in line with previous reliability research that also found high reliability scores for personality disorders (Garcia et al., 2018) (Lobbestael et al., 2011) . In the case of substance abuse disorders, comparable findings were reported after both short or more comprehensive assessment procedures (Hildebrand, 2015) (Lopez-Pelayo et al., 2015. Accordingly, our results imply that clinical profile characteristics with adequate inter-rater reliability scores can be reliable instruments in clinical practice. The case is slightly different for profile characteristics with moderate reliability scores such as living situation and trauma. Here, we recommend the use of additional criteria or more intensive training before application in clinical practice. At the other end of the scale, inter-rater reliability scores were low for profile characteristics such as premorbid level of functioning and support from relatives. This finding contrasts with the high inter-rater reliability scores for structured instruments used to assess social support from relatives or premorbid functioning (Rabinowitz et al., 2007) (Wongpakaran et al., 2011). Our findings suggest that questions may be asked about the reliable assessment of these profile characteristics in a brief procedure in an acute ward with inherent time constraints. However, it is also possible that the limited range of scoring options (three) may have led to measurement errors that preclude reliable application in acute treatment settings.
A thorough search of the relevant literature failed to identify any earlier studies of the psychometric properties of clinical profiling. We were therefore unable to compare our results with other studies of clinical profiling in schizophrenia spectrum disorders. However, in general, adequate inter-rater reliability scores based on clinical symptoms have proven difficult to attain given the absence of a clear 'golden standard'. For instance, we found comparable insufficient inter-rater reliability scores in the assessment of catatonia or the diagnosis of schizoaffective disorders or eating disorders (Santelmann et al., 2016) Berendsen, et al. Progress in Neuropsychopharmacology & Biological Psychiatry 99 (2020) 109856 (Sarkar et al., 2016) (Thomas et al., 2015). Furthermore, inter-rater reliability is also quite variable in other medical fields (such as the assessment of atopic dermatitis) (Zhao et al., 2015). Even so, these assessments are regularly used in practice.
Another important question that remains is how clinical profiling could improve treatment and recovery for patients with schizophrenia. Specific interventions are available for several profile characteristics, examples being the treatment of addiction, or interventions for reducing aggression or improving social skills (Dodell-Feder et al., 2015;Tenegra and Leebold, 2016;Darmedru et al., 2017). Clinical profiling could prompt health-care professionals to evaluate comorbidity and to intervene accordingly (Hunt et al., 2018). Furthermore, profiling may facilitate relatively easy transference of important clinical information between professionals, patients and their relatives which can be addressed in staff meetings or family gatherings. This approach represents a new perspective on diagnosis and treatment but the benefits still require further research to establish prognostic and therapeutic validity.
Our study has several limitations. Firstly, our results relate only to the clinical stages of schizophrenia spectrum disorders and not to the preclinical stages of disease since these patients were not admitted to our hospital. Secondly, although all the assessors received similar training, we did not monitor compliance with adherence or the quality of individual assessors before starting the assessments. Variations in the competence levels of the assessors may have affected the observed inter-rater reliability scores. On the other hand, variations in competence reflect real-world practice and our results could therefore be more generalisable to other regular psychiatric care settings. Thirdly, the low inter-rater reliability scores observed for the clinical profile characteristics could be attributable to the fact that medical records frequently contained incomplete information. Consequently, assessors were obliged to select one of the options on the basis of possibly inadequate data, and this may have produced variations in scores. This limitation has also been seen previously in reliability research, suggesting that differences between assessments are an artefact caused by the quality of medical records (McGorry, 2007). Indeed, incomplete medical records are a threat to the quality and continuity of care since many treatment decisions are based in part on information obtained from medical records. Incomplete records of staging or clinical profile characteristics are therefore a problem as such and improvements are required in this area. Fourthly, our results provide information about agreement between two assessors. However, an evaluation of inter-rater reliability concerning multiple assessors using identical information would enhance the generalisability of our results. Fifth, we did not correct for multiple testing. Therefore our results should be interpreted with caution, considering an increased risk for false-positive findings.
To the best of our knowledge, we are the first to report on inter-rater reliability in the clinical staging and profiling of schizophrenia spectrum disorders in regular care. The main strength of our study is that it is based on day-to-day clinical practice, with all the limitations inherent in diagnosis and treatment on an acute clinical ward. In addition, we explored the added value of a short and feasible training procedure. The practical training of assessors was not time-consuming and it is easily reproducible and so this procedure can be implemented with little effort in other acute psychiatric hospitals. Another advantage of our study is the relatively large number of consecutively recruited inpatients.

Conclusion
Our study demonstrated that adequate inter-rater reliability can be achieved when using the clinical staging model for schizophrenia spectrum disorders. However, the inter-rater reliability scores for a range of clinical profile characteristics were variable. The general implementation of clinical staging for schizophrenia spectrum disorders in regular practice is therefore warranted after a relatively short course of training. By contrast, a more cautious approach is advisable to the broad application of clinical profile characteristics in practice. We believe that an evaluation is needed of the effect of adding distinct criteria to clinical profile characteristics, more intensive training for assessors or the use of structured instruments.

Funding
No external funding or financial resources have been used for this project.

Ethic approval
The Dutch Central Medical Ethical Committee has ruled that Dutch Law regarding research with humans does not apply to the collection of anonymized information and, consequently, analyzing anonymized data for the present study does not require additional informed consent from participants.

Declaration of Competing Interest
All authors declare not to have any conflicts of interest that might be interpreted as influencing the content of the manuscript. and efforts made this project possible.