Risk or cumulative incidence of psychosis in CHR samples

As in any preventive approach, the main aim of an indicated prevention is a reduction in the overall morbidity or cumulative incidence of the disease.10 In case of the early detection of psychosis, the main outcome is therefore defined by the transition risk to frank psychosis from a CHR state.11 Thereby, cumulative incidence is usually defined as the probability that a particular event, such as occurrence of a particular disease, has occurred within a given period.12 Synonyms of cumulative incidence include incidence proportion and risk or probability of developing psychosis. It is usually calculated by the following proportion, assuming that all patients with CHR have been followed-up:

Such probabilities are commonly defined as the likelihood of an event with values ranging from 0 or 0% (no expected occurrence of the event in any case) to 1 or 100% (certain of the occurrence of the event in all cases). However, as the risk of psychosis onset in patients with CHR is the probability that psychosis will occur within a defined period of time and, consequently, heavily depends on the observation time, risk of psychosis in patients with CHR always has to be referred to in terms of a specified follow-up period.

Example A: No consideration of potential dropouts

Person-time or incidence rate of psychosis in CHR

Other than risk or cumulative incidence of psychosis that assumes that the outcome of all patients with CHR at the end of the follow-up period is known, the incidence rate or person-time rate is a measure that incorporates the true observation time directly into the denominator. Thus, it can account for the unfortunately likely persons who dropout during longer observation periods. As for the proportion used to compute the cumulative incidence, the numerator of the person-time rate is the number of transitions to psychosis during the follow-up time. However, the denominator is not the baseline number of patients with CHR but the total of the observation time (commonly given in years) of each patients with CHR (see example B below).

Example B. Consideration of potential dropouts

Thus, the 3-year person time or incidence rate of psychosis with consideration of individual observation times (=3×0.13=0.39) is considerably higher than the 3-year cumulative incidence of Example A (=0.29). This leads to the question, which measure (person-time rate vs cumulative incidence) should followed in clinical practice. Since the cumulative incidence ignores patients lost to follow-up, and conservatively assumes that they remained disease-free over the years, it might actually underestimate the true probability of psychosis transition in CHR samples. Indeed, there is evidence that about 23% of patients with CHR who disengage from CHR services (drop-outs) will later develop psychosis.13 Thus, in practice, the incidence rate relating to person-years should be preferred over the more conservative cumulative incidence or risk.

Kaplan-Meier estimates of psychosis risk in patients with CHR

An even more accurate time-to-event estimate of risk of psychosis, which also deals with incomplete observations is provided by Kaplan-Maier (survival) curves.14 A Kaplan-Maier curve is defined as the probability of surviving in a given length of time, thereby considering time in many small intervals.15 A previous meta-analysis provided a summary Kaplan-Maier estimate of psychosis risk in CHR samples (mainly by UHR criteria), and indicated that most transitions occurred within the first 2 years.5 Owing to this, it may be clinically relevant to know the exact risk of psychosis onset in patients with CHR, given psychosis will occur during this timeframe (this will apply to most transitions although later transitions are also possible, in particular in CHR samples defined by BS criteria).7

Time to transition, given transition will occur in the first 2 years, is depicted in figure 1. Such a figure—or rather such a set of figures, considering the impact of type of risk criteria, age, referral source and possibly other patient characteristics—might be of clinical interest because they might be practically used by CHR clinicians to optimise the timing of their prognostic assessments and treatments accordingly (eg, more frequent CHR reassessments in the first 2 years to not miss the chance to adapt treatment to possible symptom exacerbations16).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Cumulative chance of transition given transition occurs by 2 years. As many studies assessed transition at 2 years of follow-up,5 we plotted cumulative estimate of transition, given transition occurs by 2 years after presentation. The regression line (solid line) closely fitted the data points (R2=0.995; y=133 (1-e−0.06x)) and indicates that, among ultra-high risk individuals progressing to psychosis in the first 2 years, 25% will develop the disorder by 106 days and 50% by 240 days.

Measures of prognostic accuracy in CHR samples

Next to these risk estimates relating predominately to CHR populations, risk estimates that rather relate to CHR assessments, that is, a test's accuracy measures, are often used in diagnostic decision-making.17 These include a range of measures derived from a 2×2 contingency table, cross-classifying the binary result of a CHR assessment and the binary result of the follow-up gold standard assessment of psychosis (table 1), that will be discussed in the following.

Sensitivity and specificity of CHR assessment

In simple terms, the sensitivity of the CHR assessment is defined as the proportion of patients developing psychosis who have a positive CHR assessment result (CHR+). The specificity of a test is the proportion of patients not developing psychosis who have a negative CHR assessment result (CHR−).

Example C: Sensitivity and specificity of a CHR assessment

A perfect assessment would have 100% sensitivity and 100% specificity, yet in the likely absence of such perfect values, no clear guidelines to weigh sensitivity against specificity exist but only rules-of-thumb.18 For example, Sackett19 suggested to use the test with the highest sensitivity for ‘ruling out’ and that with the highest specificity for ‘ruling in’ the disease, irrespective of the number of false-positive or false-negative classifications. In contrast, balancing severity of the disorder against risk associated with treatment, McNeil et al20 suggested to use highly sensitive tests, even if of little specificity, when the disorder is severe but its treatment benign and safe; and highly specific tests, even if of little sensitivity, when treatments carry the risk of severe side-effects and long-term consequences.

Furthermore, clinical utility of sensitivity and specificity is limited by the fact that they cannot be directly used to estimate the individual probability of developing psychosis in a patient undergoing CHR assessment. Indeed, high sensitivity and specificity do not ensure that the test will be clinically useful, nor do low sensitivity and specificity render a test useless.21 Sensitivity and specificity may also be instable and affected by factors such as sampling bias, small samples, bias in test scores or errors in selection and use of tests.21

The sensitivity and specificity of CHR assessments have recently been estimated at meta-analytical level. The prognostic sensitivity for psychosis prediction at 38 months was 96% (95%–CI 92% to 98%), while the prognostic specificity for psychosis prediction at 38 months was 47% (95%–CI 38% to 57%).22 These values indicate that CHR assessments have an outstanding ability to rule out psychosis risk and an only modest ability to rule in subsequent psychosis.22

Sensitivity and specificity in ROC curves

Sensitivity and specificity at different data points (eg, sum scores) of continuous data can be plotted in the ROC curves. ROC curves plot (1−specificity) of a test on the x-axis against its sensitivity on the y-axis. The area under this curve (AUC) represents the overall accuracy of a test, with values of 0.9–1.0 (equal to 90–100%) considered outstanding, of 0.8–0.9 excellent and of 0.7–0.8 acceptable.23 The off-diagonal on the graph represents an AUC of 0.5, indicating only random discrimination, that is, a test whose results are no better than tossing a coin. ROC curve for CHR assessment was reported as very good (AUC=0.90; 95% CI 0.87 to 0.93).22

Positive and negative predictive values

Accuracy measures that—other than sensitivity and specificity—are not considered independent of the risk of psychosis in the population to that a particular patient with CHR belongs are the positive and negative predictive values (PPV, NPV).24 The PPV is the proportion of CHR+ patients who actually have developed psychosis (T), whereas NPV is the proportion of CHR− patients who have not developed psychosis (NT).25 The PPV of CHR assessment at different time points has been reported in an earlier meta-analysis, indicating a risk of psychosis onset among CHR+ patients between 18% (95%–CI 12% to 25%) at 6 months, and 36% (95%–CI 24% to 35%) at more than 36 months.5 Coming back to Example C, a CHR test with 93.5% sensitivity and 40.3% specificity at 3 years, the corresponding risk of psychosis, PPV and NPV are:

• Risk of psychosis the population assessed=(29+2)×100/150=20.7%

• PPV=CHR+T×100/(CHR+T+CHR+NT)=29×100/(29+71)=29.0%

• NPV=CHR−NT×100/(CHR−NT+CHR−T)=48×100/(48+2)=96.0%

Example D: Influence of the population's risk of psychosis on PPV and NPV

Although PPV and NPV may be used to estimate the individual probabilities of developing psychosis in CHR patients,26 their clinical usefulness is restricted to populations whose prevalence of disease is similar to the one from which they were estimated.26 Recent meta-analyses indicated that the prevalence of psychosis in patients undergoing CHR assessment is of 15%, with high heterogeneity (95% CI 9% to 24%) across individual sites.8

Bayes’ theorem and predictive ability of CHR assessment

Ultimately, the value of a test will depend on its ability to alter a pretest probability of a target condition into a post-test probability that will influence a clinical management decision.25 Pretest and post-test probability of psychosis in CHR patients index an individual's likelihood of the development of psychosis before and after the CHR assessment, respectively. To be clinically useful, the results of the CHR assessment should substantially change the pretest probability of the individual patient to develop the disease, that is, significantly increase or decrease his/her post-test probability.8 These concepts are linked to Bayes’ theorem that describes the probability (P) of an event (eg, transition to psychosis, T), based on conditions that might be related to the event (eg, being CHR+). In case of early detection, Bayes’ theorem can be stated mathematically as: or in words: The conditional probability of development of psychosis given a positive CHR assessment, P(T|CHR+), is the product of the conditional probability of CHR+ given that psychosis develops, P(CHR+|psychosis), and the probability of psychosis, P(T), divided by the probability of a positive CHR assessment, P(CHR+).

However, P(CHR+) and P(T) and, consequently, their related conditional probabilities are no fixed numbers but depend on sampling biases. For example, P(CHR+) and P(T) as well as P(T|CHR+) were higher in CHR samples recruited from mental health professionals than in those recruited from the community.12 The same was true for predominately or exclusively adult samples compared to child and adolescent samples.7 Thus, it is not only CHR criteria that determine the post-test probability of transition to psychosis but also the recruitment and selection of samples, which might create substantial enrichment in risk.8 These concepts are illustrated in the probability modifying plot (figure 2) which plots pretest probability of developing psychosis before the CHR assessment on the x-axis against the post-test probability of developing psychosis after CHR assessment on the y-axis. The post-test probability in turns can be positive P(T|CHR+) or negative P(T|CHR−). The positive and negative posterior probability curves visually show that CHR assessment has an excellent ability to rule out psychosis, and a modest ability to rule in.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Meta-analytical probability modifying plot, illustrating the relationship between pretest probability of psychosis onset 38 months and post-test probability of psychosis risk at 38 months based on clinical high risk (CHR) psychometric assessment in patients seeking help at high-risk services, computed as the likelihood of a positive (above diagonal line in red; LR+) or negative (below diagonal line in green, LR−) CHR assessment result over the 0–1 range of pretest probability, adapted from.22 The vertical black line indicates the average pretest probability of psychosis onset in patients referred to high-risk services (15% at 38 months8) which yields a 26% post-test probability of psychosis onset in CHR+ cases and a 1.56% post-test probability of psychosis onset at 38 months in CHR− cases.22 LR, likelihood ratio.

Likelihood ratios

Several authors have suggested the use of positive and negative likelihood ratios (LR+, LR−) instead of sensitivity and specificity24 to better account for differences in the quality between the two and for pretest probabilities. LRs combine the estimates of sensitivity and specificity and offer the advantage of existing guidelines for their quality assessment.27 Thereby, LR+ (=Sensitivity/(1—Specificity)) (table 1) gives the factor by that the pretest probability increases given a positive test result, while LR− (=(1—Sensitivity)/Specificity) gives the fraction to that the post-test probability decreases given a negative test result.

As a rough guide, LR+ >10 and LR−<0.1 generate large and often diagnostically conclusive changes from pretest to post-test probability; LR+ of 5–10 and LR− of 0.1–0.2 produce moderate shifts in probability; LR+ of 2–5 and LR− of 0.2–0.5 result in small (but sometimes important) changes in probability, while LR+ <2 and LR− >0.5 change probability to a very small (and rarely important) degree.24 A graphical presentation of LRs with the probability modifying plot (figure 2) can improve the accuracy of estimated post-test disease probability in clinical practice.17

In a previous meta-analysis, we have shown that current CHR assessment has an excellent ability to rule out psychosis (LR−=0.09), at an expense of their ability to rule in psychosis (LR+=1.82)22 and these LRs are reported in figure 2. In fact, given a pretest probability of 15% in patients referred to high-risk services,8 the post-test probability of psychosis onset at 38 months, given a CHR+ assessment is of 26% (95% CI 23% to 30%)22 and given a CHR− assessment is of 1.56% (95% CI 0.7% to 2.42%).22

Assessing psychosis risk in clinical practice: the potentials and pitfalls of real-life application

While LRs thereby offer a good tool for the probabilistic assessment of individual risk, they were shown to be the least well understood accuracy measures by clinicians.17 ,28 Yet, probabilistic reasoning makes considerable demands on the clinician even beyond understanding of LRs, for example, estimating an individual's pretest risk of psychosis onset based on available, often heterogeneous and fragmented information on population estimates, as roughly illustrated by table 2. However, probabilistic reasoning in CHR has already been very useful in advising clinicians on the use of CHR assessment. As shown in table 2, probabilistic reasoning shows that CHR assessment should be restricted to populations seeking help at high-risk services (case 3) only, to avoid small and negligible PPVs and NPVs (cases 1 and 2). Unfortunately, the pretest probability of psychosis onset in different subsamples (eg, migrants, substance users) of patients seeking help at high-risk services is still not unknown. Thus, future research should overcome the limited pretest probability data29 by providing CHR clinicians with reliable and simple, yet not over-simplistic information on pretest probabilities in different samples seeking help at high-risk services and LRs that is presented in a well-applicable, for example, graphic form and with good guidelines for their use.