Examining the Most Important Risk Factors for Predicting Youth Persistent and Distressing Psychotic-Like Experiences

,

https://doi.org/10.1016/j.bpsc.2024.05.009Psychotic-like experiences (PLEs) include unusual thought content, including unconventional beliefs, and perceptual abnormalities, including auditory and visual distortions, that lie on the lower end of the psychosis spectrum continuum (1,2).Although PLEs generally refer to nonclinical forms of psychosis spectrum symptoms, individuals who experience PLEs during childhood and adolescence show greater odds of developing diagnosable mental health concerns, including psychosis (odds ratio = 3.96), in adulthood (3).It is important to better understand as early as possible which individuals are at risk for the development of mental health concerns.Several studies, including our research using Adolescent Brain Cognitive Development (ABCD) Study data (4), have demonstrated that several characteristics may be particularly important for identifying potentially clinically relevant PLEs, including distress and persistence.
A potential factor that distinguishes more benign PLEs from PLEs that transition to psychotic disorders is whether they are persistent, which has been conceptualized as the pronenesspersistence-impairment (PPI) model (5,6).Studies that have examined PLE developmental trajectories (7-10) have generally found evidence for both persistent and transient/relatively benign subgroups (4,11).According to the expanded PPI model, most PLEs may be relatively benign and transient because exposure to additional environmental risks and/or a stronger genetic/early environmental diathesis (potentially reflected in increased expression of pathophysiology over development) is necessary for subclinical psychosis to first become distressing and persistent and second to deteriorate into a clinical psychotic state (5).
Risk factors associated with early PLEs include those that reflect relatively stable risk factors present from birth [e.g., family history of psychosis (12)], factors that emerge early in life [e.g., delayed developmental milestones, early cognitive deficits (12,13)], and factors that may emerge or worsen later in SEE COMMENTARY ON PAGE 852 development [e.g., structural and functional neural impairments (14)(15)(16), internalizing symptoms, and adverse childhood experiences (ACEs) (17)(18)(19)].Although previous research has implicated this range of risk factors in the development of PLEs, it is unclear which risk factors are the most important markers of clinically significant PLEs.Previous research examining risk factors has utilized traditional approaches such as univariate models in which each risk factor is examined independently from the other risk factors (11,12,(20)(21)(22)(23)(24).In contrast, machine learning techniques can capture multivariate patterns in the data and identify the most important markers based on the pattern of the data.
The current study represents a follow-up to our previous work in which we used univariate analyses to identify predictors of distress and persistence associated with PLEs (4).The current work expands on our previous work by aiming to use machine learning models to capture the multivariate patterns of risk factors for early PLEs.Specifically, this study aimed to use random forest classifications to identify the most important predictors of distress and persistence of PLEs measured over 3 time points when examining a range of multimodal predictors measured at baseline, with predictors spanning caregiver-and self-report symptoms and experiences, neural metrics, and neurocognition.Random forest classification was chosen because it is robust to overfitting and can include multiple measures simultaneously.Three main models were examined: 1) persistent distressing PLEs versus low-level PLEs, 2) transient distressing PLEs versus low-level PLEs, and 3) persistent distressing PLEs versus transient distressing PLEs.Alternative definitions of PLE groups were also examined to test whether findings were specific to our a priori definitions of persistence and distress.These random forest classifications incorporated best practices, including testing models in holdout samples and tuning parameters (25,26).We hypothesized that predictors from self-/parentreport variables (internalizing symptoms, family history of psychosis, motor and speech developmental delays, ACEs) would show stronger associations with distress and persistence of PLEs than predictors such as those at the behavior-(cognition) and circuit-(resting-state functional connectivity [RSFC], structural magnetic resonance imaging) levels of analysis that may emerge or worsen later in development.Importantly, the current work identified the most important risk factors at baseline prior to distressing PLEs persisting over time.

Participants
The ABCD Study is a large-scale study tracking 9-to 10-yearolds recruited from 21 research sites across the United States (27,28).Potential participants were excluded from participating in the ABCD Study for the following reasons: the child was not fluent in English; magnetic resonance imaging contraindication (e.g., irremovable ferromagnetic implants or dental appliances, claustrophobia, pregnant); major neurological disorder; gestational age ,28 weeks or birth weight ,1200 g; history of traumatic brain injury; or had a current diagnosis of schizophrenia, autism spectrum disorder (moderate or severe), mental retardation/intellectual disability, or alcohol/substance use disorder.
ABCD Data Release 4.0 (https://dx.doi.org/10.15154/1523041) includes 3 full waves of data: baseline (n = 11,878), 1year follow-up (n = 11,235), and 2-year follow-up (n = 10,416).See Table 1 for sample characteristics.All available data were utilized for measured risk factors (detailed below and in Figure 1), which were obtained at baseline.All procedures were approved by a central Institutional Review Board at the University of California, San Diego.All parents and children provided written informed consent and assent, respectively.

Measures
All measures are described in detail in the Supplemental Methods.For random forest classification analyses, the task was accuracy in distinguishing among low-level PLEs from persistent distressing PLEs, low-level PLEs from transient distressing PLEs, and transient distressing PLEs from persistent distressing PLEs (see Tables S1, S2 and Figure S1 for models examining additional PLE groups).Two types of classification models were trained.The first type, termed a priori models, utilized predictors at baseline that had been identified in previous work (4,12,(29)(30)(31): internalizing symptoms, adverse life experiences, within-network cingulo-opercular and default mode RSFC, average prefrontal cortical thickness, fluid and crystallized cognition composites, parent-reported motor delays, speech delays, and family history of psychosis.The second type, termed exploratory models, included the aforementioned predictors minus the cognitive composites and average prefrontal thickness, as well as all other circuit-level metrics (i.e., thickness and RSFC metrics) for a total of 267 exploratory metrics (7 a priori predictors 1 218 additional RSFC metrics 1 35 additional thickness metrics 1 7 individual cognitive tests) (see Supplemental Results).
Other Psychopathology and Functioning Measures.

Sums of Kiddie Structured Assessment for Affective Disorders
and Schizophrenia for DSM-5 (32) youth-rated internalizing symptoms (i.e., number of symptoms of current depression and generalized anxiety disorder) were examined.A history of psychotic disorders was scored as present if the participant had any firstor second-degree relatives with a psychotic disorder history as assessed using the Family History Assessment Module Screener (33).
Developmental Milestones.The current study examined sums of the total number of caregiver-reported motor and speech developmental milestone delays (34).30), an ACE variable was defined using the number of parent-rated child experience of traumatic experiences, a parent-rated question about whether the child was bullied at school or in the neighborhood, and 7 parentrated questions about financial adversity from a demographic questionnaire (e.g., "Were you evicted from your home for not paying the rent or mortgage?")from the Kiddie Schedule for Affective Disorders and Schizophrenia (see Supplemental Methods for all financial adversity questions and methods for creating ACE scores).

Behavior-Level Measures
Neuropsychological Test Battery.The current study utilized uncorrected National Institutes of Health Toolbox Cognitive Battery scores from the fluid and crystallized composite scores (35,36).

Circuit-Level Measures
Structural Magnetic Resonance Imaging Measures.
All participants completed T1-weighted and T2-weighted structural scans (1 mm isotropic) on a 3T scanner (Siemens, General Electric, or Phillips) with a 32-channel head coil.Structural neuroimaging processing was completed using FreeSurfer version 5.3.0 through standardized processing pipelines (37).For the current study, we included average prefrontal cortical thickness, which was the average of the bilateral caudal middle frontal, rostral middle frontal, and superior frontal gyri (see Supplemental Methods for quality control criteria; n = 42 excluded).
Resting-State Functional Connectivity.Participants completed four 5-minute resting-state blood oxygen leveldependent scans, with their eyes open and fixated on a crosshair.Resting-state images were acquired in the axial plane using an echoplanar imaging sequence.Other restingstate imaging parameters varied by 3T scanner and have been detailed previously (https://abcdstudy.org/images/Protocol_Imaging_Sequences.pdf).The data analysis pipeline has also been detailed previously (31,37).Pairwise correlations were examined for regions of interest within functionally defined parcellations (i.e., Gordon networks) and subcortical regions of interest (37,38).The Fisher z transformation of the correlation values was examined for within-network cingularopercular and within-network default mode regions of interest (see the Supplemental Methods for quality control criteria; n = 87 excluded).
Group Creation.See Figure 1 (41).These types of models have been used to explore a variety of research questions, including predicting remission of mental health diagnoses (42); predicting future mental health-relevant behavior, such as suicide attempts (43); and predicting cognitive functioning in psychiatric data (44), as well as classifying psychiatric groups [for a review of how psychiatric research has utilized machine learning techniques, see (41)].Random forest regressions are well suited to psychiatric research conducted using large-scale datasets because this technique can handle the inclusion of many predictors simultaneously and is robust in its predictive accuracy even for many predictors with small effects (45,46).However, random forest regressions are limited in their ability to handle withinperson correlations, which limits their applicability to longitudinal data.All random forest classifications were run using the randomforest package in R (47).Missing data were imputed using the rfImpute function (Table S3).All predictor variables were scaled using control normalization (48) whereby the predictor variables for the PLE groups were scaled using the averages and SDs calculated from the predictor variables for the lowlevel PLE groups.First, all models were trained on an 80% training sample.The randomforest package uses bootstrapping and averaging to deal with overfitting.An inner nested 10-fold cross-validation loop on the training sample repeated 3 times was used for tuning parameters (i.e., to find the optimal number of trees and the number of input variables that are randomly chosen at each split) (see Table S4 for more information about these tuning parameters).Model performance was evaluated in a 20% holdout sample using model accuracy (both overall and for each group).Groups were stratified across training and test samples to ensure that there were equal proportions of groups in every sample.Feature importance was quantified using the randomforest package "importance" function output, the mean decrease in the Gini index (i.e., mean decrease in impurity), and the mean decrease in the accuracy index, as well as an additional standard feature importance metric, SHAP (SHapley Additive exPlanations) values, which are considered to be standard feature importance metrics (49)(50)(51).Receiver operating characteristic curves were also examined for all models to evaluate the performance of the model at all classification thresholds.The performance of the classifier was quantified by calculating sensitivity, specificity, negative prediction error, and positive prediction error (Table 2).See Supplemental Results for follow-up analyses examining distress and persistence as factors.See Table S2 and Figure S1 for results obtained with alternative PLE group definitions.

RESULTS
A priori models performed similarly to exploratory models (see Supplemental Results).Accordingly, we focus on the results obtained using the a priori models.With these alternative definitions, the highest accuracy was found when the persistent distressing PLE group was restricted to participants who met criteria at all 3 time points (or at least at the first and third assessments) (Table S2).Importantly, regardless of PLE group definition, the most important factors distinguishing higher PLE groups from lower PLE groups were internalizing symptoms and ACEs, with some evidence for cognitive factors showing greater importance than other factors (Figure S1).

DISCUSSION
In the current study, we examined whether PLEs examined over 3 assessment waves could be classified based on baseline metrics.(52).Given this heterogeneity of the overall endorsement of PLEs, the results point to the importance of including factors that increase the clinical relevance of these endorsements, including persistence over time, when examining PLEs in large general population samples.

Classifying Persistent Distressing PLEs
The model for distinguishing persistent distressing PLEs from low-level PLEs generally performed fair/average.The persistent distressing PLE versus low-level PLE models showed relatively balanced sensitivity and specificity in the test sample, indicating the presence of both false positives and false negatives.Likewise, categorization errors were generally balanced across the persistent distressing and low-level PLE groups.
For differentiating the persistent distressing PLE group from the low-level PLE group, it appears that several markers consistently predicted persistent distressing PLEs across a priori or exploratory models (see Supplemental Results for exploratory models).These factors included greater internalizing symptoms, greater ACEs, and worse crystallized and fluid cognition, with exploratory models finding evidence specifically for picture vocabulary and list sorting working memory tests.This is consistent with our previous work on persistent distressing PLEs (4) and points to these factors as particularly useful predictive markers.Finding that internalizing symptoms was an important marker may be due in part to overlap with distress associated with PLEs (53), perhaps pointing to the role of distress and/or negative affect in classifying clinically significant PLEs.These results remained consistent regardless of how the PLE groups were defined, including if examining baseline predicting 2-year follow-up PLEs and when focusing on $1 significantly distressing PLEs (a PLE rated in the 3-5 range on the distress scale) reported at $2 of the last 3 assessment waves (Supplemental Results).Overall, persistent distressing PLEs were most distinguishable from low-level PLEs, highlighting both the severity of the persistent distressing PLE group but also the difficulty in distinguishing among PLE groups.

Classifying Transient Distressing PLEs
The Although previous work has used machine learning to distinguish between psychotic disorder groups based on neural metrics with accuracy upwards of 75% (57), consistent with the current work, there is limited evidence for the ability to use these methods to predict features from subclinical PLE scores (58).Along these lines, analyses from the current work showed few metrics that consistently distinguished the persistent distressing PLE group from other PLE groups, but these metrics included ACEs, thereby highlighting the importance of stressful life events.Evidence that ACEs were especially important for persistent distressing PLEs is consistent with the PPI model, according to which it is only in the context of greater environmental exposures that risk translates into worsening psychosis spectrum symptoms (1).Previous research indicates that traumatic life events may be an important predictor of worsening psychotic symptoms, including up to 80% of individuals at high risk for psychosis endorsing a traumatic event during childhood (59).
Overall, the performance indices, including accuracy, from models classifying persistent and transient distressing PLEs indicate that there is little evidence that these PLE groups can be distinguished from one another based on the baseline metrics examined.It may be true that factors other than baseline metrics, such as changes in functioning or symptoms, or currently unmeasured phenotypes are more important for predicting our outcome variables.Alternatively, there may be no phenotypes that can accurately distinguish between PLE groups.Future research should incorporate changes in risk factors.Additionally, regarding the second point about the role of unmeasured phenotypes, exploratory models failed to improve prediction and often found that the variables included in the models presented in this article (e.g., internalizing symptoms, ACEs, fluid cognition) were still among the most important predictors.Regarding the last point, it is possible that no cross-sectional self-report, behavioral, or circuit-level predictors can accurately classify PLE groups.Notably, circuit-level predictors were generally not among the most important predictors for distinguishing persistent distressing PLEs.This may indicate either that these predictors do not distinguish between these groups after accounting for behavioral-level predictors, or it may point to a need for more precise neural predictors.Furthermore, the overall results may point to the heterogeneity of early PLEs, which may mask our ability to uncover unifying risk factors.transient PLEs is highlighted by our poor accuracy in differentiating these groups regardless of how the groups were defined (see the Supplement).Therefore, it may not be until the worsening of symptoms or the onset of functional impairment that symptoms are of sufficient severity to be distinguishable from less severe PLEs.

Limitations
The current work has important contextualizing factors, including the fact that the models generally showed fair to poor accuracy and therefore are likely not currently useful for classifying clinically relevant PLE groups, only for distinguishing persistent distressing PLEs from low-level PLEs.It is possible that as future waves of ABCD Study data are released and group classifications can incorporate additional waves of data, persistent and transient PLE group classifications will show greater accuracy.Also, the analyses presented herein examined risk factors at baseline prior to the development of persistence of PLEs, and therefore there was overlap in the time points when we measured risk factors and PLEs.Prior to using these markers for screening purposes, efforts should be made to minimize false negatives (i.e., maximize sensitivity) (60) because the risks associated with missing anyone experiencing persistent distressing PLEs outweigh the risks of identifying a false positive.Machine learning models involve a number of choices, including decisions to use holdout samples, size of training and test sets, and choice of type of machine learning model.In the current work, we aimed to use best practices to aid in determining the most important predictors of persistent distressing PLEs (25,26).

Conclusions
Our analyses indicated that machine learning analyses can be used to distinguish persistent distressing PLE from low-level PLE groups.Additional evidence indicated that distress may be more distinguishable than persistence, including evidence that models performed poorly when trying to distinguish persistent PLEs from transient distressing PLEs.Regardless, the current work also reinforces the importance of several metrics that have been implicated previously in the development and worsening of early psychotic experiences.ACEs appeared to be especially important for persistent distressing PLEs, potentially providing support for the PPI model, according to which worsening of PLEs is linked to greater environmental loading.Both persistent and transient distressing PLEs were linked to risk factors including internalizing symptoms.Importantly, these results were robust to PLE group definitions.Overall, evidence from the current study indicates that while persistent distressing PLEs are distinguishable from low-level PLEs, there is limited evidence for the ability to distinguish between clinically relevant PLE groups.

Figure 1 .
Figure 1.Summary of findings from random forest classifications performing 3 tasks distinguishing persistent distressing psychotic-like experiences (PLEs) from low-level PLEs, persistent distressing PLEs from transient distressing PLEs, and transient distressing PLEs from lowlevel PLEs.(A) Mean decrease in accuracy for each of the predictors for each of the tasks, (B) mean decrease in the Gini coefficient for each of the predictors for each of the tasks, (C) mean SHAP (SHapley Additive exPlanations) values for each of the predictors for each of the tasks, and (D) receiver operating characteristic curves and area under the curve (AUC) for each of the tasks.Note that for (A-C), the dimensions of size and color both indicate the relative importance of the input feature such that features with larger red circles reflect greater importance.ACE, adverse childhood experience; CON, cingulo-opercular network; DT, default mode network; LP, lowlevel PLEs; PD, persistent distressing; RSFC, resting-state functional connectivity; TD, transient distressing.

Table 1 .
Participant Characteristics a 940 Biological Psychiatry: Cognitive Neuroscience and Neuroimaging September 2024; 9:939-947 www.sobp.org/BPCNNI (40)ress and persistence (4), the dichotomous distress factor was coded as 1 for individuals with a distress score on the Prodromal Questionnaire-Brief Child Version $1.96 SDs above the mean, with all other scores coded as 0. Distress thresholds were recalculated at each wave to partially account for retesting effects.These thresholds were chosen based on research in which this threshold has been used with different psychosis risk questionnaires in college students(39).A dichotomous persistence factor was coded as 1 if individuals met the above definition of distressing PLEs at 2 or more of the 3 waves of data collection (n = 305), with transient distressing PLEs coded as 0 if they met the definition of distressing PLEs for 1 time point and were at $0.50 SDs above the mean for distressing PLEs at the other 2 time points (n = 374).Participants met criteria for low-level PLEs if they scored $0.50 SDs on distressing PLEs at all 3 time points.Originally, 6589 participants met criteria for the low-level PLE group; therefore, for analyses comparing the low-level PLE group with other PLE group(s), the low-level PLE group was separately matched to each of the other PLE groups on age, sex, and race/ethnicity.See TableS1for alternative group definitions and Supplemental Results for findings using data-driven definitions of PLE groups, created using growth mixture modeling [package lcmm(40)] and then used as outcomes in random forest classification models.
for group definitions.PLEs in middle childhood may encompass some developmentally appropriate and/or transient experiences, and therefore focusing on more clinically relevant features of distress and persistence can enhance the clinical significance of this construct.In accordance with our previously generated definitions of Random Forest Classifications.Machine learning techniques, including random forest regression, are becoming more widespread in psychiatric research