Integrated Multisystem Analysis in a Mental Health and Criminal Justice Ecosystem

Patients with a serious mental illness often receive care that is fragmented due to reduced availability of or access to resources, and inadequate, discontinuous, and uncoordinated care across health, social services, and criminal justice organizations. These gaps in care may lead to increased mental health disease burden and relapse, as well as repeated incarcerations. Further, the complex health, social service, and criminal justice ecosystem within which the patient may be embedded makes it difficult to examine the role of modifiable risk factors and delivered services on patient outcomes, particularly given that agencies often maintain isolated sets of relevant data. Here we describe an approach to creating a multisystem analysis that derives insights from an integrated data set including patient access to case management services, medical services, and interactions with the criminal justice system. We combined data from electronic systems within a US mental health ecosystem that included mental health and substance abuse services, as well as data from the criminal justice system. We applied Cox models to test the associations between delivery of services and re-incarceration. Using this approach, we found an association between arrests and crisis stabilization services in this population. We also found that delivery of case management or medical services provided after release from jail was associated with a reduced risk for re-arrest. Additionally, we used machine learning to train and validate a predictive model linking non-modifiable and modifiable risk factors and outcomes. A predictive model, constructed using elastic net regularized logistic regression, and considering age, past arrests, mental health diagnosis, as well as use of a jail diversion program, outpatient, medical and case management services predicted the probability of re-arrests with fair accuracy (AUC=.67). By modeling the complex interactions between risk factors, service and


Introduction
The mental healthcare system in the United States is fundamentally broken -it is fragmented, inconsistent, underfunded, and rapidly deteriorating. Dominated by a lack of both consistency and continuity of care, the system allows many clients to mentally decompensate, ultimately leading to three negative crisis outcomes -homelessness, emergency hospitalization and incarceration. This fragmentation in the treatment of the mentally ill is particularly evident when one considers that it has been estimated that between 44 and 61 percent of jail inmates in the United States have a mental health problem (James et al Mental health problems in prison). Many individuals with severe mental illness are released from prison every year in the United States and re-enter the community with a need to continue treatment for their mental health issues. Continuous mental health treatment of these individuals may help prevent relapse and recidivism.
Lack of continuous care for adults with serious mental illness may not only result in more decompensation and crisis as the individual navigates the mental health, social, and criminal justice systems, but it also limits our understanding of the impact and interaction between modifiable risk factors and access to multiple services on patient outcomes such as re-arrest. Given that different agencies often maintain isolated datasets, the fragmentation of data systems and lack of access to continuous patient-level data means that it is difficult to collate data across health, social, and criminal justice agencies to evaluate the interplay between multiple services and outcomes. There is a critical need to evaluate patient-level and service-level data across multiple agencies in order to understand the mechanisms through which we may intervene to prevent or delay psychiatric crises.
Previous work evaluated data from US Medicaid claims files and arrest records and found a reduced risk of re-arrest with receipt of outpatient services 1,2,3 and psychotropic medication possession 3 in adults with mental illness. Other research using county and statewide criminal justice records and archival data from health and social services found that individual risk factors including being homeless, not having outpatient mental health treatment, and having involuntary psychiatric evaluation in the previous quarter, and being black, younger than 21 years and having a cooccurring substance abuse problem increased the odds of arrest 4 .
Recent studies for other medical applications have used electronic medical records data to establish predictive models for illness severity in various disease domains, including preterm infants 5 , congestive heart failure 6 , septic shock 7 and HIV 8 .
In this study, we describe an approach to modeling the interplay among services and outcomes across an ecosystem of medical and social services providers and the criminal justice system in which there is a constant flow of individuals with serious mental illness in and out of the criminal justice system. We explored associations between the occurrence of arrest and crisis services outcomes tested using hazard modeling with both fixed-and timedependent covariates. We demonstrate the utility of such a combined dataset for predictive modeling by training and testing a model for arrest prediction.

Preliminaries
In this study, the sets of covariates used for prediction include both basic risk factors as well as indicators of access to specific services. Cox models were applied to test the associations between access to services and outcomes. Predictive models were constructed using elastic net regularized logistic regression.

Association analysis using Cox models
Cox proportional hazard models were used for testing associations between risk factors and the expected time for failure events to occur 9 .This association is modeled using a hazard rate that represents the amount of risk as a function of time. The effect of each risk factor is assumed to be multiplicative with respect to the hazard rate.
In addition to predicting the effect of services given immediately after release, we examine the effect of continuous access to services. These tests involve time dependent covariates such as access to services in every month after release from jail. Association tests with such covariates were performed using extended Cox models (see 11 for more details).

Predictive modeling using elastic nets
An elastic net is a method that allows classical regression models to deal with high dimensionality of observations. This method performs data-driven variable selection and results in a sparse model that includes the most informative covariates. Learning in such models involves tuning of two parameters: (1) alpha -which controls the sparseness and stability of the model, where a higher alpha increases the tendency of the learning algorithm to filter out noninformative covariates; (2) beta -a regularization parameter that prevents over-fitting of the model to the data which is employed to obtain optimal generalization performance. These parameters are usually tuned using internal crossvalidation on a training data set. The accuracy of the model is assessed on a test set. For more details see 12 .

Framework -integrated view of patients ecosystem
This study uses information extracted from electronic systems resident within a mental health ecosystem in the southern US. This included data used to support the claims process for medical and social services delivered to mental health and substance abuse patients, and data collected by the criminal justice system. All individual patient data used for the analysis were collected after obtaining appropriate consents and agreements, and personal information was removed to protect patient privacy.
Patient data collected for this analysis span 21 months and describe the engagements that patients have with service providers in the ecosystem. An essential part of analyzing such longitudinal data is defining an index date, where data collected before that date serve as input to decision making and data from after that point in time define the outcome values and characterizes the treatment. We then define the target populations on which to focus and a quantitative outcome measure. Finally, we extract and filter features and risk factors to drive the modeling ( Figure  1).
The main outcome addressed in this study was a re-arrest, as experienced by individuals that had become involved in the criminal justice system. The study explored two main questions: (i) What are the modifiable and nonmodifiable risk factors associated with this outcome? (ii) How well can we predict the likelihood of re-arrest using such risk factors? Our approach employed an association analysis to explore the answers to the first question and a machine learning analysis to explore the answers to the second question.
We further test the association of various risk factors to admission to acute service, namely crisis stabilization unit.
We hypothesized that the relationship between mental health services and the criminal justice system may be bidirectional; to explore this hypothesis, we test associations between various risk factors including previous arrest records and the risk of admission to acute mental health treatment facilities (namely crisis stabilization unit). Arrest data were supplied by the Department of Law Enforcement and was extracted from the Criminal Justice Information Services (CJIS). These data span a period from January 1, 2007 to September 6, 2012, and include records on 184,470 individuals. Out of these, 5,148 overlap with the SPMI population in the health ecosystem studied. The court runs a program that helps identify and divert detainees with a mental illness into a Jail Diversion Program (JDP), which seeks to reintroduce individuals into a sustained care environment, combining mental health and housing services as part of a structured year-long engagement. The court provided a list of participants for approximately ten years, overlapping the data contained in the other data sources.

Population Selection Criteria
To analyze the relations between arrest and behavioral health service events we focused on a subset of the adult population having records both in the CJIS and mental health ecosystem datasets. We excluded 281 individuals from this cohort because of inaccurate and inconsistent timeline data. Out of the remaining individuals, a total of 3,274 were released from an arrest after October 1 st , 2010, which is the starting date of the mental health services recorded in the dataset. Of these, 3,171 were adults at the time of release a .
In addition to viewing arrest as an outcome, we analyzed the association of past arrests with the outcome of admission to an acute mental health treatment using the SPMI cohort. We excluded individuals whose first recorded admission had ambiguous or unknown dates, creating a subset of 15,930 subjects. Of these, we further focused on the adult population (N=14,228).

Statistical Analysis
Association between non-modifiable risk factors, receiving services after release from jail, and the risk of re-arrest The initial association analysis examined non-modifiable risk factors including gender, age, race, mental health diagnosis and past arrests using a Cox proportional hazard model. The association of receiving different service types with the risk of re-arrest was evaluated, adjusting for these non-modifiable risk factors. The dataset contains thirty nine types of services, out of which fourteen service types were given to more than 20 patients in the cohort. Each of these service types was represented using an indicator covariate equal to one if an individual received the service at least once in the first quarter after release from jail and equal to zero otherwise.

Continuous access to services
Extended Cox models were used to examine the association of services given throughout the entire period after release from jail to the risk of re-arrest. More specifically, for each patient, all release dates after Oct 1, 2010 were listed and corresponding re-arrest dates were identified (or, if the patient was not re-arrested, the end-of-study date was determined). Starting from each such release date, the number of times each service was given to the patient in each consecutive 90 day time period was tabulated.
Subsets of these time-varying covariates, in addition to the non-modifiable factors, were then used to infer the parameters of extended Cox models. In particular, models were constructed with the following features: 1. An indicator covariate identifying whether or not a specific service was given within the last ninety days (including non-modifiable risk factors) to predict re-arrest within the coming ninety days. 2. An indicator covariate identifying whether or not a specific service was given since the last release from jail (including non-modifiable risk factors) to predict re-arrest within the coming ninety days.

Predictive modeling using elastic nets
To test the predictability of the arrest outcome, data were partitioned into a training set which contained approximately 80% of the cohort and a test set which contained the remaining 20%. An elastic net regularized regression model was used where alpha was tuned to balance sparseness and stability on the training set.
Because the goals of the analysis were set to predict re-arrest probability in the second quarter after release, the target population was similar to the one described in re-arrest risk factor analysis. However, individuals were excluded for which data were not available for two quarters. Applying this additional criterion, the cohort size was established as 1,679 individuals in the training set and 421 in the test set. We evaluated the predictive power of the model using a receiver operating characteristic (ROC) curve which compares the likelihood of correctly and incorrectly predicting re-arrest.
a Due to the removal of exact birth dates, we use estimated ages at different time points. Here we include individuals with estimated age at release > 18.

Preliminary associations of demographic and historical factors with re-arrest
The association between non-modifiable risk factors was estimated using Cox proportional hazard models. Past arrests factors were modeled as indicator variables whose value was one if the individual was arrested and released from jail between January 2007 (start date of the CJIS data) and October 2010, and zero otherwise.
Preliminary associations (i.e., without adjusting for other variables) between these factors and the risks of re-arrest are summarized in Table 1. Associations with p-value < 1e-4, will remain significant at a 0.05 level after a Bonferroni correction for 500 hypotheses. In particular, schizophrenia, history of arrests, male gender, black race, and younger age are shown to be risk factors for increased likelihood of re-arrest, in agreement with previous studies 4 .

Association between arrests and crisis services
In the ecosystem studied, a large proportion of individuals were first admitted into the system of care through a Crisis Stabilization Unit (CSU; representing more than 30% within a week from first admission). A subset of the population was examined that did not record an admission to a CSU in the first quarter (N=10,307). In this sample, the association of a later CSU admission with past arrests, adjusted for age, gender, race and mental health diagnosis was significant (p<1e-4) with a high hazard ratio of 2.46 (95% confidence interval 2.00-3.02).

Services associated with reduced risk of re-arrest
To test associations with services given in the first quarter after release inmates that were re-arrested within the quarter were excluded. Out of 3171 adults, a total of 2,377 (~75%) remained out of jail during this period. This test therefore was able to examine conditional probabilities of future re-arrests given that the individual remained out of jail in the first quarter.
Associations between service indicator variables and the risk of re-arrest adjusting for gender, age, race, mental health diagnosis and past arrests as defined in the baseline test model were also tested. Results, summarized in Table  2, indicate an association of case management and medical services with a reduced risk of re-arrest b . Figure 2 presents Kaplan-Meyer plots of arrest probability for individuals that stayed out of jail in the first quarter after release, given their access to these services in this quarter.

Continuous access to care and continuous monitoring of patients states
Extended Cox model analysis shows that the indicator for Medical Services, either in the past 90 days or since release, is significantly associated with a reduced risk for re-arrest, with hazard ratios of 0.68 (confidence interval 0.58-0.80, p-value<1e-5) and 0.67 (0.58-0.78, p-value<1e-7), respectively. Conversely, the indicator for Crisis Stabilization, in both time periods, is associated with an elevated effect on the risk of re-arrest, with hazard ratios of 1.43 (confidence interval 1.22-1.69, p-value<1e-4) and 1.23 (1.07-1.42, p-value=0.003), respectively. Schonenfeld residuals for all these indicators, except for Crisis Stabilization in the past 90 days, attested to the correctness of the proportional hazard assumption.

Predictive Modeling of Re-Arrests
Elastic net regularized logistic regression models were trained using a training set containing 1,679 individuals. The regularization parameters alpha and beta were tuned using cross validation and the model was retrained on the entire training set with optimal parameters. Testing this model on a test set of 421 individuals resulted in an AUC of 0.67 (see 'Full model' in Figure 3 below). Informative covariates selected by the training procedure included age, past arrests, mental health diagnosis, enrollment to the JDP as well as utilization of outpatient group services, medical services and case management. The probability of re-arrest is modeled as function of a weighted sum of these factors. As the ROC curve in Figure 3 indicates, the model correctly predicts 50% of individuals in the ecosystem at risk for re-arrest based on the defined risk factors, while mis-characterizing 30% of individuals at risk. To assess the predictability of re-arrest from basic demographic data, namely, age, gender and race, we trained a simpler model using the same cohort and an elastic net model. This model was inferior to the full model, with an AUC of 0.60 and 42% true positive rate at the 30% false positive threshold ('Basic model' in Figure 3). The difference between the two ROCs illustrates the additional predictive power of the judicial and mental health related factors.

Discussion
In this analysis, we found that characteristics including schizophrenia, history of arrests, male gender, black race, and younger age were risk factors for increased likelihood of re-arrest. Further, factors such as past arrests increased the risk of having a crisis stabilization event, whereas receipt of case management services or medical services following release from jail reduced the likelihood of re-arrest. Using this knowledge of non-modifiable and modifiable core risk factors associated with re-arrest, we developed a model that predicted the probability of rearrest after the first 90 days of release from jail with reasonable accuracy.
These findings support previous work that examined US Medicaid claims and arrests records, and other research, which found that factors associated with greater risk of arrest include minority racial-ethnic status 3 or African American race 1,4 , male gender, 3 younger age. 3 and receipt of outpatient services (which may have included individual or group behavioral health services, medication checks, 2 and/or case management 1,3 ). While we found that the receipt of medical and case management services predicted a reduced risk of arrest, the receipt of other behavioral health services such as outpatient group therapy was not considered significant. These findings therefore extend previous research by considering the independent contributions of case management and medical services in predicting re-arrest. However, while our findings suggest that providing case management and medical services within 90 days of release from jail may reduce the chance of re-arrest in adults with serious mental illness, it is important to remember that these are correlational results, and differential assignment to services may be due to some unmeasured variable associated with a better prognosis.
In the current analysis, we chose to include individuals with major mental illness including depression, bipolar disorder, and schizophrenia because these diagnoses are found at a high rate in the criminal justice system. 13 Future work should explore the role of other mental health diagnoses in predictive models, as well as the effect of comorbid conditions. Future studies should also be conducted to refine the model by integrating other sources of data (e.g. additional medical claims, pharmacy and hospitalization data). In the current analysis, medical services included primary medical care, psychiatric assessment and services, and administration of psychotropic drugs and other medications. It would be of particular interest for future analyses to examine the specific role of pharmacy/medication administration in predicting re-arrest, particularly given recent data showing that posthospitalization medication possession reduced the likelihood of arrest in adults with serious mental illness in a Florida Medicaid population. 3 Understanding how we can best intervene to address modifiable risk for incarcerations may be important not only for improving the quality of life of adults with serious mental illness, but also for reducing costs within the systems that support their care. Using Florida Medicaid data and records from Florida's Department of Children and Families (DCF) and the Florida Department of Law Enforcement (FDLE), Van Dorn et al (2013) compared the costs associated with criminal justice system involvement with those for mental health treatment, and found that overall system costs were lower for adults with serious mental illness who did not get arrested. 3 Taken together with our current findings, the results suggest that increasing the provision of case management and medical services in a SPMI population at risk for arrest may be an important strategy for reducing overall system cost burden. This should be explored in future research.

Conclusions
In conclusion, our findings illustrate the complex interactions between modifiable and non-modifiable risk factors and delivery of services on outcomes in adults with serious mental illness. The data-driven approach defined in this analysis demonstrates the value of integrating data across disparate datasets from healthcare, social services, and criminal justice agencies. Further development of this predictive model may help us to identify those individuals who are at greater risk for re-arrest and crisis, and to intervene in a timely manner to help improve outcomes for the mentally ill. A reduction in arrests in this seriously mentally ill population may not only improve patient outcomes, but also diminish the burden on the judicial and health systems.