https://orcid.org/0000-0002-2524-1113
Figures
Abstract
Background
Treatment of surgical pain is a common reason for opioid prescriptions. Being able to predict which patients are at risk for opioid abuse, dependence, and overdose (opioid-related adverse outcomes [OR-AE]) could help physicians make safer prescription decisions. We aimed to develop a machine-learning algorithm to predict the risk of OR-AE following surgery using Medicaid data with external validation across states.
Methods
Five machine learning models were developed and validated across seven US states (90–10 data split). The model output was the risk of OR-AE 6-months following surgery. The models were evaluated using standard metrics and area under the receiver operating characteristic curve (AUC) was used for model comparison. We assessed calibration for the top performing model and generated bootstrap estimations for standard deviations. Decision curves were generated for the top-performing model and logistic regression.
Results
We evaluated 96,974 surgical patients aged 15 and 64. During the 6-month period following surgery, 10,464 (10.8%) patients had an OR-AE. Outcome rates were significantly higher for patients with depression (17.5%), diabetes (13.1%) or obesity (11.1%). The random forest model achieved the best predictive performance (AUC: 0.877; F1-score: 0.57; recall: 0.69; precision:0.48). An opioid disorder diagnosis prior to surgery was the most important feature for the model, which was well calibrated and had good discrimination.
Author summary
The opioid epidemic continues in the United States and prescription opioids contribute to opioid-related overdose deaths. The study aimed to develop a machine-learning algorithm to predict the risk of opioid-related adverse outcomes (OR-AE) following surgery using Medicaid data with external validation across seven US states. This retrospective study of 96,974 patients presents seven machine learning models to predict opioid-related adverse outcomes (OAO) following surgery. The best predictive performance was achieved by the random forest model, with an AUC of 0.877. Patients with depression, diabetes, or obesity had higher rates of OR-AE. The model was well calibrated and had good discrimination, with an opioid disorder diagnosis prior to surgery being the most important feature. This work provides evidence that an automated risk prediction can potentially guide postoperative opiate therapy and monitoring.
Citation: El Hajouji O, Sun RS, Zammit A, Humphreys K, Asch SM, Carroll I, et al. (2023) Prediction of opioid-related outcomes in a medicaid surgical population: Evidence to guide postoperative opiate therapy and monitoring. PLoS Comput Biol 19(8): e1011376. https://doi.org/10.1371/journal.pcbi.1011376
Editor: Benjamin Althouse, University of Washington, UNITED STATES
Received: January 18, 2023; Accepted: July 20, 2023; Published: August 14, 2023
Copyright: © 2023 El Hajouji et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The datasets analyzed during the current study are not publicly available due to reasonable privacy and security concerns. The datasets may be available through Gainwell Technologies, https://www.gainwelltechnologies.com/contact-us/.
Funding: Data and support for this work were provided by HMS (now part of Gainwell Technologies) as part of the Digital Health CRC (Cooperative Research Centre). The DHCRC is established and supported under the Australian Government’s Cooperative Research Centres Program. Research reported in this publication was supported by the National Library Of Medicine of the National Institutes of Health under Award Number R01LM013362 (THB). The content is solely the responsibility of the authors and does not necessarily represent the official views of the DHCRC or the National Institutes of Health. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The risks of opioids including dependence, misuse and overdose death are now well documented. In the United States, the opioid epidemic is a national public health emergency, with recent estimates suggesting that more than 10 million Americans aged 12 or older misused opioids in 2020 [1]. Prescription opioids are a major contributor to misuse, which can lead to addiction and death. Drug overdose deaths involving prescription opioids rose from 3,442 in 1999 to 16,416 in 2020 [2].
Treatment of surgical pain is the second most common reason patients receive opioids, and thus is an important avenue of exposure to opioid-related risks [3,4]. Postoperative pain management attempts to balance opioid-facilitated analgesia against the inherent risks of opioids. Inadequate pain management besides causing human suffering, increases risk of complications including the development of chronic pain [5]. However, the receipt of prescription opioids after surgery is associated with increased risks of long-term opioid use disorders [6–8]. Overprescribing can also lead to opioid diversion with the excess opioids [9]. A recent meta-analysis suggests that surgical procedures are associated with nearly 7% of patients experiencing persistent opioid use after surgery [10]. Understanding how to provide adequate pain management and mitigate the risk of developing opioid use disorders remains a top question in the surgical field.
The Medicaid population presents unique challenges in perioperative pain management because multiple stressors can complicate their pain outcomes [11–13]. For example, opioid use disorder (well known to adversely impact pain outcomes) is higher among Medicaid beneficiaries [14]. Among the two million nonelderly US adults with opioid use disorder, nearly 40% were covered by Medicaid [15]. Increased socioeconomic stressors and pre-existing opioid exposures makes perioperative pain management and use of opioids more fraught in the Medicaid population.
The prevalence of opioid-related adverse outcomes (OAO) in the Medicaid population is not well-documented. Predicting risks within the covered population can help state Medicaid programs implement interventions targeting individuals at high-risk for adverse opioid-related outcomes and to promote precision medicine. The aim of this paper is to develop a risk score for Medicaid patients who are at high risk for opioid abuse, dependence and overdose following surgery. This score can be used to identify patients at risk who might benefit from alternative pain management modalities or need additional monitoring and support. In addition, understanding the risk of OAO in vulnerable populations (i.e., patients with depression, or previous substance abuse can inform prescribing practices by clinicians to these higher-risk patients.
Methods
Ethics statement
We received the HIPAA patient waiver allowing us to access and use the patient’s medical records for research purposes. This study received the approval from the Stanford University’s Institutional Review Board (IRB), #47644.
Data source
This retrospective cohort study consisted of de-identified enrollee data obtained from Medicaid enrollment and claims provided by the Digital Health Cooperative Research Center. Data collected were from three western states, two midwestern states and two southeastern states between January 2016 and June 2020. Two states provide data on the entire Medicaid population including both the managed care plan and the fee-for-service patients. Four states provide data from a managed care plan and one state had only fee-for-service Medicaid beneficiaries. Data were reported according to MINIMAR guidelines [16].
Cohort selection
We included Medicaid enrollees who underwent one of 19 common types of surgery as identified by the International Classification of Diseases 9 and 10 codes and Current Procedural Terminology (CPT) codes. We categorized surgeries using the Clinical Classifications Software for Services and Procedures (CCS-Services and Procedures). (S1 Table). We included the most common major surgeries but excluded those that would likely have a low pain profile such as cataracts, breast biopsies, etc. We excluded individuals who underwent more than one surgery to ensure that any opioid event (e.g., prescription, opioid disorder diagnosis) is not related to another surgery. We included enrollees with continuous Medicaid coverage for at least 6 months prior and 12 months after the surgery event. This ensured data observability and a complete capture of persons’ characteristics and outcomes needed for prediction. We excluded enrollees who were 65 years of age and older due to their dual eligibility for Medicare, for which we had no claim data. The starting age of inclusion was 15 years, because prescription and illicit opioid use among the 15–19 age group is high [17]. Opioid prescriptions were identified using RxNorm identifiers from the Unified Medical Language system (opioid prescriptions are listed in Appendix A). Opioid prescriptions were also converted to Morphine Milligram Equivalent (MME) associated following CDC recommendations [18].
Feature construction
Our primary outcome was a binary indicator of whether a person had opioid abuse, dependence, or an overdose event (opioid-related adverse outcomes) during the 6-month period after surgery. OAO were identified by the International Classification of Diseases, 9th Revision (ICD-9-CM) codes and the International Classification of Diseases, 10th Revision (ICD-10-CM) codes summarized in S2 Table. We did not separate these adverse outcomes due to the limited event incidence of opioid overdose. Our secondary outcome was persistent opioid use, a binary variable indicating whether a patient had a new opioid prescription between 3 and 6 months after discharge from surgery [4].
Sociodemographic features, including age, gender and urbanization status based on an enrollee’s zip code were included. These variables were in a standard format and therefore comparable across states. We did not include race and ethnicity features due to lack of data availability. Based on the diagnosis codes available in the claims during the 6 months pre-surgery, we extracted 544 diagnosis features corresponding to the Clinical Classifications Software Refined (CCSR) ICD-10-CM diagnosis categories (S3 Table) and used the CCSR to identify diabetic, obese and depressed populations [19]. Similarly, 326 procedure features were extracted corresponding to the CCSR ICD-10-PCS procedure categories. We excluded those that were reported in less than 2% of patients. With a threshold of less than 2%, we are able to exclude features with very low prevalence, as they may not have enough variation to be useful for prediction and including them in the model may actually decrease its performance. This feature selection process resulted in 227 diagnosis features and 38 procedure features used as model inputs. In addition, type of surgery was used as a feature. Based on a combination of opioid and non-opioid prescription claims 6-months preceding surgery, a total of 46 prescription features were extracted. Patients who were not exposed to opioids during the 6 months period before surgery were classified as opioid naïve. Morphine milligram equivalent (MME) were computed using the CDC formula, as previously described in detail.18 All features are described in S4 Table.
Model development and construction
The primary goal was to predict 6-month postoperative risk of an OAO and the secondary goal was to predict risk of persistent opioid use. The steps for both prediction goals were the same. Medicaid data from 7 states were combined and then divided randomly into a training set comprising 90% of the data and a test set comprising 10%. We randomly sampled (stratifying by outcome) a training set from the cohort consisting of 90% of the data, the rest of the data being used as a testing set. Due to class imbalance, the pipelines consisted of a step of median imputation for missing values (for age, gender and urbanization status features), a step of standardization for numeric features, a step of minority class over-sampling with ratio 0.3, a step of majority class under-sampling with ratio 0.7 (resulting in a 41% outcome rate), followed by the machine learning model. We built our prediction models using multiple machine learning as well as deep learning methods: logistic regression and its variants Ridge, Lasso and Elastic Net, random forest, extreme gradient boosted trees (xgboost) and a deep neural network approach. For all pipelines, hyperparameters were selected using a grid search with 5-fold cross-validation on the training data, optimizing for area under the receiver operating characteristic curve (AUC). Hyperparameters are reported in S5 Table. In practice, identifying all patients with high risk of having OAO is desired, thus we favor recall over precision in the precision-recall trade-off.
Model evaluation
We ran the models on 5 subsets of the held-out test set and computed means and standard deviations of several metrics obtained on each subset. Evaluation was done by comparing the means of these metrics. We reported AUC, precision, recall and F1 score to show the prediction ability. We prioritized AUC for comparison since it is threshold-independent, so that it measures the overall prediction ability of the model. The other metrics measure a snapshot of the model in the precision-recall trade-off. A precision-recall curve was generated for the best performing model for the primary outcome prediction.
To assess the calibration of the top performing model, we conducted several analyses. First, we calculated the calibration intercept and slope and generated calibration curves specific to each population. Additionally, we generated bootstrapped estimates of standard deviations for the overall calibration curve. The calibration intercept assesses whether risks are overestimated (intercept <0) or underestimated (intercept >0). An extreme slope leads to underestimation of low risks and overestimation of high risks, whereas a moderate slope results in overestimation of low risks and underestimation of high risks [20]. We used these metrics to evaluate the performance of our model. Finally, we generated overall decision curves for both the top performing model and the logistic regression to visualize the potential clinical impact of using each model. By presenting these curves, we aimed to provide clinicians with an intuitive way to understand the potential benefits and harms of using each model.
Feature importance
For the primary outcome, to better understand the prediction and how different features impact the model result, we generated the top 20 important features using Shapley-value feature importance obtained from the best performing model [21]. We applied hierarchical clustering to the original features based on their Spearman correlations so that correlated features can be grouped in clusters. We then chose one feature per cluster and ran the feature importance analysis based on those newly selected features. Without such a subset selection procedure, the model could not be interpreted accurately because of feature correlations. Indeed, the estimated importance of a feature correlated with a lot of other features is an underestimation of the actual importance.
Statistical analysis
Descriptive statistics were used for the entire sample as well as for individuals with OAO during the 6-month period after surgery. We measured the bivariate association of the categorical variables with the primary outcome using the chi-squared test of independence and we measured the association of age to the primary outcome using Welch’s t-test. Additionally, we evaluated the primary outcome and average discharge MME within the diabetic, obese and depressed population subsets of the cohort overall and across all years, using a t-test for regression slope comparison with the control population. Statistical significance was set as 2-tailed p<0.05.
Results
Cohort characteristics
A total of 96,974 enrollees were included in the study (S1 Fig). Table 1 presents the characteristics of the study cohort, overall and by outcome category. The cohort was on average 40.5 (SD 13.5) years old with 69.4% female. Opioid naive patients account for 66.2% of all enrollees. A higher proportion of males had an opioid-related adverse outcome compared to females (12.2% vs. 10.2%, p<0.001). By 6-month post-surgery, 10,018 (10.3%) patients had opioid abuse or dependence and 908 (0.9%) had at least one overdose event, totaling a 10.8% OAO rate. 20,075 (20.7%) patients had persistent opioid use. Patients with diabetes, depression, and obesity had a higher rate of OAO compared to non-vulnerable patients. We report the evolution, between 2016 and 2020, of the discharge daily MME (Fig 1), persistent opioid use rate (Fig 2) and OAO rate (Fig 3) for these populations, as well as the control population. The average daily MME discharge prescription across the populations was not significantly different. As for persistent opioid use rates, those are significantly higher for patients with diabetes, depression, and obesity than for the control population (p<0.001), but all the rates decrease between 2016 and 2020. The difference of trend was statistically significant for the depressed (p = 0.012) and diabetic (p = 0.009) populations. The year-on-year changes of OAO rates are 0.75 percentage points for depression (p = 0.002), 1.14 percentage points for diabetes (p = 0.004) and 0.57 percentage points for obesity (p = 0.010).
Model performance
The results for the models for both outcomes are reported in Table 2, S6 and S7 Tables. The random forest model achieved the highest score for all metrics: 0.877 for AUC, 0.57 for F1 score, 0.69 for recall and 0.48 for precision. A precision-recall curve for the random forest model is displayed in S2 Fig. For the persistent opioid use prediction, the F1 scores are overall comparable to the best F1 score for OAO prediction.
The random forest model had an overall calibration intercept of -0.01 and a slope of 1.12. Risk estimates were slightly underestimated (Figs 4 and 5). Calibration for the diabetes subpopulation was marginally better than the overall cohort. The random forest model had improved clinical utility to select patients at high risk of OAO compared to the logistic regression, as defined by the decision curve analysis; the higher the curve, the better the clinical utility at the chosen risk threshold.
Model specification
The top 20 features obtained from the Shapley-value method are reported in Fig 6. The most important feature associated with a higher likelihood of OAO is diagnosis of opioid-related disorders (pre-surgery). The second most important feature is cumulative daily MME (pre-surgery). Some factors were complex, influencing different populations in different manners. For example, a high discharge daily MME can lead both to predicting OAO and predicting no OAO. For the MME of hydrocodone prescriptions, most high values lead to OAO, but some high values are associated with a prediction of no OAO.
Age is the third most important feature, with young patients having less adverse outcomes. Gender is also part of the important features; males being more often associated with OAO. Urban influence code is also identified as one of the most important features: rural regions were predictive of both OAO and no OAO.
Discussion
In this study we found more than 10% of Medicaid surgical patients were diagnosed with opioid abuse, dependence, or overdose after surgery, and more than 20% had persistent opioid use—requesting a new opioid prescription 3–6 months after surgery. We developed a machine learning approach to predict postoperative OAO in the Medicaid population. The models were reasonably calibrated, had good discrimination, and identified key features associated with OAO. The random forest model was best to distinguish between low and high-risk patients and thus had the highest potential clinical utility. If confirmed, these models could identify patients with a high risk of an OAO prior to surgery, increasing the care systems ability to change postoperative pain management among these high-risk patients to improve care outcomes, reduce unnecessary costs, and potentially reduce opioid misuse among this vulnerable population.
The models developed to predict OAO performed well, with AUC over 0.8. Previous opioid exposures (i.e., diagnosed opioid use disorder or high cumulative MME 6 months prior to surgery) were the strongest features associated with OAO. Although this is consistent with available literature,3,8 the machine learning approach builds on this evidence to generate a risk score for OAO in Medicaid patients that can be used prior to surgery to improve patient outcomes. Importantly, the machine learning model highlights a combination of features predicting adverse outcomes far better than any one individual feature.
Other diagnoses also put the patients at higher risk for OAO. A prior history of diabetes, depression, and obesity increased the risk by 2-fold. Moreover, we found that OAO rates increased across the study period in all vulnerable populations whereas the rest of the population had decreasing OAO. There are several potential explanations for this finding. First, these populations have increased healthcare needs and are at higher risk of adverse results after surgery that might require opiates [22–25]. Second, many of these patients are taking medications or have metabolic derangements that can complicate perioperative pain management [26,27]. This medical complexity could impact their pain management and opioid risk. These findings highlight how a machine learning approach can inform a practitioners approach pain management in unique populations and identify those that would most benefit from either increased monitoring, increased patient education, or reduced opioid dosing to improve outcomes, congruent with precision medicine initiatives.
Precision medicine considers the many variables that could affect the individual patient’s course to more accurately match their needs to therapy. The importance of these complex relationships is demonstrated by the machine learning models from this study, which are trained on 316 features. Machine learning gives a more nuanced picture than a simple binary outcome of a linear regression model. Models like random forests capture highly non-linear interactions of features and can take advantage of those interactions to predict the outcome. Therefore, one feature can have a positive influence on the outcome for some patients and a negative influence for others, depending on the feature’s complex interactions with other features. Shapley values provide a framework to explain the impacts on model outcomes of features while at the same time ranking features by importance. One example of this complexity was discharge daily MME, which is both a positive and negative predictor of OAO. This result means that discharge daily MME is important in pain response to surgery, but other factors importantly interact with discharge daily MME, influencing the outcomes.
There are several limitations to this study. First claims data do not provide functional measures, such as pain scores. However, we use a combination of prescriptions, encounters, and ICD codes to capture the patient care episode. Second, ICD codes for opioid abuse or overdose may be under-reported and do not distinguish between type of opioids, thus we cannot examine problems with prescribed opioid vs. illicit opioids. This likely means that models would perform better if there was less subjectivity in the outcomes. Finally, this study is based on data from seven states, therefore the results may not be generalizable to the entire Medicaid population. However, the data sets we have included are state-wide, geographically diverse and represent millions of Medicaid recipients. This allowed us to test the generalizability and robustness of our models across different states that were geographically diverse and had varying in demographic and socioeconomic characteristics. This allowed us to test the performance of our models in a range of different contexts and to assess their ability to generalize to new settings.
In this study, we characterize OAO following surgery using Medicaid administrative data and highlight the increased rates of these events in vulnerable populations. We present a risk score developed on pre-operative data that has good discrimination and calibration, which could allow one to identify patients at high risk of OAO prior to surgery and allow for either tailored pain management or increased monitoring following surgery. The approach highlights known prediction factors but also leverages the combination of interactions between variables to improve patient discrimination. Developing a risk tool flagging patients at high risk for developing OAO in clinicians’ workflow is crucial for optimizing treatment options and improving patient outcomes, particularly among vulnerable populations.
Supporting information
S1 Table. Type of surgical procedures included in the study with corresponding CCS procedure category.
https://doi.org/10.1371/journal.pcbi.1011376.s001
(DOCX)
S2 Table. Diagnosis Codes for Opioid Abuse, Dependence and Overdose from International Classification of Diseases (ICD) 9th and 10th Revision.
https://doi.org/10.1371/journal.pcbi.1011376.s002
(DOCX)
S3 Table. CCSR diagnosis categories for the 3 vulnerable populations.
https://doi.org/10.1371/journal.pcbi.1011376.s003
(DOCX)
S5 Table. Hyperparameters found by grid search with 5-fold cross-validation (Python 3.6.9).
https://doi.org/10.1371/journal.pcbi.1011376.s005
(DOCX)
S6 Table. Model performance metrics for opioid abuse/dependence/overdose prediction.
https://doi.org/10.1371/journal.pcbi.1011376.s006
(DOCX)
S7 Table. Model performance metrics for persistent opioid use prediction.
https://doi.org/10.1371/journal.pcbi.1011376.s007
(DOCX)
S2 Fig. Precision-recall curve for random forest model.
https://doi.org/10.1371/journal.pcbi.1011376.s009
(DOCX)
References
- 1.
Abuse S. Key substance use and mental health indicators in the United States: results from the 2019 National Survey on Drug Use and Health. 2020.
- 2. Rudd RA, Aleshire N, Zibbell JE, Gladden RM. Increases in Drug and Opioid Overdose Deaths—United States, 2000–2014. MMWR Morb Mortal Wkly Rep. 2016;64(50–51):1378–82. pmid:26720857.
- 3. Shah A, Hayes CJ, Martin BC. Characteristics of Initial Prescription Episodes and Likelihood of Long-Term Opioid Use—United States, 2006–2015. MMWR Morb Mortal Wkly Rep. 2017;66(10):265–9. Epub 20170317. pmid:28301454.
- 4. Jivraj NK, Raghavji F, Bethell J, Wijeysundera DN, Ladha KS, Bateman BT, et al. Persistent Postoperative Opioid Use: A Systematic Literature Search of Definitions and Population-based Cohort Study. Anesthesiology. 2020;132(6):1528–39. pmid:32243330.
- 5. Sinatra R. Causes and consequences of inadequate management of acute pain. Pain Med. 2010;11(12):1859–71. Epub 20101028. pmid:21040438.
- 6. Alam A, Gomes T, Zheng H, Mamdani MM, Juurlink DN, Bell CM. Long-term analgesic use after low-risk surgery: a retrospective cohort study. Arch Intern Med. 2012;172(5):425–30. pmid:22412106.
- 7. Brummett CM, Waljee JF, Goesling J, Moser S, Lin P, Englesbe MJ, et al. New Persistent Opioid Use After Minor and Major Surgical Procedures in US Adults. JAMA Surg. 2017;152(6):e170504. pmid:28403427.
- 8. Brat GA, Agniel D, Beam A, Yorkgitis B, Bicket M, Homer M, et al. Postsurgical prescriptions for opioid naive patients and association with overdose and misuse: retrospective cohort study. BMJ. 2018;360:j5790. Epub 2018/01/17. pmid:29343479 at www.icmje.org/coi_disclosure.pdf and declare: no support from any organization for the submitted work; no financial relationships with any organizations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
- 9. Humphreys K, Shover CL, Andrews CM, Bohnert ASB, Brandeau ML, Caulkins JP, et al. Responding to the opioid crisis in North America and beyond: recommendations of the Stanford-Lancet Commission. Lancet. 2022;399(10324):555–604. Epub 20220202. pmid:35122753.
- 10. Lawal OD, Gold J, Murthy A, Ruchi R, Bavry E, Hume AL, et al. Rate and Risk Factors Associated With Prolonged Opioid Use After Surgery: A Systematic Review and Meta-analysis. JAMA Netw Open. 2020;3(6):e207367. Epub 20200601. pmid:32584407.
- 11. Nam YH, Bilker WB, DeMayo FJ, Neuman MD, Hennessy S. Incidence rates of and risk factors for opioid overdose in new users of prescription opioids among US Medicaid enrollees: A cohort study. Pharmacoepidemiol Drug Saf. 2020;29(8):931–8. Epub 20200710. pmid:32648636.
- 12. Dilokthornsakul P, Moore G, Campbell JD, Lodge R, Traugott C, Zerzan J, et al. Risk Factors of Prescription Opioid Overdose Among Colorado Medicaid Beneficiaries. J Pain. 2016;17(4):436–43. Epub 20151222. pmid:26721613.
- 13. Johnson D, Ho VT, Hah JM, Humphreys K, Carroll I, Curtin C, et al. Prescription Quantity and Duration Predict Progression from Acute to Chronic Opioid Use in Opioid-Naïve Medicaid Patients. PLOS Digital Health. 2022;in press.
- 14. O’Brien P, Henke RM, Schaefer MB, Lin J, Creedon TB. Utilization of treatment by Medicaid enrollees with opioid use disorder and co-occurring substance use disorders. Drug Alcohol Depend. 2020;217:108261. Epub 20200828. pmid:32979735.
- 15.
Orgera K, Tolbert J. Key facts about uninsured adults with opioid use disorder. KFF; 2019.
- 16. Hernandez-Boussard T, Bozkurt S, Ioannidis JPA, Shah NH. MINIMAR (MINimum Information for Medical AI Reporting): Developing reporting standards for artificial intelligence in health care. J Am Med Inform Assoc. 2020. Epub 2020/06/28. pmid:32594179.
- 17. Gaither JR, Shabanova V, Leventhal JM. US National Trends in Pediatric Deaths From Prescription and Illicit Opioids, 1999–2016. JAMA Netw Open. 2018;1(8):e186558. Epub 20181207. pmid:30646334.
- 18. Lossio-Ventura JA, Song W, Sainlaire M, Dykes PC, Hernandez-Boussard T. Opioid2MME: Standardizing opioid prescriptions to morphine milligram equivalents from electronic health records. Int J Med Inform. 2022;162:104739. Epub 20220316. pmid:35325663.
- 19.
Elixhauser A, Steiner C, Palmer L. Clinical classifications software (CCS). Rockville, MD: Agency for Healthcare Research and Quality,; 2014.
- 20. Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol. 2016;74:167–76. Epub 20160106. pmid:26772608.
- 21. Williamson BD, Feng J. Efficient nonparametric statistical inference on population feature importance using Shapley values. Proc Mach Learn Res. 2020;119:10282–91. pmid:33884372.
- 22. Ekström W, Al-Ani AN, Sääf M, Cederholm T, Ponzer S, Hedström M. Health related quality of life, reoperation rate and function in patients with diabetes mellitus and hip fracture—a 2 year follow-up study. Injury. 2013;44(6):769–75. Epub 20121102. pmid:23122996.
- 23. Karci A, Tasdogen A, Erkin Y, Aktaş G, Elar Z. The analgesic effect of morphine on postoperative pain in diabetic patients. Acta Anaesthesiol Scand. 2004;48(5):619–24. pmid:15101859.
- 24. Ri M, Aikou S, Seto Y. Obesity as a surgical risk factor. Ann Gastroenterol Surg. 2018;2(1):13–21. Epub 20171028. pmid:29863119.
- 25. Ghoneim MM, O’Hara MW. Depression and postoperative complications: an overview. BMC Surg. 2016;16:5. Epub 20160202. pmid:26830195.
- 26. Regnier SM, Khatibi B, Gabriel RA. Association of diabetes mellitus with postoperative opioid consumption in patients undergoing primary unilateral hip arthroplasty. J Perioper Pract. 2021;31(3):96–101. Epub 20200225. pmid:32096444.
- 27. Parthipan A, Banerjee I, Humphreys K, Asch SM, Curtin C, Carroll I, et al. Predicting inadequate postoperative pain management in depressed patients: A machine learning approach. PLoS One. 2019;14(2):e0210575. Epub 2019/02/06. pmid:30726237