Predictive value of machine learning for the risk of acute kidney injury (AKI) in hospital intensive care units (ICU) patients: a systematic review and meta-analysis

Search strategy and inclusion criteria

We conduct a comprehensive and systematic search of PubMed, Embase, the Cochrane Library, and Web of Science from inception through 25 June 2022. Only literature in the English language was included. The search utilized a combination of medical subject headings (MeSH) terms and text words related to intensive care units, acute kidney injury, and machine learning. Two researchers (Yuanhong Du, Ping Gan) independently searched the literature, with any discrepancies resolved by a third researcher. Furthermore, the references of each eligible article were meticulously reviewed for potential additional relevant research.

Literature screening and data extraction

The retrieved results were uploaded into the document management tool endnote.X9 for screening process. Two investigators independently assessed the studies for inclusion based on their titles, abstracts, full texts, and any Supplementary Materials. Any disagreements that arose were resolved through consultation with a third party.

Before starting data extraction, we developed a standardized data extraction form. This form includes the following information: first author, publication year, author affiliation, study type, patient sources, AKI diagnostic criteria, total number of AKI patients, overall sample size, AKI sample size in the training set, training set sample size, overfitting methods, AKI sample size in the validation set, validation set sample size, approaches for addressing missing data, variable screening and feature selection methods, categories of machine learning models utilized in the literature, and variables chosen for modeling, etc. Two investigators (Yuanhong Du, Lin-Yu Li) independently conducted the screening of studies and data extraction from the included studies based on the provided documents. Any disagreements were resolved with the assistance of a third party.

Quality assessment

Probast (Wolff et al., 2019) was used to assess the risk of bias in the included original research. This tool encompasses several questions distributed across four distinct domains: participants, predictor variables, outcomes, and statistical analyses. These questions collectively assess the overall risk of bias and the overall usability of the studies. Each of the four domains contains two, three, six and nine specific questions, respectively, and each question offers three response options (yes/probably yes, no/probably no, and no information). A domain was categorized as high risk if at least one question indicated “no” or “probably no”, while it was considered low risk if all questions within the domain were answered with “yes” or “probably yes”. The overall risk of bias was rated as low when all domains were considered low risk, and high risk when at least one domain was considered high risk. Two investigators (Yuanhong Du, Chengjing Guan) independently assessed the risk of bias based on Probast and cross-checked their assessments. In cases of disagreement, a third investigator was consulted to assist in resolving the discrepancies.

Data synthesis and statistical analysis

We performed a meta-analysis for evaluation metrics (c-index and accuracy) of machine learning models. In cases where the c-index lacked 95% confidence intervals (CI) and standard errors, we followed the approach described in Debray et al. (2019) to estimate standard errors. Given the discrepancies in the included variables and inconsistent parameters among the machine-learning models, we primarily employed the random effects model for the c-index meta-analysis. In addition, a bivariate mixed-effect model was used for the meta-analysis of sensitivity and specificity. The meta-analysis of this study was performed in R4.2.0 (R Core Team, 2022).

Study selection

The study selection process is described in Fig. 1. Initially, we identified 1,229 distinct records, subsequently evaluating 54 full-text papers. Ultimately, we incorporated 29 studies that met our inclusion criteria. For a comprehensive list of excluded studies that satisfied the inclusion criteria, please refer to the Supplemental Material. Figure 1 provides a visual representation of the specific screening process.

Characteristics of included studies

The 29 included studies covered 13 different types of models (NB (naive Bayes), RF (random forest), CNN (convolutional neural networks), LR (logistic regression), SVM (support vector machine), ANN (Artificial Neural Networks), GBDT (gradient boosting decision tree), DT (decision tree), DL (deep learning), XGBoost (eXtreme gradient boosting), AdaBoost (adaptive boosting), KNN (k-nearest neighbor)), with a total of 860,852 participants. These studies comprised one cohort study and 28 case-control studies retrieved from four databases: PubMed (n = 185), Embase (n = 357), Web of Science (n = 643), and Cochrane (n = 44). There were 13 studies from the United States, 13 from China, one from the Netherlands, one from Singapore, and one from Belgium within the included literature. See Table 1 for characteristics of included studies.

In this study, we utilized 86 models for training, encompassing NB, RF, CNN, LR, SVM, ANN, GBDT, DT, DL, XGBoost, AdaBoost, KNN, and Time Series Models. Additionally, we employed 65 models for testing, which included NB, RF, CNN, LR, SVM, ANN, GBDT, DT, XGBoost, and AdaBoost. The variable screening and feature selection methods primarily relied on various techniques, such as LASSO regression, stepwise regression, variable importance analysis, univariate and multivariate logistic regression models, the XGBoost package in Python, feature importance assessment, Nadam optimizer, bootstrapped backward elimination, least absolute shrinkage, least-squares-fit linear equations, multivariable linear and logistic regression, and backward selection. Furthermore, we employed ten distinct testing models, namely NB, RF, CNN, LR, SVM, ANN, GBDT, DT, XGBoost, and AdaBoost.

Quality of evidence and risk of bias

The high risk is mainly concentrated in the research object and the part of statistical analysis. This is predominantly attributed to several factors, including a certain number of case-control studies in the sample source of the included articles, the design of the original study sample size in the statistical analysis part, the way of screening the prediction factors, and the consideration of over-fitting, under-fitting, and optimal fitting of the prediction model.

Among the 86 included models, the machine learning models that relied on demographic information like gender, age, and clinical examination data, including creatinine, central venous pressure, and urine volume, exhibited a reasonable approach. These models were particularly well-suited for analyzing records that contained essential sample characteristics, enhancing the reliability of case records. In the process of data generation, five studies used multivariate imputation to deal with missing values.

In the modeling process, the validation set primarily consists of internal validation generated through random sampling. Additionally, among the 10 studies conducted, there is a subset of studies that include independent external validation sets. Figure 2 shows the probast quality evaluation results.

Results of individual studies

C-index

The systematic review included 29 eligible machine learning models, totaling 86 Models, with 13 types of models in the training set and 10 types of models in the validation set. In this meta-analysis, the training set related to C-index involves 16 articles, including 86 models, resulting in a final combined result of 0.767 (95% CI [0.746, 0.788]), while the validation set related to C-index involves 27 articles, including 57 models, with the combined result of 0.773 (95% CI [0.741, 0.804]) (see Fig. 3). Among the individual models included, the training set encompassed five NB, 12 RF, five CNN, 20 LR, seven SVM, eight ANN, 10 GBDT, two DT, five DL, six XGBoost, two AdaBoost, 2 KNN, and two Time Series Models. The validation set included 13 RF, 13 LR, three SVM, four ANN, 12 GBDT, one DT, four DL, four XGBoost, and three AdaBoost models. For a comprehensive overview of the C-index results for individual models and the overall C-index results, please refer to Table 2.

Table 2:

The outcome of the C-index.

	Train		Test
	Number	cindex (95% CI)	Number	cindex (95% CI)
Model
NB	5	0.660 (0.550∼0.770)
RF	12	0.775 (0.732∼0.817)	13	0.791 (0.741∼0.841)
CNN	5	0.770 (0.710∼0.830)
LR	20	0.750 (0.716∼0.784)	13	0.715 (0.676∼0.753)
SVM	7	0.737 (0.645∼0.829)	3	0.638 (0.555-0.721)
ANN	8	0.783 (0.693∼0.873)	4	0.780 (0.641∼0.919)
GBDT	10	0.805 (0.749∼0.861)	12	0.787 (0.727∼0.848)
DT	2	0.843 (0.627∼1.000)	1	0.872 (0.844∼0.900)
DL	5	0.823 (0.757∼0.888)	4	0.824 (0.739–0.910)
XGBoost	6	0.792 (0.737∼0.846)	4	0.823 (0.778∼0.868)
AdaBoost	2	0.793 (0.767∼0.818)	3	0.832 (0.804∼0.860)
KNN	2	0.631 (0.626–0.635)
Time Series Model	2	0.810 (0.806∼0.814)
Overrall	86	0.767 (0.746∼0.788)	57	0.773 (0.741∼0.804)

DOI: 10.7717/peerj.16405/table-2

Sensitivity and specificity

The training set related to sensitivity encompasses 18 articles, with a sensitivity of 0.66 (95% CI [0.59, 0.73]). In contrast, the sensitivity related to the validation set involves 10 articles, resulting in a sensitivity of 0.73 (95% CI [0.68, 0.77]).

As for specificity, the training set consists of 18 articles with a specificity of 0.83 (95% CI [0.78, 0.87]), while the validation set related to specificity includes 10 articles, yielding a specificity of 0.75 (95% CI [0.71, 0.79]) (see Figs. 4 and 5).