Machine Learning for Prediction of Technical Results of Percutaneous Coronary Intervention for Chronic Total Occlusion

(1) Background: The probability of technical success in percutaneous coronary intervention (PCI) for chronic total occlusion (CTO) represents essential information for specifying the priority of PCI for treatment selection in patients with CTO. However, the predictabilities of existing scores based on conventional regression analysis remain modest, leaving room for improvements in model discrimination. Recently, machine learning (ML) techniques have emerged as highly effective methods for prediction and decision-making in various disciplines. We therefore investigated the predictability of ML models for technical results of CTO-PCI and compared their performances to the results from existing scores, including J-CTO, CL, and CASTLE scores. (2) Methods: This analysis used data from the Japanese CTO-PCI expert registry, which enrolled 8760 consecutive patients undergoing CTO-PCI. The performance of prediction models was assessed using the area under the receiver operating curve (ROC-AUC). (3) Results: Technical success was achieved in 7990 procedures, accounting for an overall success rate of 91.2%. The best ML model, extreme gradient boosting (XGBoost), outperformed the conventional prediction scores with ROC-AUC (XGBoost 0.760 [95% confidence interval {CI}: 0.740–0.780] vs. J-CTO 0.697 [95%CI: 0.675–0.719], CL 0.662 [95%CI: 0.639–0.684], CASTLE 0.659 [95%CI: 0.636–0.681]; p < 0.005 for all). The XGBoost model demonstrated acceptable concordance between the observed and predicted probabilities of CTO-PCI failure. Calcification was the leading predictor. (4) Conclusions: ML techniques provide accurate, specific information regarding the likelihood of success in CTO-PCI, which would help select the best treatment for individual patients with CTO.


Introduction
Despite progressive declines in cardiovascular mortality, coronary artery disease (CAD) remains the leading cause of death in developed countries [1]. For CAD, accumulated evidence has led to the standardization of treatment selection among percutaneous coronary intervention (PCI), coronary artery bypass grafting, or optimal medical treatment alone. However, due to the broad and heterogeneous spectrum of CAD patients, complex cases should be discussed individually to identify the optimal solution for each specific patient. The probability of technical success for CTO-PCI represents essential information for specifying the priority of PCI in preprocedural discussions regarding treatment selection for CAD patients with CTO. Indeed, numerous scores for predicting CTO-PCI results have been derived based on regression analyses [2][3][4][5][6]. Nevertheless, the predictive ability of those scores remains modest at best [7,8], leaving room for improvements in model discrimination.
Machine learning (ML) techniques have emerged as highly effective methods for prediction and decision-making in a multitude of disciplines, including internet search engines, customized advertising, finance trending, and natural language processing [9,10]. When the goal is to generate a model that most accurately predicts an outcome, ML algorithms can prove quite advantageous over traditional regression methods. To date, the benefits of utilizing ML for predicting the technical results of CTO-PCI have not been evaluated on a large scale.
We therefore investigated the feasibility and accuracy of ML models for predicting the technical outcomes of CTO-PCI and compared their performances to the results from existing scores, including J-CTO [2], CL [4], and CASTLE [5] scores.

Study Population
This analysis used data from the Japanese CTO-PCI expert registry. This registry is a prospective, non-randomized study enrolling consecutive patients who are undergoing CTO-PCI performed by 46 highly experienced Japanese operators, all certified by the Japanese Board of CTO Interventional Specialists.
The requirements for certification are that the PCI operator has performed more than 300 CTO-PCIs and performs more than 50 CTO-PCIs per year. Certified specialists need to enrol all consecutive CTO-PCI datasets into the registry. The planned patient enrollment is from January 2014 to December 2022, and clinical follow-up will continue until December 2027. The design and enrollment status have been reported in detail [11,12]. Notably, an independent body of researchers (Clinical Research Center, Kurashiki Central Hospital, Ohara Healthcare Foundation, Okayama, Japan) monitors and controls data analysis, and procedural-related images (PCI angiograms, computed tomography images, and intravascular ultrasound images) are all uploaded into the central server of the core laboratory (Cardiovascular Imaging Center, Aichi, Japan) where independent physicians and technicians validate the content. This study protocol was approved by the review board of each institution, and written informed consent was obtained from all participants.
The study population was randomly divided into a training set (80%), from which ML models for predicting CTO-PCI results were derived, and a test set (20%), in which ML models and the existing scores were evaluated.

Definitions and Study Endpoint
Hyperlipidemia was defined as a total cholesterol level ≥220 mg/dL, a low-density lipoprotein cholesterol level ≥140 mg/dL, a high-density lipoprotein cholesterol level <40 mg/dL, a triglyceride level ≥150 mg/dL, or treatment for hyperlipidemia. Hemodialysis was defined as undergoing regular hemodialysis. The definition of CTO and the angiographic analysis of the target procedures have already been described [11,12]. The indication for CTO-PCI was completely left to the discretion of each operator and the discussion among the heart team of each institution. The selection of a CTO-PCI strategy depended on the operator's discretion. The definitions of predictor variables for angiographic findings are provided in Supplemental Table S1. Viable CTO territory was defined as the presence of viability of myocardium in the perfusion territory of the target CTO lesion based on the findings of imaging modalities such as echocardiography, single photon emission computed tomography, cardiovascular magnetic resonance, or left ventriculography. Technical success was defined as successful guidewire CTO achieving <50% residual diameter stenosis without major side branch occlusion and thrombolysis in myocardial infarction flow grade 3. According to CTO-ARC consensus recommendations [13], in-hospital major adverse cardiovascular event included any of the following adverse events prior to hospital discharge: death, myocardial infarction, or clinically driven target vessel revascularization with PCI or coronary artery bypass grafting. Procedural success was defined as technical success plus the absence of an in-hospital major adverse cardiovascular event.

Predictor Variables
To ensure the availability of all predictor variables in prediction model development, we excluded variables with a missing data rate exceeding 20%. Missing values were filled with the median and mode of each continuous and categorical variable, respectively. To handle overfitting with regularization, continuous variables were normalized by z-scoring so that each continuous variable had both a mean of zero and a standard deviation of one. Multicategory variables were one-hot encoded in binary variables. Finally, a total of 65 predictor variables consisting of clinical and angiographic characteristics were used as independent predictor variables for model development.

ML Algorithm Models
To develop the prediction model for technical failure of CTO-PCI, we applied and compared the performances of 5 ML classifiers that are widely used in the literature: random forest; extreme gradient boosting (XGBoost); deep neural networks; support vector machine classifier; and L2-regularized logistic regression. For hyperparameter selection, a stratified 10-fold cross-validation and grid search was performed. The ranges of optimized hyperparameters for each classification algorithm are provided in Supplemental Table S2.

Comparison of Results from ML Models and Conventional Prediction Scores
We compared the performance of the developed ML algorithms with standard predictive multivariate logistic regression models: J-CTO, CL, and CASTLE scores. With the CASTLE score [5], the score component of "tortuosity" was defined as either 2 or more pre-occlusive bends of >90 • or at least one bend of >120 • in the CTO vessel. Because of the absence of identical findings obtained in the current registry, we used the finding "lesion bending", defined as at least one bend of >45 • throughout the occluded segment, as a substitute.
To compare ML models with those existing scores, we evaluated the existing scores directly on the test dataset, essentially performing an external validation of the prediction rules. However, comparing the external performance of those regression-based scores with the internal performance of ML algorithms could provide an unfair advantage to the ML algorithms. We therefore further developed a prediction score for technical failure of CTO-PCI in the training dataset using multivariate logistic regression analysis in a similar way with the existing scores. Potential predictive factors for CTO-PCI failure showing values of p < 0.005 in the univariate model were entered into the multivariate analysis. An integer scoring system (the CURRENT score) was developed by assigning points for each strong and independent predictor according to the beta coefficient and summing all points accrued. We also compared the predictive performance of ML models with that of the CURRENT score.

Evaluation Metrics
The models were evaluated in the test dataset, which was independent from the training dataset. Receiver operating characteristics (ROC) and precision/recall (PR) curve analysis were performed to assess the discriminatory ability of each ML model and the conventional prediction scores. Pairwise comparisons of the area under the ROC curves (ROC-AUC) were performed as described by Delong et al. [14].
Calibration of the best model (XGBoost) was evaluated using the Brier score method (range, 0-1) [15] and a figure comparing the observed and predicted risk of CTO-PCI failure.

Variable Importance
We also computed the variable importance of the best model (XGBoost) by measuring the average gain of splits using the variable across all decision trees within the model.

Software
Model development codes were developed in Python 3.6.6 (Python Software Foundation, Wilmington, DE, USA). The open-source library scikit-learn was used for the implementation of ML classifiers. The XGBoost 0.90 was used to build the XGBoost model.

Statistical Analysis
Data were statistically analyzed using SPSS Statistics version 24 (IBM, Armonk, NY, USA) and Medcalc version 20.110 statistical program (Medcalc, Ghent, Belgium). Continuous variables are presented as mean ± standard deviation. Categorical data are presented as frequencies and percentages. Normality was evaluated using the Shapiro-Wilk test. Normally distributed values were compared by unpaired t-test, and non-normally distributed values were compared by the Mann-Whitney U test. Categorical data were compared using the χ 2 test or Fisher's exact test.
We used logistic regression models for the training dataset to extract the score component of the CURRENT score by uni-and multivariate analyses. Given many variables, strong and independent predictor variables were identified using a stepwise approach with p < 0.005 as the inclusion criterion. All statistical tests were two-tailed and values of p < 0.05 were considered significant.

Patient Characteristics
Among the 8760 CTO-PCI procedures performed between January 2014 and December 2019, technical success was achieved in 7990, representing an overall success rate of 91.2%. Each patient was randomly assigned to either the training cohort (80%, 7008 procedures) or the test cohort (20%, 1752 procedures). Patient characteristics in the training and test datasets are shown in Table 1 and Supplemental Table S3. Except for sex, smoking, hemodialysis, and viability of CTO territory, no significant differences in clinical or lesion-related characteristics were identified between the training and test cohorts.
Each training and test cohort was divided according to the technical outcome, and patient characteristics were analyzed (Supplemental Tables S4 and S5). In univariate analyses for the training dataset, patients with failed CTO-PCI were significantly more likely to have the following clinical characteristics: hypertension; diabetes; prior CABG; prior PCI; chronic occlusive pulmonary disease; arteriosclerosis obliterans; higher serum creatinine; and lower estimated glomerular filtration rate.
On uni-and multivariate logistic regression analysis for the training dataset, 8 variables were identified as strong independent predictors of the failure of CTO-PCI, collectively forming the CURRENT score (       The Brier score for XGBoost was 0.074, indicating good calibration between the estimated predicted risk and observed risk of CTO-PCI failure. Calibration was also assessed by comparing estimated predicted and observed risk of CTO-PCI failure stratified by decile of predicted risk (Figure 3). A high correlation of predicted versus observed CTO-PCI failure was found (r = 0.97; p < 0.001). The Brier score for XGBoost was 0.074, indicating good calibration between the estimated predicted risk and observed risk of CTO-PCI failure. Calibration was also assessed by comparing estimated predicted and observed risk of CTO-PCI failure stratified by decile of predicted risk (Figure 3). A high correlation of predicted versus observed CTO-PCI failure was found (r = 0.97; p < 0.001).

Comparison of Prediction Models
(a) (b) Figure 2. Areas under the ROC (a) and PR (b) curves comparing the best ML model with existing prediction scores for CTO-PCI failure. Abbreviations: CTO, chronic total occlusion; ML, machine learning; PCI, percutaneous coronary intervention; PR, precision/recall; ROC, receiver operating characteristics; XGBoost, extreme gradient boosting.
The Brier score for XGBoost was 0.074, indicating good calibration between the estimated predicted risk and observed risk of CTO-PCI failure. Calibration was also assessed by comparing estimated predicted and observed risk of CTO-PCI failure stratified by decile of predicted risk (Figure 3). A high correlation of predicted versus observed CTO-PCI failure was found (r = 0.97; p < 0.001).

Variable Importance
The importance matrix plot for XGBoost is shown in Figure 4. The first 6 variables contributing to the predictive performance of the XGBoost model were as follows: calcification; hyperlipidemia; reattempted by another operator; CTO distal diameter; lesion bending; and hemodialysis.

Variable Importance
The importance matrix plot for XGBoost is shown in Figure 4. The first 6 variables contributing to the predictive performance of the XGBoost model were as follows: calcification; hyperlipidemia; reattempted by another operator; CTO distal diameter; lesion bending; and hemodialysis.

Discussion
This study had two major findings. First, XGBoost was our best-performing ML model for predicting CTO-PCI results. Second, XGBoost showed significantly better performance than existing scores for predicting the technical outcomes of CTO-PCI.

Discussion
This study had two major findings. First, XGBoost was our best-performing ML model for predicting CTO-PCI results. Second, XGBoost showed significantly better performance than existing scores for predicting the technical outcomes of CTO-PCI.
To the best of our knowledge, the present study represents the first large-scale, multicenter evaluation of ML for predicting the technical results of CTO-PCI. ML techniques provide accurate, specific information regarding the likelihood of success in CTO-PCI, which would optimize treatment selection for CAD patients with CTO in preprocedural discussions.

Prediction Accuracy of CTO-PCI Results
In the decision-making process for treatment selection when managing patients with complex CAD, recent revascularization guidelines have advocated a 'Heart Team' approach, referring to non-invasive cardiologists, anesthetists, and other specialists if deemed necessary [16]. A Heart Team approach facilitates more transparent decision-making but requires specific and accurate information regarding the likelihood of a successful result for each candidate's procedural treatment instead.
To date, numerous risk prediction models for the results of CTO-PCI have been developed based on regression analysis, but the accuracy of those scores is modest at best [2,6]. Attempts to create new or additional scores have thus been made by integrating procedural algorithms [3] or increasing the number of patients included [5]. However, the predictive ability of scores has not been markedly improved through such efforts [7,8], emphasizing the need for improvements in model discrimination.

Advantages of ML Methods
ML models have been shown to work well when provided with large amounts of data [17][18][19][20], and the current registry provided data from 8760 procedures and 65 variables. Moreover, ML methods usually offer incremental gains in predictive performance while handling vast numbers of variable-variable interactions in each patient, effectively individualizing risk assessments and overcoming many limitations of standard statistical approaches using regression-based analysis [21]. In conventional prediction models for CTO-PCI results, most score components comprise angiographic findings of the CTO lesion and, in particular, the finding of severe calcification has been the most consistently included variable. However, the decision tree for CTO-PCI success in our previous report [22] showed that, among patients suffering CTO with severe calcification, no other angiographic findings affected CTO-PCI results. Additionally, a recent report showed apparent differences in the performance of prediction scores according to the procedural techniques applied, with higher predictability for patients who underwent CTO-PCI with antegrade-only procedures compared to those with bidirectional procedures [7]. Moreover, in the current study, hyperlipidemia was one of the leading predictors of CTO-PCI results. Hyperlipidemia has not been included as a score component among recently developed scoring systems based on regression analysis to gauge the likelihood of success in CTO-PCI. However, our previous report [11] showed that hyperlipidemia was an independent predictor of successful CTO-PCI in a primary retrograde approach, but not in overall or primary antegrade procedures. The effects of statin treatment on endothelium-mediated responses [23] and collateral development [24,25] of the coronary arteries might be beneficial for collateral channel crossing and retrograde procedures. Such findings suggest complex variable-variable interactions in clinical data for CTO-PCI, indicating incremental predictive performance by using ML methods, particularly for tree-based models.

Disadvantages of ML Methods
ML methods are usually more time-consuming than conventional regression analysis. Further, attention should be paid to the interpretation of the results of ML models. Important predictor variables may not be causal factors but just useful markers. The conversion to a points-based score based on coefficients for each variable obtained from conventional regression analysis cannot be applied to many ML methods.
To date, prediction models for CTO-PCI results have been developed to be as simple as possible, prioritizing the ease of remembering and calculation [2][3][4][5][6]. However, unlike those traditional scores from regression analysis, ML models require a computer for calculation and cannot be converted to a bedside arithmetical risk score. While the need to favor simplicity over accuracy might have been reasonable in the past, such considerations are no longer relevant within computerized medical care. Sufficient simplicity to be handcalculable would not be acceptable if the trade-off were the sacrifice of accuracy in the prediction model providing critical information for treatment selection.

Future Directions
As described in the original report of the J-CTO score [2], which was originally developed to predict guidewire crossing within 30 min and remains the most widely applied score for technical results, clinical prediction systems should be continually updated to improve predictive performance by handling new data and optimizing algorithms. Recently, prediction models for CTO-PCI results, including coronary computed tomographic angiography (CCTA) findings, have shown relatively high performance [26,27]. CCTA offers advantages over CAG for direct visualization of CTO vessel trajectory, three-dimensional depiction of lesion bending, the distribution of calcification, and the presence of multiple occlusions. CCTA was not routinely obtained in the current registry, and ML prediction models for CTO-PCI results using CCTA findings have not yet been developed using a large-scale dataset. However, such analyses should be carried out in the future. Although experienced specialists have interpreted CCTA findings for developing prediction models, ML models such as neural networks might facilitate image interpretation and improve the predictive performance based on much larger datasets [28,29].

Limitations
This study has several potential limitations. First, the developed ML models have not been externally validated on a separate cohort. Second, in the CASTLE score [5], the score component of "tortuosity" was defined as either two or more pre-occlusive bends of >90 • or at least one bend of >120 • in the CTO vessel. Because of the absence of identical findings obtained in the current registry, we used the finding of "lesion bending", defined as at least one bend of >45 • throughout the occluded segment, as a substitute. This may have resulted in an underestimation of the performance of the CASTLE score. Finally, all CTO-PCIs were performed by highly experienced specialists, and the results may not be generalizable to the daily clinical practice of less-experienced operators. However, previous consensus reports on CTO have specified that CTO-PCI should be aggressively referred to a skilled operator [30]. As recent guidelines indicate, success rates for CTO-PCI are strongly associated with operator skillset, procedural volume, and the availability of dedicated equipment [16].

Conclusions
ML techniques improve the prediction of technical results of CTO-PCI. These techniques may help select the best treatment for individual patients with CTO in the standardized preprocedural discussion.
Supplementary Materials: The following supporting information can be downloaded at: https: //www.mdpi.com/article/10.3390/jcm12103354/s1; Table S1: Definitions of the predictor variables for angiographic findings; Table S2: The ranges of optimized hyperparameters for machine learning models; Table S3: Patient characteristic in the training and test cohort; Table S4: Patient characteristics in the training cohort; Table S5: Patient characteristics in the test cohort.