Machine learning outperforms traditional logistic regression and offers new possibilities for cardiovascular risk prediction: A study involving 143,043 Chinese patients with hypertension

Introduction

Cardiovascular disease (CVD) is the leading burden of disease in China, with a prevalence of approximately one in five adults and accounting for more than 40% of the total deaths (1, 2). According to the latest CVD report in 2018, there were 290 million CVD patients in China, including 1.3 million cases of stroke and 1.1 million cases of coronary heart disease (CHD) (1).

Cardiovascular disease (CVDs) is highly preventable—it was estimated that up to 90% of CVDs could be prevented (3, 4). Early prevention and screening for high-risk populations are key strategies for reducing the burden of CVDs (3, 5). At the individual level, some risk factors for CVDs have been well-established, such as smoking, blood pressure, diabetes and obesity, air pollution, and social determinants including health system and health policies (6). As early as 1996, the concept of matching the intensity of risk factor management to the hazard of CVDs was first proposed (7). Currently, there is an increasing emphasis on stratifying the risk of CVDs to guide the prevention and treatment schedules (8–10).

Several CVD risk assessment tools have been developed from different populations, such as the American College of Cardiology/American Heart Association atherosclerotic cardiovascular disease (ASCVD) pooled cohort equations, the Framingham Risk Score, the Systematic Coronary Risk Evaluation in Europe, the Global Registry of Acute Coronary Events acute coronary syndrome (ACS) risk and mortality calculator, the Thrombolysis In Myocardial Infarction Risk Score, and the QRISK in the United Kingdom and Scottish ASSIGN risk score (11–16). The applicability of these tools in China, however, is limited by the fact that they originated from Western countries, where the disease pattern of CVDs may substantially differ from that of China. Currently, Chinese guidelines recommend a simplified risk scoring table with common predictors including age groups, low-density lipoprotein cholesterol (LDL-C) or total cholesterol (TC), smoking, body mass index (BMI), blood pressure, hypertension, and diabetes mellitus (DM) to predict ASCVD risks (10). Some other CVD risk prediction tools are also available for the Chinese population, such as the in-hospital mortality risk prediction tool for ACS patients, the 10-year ASCVD risk prediction tool from the China-PAR Project, the CVD-death risk prediction tool, and the 5-year CVD risk prediction tool for patients with DM (17–20). To date, there is no prediction tool specific to Chinese patients with hypertension and the accuracy of CVD risk assessment remains an issue of concern (21, 22).

Machine learning (ML), a technique that allows computer systems to learn from data and effectively perform a specific task without explicit instructions, offers an alternative approach to predict an individual’s CVD risk (23). Previous studies have demonstrated that ML can significantly improve the model performance and the accuracy of the CVD risk prediction (24, 25). Given the present availability of individual-level data in China, ML is also expected to improve the CVD risk prediction for the Chinese hypertensive population.

On this basis, our study aimed to explore the potential of using ML to predict CVD risk for the Chinese hypertensive population based on the information routinely collected in clinical settings and to evaluate the performance of each ML model.

Materials and methods

Data source

The study dataset was based on the Microalbuminuria Screening in Hypertensive Patients Project, a multicenter, cross-sectional study initiated by the China International Exchange and Promotive Association for Medical and Healthcare. A total of 143,043 inpatients and outpatients with hypertension from 600 tertiary, secondary, or community hospitals in China from November 2016 to August 2017 were included in the analysis. The flowchart of patient selection is shown in Figure 1. The project was carried out in accordance with the Good Pharmacoepidemiology Practice. All patients signed informed consent before participation. The study was approved by the Ethical Committee of Peking University People’s Hospital [2013–17].

Figure 1. Flow diagram showing numbers of included and excluded patients.

Variables

Eighteen variables were included to develop the models, including demographics (gender, age, and geographic information), health indicators [body mass index (BMI), systolic blood pressure (SBP), diastolic blood pressure (DBP), carotid artery thickening, left ventricular hypertrophy, history of smoking, and family history of hypertension], results of laboratory tests [levels of low-density lipoprotein cholesterol (LDL-C) and urine albumin to creatinine ratio (UACR)], current treatment regimen, type of visit (outpatient or inpatient), whether hypertension was newly diagnosed, whether UACR was newly detected, presence of diabetes, and length of use of renin-angiotensin system inhibitor (RASI). Categorical variables were further converted into dummy variables. Microalbuminuria was defined as UACR between 30 and 300 mg/g.

Machine learning algorithms

The primary outcome was CVD events defined as the development of CHD and/or stroke. To identify the best model performance, a total of six machine learning methods were compared, including the Logistic Regression (LR) with Elastic-Net, Random Forest, XGBoost, Support Vector Machine, Deep Learning, and Ensemble models. The LR model was used as the benchmark reference. These algorithms were selected based on the ease of implementation into current datasets.

Each algorithm was trained and evaluated using 5-fold cross-validation. Specifically, the final data sample was randomly stratified and split into the modeling dataset (80%) and the hold-out dataset (20%). Then the modeling dataset was randomly stratified into five evaluation folds/samples of equal size: one evaluation fold as the test dataset and the other four evaluation folds as the training dataset. The area under receiver operating characteristic curves (AUROCs) and area under precision-recall curves (AUPRCs) were used to evaluate the performance of the algorithms.

The hold-out dataset was used to evaluate the optimal threshold for predicting short-term CVD events in patients with hypertension. Two strategies, respectively, from a clinical perspective and a data science perspective, were used to explore the effect of prediction threshold values on false positive/negative cases and rates. We chose three clinical thresholds, 0.05 (T1), 0.10 (T2), and 0.15 (T3), based on the high-risk rating for ASCVD within 10 years from the 2016 Chinese Adult Dyslipidemia Prevention and Treatment Guideline (26) and the extreme high-risk rating of ASCVD from the 2013 ACC/AHA Guideline on the Assessment of Cardiovascular Risk (27) and the European SCORE (28). Two additional thresholds were chosen from a data science perspective—the first was set to fix the sensitivity at 90% (T4); the second was ROC01, which was designed to minimize the distance between the ROC plot and the point (0,1) to balance sensitivity and specificity (T5).

All analyses were performed using Python 3.6. P values of less than 0.05 were considered statistically significant.

Results

Study population characteristics

Briefly, a total of 143,043 patients were included in the analysis, of whom 26.2% were newly diagnosed cases, 30.1% controlled their blood pressure by treatment, and 21.6% experienced at least one CVD event. The patient characteristics are shown in Table 1.

Table 1. Characteristics of included patients.

Model performance comparison

The performance of all algorithms was reasonably good (Figure 2 and Table 2), indicated by AUROCs between 0.7 and 0.8. The Ensemble model was the best performing model with the highest AUROC, compared with the LR model [Ensemble model: AUROC = 0.760, 95% confidence interval (CI): 0.754–0.766]. The machine learning models generally outperformed the LR model (Figure 2 and Table 2). However, the LR with Elastic-Net model showed poorer performance compared to the LR model (LR: AUROC = 0.737, 95% CI: 0.730–0.742; LR with Elastic-Net: AUROC = 0.728, 95% CI: 0.723–0.733). The Ensemble model was also the best performing model based on AUPRC (AUPRC = 0.465, 95% CI: 0.454–0.475, Figure 3 and Table 3).

Figure 2. Receiver operating characteristic curves of all models.

Table 2. Model performance measured by area under receiver operating characteristic curve.

Figure 3. Precision-recall curves of ensemble model and traditional logistic regression model.

Table 3. Model performance measured by area under precision-recall curves.

Error and sensitivity analysis of the ensemble model

We compared the Ensemble model with the LR model to determine the optimal threshold for classifying patients as likely or unlikely to develop CVDs in a short term. Table 4 shows the results of altering the threshold on false positive rates and false negative rates. T1 was the most sensitive threshold where the Ensemble model incorrectly classified 32 patients as low risk of developing CVDs (LR = 99.03%; Ensemble = 99.48%). The most specific threshold was T5 where the Ensemble model misclassified 7,075 patients as high-risk (LR = 65.84%; Ensemble = 68.45%). Of all the thresholds tested, T5 showed the best performance because it misclassified the least number of patients (LR = 9,505; Ensemble = 8,995), compared with T1 which misclassified the greatest number of patients (LR = 21,259; Ensemble = 21,100). Overall, using T5 on the Ensemble model showed the best precision, with correct identification of 37.7% of patients (Figure 4). However, this 8.6% increase in precision of using T5 over T4 came at a 20% reduction in sensitivity.

Table 4. Sensitivity analysis of Ensemble model and its comparison with benchmark.

Figure 4. Precision based on different thresholds.

Discussion

Although CVDs is the leading burden of disease in China that affects 290 million people (1, 2), up to 90% of CVDs can be prevented (3–5). There is an increasing emphasis on risk stratification for CVDs to guide the prevention and treatment schedules (8–10).

This study is a cross-sectional survey of hypertensive patients with or without CVDs. In this study, we established ML models for predicting CVD risk based on a large-sample database with comprehensive clinical information. The ML models, except for LR with Elastic-Net model, all showed better performance than traditional LR, as measured by both AUROC and AUPRC. Since the AUROCs are between 0.7 and 0.8, the performance of all models can be considered reasonably good. In addition to AUROC, we also calculated AUPRC as the model performance measure. In our data, 112,140 (78.4%) patients were non-CVD as negative cases. AUPRC is, therefore, suitable to measure the model performance. Both AUROC and AUPRC showed that the Ensemble model was the best performing model.

A low threshold can help identify as many patients as possible who would develop CVDs in a short term. In contrast, the optimal T5 threshold will reduce misclassification and minimize patients’ overall anxiety about future risk. In practice, which threshold should be chosen also depends on physician preference.

Previous CVD risk assessment models were mainly based on the simplest parameters, such as blood pressure, age, lipids, BMI, and drug treatment. The atherosclerotic cardiovascular disease (ASCVD) cohort equation of ACC/AHA in the United States, the Framingham risk score, the European Systemic coronary risk assessment model (10–16), the 10-year ASCVD risk prediction tool of the PAR project in China, and the CVD death risk prediction tool (17–20) are all in this case. Compared with the previous models, our machine learning prediction model adds some new parameters commonly detected in patients with hypertension at present, including blood glucose, creatinine, microalbuminuria, and carotid intima-media thickening, so our model has good availability and can provide more accurate risk predictions of CVD. In addition, the present study was based on a large-sample cross-sectional investigation, which provided sufficient statistical power for the construction of the machine learning model.

There are several limitations in this study, mainly due to issues related to data quality. First, the number of variables collected at baseline was limited. Only eighteen variables were collected by the survey, which may not cover all risk factors associated with CVDs. Nevertheless, our models with these variables showed reasonably good performance indicated by AUROC over 0.7. Second, some variables were not clearly defined during data collection. For example, in terms of the blood pressure readings, we were unable to determine whether the measurements were before or after medication, which may cause heterogeneity and affect the accuracy of the effect estimates. Thirdly, several key variables were collected through open-ended questions. This resulted in a high rate of missing values. For example, nearly 90% of participants did not provide valid information on the duration of hypertension.

Our model can be adopted in healthcare settings where key clinical information is available. In future work, we will improve the model by collecting additional information related to cardiovascular risk and comparing it with other models being developed. We will validate the model with other datasets to evaluate its generalizability. We are also following up a part of patients and plan on adding microalbuminuria and other new parameters through a large external cohort to verify the accuracy of the machine learning model. In addition, disease prediction models trained by longitudinal data may predict long-term CVD risks, to guide CVD prevention.

Data availability statement

The original contributions presented in this study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Ethics statement

The studies involving human participants were reviewed and approved by Ethical Committee of Peking University People’s Hospital. The patients/participants provided their written informed consent to participate in this study.

Author contributions

All authors listed have made a substantial, direct, and intellectual contribution to the work, and approved it for publication.

Funding

This project was sponsored by Sanofi, China. The funder has no role in study design, data collection and management, and statistical analysis. Funding was not contingent upon the publication of the manuscript. No patents have been applied for relating to the content of the manuscript. The authors are responsible for all content and editorial decisions and did not receive honoraria related to the development of this publication.

Acknowledgments

The authors gratefully acknowledge the participation of all patients and the contribution of the investigators. Medical writing support under the authors’ direction was provided by Zheng Yin, Ph.D. and Xinran Zhao, MHS (IQVIA), according to Good Publication Practice guidelines.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References