Risk prediction models for atherosclerotic cardiovascular disease: A systematic assessment with particular reference to Qatar

Background: Atherosclerotic cardiovascular disease (ASCVD) is a common disease in the State of Qatar and results in considerable morbidity, impairment of quality of life and mortality. The American College of Cardiology/American Heart Association Pooled Cohort Equations (PCE) is currently used in Qatar to identify those at high risk of ASCVD. However, it is unclear if this is the optimal ASCVD risk prediction model for use in Qatar's ethnically diverse population. Aims: This systematic review aimed to identify, assess the methodological quality of and compare the properties of established ASCVD risk prediction models for the Qatari population. Methods: Two reviewers performed head-to-head comparisons of established ASCVD risk calculators systematically. Studies were independently screened according to predefined eligibility criteria and critically appraised using Prediction Model Risk Of Bias Assessment Tool. Data were descriptively summarized and narratively synthesized with reporting of key statistical properties of the models. Results: We identified 20,487 studies, of which 41 studies met our eligibility criteria. We identified 16 unique risk prediction models. Overall, 50% (n = 8) of the risk prediction models were judged to be at low risk of bias. Only 13% of the studies (n = 2) were judged at low risk of bias for applicability, namely, PREDICT and QRISK3.Only the PREDICT risk calculator scored low risk in both domains. Conclusions: There is no existing ASCVD risk calculator particularly well suited for use in Qatar's ethnically diverse population. Of the available models, PREDICT and QRISK3 appear most appropriate because of their inclusion of ethnicity. In the absence of a locally derived ASCVD for Qatar, there is merit in a formal head-to-head comparison between PCE, which is currently in use, and PREDICT and QRISK3.


INTRODUCTION
Atherosclerotic cardiovascular disease (ASCVD) is common and results in considerable morbidity, impairment of quality of life and mortality. 1,2 A number of ASCVD prediction models have been developed, which were typically being used to generate risk scores over a 10-year period. Some more recent models have estimated longer-term risks. 3 The first ASCVD model was the Framingham Risk Score. 4 Subsequent models have broadened the range of risk factors used to include, for example, social deprivation and ethnicity. 5,6 Many clinical guidelines have recommended the use of ASCVD prediction models 7 however, deciding on which model(s) to use is not straightforward, as there is a need to take into account background risk factors and the applicability of the model(s) to the target population. Furthermore, studies have shown that models do not necessarily align in terms of their risk estimation. 8 Choosing the optimal model is of crucial importance given that ASCVD is very prevalent and the leading cause of mortality in Qatar. 9,10 Between 2009 and 2019, the number of deaths attributable to ischemic heart disease increased substantially. Based on 2019 United Nation estimates that Qatar has a population of 2.8 million people. Approximately 12% are Qatari and 88% are foreign nationals. These foreign nationals come mainly from Asia, with smaller numbers from Africa and Europe.11 Ideally, a risk prediction model would be derived using data from the Qatari population; however, this is not available. Currently, Qatari experts have concluded that the American College of Cardiology (ACC)/American Heart Association (AHA) Pooled Cohort Equations (PCE) 12,13 should be used to evaluate ASCVD risk. This study sought to identify, assess the methodological quality of and compare the properties of established ASCVD risk prediction models through head-to-head comparisons of established risk prediction models for use in Qatar.

METHODS
We drew on guidance from the Cochrane Prognosis Methods Group, which produced the CHecklist for critical Appraisal and data extraction for systematic reviews of prediction Modeling Studies (CHARMS). 14 Detailed methods are available in the protocol for this review from PROSPERO with Registration no. CRD42020176981.

Search strategy
We built on the search strategies developed by Damen et al., 15 and Siontis et al., 16 which were modified by adding terms to capture newly developed risk prediction models (such as the PCE and GLOB-ORISK). 17,18 A sensitive search strategy was developed, and validated study design filters were applied to retrieve articles from MEDLINE (OVID), Embase (OVID) and CINAHL (Ebscohost) (Appendix 1). Databases were searched from 1 July 2013 to 31 July 2019 with earlier relevant studies being identified as per the studies retrieved by Damen et al., 12 and Siontis et al., 13 Additional studies were identified by searching references cited by included studies. Unpublished work and research in progress were identified through searches on Google and Google Scholar. All searches were undertaken in English. Study designs Retrospective and prospective cohort studies were eligible.

Study selection
All references were uploaded into the systematic review software DistillerSR. Study titles were independently checked by two reviewers according to the above selection criteria. Studies were eligible if they included at least two models in populations without preexisting cardiovascular disease. Full-text copies of potentially relevant studies were obtained, and their eligibility for inclusion was then independently assessed. Any discrepancies were resolved through discussion, and if necessary, a third reviewer was consulted.

Quality assessment
Quality assessments were independently carried out on each model by two reviewers using the Prediction Model Risk Of Bias Assessment Tool (PROBAST), which have specifically been developed for assessing the risk of bias (ROB) and the applicability of prediction modeling studies. 19 The latest version of the model was used, although earlier versions were often referred to for details of the derivation and validation cohort(s).

Data extraction, analysis, and synthesis
Two reviewers independently extracted data onto a customized data extraction sheet in DistillerSR. Methods for quantitatively synthesizing evidence from prognostic modeling studies have yet to be developed. We therefore undertook a detailed descriptive summary with data tables and narrative synthesis of data.

Discrimination
For each model, discrimination, defined as the ability of the prediction model to identify individuals who developed the event of interest from those who did not, was measured by recording the area under the receiver operating curve (AUROC or AUC). We used a threshold of .5% for each paired comparison to indicate a significant difference. 15 The C-statistic, D-statistic, Brier score, p values and 95% confidence intervals (95% CIs) were also recorded (key definitions are shown in Appendix 2). 15 Calibration Calibration performance, defined as the agreement between the observed and predicted risks, was established for each study by recording the number of observed and predicted events.

Risk reclassification
Where a risk prediction model was compared with another model, but an additional predictor had been added, risk reclassification analysis should be performed to quantify how well the new model reclassified the subjects. To report data, we recorded the net reclassification index (NRI) and the absolute net reclassification index. The NRI consists of two components, namely, subjects with events and those without events, and then scores subjects on whether their risk was reclassified as higher or lower or not changed using the new prediction model. 20 Outcome selection and optimism bias Some risk prediction models were developed for one cardiovascular outcome, but were then evaluated for a different outcome, which can introduce bias. We recorded the outcomes originally of interest and the outcomes from the models. Some models included investigated a new risk model. Optimism bias can be a relevant consideration in such cases, as the new models can perform better when initially described than older models, but this may not be applicable in subsequent comparisons. 21

Risk of bias
We critically appraised each unique risk calculator identified (n ¼16) ( ? ? (þ) ¼low risk of bias or applicability concern, (?) ¼unclear risk of bias or applicability concern, (-) ¼high risk of bias or applicability concern PROBAST Prediction model Risk of Bias ASsessment Tool scored at low ROB (n ¼12, 75%). The predictors, outcome and analysis were often at low ROB (n¼ 15, 94%; n ¼14, 87%; n ¼12, 75%), respectively. The key concern regarding analysis is related to handling of missing data. Only 25% of the studies (n ¼4) were at low ROB for participation selection in relation to the applicability concern, as it appeared that the study population did not reflect the Qatari population. All studies were at low ROB for applicability of predictors chosen; 87% (n ¼ 14) of studies were at low ROB for outcome measures in the applicability domain. Overall, 50% (n ¼8) of the risk prediction models were judged to be at low ROB. These models were the ACC/AHA (PCE), American CVH, FINRISK, FRS, Health 2000, PREDICT, PROCAM, and RRS. However, only 13% of the studies (n ¼2) were judged at low ROB for applicability concern, including QRISK and PREDICT ( Table 2). Only the PREDICT risk calculator were at low ROB in both domains.

Risk reclassification
Information on risk classification and reclassification was only available for four (10%) studies. In one study, 29
In nine studies, the outcome of interest was CHDrelated events 28,32,[35][36][37]42,44,49,51 ; in six studies, allstroke or ischemic stroke incidences were the outcome of interest. 27,[29][30]38,42,50 In one study, the outcome was the agreement with predicted CVD risk using Lin's concordance correlation coefficient. 21 Another study assessed differences in absolute risk of CV events, 25 and in one study, the outcome was defined as the comparison of ACC/AHA guidelines to the European Society of Cardiology/European Atherosclerosis Society Guidelines for Primary Prevention of ASCVD for accurately assigning statin therapy to those who would benefit. 48 Optimism bias  61 However, none of these differences exceeded 5%.

Principal findings
We systematically examined head-to-head comparisons of established risk prediction models for the primary prevention of ASCVD. Through this process, we identified 41 studies reporting on 16 unique cardiovascular risk prediction models that have been deployed in clinical practice. The majority (54%) of these models were derived from Europe and USA. None had been developed specifically for use in Arab populations. Careful comparisons of these models have shown the lack of overall consistent findings with studies showing in some cases comparable performance and in others superior or inferior performance. In some studies, new models appear to perform better than old established models, but these may be subject to optimism bias, and further studies are needed to verify these results.

Strengths and limitations
We performed a formal systematic comparison using state-of-the art methods. This study builds on  Table 4 continued  previous work that has been carried out in this field. 14,15 The newly developed PROBAST tool was used to assess both ROB and applicability of each calculator. 18 The limitations of this review include the fact that we may not have identified all relevant studies. In some studies, there was poor reporting of data, which made it difficult to assess study quality. Furthermore, as most studies were conducted in Europe or the USA, there were challenges in inferring which risk prediction model(s) would work best for the Qatari population.

Comparison with other studies
Our results are in keeping with previous systematic reviews. 14,15,77 Most reviews concluded that there is now an abundance of cardiovascular risk calculators, but reported difficulty in deciding which is most appropriate to use. Moreover, the majority of the risk calculators have been developed in predominantly White European-origin populations limiting their usefulness for other ethnic groups. The heterogeneity and lack of reporting of discrimination statistics have previously been highlighted 4,15 ; for example, we found that 30% of studies reported no statistics whatsoever.

Implications for policy, practice, and research
We were able to identify models that included ethnically diverse populations in their derivation and validation cohorts. No model closely resembled Qatar's diverse ethnic profile. That said, the risk calculators that incorporated ethnicity within their development, i.e. in both the derivation and validation phases, were PREDICT and QRISK3. Furthermore, using PROBAST to assess ROB and applicability of each of the individual models, we were able to identify PREDICT and http://chd.bestsciencemedicine.com/ calc2.html) and QRISK3 (https://www.qrisk.org/ three/) as potential candidates for use in Qatar. 50,69 Qatar is currently using the ACC/AHA PCE, which although judged to be at a low ROB was not found to be applicable to Qatar's ethnically diverse population. These results were discussed with clinical and policy leaders across Qatar in a workshop and will be used to inform deliberations on the need for formal validation studies in Qatar. These findings may also be applicable to other Arab countries with similar ethnically diverse populations.

CONCLUSIONS
This study commissioned by the Qatari Ministry of Public Health has shown that there is no existing ASCVD risk calculator particularly well suited for use in the ethnically diverse Qatari population. Of the available risk calculators, PREDICT and QRISK3 appear to be best suited for use in Qatar because of their inclusion of ethnicity. In the absence of a locally derived ASCVD for Qatar, there is merit in a formal head-to-head comparison between the currently used PCE, and PREDICT and QRISK3.

Registration
This systematic review is registered with PROSPERO (Registration no. CRD4202017698).

Funding
This study was funded by the Qatar Ministry of Public Health.

Conflicts of interest
AS was involved in the development of the QRISK2 algorithm.