Evaluation of the Canadian Clinical Practice Guidelines Risk Prediction Tool for Acute Aortic Syndrome: The RIPP Score

Introduction Acute aortic syndrome (AAS) is a rare clinical syndrome with a high mortality rate. The Canadian clinical practice guideline for the diagnosis of AAS was developed in order to reduce the frequency of misdiagnoses. As part of the guideline, a clinical decision aid was developed to facilitate clinician decision-making (RIPP score). The aim of this study is to validate the diagnostic accuracy of this tool and assess its performance in comparison to other risk prediction tools that have been developed. Methods This was a historical case-control study. Consecutive cases and controls were recruited from three academic emergency departments from 2002–2020. Cases were identified through an admission, discharge, or death certificated diagnosis of acute aortic syndrome. Controls were identified through presenting complaint of chest, abdominal, flank, back pain, and/or perfusion deficit. We compared the clinical decision tools' C statistic and used the DeLong method to test for the significance of these differences and report sensitivity and specificity with 95% confidence intervals. Results We collected data on 379 cases of acute aortic syndrome and 1340 potential eligible controls; 379 patients were randomly selected from the final population. The RIPP score had a sensitivity of 99.7% (98.54–99.99). This higher sensitivity resulted in a lower specificity (53%) compared to the other clinical decision aids (63–86%). The DeLong comparison of the C statistics found that the RIPP score had a higher C statistic than the ADDRS (−0.0423 (95% confidence interval −0.07–0.02); P < 0.0009) and the AORTAs score (−0.05 (−0.07 to −0.02); P = 0.0002), no difference compared to the Lovy decision tool (0.02 (95% CI −0.01–0.05 P < 0.25)) and decreased compared to the Von Kodolitsch decision tool (0.04 (95% CI 0.01–0.07 P < 0.008)). Conclusion The Canadian clinical practice guideline's AAS clinical decision aid is a highly sensitive tool that uses readily available clinical information. It has the potential to improve diagnosis of AAS in the emergency department.


Introduction
Acute aortic syndrome is a time-sensitive emergency defned by three distinct diagnoses: aortic dissection, intramural hematoma, and penetrating atherosclerotic ulcer. Acute aortic syndrome and each of its component diagnoses involve blood leaking into the wall of the aorta, which is the major artery that supplies blood to the entire body [1]. Tis blood separates the layers of the wall of the aorta and can block blood fow to vital organs (heart, muscle, brain, or limbs) or lead to a rupture of the aorta and catastrophic blood loss [2][3][4][5]. Te risk of death from acute aortic syndrome increases by 2% per hour, reaching 90% if undiagnosed [6]. Currently, there is signifcant variation in how physicians investigate acute aortic syndrome. Tis variation has led to inefcient use of computed tomographic imaging and a high miss rate. Health care providers miss 1 in 4 cases on the frst presentation [2,[7][8][9]. Computed tomography is the investigation of choice to diagnose acute aortic syndrome. Its use in the emergency department has increased exponentially over the past number of years without any impact on the number of missed cases [10].
We have developed the Canadian clinical practice guideline for the diagnosis of acute aortic syndrome in the emergency department [11]. Te goal of this guideline is to ofer practical recommendations to guide the investigation of acute aortic syndrome, reducing misdiagnosis and improving efciency of resource utilization. Te guideline committee proposed a consensus-based risk stratifcation tool to aid in pretest probability assessment for those at risk of acute aortic syndrome (RIPP score shown in Table 1).
Te aim of this study is to validate the diagnostic accuracy of this tool and assess its performance in comparison to other risk prediction tools that have been developed.

Methods
Tis was a historical case-control study. Consecutive cases were recruited from three academic emergency departments from 2002-2020, with an annual census ranging from 75,000-120,000. Consecutive controls were recruited from a single academic emergency department. Tese departments were geographically distributed across Ontario, with one northern and two southern sites. Tis study follows the methodological and reporting recommendations outlined in the Standards for eporting of Diagnostic Accuracy Studies (STARD) criteria. [12] Te study was approved by the research ethics boards of all participating institutions.

Data Extraction.
Te data were extracted and verifed from multiple sources, including emergency department records and progress notes. Trained reviewers used electronic data forms and underwent training [13]. Te data were analysed for agreement using the Kappa statistic with oversight. Interobserver agreement was calculated for at least 20% of the total charts. Reviewers were not informed of the study objective but were aware of the patient's case or control status.

Variables.
Te full data dictionary is included in the appendix. Abrupt onset pain was defned as pain described as sudden, unexpected, or onset at a specifc time point. Severe was a pain score greater than 6 or required opioids for pain relief [14,15]. Migrating/radiating pain was defned as pain that moved from one location to another or pain reported in two distinct anatomical locations, i.e., chest and back. Hypotension was defned by a systolic blood pressure <90 mmHg [15]. Te clinical suspicion variable had three levels: AAS as most likely diagnosis, alternative diagnosis as more likely, and unsure. AAS was considered the most likely diagnosis if, after the initial assessment, a CT aorta was ordered to rule out AAS. An alternative diagnosis was considered most likely if documentation of a specifc discharge or admission diagnosis was present. Te clinical impression was deemed unsure if the diagnosis was unspecifed or not yet diagnosed (i.e., chest pain not yet diagnosed).Where a score ofered three risk levels, we dichotomized them into high risk and low risk.

Participants
2.3.1. Case. We included consecutive eligible cases of acute aortic syndrome from 2002 to 2020. Cases were identifed through an admission, discharge, or death certifcated diagnosis of acute aortic syndrome.

2.3.2.
Control. Controls were identifed through the presenting complaint of chest, abdominal, fank, back pain, and/ or perfusion defcit (limb ischemia, cerebrovascular accident possible, neurological defcit, syncope, or altered level of consciousness). Tese were based on the Canadian Emergency Department Information System (CEDIS), which presents complaints that are indexed in our electronic health records. We randomly selected an equal number of unmatched controls (1 : 1) to be included in the fnal population.

Exclusion.
We excluded patients <18 years old, with a pain duration of >14 days, or with trauma within 24 hours of the onset of pain. No documented chest, abdomen, back pain, or perfusion defcit. Left without being seen or with no documentation available for the patient. A clear alternative diagnosis after initial clinical assessment, i.e., subcutaneous abscess, urinary tract infection, trapezius muscle strain, panic attack, cannabis hyperemesis syndrome, gastroenteritis, upper respiratory tract infection, uterine prolapse, upper gastrointestinal bleed, etc. Follow-up visits to hospitals without a diagnosis of acute aortic syndrome or imaging that did not reveal acute aortic syndrome were used to confrm the absence of the condition. In cases where patients did not return to study hospitals or undergo further imaging, publicly available sources such as obituaries were searched to determine if they had passed away.

Data Analyses.
We calculated the classifcation performance of the RIPP score using sensitivity and specifcity together with 95% confdence intervals. We also assessed the classifcation of the acute aortic dissection detection risk score (ADDRS), the Lovy clinical decision tool, the Von Kodolitsch decision tool, and the AORTAs score according to their ability to classify patients as either low or high risk [16][17][18][19]. We compared the clinical decision tools' C statistic (area under the curve) and used the DeLong method to test for the signifcance of these diferences. We calculated the absolute net reclassifcation indices by comparing the RIPP score with the other clinical decision tools [20]. Te net reclassifcation index quantifes the improvement in prediction when comparing prediction tools. Te sample size was based on an estimation of the precision of the classifcation performance of the risk scale. Our goal was to ascertain sufcient cases to evaluate the sensitivity with 95% confdence bands plus/minus 5%, corresponding to 200 cases of acute aortic syndrome.

Patient and Public Involvement.
Neither patients nor the public were formally involved in the planning of the study. We plan to involve patients before assessing the efects of implementing this rule in clinical practice.

Results
Data were collected from 2002 to 2021, yielding 379 cases of acute aortic syndrome. We found 1340 potential eligible controls over a 1-month period; 580 were excluded. Of the remaining controls, 379 patients were randomly selected for the fnal population ( Figure 1). Te Kappa between data extractors after chart training was 0.87. Chest pain unspecifed (20.7%), abdominal pain unspecifed (9.9%), and acute coronary syndrome (8.7%) were the top three diagnoses in the control population. Table 2 shows the clinical features of our cohort. Te patients had a mean age of 68.5 years, and 52.3% were female.
Te RIPP score had a sensitivity of 99.7%. Tis was higher than all the other scores. Tis higher sensitivity resulted in a lower specifcity (53%) compared to the other clinical decision aids (63-86%) ( Table 3). Te DeLong comparison of the C statistics found the RIPP score had a higher C statistic compared to the ADDRS (−0.0423 (95% confdence interval −0.07-0.02); P < 0.0009) and the AORTAs score (−0.05 (−0.07 to −0.02); P � 0.0002), no diference compared to the Lovy decision tool (0.02 (95% CI −0.01-0.05 P < 0.25)), and decreased compared to the Von Kodolitsch decision tool (0.04 (95% CI 0.01-0.07 P < 0.008)) ( Figure 2. Table 4). Table 5 shows the absolute net reclassifcation index between the RIPP score and the other clinical decision tools. Tis ranged from 0.03 compared to the AORTAs to 0.20 for the Von Kodolitsch score. Te RIPP score had the highest number of cases of acute aortic syndrome that were correctly classifed as high-risk.

Discussion
Our study provisionally validates the predictive performance of the RIPP score in a broad sample of patients identifed in the emergency department with a diagnosis of acute aortic syndrome. To improve the generalisability of the score, we included both large and smaller volume centres, including sites that were not involved in the derivation of any of the assessed scores. Te score was able to correctly stratify many more patients with acute aortic syndrome into an appropriate high-risk category. Tis rule meets the prespecifed sensitivity identifed by emergency medicine clinicians to successfully be used in guiding investigations for patients with acute aortic syndrome [21].
It should be noted that this is a retrospective case-control study and sufers from signifcant bias in the assessment of diagnostic accuracy estimates. Tis bias should afect all scores equally and therefore be nondiferential. However, this study only provides provisional evidence of diagnostic accuracy; it is not sufcient in isolation to change practice.

Comparison with Other Studies.
We had previously surveyed emergency physicians to identify thresholds of acute aortic syndrome risk that would alter clinical decisions. In this study, respondents indicated that patients with a subsequent risk of acute aortic syndrome below 1% were most appropriate for no further investigation [21].
In our study, the RIPP score was the only decision tool that had sufciently high sensitivity to be able to defne a no testing group. Te ADDRS and the AORTA scores are designed to be used with D-dimer and allow the user to diferentiate between D-dimer versus computed tomography as the frst test in an investigation [14,19]. Our study confrms that they are unable to identify a low-risk group that requires no further testing, and their use should be confned to assessing D-dimer versus computed tomography as the initial investigation. It was expected that there would be a high level of missing data for D-dimer within our cohort; therefore, a priori, we decided to assess each decision aids ability to defne a no testing cohort.
Te Lovy and Von Kodolitsch decision aids both use clinical variables to defne a low-or a high-risk group of patients. To our knowledge, this is the frst evaluation of these decision aids. We found that neither had sufcient sensitivity to defne a no testing group. Excluded -random (n=203) Figure 1: Flowchart of included patients.    Terefore, baseline risks, hazards, and absolute probabilities cannot be correctly adjusted. However, the risk of bias is applicable to all decision tools tested, and the comparative diferences are not afected. One variable of the score was clinical impression. Tis was extracted from the patient chart; therefore, a physician may have had a clinical suspicion for AAS but not documented the diagnosis. Tis may artifcially infate the specifcity of the clinical risk score. We were conservative in our estimate of the presence of a clinical impression for AAS, only coding as yes if a CT aorta was ordered to rule out AAS; therefore, the inherent fragility of this variable should not afect the reported sensitivity.

Emergency Medicine International
Our aim was to select a control population that refected those in whom there would be a reasonable consideration of acute aortic syndrome. We excluded those who had an image-proven alternative diagnosis or those who underwent computed tomography with intravenous contrast for another reason and would have been diagnosed with acute aortic syndrome. Because of this, we were left with a population that was less severe in presentation than those who underwent imaging for other reasons. Tis likely artifcially infates our specifcity; however, the comparative diferences in specifcity between tools should be independent of this bias.
Tis was a retrospective chart review where missing values were defaulted to negative. Tis was done to bias our results towards the lowest possible sensitivity. In addition, no assessment of the interrater reliability of the clinical variables could be performed.
Our cases are patients in whom acute aortic syndrome was identifed; patients who were never identifed, discharged, or died will not be identifed. Tese patients are likely to have a diferent presentation, and thus the RIPP score may not perform as well in this patient population.
Not all patients underwent the reference standard of computed tomography, MRI, or transesophageal echocardiography. Terefore, there is a possibility of misclassifying cases as controls. Tis could artifcially increase our sensitivity and specifcity. We could potentially have chosen a control population from patients who underwent imaging to rule out acute aortic syndrome; however, that would have resulted in partial verifcation bias and artifcially lowered the specifcity.

Research and Clinical Implications.
A prospective multicentre implementation study following established implementation guidelines is now needed to assess the impact of the RIPP score when applied in clinical practice. Although a tool may have sufcient accuracy to classify cases and controls, it still may not lead to the desired decrease in missed cases of AAS or the efcient use of advanced imaging.
Te major concern with all of the clinical risk scores is the likely low specifcity in clinical practice [22]. We found that the RIPP score had the lowest specifcity. In order for the RIPP score to achieve sufcient sensitivity to meet prespecifed criteria, there is a corresponding increase in false positives. Te question remains: is the potential decrease in the missed rate of this rare diagnosis adequately balanced with the likely increase in CT usage? Although clinicians may use the score to help guide them in the collection of important clinical variables to help defne risk for AAS, caution should be used until an implementation study is completed in order to confrm its use leads to better patient outcomes. Clinical decision rules and risk scores, when implemented, may not lead to a clinically important change in patient-oriented outcomes [22].

Conclusion
Te RIPP score identifes those who require further investigation for acute aortic syndrome among those presenting to the emergency department with undiferentiated symptoms. Incorporating this validated clinical decision tool into clinical practice has the potential to improve decisions regarding the investigation of patients for acute aortic syndrome in the emergency department.

Data Availability
Te data used to support the fndings of this study are available from the corresponding author upon reasonable request.