Clinical Evaluation of the Automatic Coronary Artery Disease Reporting and Data System (CAD-RADS) in Coronary Computed Tomography Angiography Using Convolutional Neural Networks

Rationale and Objectives: The coronary artery disease reporting and data system (CAD-RADS TM ) was recently introduced to standardise reporting. We aimed to evaluate the utility of an automatic postprocessing and reporting system based on CAD-RADS TM in suspected coronary artery disease (CAD) patients. Materials and Methods: Clinical evaluation was performed in 346 patients who underwent coronary computed tomography angiography (CCTA). We compared deep learning (DL)-based CCTA with human readers for evaluation of CAD-RADS TM with commercially-available automated segmentation and manual postprocessing in a retrospective validation cohort. Results: Compared with invasive coronary angiography, the sensitivity, speci ﬁ city, positive predictive value, negative predictive value, and accuracy of the DL model for diagnosis of CAD were 79.02%, 86.52%, 89.50%, 73.94%, and 82.08%, respectively. There was no signi ﬁ cant difference between the DL-based and the reader-based CAD-RADS TM grading of CCTA results. Consistency testing showed that the Kappa value between the model and the readers was 0.775 ( 95% con ﬁ dence interval [CI] : 0.728 (cid:1) 0.823, p < 0.001), 0.802 ( 95% CI : 0.756 (cid:1) 0.847, p < 0.001), and 0.796 ( 95% CI : 0.750 (cid:1) 0.843, p < 0.001), respectively. This system reduces the time taken from 14.97 § 1.80 min to 5.02 § 0.8 min ( p < 0.001). Conclusion: The standardised reporting of DL-based CAD-RADS TM in CCTA can accurately and rapidly evaluate suspected CAD patients, and has good consistency with grading by radiologists


INTRODUCTION
C oronary artery disease (CAD) is a global medical problem and a leading cause of morbidity and mortality (1). Patients with CAD are at risk of ischaemic stroke, myocardial infarction, and cardiovascular death (2). Coronary computed tomography angiography (CCTA) is increasingly being used as a noninvasive evaluation method to assess patients with chest pain because it has high sensitivity and specificity for the detection of CAD (3,4). Moreover, unlike functional stress testing, CCTA provides an anatomical assessment of the coronary arteries, allowing for the detection and quantification of obstructive and non-obstructive atherosclerotic plaques (5).
The coronary artery disease reporting and data system (CAD-RADS TM ) was introduced in 2016 to standardise reporting and decision-making (6). Unlike prior classifications of CAD that were based on the number of affected vessels, CAD-RADS TM classifies patients based on the highest grade of coronary artery stenosis. Xie et al. recently reported that CAD-RADS TM effectively identified patients at risk of myocardial infarctions and all-cause mortality to a similar degree as the CAD extent classification and the Duke CAD Index (7).
Computer-aided diagnosis has been rapidly growing and is increasingly used in radiological assessment (8,9). Deep convolutional neural networks (CNN) are a newly-emerging form of computer-aided diagnostic analysis that allows the automatic extraction of features and supervised learning of large amounts of data to make quantitative decisions (10). Accumulated evidence suggests that deep learning (DL) analysis might be a potential alternative to conventional handcrafted methodologies for solving pattern-recognition and imaging-classification problems (11,12). To date, there has been no research focusing on the combination of CNN and CCTA to solve lesion detection and classification problems based on CAD-RADS TM .
Thus, we aimed to assess the utility of an automatic postprocessing and reporting system based on CAD-RADS TM in CAD or suspected CAD patients. We also sought to investigate the performance of DL-based CCTA to assist radiologists in their daily work and establish a time-saving work process.

MATERIALS AND METHODS
The present study was approved by the local institutional review board and was conducted in compliance with the Health Insurance Portability and Accountability Act of 1996.

Patients
We retrospectively searched a database of CCTA data from patients who were examined between July 2017 and December 2019. The study population consisted of 346 consecutive patients who were confirmed or suspected to have CAD (Fig 1). Scans affected by stair-step artifacts, respiratory motion artifacts, excessive image noise, or artifacts caused by arrhythmia, or poor quality images were excluded from analysis. We also excluded patients without complete records and those who previously underwent percutaneous coronary intervention or coronary artery bypass grafting or other cardio surgery from analysis.

CCTA Image Acquisition and Analysis
Multidetector row computed tomography (CT) imaging was performed with a dual-source CT scanner (Somatom Definition, Siemens Medical Solutions, Forchheim, Germany) and a 256-slice CT scanner (Brilliance iCT; Philips Healthcare, Cleveland, OH). Heart rate (HR) control ( 65 beats/min) was performed with beta-blockers before the scan. For contrast enhancement, 60À80 mL of iopromide (370 mgI/mL, Bayer Schering Pharma, Leverkusen, Germany) followed by 30À40 mL of pure saline was administered with a flow rate of 4À5 mL/s. The iodine contrast agent was automatically triggered into the descending aorta of 100 Hounsfield units threshold units. Then the scanning was performed during an inspiratory breath hold of 8À14 s after a delay of 2 s. The reconstructed images were automatically sent to a workstation (CoronaryDoc, Shukun technology, Beijing, China) equipped with a coronary analysis software tool (Computer Aided Diagnosis of Coronary Artery, Version 1.8, Shukun technology).

Deep Learning (DL)
The patients in the test set were examined by invasive coronary angiography (ICA) with an interval of fewer than 30 days after the CCTA procedure. We have previously reported the validation of our DL system (13,14). Before training, the aorta, coronary artery and plaques were labeled on each image using a multi-layer manual annotation system consisting of multiple layers of trained graders. The first layer of graders consisted of radiologists who had knowledge of medical imaging and coronary anatomy. The second layer of graders comprised radiologists with more than 3 years of work experience in radiology, who performed a preliminary inspection of the accuracy of the label. The third and final layer of graders was made up of experienced experts with over 5 years of work experience who verified the correctness of the label of each image. In this study, we adopted an improved three-dimensional (3D) U-Net architecture and added a bottle-neck model for segmentation of coronary arteries and aorta, then a growing iterative prediction network model was developed to solve the problem of vascular segmentation fracture, and finally full coronary tree segmentation was obtained (15). Based on coronary tree segmentation, multiple planar reformat (MPR), curve planar reformat (CPR), maximum intensity projection, and volume rendering images were reconstructed. To detect stenosis, we developed a 3D segmentation neural network and a onedimensional sequence checking hybrid technique (16). Firstly, a 3D segmentation neural network was applied to MPR and CPR images to detect stenosis, and then a onedimensional sequence-checking algorithm was used to reduce false-positive results (Fig 2).
Three readers with 5 years of experience in cardiac CT imaging diagnosis recorded the CAD-RADS TM classification based on the degree of coronary stenosis (CAD-RADS 0: 0%, CAD-RADS 1: 1À24%, CAD-RADS 2: 25À49%, CAD-RADS 3: 50À69%, CAD-RADS 4A: 70À99%, CAD-RADS 4B: Left main > 50% or 3-vessel disease, 70À99%, CAD-RADS 5: 100%) according to the CAD-RADS TM consensus document (6). A structured report including CAD-RADS TM category was created based on the model independently. All the readers and the model provided CAD-RADS TM scores independently, which remained hidden to each. Coronary artery lesions were evaluated for stenosis on a diameter-based approach. The readers, who were blinded to the results of each other and to the model, manually measured the relevant lesions independently. One hundred patients were randomly selected for time consumption (including postprocessing, report-writing, typesetting and print) analysis. CAD was defined as stenosis > 50% in a coronary artery segment 2mm in diameter.

Statistical Analysis
Continuous variables are presented as mean § SD. Categorical variables are presented as percentages or absolute values. We used either the chi-square test, or Fisher's exact test as appropriate for categorical variables. The diagnostic performance of CAD with DL-based CCTA was determined with regard to sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy by comparison with ICA as the standard of reference. The agreement of the CAD-RADS TM categories was compared between readers and the model using the linear weighted kappa identity test. p < 0.05 was considered to represent statistical significance. All statistical analysis was performed using SPSS version 18 (SPSS, Inc., Chicago, IL, USA) and MedCalc Statistical Software version 16.8.4.0 (Ostend, Belgium). Table 1 shows the baseline characteristics of the 346 study participants (mean age: 62.9 years; range from 33 to 85 years). Of the 346 patients, 181 (52.3%), 181 (52.3%), 180 (52.0%), and 180 (52.0%) were diagnosed with CAD by the DL model, Reader 1, Reader 2, and Reader 3, respectively. There were no significant differences between readers and the DL model in the detection of CAD. The PPV, NPV, sensitivity, specificity, and accuracy of the DL model for diagnosis of CAD were 79.02%, 86.52%, 89.50%, 73.94%, and 82.08%, respectively when using ICA as the reference standard ( Table 2). The area under the receiver operating characteristic curve (AUC) of the DL model for CAD diagnosis was 0.83, which was similar to those of Reader 1 (0.84), Reader 2 (0.82) and Reader 3 (0.84) (all p > 0.05, Fig 3). In addition, the diagnostic performance of CAD between readers was similar (all p > 0.05, Fig 3).

RESULTS
The distribution classification of CAD-RADS TM assessed by the DL model was 38 as CAD-RADS 0, 19 as CAD-RADS 1, 108 as CAD-RADS 2, 90 as CAD-RADS 3, 58 as CAD-RADS 4A, 24 as CAD-RADS 4B and 9 as CAD-RADS 5 (Fig 4). As shown in Table 3 Fig 5). We also found that the time taken differed between manually-recognised CAD patients and non-CAD patients, while there was no significant difference when DL was used. Colored arrows indicate different operations: yellow = convolution and rectified linear unit (ReLU), orange = copy and concatenate from downsampling path to up-sampling path, green = maximum pooling, blue = up-convolution and purple = convolution. The input was images from CCTA acquisition, whereas the output was the aorta, coronary artery and plaque contours if these are detected on the image. (b) CNN networks for coronary artery disease reporting and data system (CAD-RADS TM ) classification. Plaque segmentation was applied by a 3D UNET network on a curve planar reformat (CPR) image and a straightened image respectively, and then the output results were modified and classified by a one-dimensional back-detection model. Finally, the CAD-RADS TM classification output was determined according to the degree of stenosis. C&D, DL-based CCTA assistant diagnostic system. (Color version of figure is available online.)

DISCUSSION
In this study, we first developed a novel CNN model using CCTA images and evaluated its performance in categorization of coronary artery stenosis based on CAD-RADS TM . Then, the utilisation of an automatic post-processing and reporting system based on CAD-RADS was assessed in suspected CAD patients.
As a standardised reporting system of CCTA, the primary aim of CAD-RADS TM is to facilitate consistent assessment among physicians, including recommendations for further investigations and management (6,17). Moreover, the common language of coronary artery stenosis provided by this reporting system increases the clarity between radiologists and clinicians in the diagnosis and treatment planning of suspected CAD patients. In addition, the achieved standardisation of reporting will be of benefit in education, research, peer review, and quality assurance, ultimately resulting in improved quality of care (18).
Cancer imaging standardised reporting systems, such as Lung-RADS and BI-RADS have been widely accepted by offering reliable and consistent assessment categories to radiologists and physicians who can then supply appropriate management recommendations for specific patients. Structured reporting tools have been applied in cardiac imaging to improve imaging data integrity and then establish standard databases for education, patient care and research purposes (19). Structured reporting platforms based on automated CAD-RADS TM calculations have been proposed by Dewey et al. (20), and Szilveszter et al. have revealed good agreement between manual and automated CAD-RADS TM classifications using a structured reporting platform (21). Previous methods for automated CAD or CAD-RADS TM assessment showed strong agreement and inter-observer reproducibility (22,23). However, no machine learning or DL models have been used in these studies.
Recently, promising applications of machine learning or artificial intelligence (AI) have been used in cardiovascular imaging and cardiology to process several algorithms that are already in routine clinical use (24,25). These algorithms are likely to provide a process to improve the quality and speed of acquisition, reduce manual postprocessing, measurement and manual report writing time, and allow prompt diagnoses, which in turn would improve workflow and patient care. Moreover, these could facilitate screening programs and promote establishment of more effective referral mechanisms in medicine, particularly in remote or low-resource areas, leading to a broad, positive clinical, and public health impact (26). In addition, they encourage researchers to improve the performance of future models and help drive the field forward. BMI, body mass index; BUN, blood urea nitrogen; CKD, chronic kidney disease; LDL-C, low density lipoprotein-C; DM, diabetes mellitus; EF, ejection fraction; HDL-C, high density lipoprotein-C; SCR, serum creatinine; TC, total cholesterol; TG, triglyceride. Machine learning algorithms have been extensively used for the optimisation of information extraction from CCTA. A two-step support vector machine based on learning with CCTA was established by Kang et al. (27) for detecting nonobstructive and obstructive CAD, and resulted in an accuracy of 94% and an AUC of 0.94 and thus had a higher accuracy than obtained in the present study. However, only two classifications were used in the study and obstructive CAD was defined as lesion with stenosis 25%. However in our study the CAD-RADS classification system was applied and we used the widely-accepted definition of CAD (lesion with stenosis 50%). Moreover, a small CCTA dataset with only 42 patients were included in the study by Kang et al., while our study included a relatively larger sample size of 346

ARTICLE IN PRESS
patients. In addition, our DL platform was trained and validated using the Digital Imaging and Communications in Medicine (DICOM) standards, which makes CCTA images from different manufacturers (dual-source CT and Philips ICT) reasonably consistent. Finally, our study developed a DL model which was able to automatically extract coronary arteries and detect stenosis.
Accumulated researches revealed good automatic detection and quantification with CCTA images. Schuhbaeck et al. evaluated the interscan reproducibility of quantitative measurements of coronary plaque volumes using a standardised automated method (28). A recent study showed that CCTAderived plaque markers combined with a DL-based CT-FFR had a good predictive value in identifying lesion-specific ischaemia when compared to CCTA stenosis grading alone (29). Quantification of coronary artery calcium score derived from CCTA with a DL algorithm has also been validated to be accurate (30,31). A previous study demonstrated that a machine learning algorithm-based CT-FFR value was able to assess the functional significance of CAD and make a therapeutic decision. This algorithm showed excellent performance (sensitivity of 97%, specificity of 100%, PPV 100%, NPV of 97%, and accuracy of 99%) of ML-based CT-FFR with CCTA in determining an appropriate treatment strategy (32). However, few studies have focussed on DL-based CCTA using CAD-RADS TM for stratifying the risk of suspected CAD patients. The present study was designed to evaluate the feasibility of CAD-RADS TM classifications by using DL based on CCTA images (Fig 6).
Some limitations of this study should be addressed. First, this was a retrospective analysis of a relatively small sample size from a single centre. Prospective multicentre studies of large samples will enhance the application of CAD-RADS TM among cardiologists and radiologists. Second, in order to restrict our analysis, we did not include CAD-RADS TM   modifiers to describe patients with stents (modifier S), vulnerable plaque features (modifier V), or grafts (modifier G). Further studies should include more patients in the training set and testing set of DL to improve the CAD-RADS TM classification scheme. Third, the DL model slightly underestimated CAD-RADS 4A and CAD-RADS 5 patients compared to the readers as shown in Figure 4. As a result, this would impact clinical decision-making as the consideration of ICA would be reduced in a portion of patients that might need them. It should be noted that the sensitivity of the DL model for diagnosing CAD is 79%, which is lower than previous studies with a sensitivity of 84À94% using CCTA (33). Further improvement of this model is still needed to reduce the possibility of underdiagnosing CAD. In addition, visual evaluation by ICA may overestimate stenosis grade, as has been demonstrated in previous research (34).

CONCLUSION
The standardised reporting of DL-based CAD-RADS TM in CCTA images can accurately evaluate suspected CAD patients while saving time, and has excellent consistency with radiologists.

FUNDING
Not applicable.