Automated identification of thrombectomy amenable vessel occlusion on computed tomography angiography using deep learning

Introduction We developed and externally validated a fully automated algorithm using deep learning to detect large vessel occlusion (LVO) in computed tomography angiography (CTA). Method A total of 2,045 patients with acute ischemic stroke who underwent CTA were included in the development of our model. We validated the algorithm using two separate external datasets: one with 64 patients (external 1) and another with 313 patients (external 2), with ischemic stroke. In the context of current clinical practice, thrombectomy amenable vessel occlusion (TAVO) was defined as an occlusion in the intracranial internal carotid artery (ICA), or in the M1 or M2 segment of the middle cerebral artery (MCA). We employed the U-Net for vessel segmentation on the maximum intensity projection images, followed by the application of the EfficientNetV2 to predict TAVO. The algorithm’s diagnostic performance was evaluated by calculating the area under the receiver operating characteristics curve (AUC), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). Results The mean age in the training and validation dataset was 68.7 ± 12.6; 56.3% of participants were men, and 18.0% had TAVO. The algorithm achieved AUC of 0.950 (95% CI, 0.915–0.971) in the internal test. For the external datasets 1 and 2, the AUCs were 0.970 (0.897–0.997) and 0.971 (0.924–0.990), respectively. With a fixed sensitivity of 0.900, the specificities and PPVs for the internal test, external test 1, and external test 2 were 0.891, 0.796, and 0.930, and 0.665, 0.583, and 0.667, respectively. The algorithm demonstrated a sensitivity and specificity of approximately 0.95 in both internal and external datasets, specifically for cases involving intracranial ICA or M1-MCA occlusion. However, the diagnostic performance was somewhat reduced for isolated M2-MCA occlusion; the AUC for the internal and combined external datasets were 0.903 (0.812–0.944) and 0.916 (0.816–0.963), respectively. Conclusion We developed and externally validated a fully automated algorithm that identifies TAVO. Further research is needed to evaluate its effectiveness in real-world clinical settings. This validated algorithm has the potential to assist early-career physicians, thereby streamlining the treatment process for patients who can benefit from endovascular treatment.


Introduction
Advancements in stroke imaging and procedural devices have extended the endovascular treatment (EVT) window for patients with hyperacute ischemic stroke (1,2).The DAWN (DWI or CTP Assessment with Clinical Mismatch in the Triage of Wake-Up and Late Presenting Strokes Undergoing Neurointervention with Trevo) and DFFUSE 3 (Endovascular Therapy Following Imaging Evaluation for Ischemic Stroke) trials have changed the standard of care for ischemic stroke patients who present within 6 to 24 h of their last well known status.
Triage entry for clinical trials primarily relies on magnetic resonance imaging (MRI) or computed tomography perfusion (CTP) to identify clinical or tissue mismatch and is now endorsed in guidelines.However, access to acute MRI or CTP is limited and not commonly available in the majority of primary stroke centers worldwide.Recent studies have brought attention to more readily available imaging techniques, such as CT angiography (3).The CT for Late Endovascular Reperfusion (CLEAR) trial (4) revealed no significant differences in clinical outcomes between patients selected using non-contrast CT with CT angiography and those selected using CTP or MRI.In addition, a sub-study (5) of the HERMES collaboration (Highly Effective Reperfusion Evaluated in Multiple Endovascular Stroke Trials) has extended this approach to the early time window (0-6 h) by demonstrating that the rates of favorable functional outcomes were comparable between patients who underwent CTP and those who did not.
Two-thirds of EVT candidates were initially routed to centers not equipped for EVT (6), despite better outcomes and higher chances of receiving EVT at EVT-capable centers.Consequently, non-EVTcapable centers must consistently identify large vessel occlusion (LVO) around the clock, ensuring quick reporting to facilitate patient transfer to EVT-capable centers.However, a lack of vascular specialists poses challenges for many smaller, non-EVT-capable centers.Even in EVT-capable centers, the ability to screen CTAs for the presence of LVOs can streamline workflow, staffing, and door-to-puncture times by facilitating LVO detection.Machine learning has been employed to automate LVO detection in CTA, which is now in clinical use in a few countries (7,8).However, independent external evaluation of these automated LVO detection algorithms have shown only modest sensitivity (7,8).Moreover, machine learning algorithms contingent on Hounsfield unit and hemisphere asymmetry information have limited its applicability in patients with bilateral occlusions, such as Moyamoya disease (9).
In contrast to initial focus on intracranial LVO, recent advancements in neurointerventional devices and cerebrovascular imaging have expanded the application of EVT to medium vessel occlusions.Consequently, the selection of appropriate EVT candidates has become more complex, necessitating advanced imaging.This complexity poses a challenge for early-career physicians in making treatment decision (10).
In the context of current clinical practice, we have developed and validated a fully automated deep learning algorithm to detect comprehensive EVT target vessels.This includes not only the wellknown LVOs but also other relevant vessel occlusions.We utilized a dataset from multiple centers in Korea, encompassing 2,441 patients with acute ischemic stroke.The deep learning algorithms were specifically designed for (1) selecting the appropriate slices for consistent maximal intensity projection (MIP) image generation, (2) segmenting vessels on MIP images, and (3) identifying vessel occlusions using the vessel segmentation mask.We further validated the algorithm with two independent external datasets.

Datasets Training validation and internal test
From May 2011 and June 2013, 1,745 patients were collected from two hospitals.Additionally, 389 patients were included from a university hospital between August 2020 and May 2021.After combining these groups, a total of 2,134 patients who were suspected with ischemic stroke and who underwent CT angiography were initially considered (refer to Supplemental Figure 1).Following the exclusion of 89 patients, the final cohort consisted of 2,045 patients, who were randomly divided into training (1,277), validation (144), and internal test (624) datasets with a 63:7:30 ratio.

External test
Between April 2011 and July 2013, a total of 71 patients were included from two tertiary hospitals.After excluding 7 patients, 64 were assigned to external test dataset 1.In addition, 337 patients from a university hospital were included for the period between February 2017 and March 2022.After excluding 24 patients from this group, 313 were assigned to external test dataset 2. The study procedure was approved by the institutional review boards of each hospital, and written consent was waived due to the retrospective and anonymised nature of the study design.

Definition of thrombectomy amenable vessel occlusion
In our study, the vascular occlusions we aimed to investigate are termed as thrombectomy amenable vessel occlusion (TAVO) which include any arterial occlusion involving the intracranial internal carotid artery (ICA), middle cerebral artery (MCA)-M1, and MCA-M2 segments.Intracranial ICA is defined as the segment of the ICA from the petrous part to the MCA-ACA (anterior cerebral artery) bifurcation.MCA-M1 indicates the MCA segment from the MCA-ACA bifurcation to the MCA branching point, and the MCA-M2 refers to the segment of the MCA ascending vertically along with Sylvian fissure from its branching point.For the subsequent analysis, TAVO patients were divided into intracranial LVO and isolated MCA-M2 occlusion in our study (11).In cases of early division of the MCA, a functional rather than traditional angiographic definition was adopted; the short proximal trunk was called the M1 segment and the branches distal to division were defined as M2 segments.To determine the presence or absence of TAVO (ground truth label), each image used in the study was reviewed by an experienced neurologist along with the subject's MR image (MRI) scans and patient symptom data.The neurologist's TAVO diagnosis was cross-referenced with the stroke registry, which was independently verified by attending vascular neurologists at each hospital.In case of disagreement, a consensus was made.

Slices selection for maximal intensity projection image generation
In order to maintain the uniformity of the input image for the deep learning model, we have developed an automated approach for selecting slices from the source images (Figure 1).These slices are then used to create a maximum intensity projection (MIP) image.We constructed a sagittal bone array by summing the pixel values of the source images.This process automatically outlines the skull, enabling the automatic determination of the vertex and the C1 atlas (12).Using this method on the training data, we effectively identified target slices in 1,240 cases, which accounts for 97.1% of the total.We calculated the mean and standard deviation of the distance between the vertex and C1 atlas in these patients.For cases falling outside the mean ± 2SD range, we utilized the second method, a deep learning-based algorithm.This model, modified Inception (13), classifies CTA source images from the vertex to the C1 atlas as target regions.We used 100 randomly selected cases (4,020 CTA source images), where the first method correctly identified the vertex and C1 atlas.Of these, 89 cases were used for training and 11 cases for validation.The method accurately identified target slices within a 5% margin of error in all cases in the validation dataset.By combining these two techniques, we successfully produced MIP images with a uniform range across all patients in the training and validation dataset.Following skull stripping using our in-house algorithm, we created axial MIP images suitable for the deep learning method, utilizing software developed at our institution (14).

Vessel segmentation on maximal intensity projection image
We developed a 2D U-Net based on the Inception Module specifically for vessel segmentation in axial MIP images (15).This model was trained to segment vessels from the generated MIP images.The U-Net architecture integrates structural information from the network with the semantic information from the Inception Module, enabling more precise segmentation of vessels in MIP images.After generating MIP images, researchers manually segmented intracranial arteries in 208 randomly selected patients (16% of whom had TAVO) from the training (n = 189) and validation dataset (n = 19).For the training of the vessel segmentation model, we manually outlined all discernible anterior circulation intracranial arteries on MIP, including the distal ICA, MCA, ACA, posterior cerebral artery, and their visible branches.This manual segmentation was conducted under the supervision of an experienced vascular neurologist (W-S Ryu).The trained model achieved a Dice similarity coefficient of 0.80, indicating strong agreement with the manual segmentation performed by researchers supervised by a vascular neurologist.Two representative cases comparing automated intracranial vessel segmentation with manual segmentation is provided in Supplementary Figure 2.

Vessel occlusion detection algorithm
After automatically segmenting blood vessels on the axial MIP image, the vessel masks from each slice were combined to create a two-dimensional compressed image of the vessel masks.These compressed images serve as inputs for our deep learning model, trained for TAVO classification.We observed that using multiple compressed images of vessel masks at constant intervals as input significantly reduced model's performance compared to using a single compressed image.This was primarily due to overfitting of the algorithm on MCA slices where most TAVOs are located in a specific slice of the 3D volume.The issue of overfitting became more pronounced with an increase in the number of compressed images.Therefore, to mitigate this overfitting, we compressed the segmented vessel masks into a single image for this study.EfficientNetV2 was employed to train the TAVO classification model (16).Data augmentation was applied during the training process to prevent overfitting and alleviate domain shift problems.The augmentation algorithm was implemented using albumentations, a Python library for image augmentations (17).A batch size of 32 was maintained, with TAVO and non-TAVO cases sampled at a 1:1 ratio in each batch.For the deep learning training, intracranial LVO and isolated MCA-M2 occlusion were combined into a single category as TAVO.
For the training process, we used the AdamW optimizer with a batch size of 32, and a StepLR learning rate Scheduler with a step size of 7 and gamma of 0.1.Additionally, we addressed class imbalance by using a Weighted Random Sampler to sample the TAVO cases more frequently.The training utilized libraries including Python, PyTorch, TensorFlow, Pydicom, OpenCV, ITK, and was performed on an NVIDIA RTX A6000 GPU.The developed software operates on Window 10 or higher.

Statistical analysis
Data were presented as mean ± standard deviation, median (interquartile range), or number (percentage).To compare baseline characteristics between training and validation, internal test, external test 1, external test 2, we employed ANOVA or Kruskal-Wallis test for continuous variables and chi-square test or Fisher exact test, as appropriate.The diagnostic performance of the algorithm for detecting TAVO was assessed using sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV), calculated through receiver operating characteristics (ROC) analysis.The Youden index was utilized to determine the optimal threshold (18).We then stratified the TAVO patients into intracranial LVO and isolated MCA-M2 occlusion groups and conducted two subgroup analyses: one excluding subjects with isolated MCA-M2 occlusion and another excluding intracranial LVO.In this subgroup analysis, external test 1 and external test 2 dataset were combined due to small number of subjects in the external test 1 dataset.The DeLong test was used to calculate 95% confidence interval of the area under the curve (AUC) (19).For other parameters, bootstrap analysis with 1,000 repetitions was conducted to calculate 95% confidence intervals.All statistical analyses were performed using STATA 16.0 (STATA Corp., Texas, United States), with p < 0.05 considered statistically significant.

Baseline characteristics
The mean ages for the training and validation, internal test, external test 1, and external test 2 were 68.7, 68.3, 68.8, and 67.1 years, respectively, as shown in Table 1.The prevalence of male, atrial fibrillation, and history of prior stroke were similar across all groups.However, the occurrence of TAVO was less frequent in the external test 2 dataset compared to the others (13.4% vs. 18.0 to 23.4%).Notably, the CT vendors and imaging parameters, including slick thickness and pixel spacing, significantly varied between the groups (see Supplementary Table 1).For 100 randomly selected cases from the internal test dataset, the mean processing time from the input of source images to the output of results was 178 ± 11 s.

Diagnostic performance for overall TAVO
Representative examples of TAVO detection using deep learning in four patients with intracranial LVO or isolated MCA-M2 occlusion are illustrated in Figure 2. In the internal test dataset, the deep learning algorithm achieved an area under the AUC of 0.950 (95% CI, 0.915-0.971,see Figure 3).For the external test datasets 1 and 2, the AUCs were 0.970 (0.897-0.997) and 0.971 (0.924-0.990), respectively.Using a cutoff threshold of 0.5, the sensitivity ranged from 0.800 to 0.860 (Table 2), and specificity varied from 0.956 to 1.000.The result of the combined external test dataset is visualized in Supplementary Figure 3. Density plots of TAVO probability demonstrated that the deep learning algorithm could robustly differentiate between TAVO and non-TAVO in both internal and external datasets (as shown in Supplementary Figures 4A,B).

Diagnostic performance for intracranial large vessel occlusion
Excluding subjects with isolated MCA-M2 occlusion from the analysis significantly improved the performance of the deep learning algorithms.The AUC increased up to 0.967 in the internal dataset and to 0.993 in the combined external datasets, as shown in Table 3 and Supplementary Figure 5.With a cutoff point of 0.5, the sensitivity and specificity in the internal test were 0.943 (0.872-0.981) and 0.956 (0.935-0.972), respectively.The corresponding values in the combined external dataset were 0.947 (0.823-0.994) and 0.975 (0.951-0.989).

Diagnostic performance for isolated MCA-M2 occlusion
When subjects with intracranial LVO were excluded from the analysis, the diagnostic performance of deep learning algorithm was somewhat diminished.The AUC of internal and combined external datasets were 0.903 (0.812-0.944) and 0.916 (0.816-0.963), respectively.The sensitivities were 0.636 in the internal test and 0.632 in the combined external dataset.However, the specificities remained high at 0.956 and 0.975, respectively.

False positive and false negative samples in external test datasets
In the combined external test datasets, eight cases were incorrectly classified as TAVO (Supplementary Table 3).Among these, three patients had occlusions in the MCA-M3 or MCA-M4 segments, which are not considered candidate for EVT.Of the nine patients misclassified as non-TAVO, seven had isolated occlusions in the MCA-M2 segment.Two patients with false negative results and intracranial LVO had marginal TAVO probability score of 0.155 and 0.355, respectively.

Discussion
In this study, we developed a fully automated deep learning algorithm to detect intracranial anterior circulation arterial occlusions in CTA, which are likely candidates for EVT in hyperacute ischemic stroke.The algorithm underwent external validation in two different datasets and demonstrated high diagnostic sensitivity and specificity.When analyzing occlusion sites separately, the algorithm exhibited an excellent diagnostic performance for intracranial LVO.Although the performance for isolated MCA-M2 occlusion was slightly lower than for intracranial LVO, it remained competitive.
Until recently, a number of studies have reported on AI detection of LVO or TAVO (see Supplementary Table 4).However, these algorithms were either lacked external validation in previous research, or if validated, it was in a limited number of cases (20)(21)(22)(23).Moreover, earlier studies reported that the artificial intelligence algorithms achieved AUC scores ranging from 0.74 to 0.86, which may not sufficiently support early-career physicians (20)(21)(22)(23).Notably, in external validation sets with adequate sample sizes, our deep learning algorithm achieved AUCs of 0.961, 0.993, and 0.913 for total TAVO, intracranial LVO, and isolated MCA-M2 occlusion, respectively.
The ability of our deep learning algorithm to accurately predict isolated MCA-M2 occlusion with good performance is of significant importance.Although isolated MCA-M2 occlusion is an important and emerging target of the EVT (24), it was reported that the rate of misdiagnosis for isolated MCA-M2 occlusion is substantially higher than that for intracranial LVO even among the neuroimaging specialists (35.0% vs. 9.7%, respectively) (25), and the use of previously developed deep learning models has yielded worse results (50.8%) (20).This characteristic of isolated MCA-M2 occlusion have influenced our model as well, leading to improved specificity and NPV compared to previous studies (20,21), despite with somewhat less satisfactory sensitivity and PPV.This issue was largely due to false negatives in cases of short-segment (where collaterals reconstituted the M2 segment immediately distal to the occlusion) and incomplete (with antegrade flow) occlusions, where the reduction in the interhemispheric vessel density was too small for the algorithm to detect.However, in terms of TAVO detection, these occlusions missed due to false negative may not be ideal candidates for emergent intraarterial thrombectomy (26).
Our study may have a few potential clinical implications.First, our deep learning model can provide decent assistance to healthcare professionals who may not have much experience with ischemic strokes.Regardless of the reason for admission, the majority of ischemic strokes are initially encountered by non-specialized physicians (27).In such scenarios, identifying vascular occlusion and its location in brain imaging is a complex and demanding task, potentially delaying appropriate stroke treatment.Second, considering that our model processes vascular images in less than 180 s, it could significantly reduce the time taken to make EVT decision.In clinical practice, formal interpretation of brain imaging often requires several hours, and sometimes even more than a day (28).Therefore, rapid primary interpretation of vascular status through our model can play a crucial role in shortening the time to initiate EVT.Third, the objective and reproducible nature of our artificial intelligence software can aid in refining EVT process.This includes the preparation protocol for interventional devices based on occlusion patterns, which is another important factor in reducing reperfusion time.

Limitation of the study
One limitation of our study is the relatively small dataset size used for training the deep learning algorithm.This is particularly true for isolated MCA-M2 occlusion, where fewer cases were collected  compared to intracranial LVO, influencing the sensitivity and PPV performance of the model.The second caveat is that this study was conducted within a single country and given the higher prevalence of intracranial atherosclerosis in East Asian populations compared to Western ones, the assessment of the complete occlusion over pre-existing stenosis may have been challenging (29).Such factors could contribute to model's less favorable performance, underscoring the need for further research involving diverse ethnic groups.Therefore, radiologists and neurologists should be aware of the potential and causes of false negatives in the algorithm's output.Third, our algorithm does not include neck CT angiography and, therefore, is unable to detect tandem lesions in the cervical ICA.

Future directions
Collateral score and collateral status on CTA serve as important tools in evaluating collateral circulation in patients with TAVO (26).In CTA in patients with TAVO, collateral score and collateral status are used as important parameters to evaluate cerebral hemodynamics and tissue perfusion.These assessments help understand the adequacy of compensatory collateral circulation, which is essential for determining treatment strategies and predicting tissue viability in patients with ischemic stroke.Our developed system can estimate the course of collateral circulation by detecting and displaying the location of TAVO as a heat map.Collateral status involves a comprehensive assessment of the collateral circulation, considering factors such as collateral filling rate, extent of collateral vessels, and final tissue perfusion achieved through collateral flow (30).This assessment plays a pivotal role in predicting the likelihood of ischemic tissue salvage.Appropriate adjuvants contribute to increasing the likelihood of viable tissue despite vascular occlusion, potentially influencing treatment decisions such as endovascular reperfusion therapy.Therefore, auxiliary evaluation of collateral status may be considered in future studies.

Conclusion
We developed and validated a novel, fully automated deep learning algorithm derived from CTA to detect vascular occlusion suitable for EVT.While the algorithm could benefit from further improvements and real-world clinical evaluations, its potential as a tool to assist in the diagnosis of acute ischemic stroke in patients through detection of TAVO has been firmly established.

FIGURE 1
FIGURE 1 Depiction of the algorithm used to detect automatic thrombectomy amenable vessel occlusion (TAVO).(I) the acquisition of unprocessed CT angiogram images in DICOM (Digital Imaging and Communications in Medicine) format, accompanied by the automated selection of slices from the vertex and C1 atlas.(II) Skull removal, standard template registration, and maximal intensity projection (MIP) image rendering.(III) Using a rendered axial MIP image to segment blood vessels using deep learning (DL) and merging the blood vessel masks.(IV) Predicting TAVO using the merged blood vessel mask and generating a heatmap that identifies the region that influences the DL decision the most.

FIGURE 2
FIGURE 2 Representative cases for deep learning based thrombectomy amenable vessel occlusion (TAVO) detection.(A) 71-years old women with cervical internal carotid artery (ICA) occlusion (red arrows) without reconstruction of distal flow.(B) 53-years old man with left distal carotid and left proximal middle cerebral artery (MCA)-M1 occlusion (blue arrows).(C) 77-years old women with bilateral proximal MCA-M1 occlusion (green arrows).(D) 68-years old women with occlusion of the proximal left inferior M2 division (yellow arrows).In all cases, the heatmap visualize the occlusion site or a paucity of distal flow.

FIGURE 3 ROC
FIGURE 3 ROC analysis.ROC curves for detection of thrombectomy amenable vessel occlusion (TAVO) occlusions.(A) Internal test.(B) External test 1.(C) External test 2. Red dots indicate optimal cutoff points with the maximum Youden index.ROC, receiver operating characteristics; AUC, area under the curve.

TABLE 1
Baseline characteristics of training and validation, internal test, and external test datasets.
a Clinical data were missing in 4, 5, and 7 patients in training and validation, internal test, and external test 1 datasets, respectively.b Isolated M2-MCA occlusion was considered as TAVO.c Kruskal-Wallis test was used.MCA, middle cerebral artery; NIHSS, National Institutes of Health Stroke Scale; TAVO, thrombectomy amenable vessel occlusion.

TABLE 2
Diagnostic performance of deep learning algorithm detecting TAVO.
a Jmax represents, across all thresholds, the maximum Youden index (sensitivity + specificity − 1).As a secondary reference point, Jmax provides an optimality criterion with equal weighting for sensitivity and specificity.CI, confidence interval; GT, ground truth; PPV, positive predictive value; NPV, negative predictive value; AUC, area under the curve; TAVO, thrombectomy amenable vessel occlusion.

TABLE 3
Diagnostic performance of deep learning algorithm stratified by location of occlusion., confidence interval; GT, ground truth; NPV, negative predictive value; PPV, positive predictive value; TAVO, thrombectomy amenable vessel occlusion. CI