The diagnostic reliability and validity of noninvasive imaging modalities to assess leptomeningeal collateral flow for ischemic stroke patients

Abstract Leptomeningeal collateral flow (LMF) is associated with infarct area and clinical outcome for ischemic stroke patients. Although LMF can be detected by multiple imaging methods, but their diagnostic performance is uncertain. The aim of this study was to evaluate the diagnostic validity or reliability of noninvasive image methods in assessing LMF. Databases included PubMed, Web of Science, Embase, and Cochrane Library. Original observational cohort studies. Ischemic stroke patients. Different noninvasive image methods to assess LMF. Newcastle–Ottawa Scale to evaluate the quality of the studies; forest plot to show pooled results; I2 and Egger test to evaluate the heterogeneity and publication bias. Thirty of the 126 selected studies were eligible. For CT angiography, the interobserver agreement ranged from 0.494 to 0.93 and weighted kappa was 0.888; for patients receiving thrombolysis or endovascular treatment, 0.68 to 0.91; 0.494 to 0.89 for the 2-point system, 0.60 to 0.93 for the 3-point system, 0.68 to 0.87 for the system of >4 points; area under the curve (AUC) was 0.78. For perfusion computed tomography (CTP), the interobserver agreement ranged from 0.724 to 0.872; for patients receiving thrombolysis or endovascular treatment, 0.74 to 0.872; 0.724 for the 2-point system, 0.783 to 0.953 for the 3-point system; the intraobserver agreement was 0.884; AUC was 0.826. For MRI-fluid attenuated inversion recovery (FLAIR), the interobserver agreement ranged from 0.58 to 0.86; for patients receiving thrombolysis or endovascular treatment, 0.75 to 0.86; 0.86 for the two-point system, 0.77 to 0.87 for the system of more than 5 points; AUC was 0.82. No pooled data of CTP and FLAIR. The difference cohort study had difference bias. The unpublished data were not included. CT angiography is a good tool for assessing LMF. CTP shows a good validity and reliability, but its diagnostic value needs more evidence. FLAIR is a good modality to assess LMF. These image methods had better validity and reliability to evaluate LMF of patients receiving thrombolysis or endovascular treatment than all ischemic stroke patients.


Introduction
For ischemic stroke patients, efficient leptomeningeal collateral flow (LMF) can improve the clinical outcome by reducing the volume of penumbra. [1][2][3] The LMF condition can be used to evaluate the ischemic stroke patient's outcome and prognosis. [3] Presently, multiple imaging techniques are available to examine LMF. The application of digital subtraction angiogra-phy (DSA), the golden standard to assess LMF, is limited by its invasiveness and complexity, as well as the patient's or the hospital's condition. [4] In clinical practice, noninvasive methods with simple operation and wide application, such as CT angiography (CTA) and MRI-fluid attenuated inversion recovery (FLAIR), have also been introduced to assess LMF. [3,[5][6][7][8][9] Normally the effectiveness of an imaging method should be evaluated by its validity and reliability. The validity indicates the veracity of a method. The reliability, which refers to the agreement reached between different observers or at difference times by the same observers, indicates the stability and credibility of a method. However, the effectiveness of these noninvasive methods in assessing LMF was uncertain, [5][6][7][8][9] and the different grading systems of these methods further intensify the uncertainty. [5][6][7][8][9] A review published in 2012 showed the difference in reliability among the different imaging methods. [10] But the study had not evaluated the validity of different image methods.
To find out a better tool to assess LMF, We conducted a systematic review and meta-analysis using the recently published studies evaluating the diagnostic validity and reliability of multiple imaging methods. [3,[5][6][7][8][9]

Methods
The study design, data collection and analysis, and result report all followed the MOOSE Guidelines.

Search strategy
We searched the databases (PubMed, Web of Science, Embase, and Cochrane Library) for studies published before October 1, 2019. The key words of search were ([leptomeningeal collateral] AND stroke). Then we obtained articles evaluating the diagnostic validity, reliability, and consistency of LMF-assessing methods.

Study selection
We (CC and YH) independently screened the articles. We downloaded the articles after reading the titles and abstracts. After reading full text, we further excluded the articles of the unwanted type or with unrelated outcomes. From the references of the articles, we supplemented the qualitative articles. The search process was shown in Figure 1. We did not exclude the studies for language and geographic reasons.

Inclusion criteria
Inclusion criteria were: original observational cohort studies; studies on ischemic stroke patients; studies on diagnostic validity or reliability of different imaging methods to assess LMF; studies demonstrated validity by area under curve (AUC) and reliability by kappa coefficient.

Exclusion criteria
Exclusion criteria were: studies with the number of patients <20; animal studies; letters, meeting reports, unpublished studies.

Data collection and quality evaluation
We (CC and JB) independently extracted the data, including name of the author, date of publication, region, number, age and type of subjects, study design, imaging techniques, grading system, value of AUC, and value of kappa. We independently evaluated the quality of the studies by Newcastle-Ottawa Scale and gave a score ranging from 1 to 5 points. A higher score was given to the study with a scientific design, appropriate inclusion and exclusion criteria, well-handled confounding factors and bias, rationally assessed risk factors and outcome. The disagreement on the same study was solved through discussion.
The study was performed in accordance with the Declaration of Helsinki and the ethical standards of the institutional and/or national research committee. The study was approved by the Ethics Committee of West China Hospital, Sichuan University.

Statistical analysis
The diagnostic validity was evaluated using the extracted values of AUC and 95% confidence interval (CI) (available data). The reliability was evaluated with the extracted kappa and 95% CI (available data). The reliability was explained by the interobserver agreement (the consistency of observation results of different observers for the same group of objects in the same situation) and the intraobserver agreement (the consistency of observation results of the same observer for the same group of objects in the same situation). When assessing intraobserver agreement, the value of intraclass correlation coefficient was equal to the value of kappa. [11] So we extracted value of intraclass correlation coefficient as value of kappa when a study showed intraobserver agreement.
We pooled a weighted kappa value and 95% CI using Fleiss et al's methods. [11] We demonstrated the kappa value of each eligible study and pooled kappa value in a forest plot. We evaluated the reliability by Landis and Koch' criteria [12] (value of kappa: <0, poor; 0.1-0.2, slight; 0.21-0.40, fair; 41-0.60, moderate; 0.61-0.8, substantial; 0.81-1, almost perfect). The heterogeneity of the studies was evaluated by I 2 . When I 2 value was not >0.5, these studies were considered to have no significant heterogeneity. When available, the different grading system and subgroup of modalities, data of stroke patients receiving thrombolysis or endovascular treatment were further analyzed.
All the data were processed by StataMP 14 software (Stata Corp, College Station, TX).

Study identification and description
Among the 653 searched articles from the databases, we downloaded 156 articles. Finally, 28 eligible articles were selected for this meta-analysis. The search process was showed in Figure 1. Among these articles, 18 evaluated the reliability or consistency of CTA in assessing LMF; 4 articles evaluated CTP in assessing LMF (Table 1); 5 articles evaluated FLAIR in assessing LMF; 2 articles evaluated other magnetic resonance imaging (MRI) modalities in assessing LMF (Table 2).

Discussion
This meta-analysis reviewed the diagnostic performance of different image modalities in assessing LMF. For CT methods, CTA showed an unstable interobserver agreement, a substantial to perfect intraobserver agreement, and a better validity. For systems of different grades, the systems of 3 points or >4 points had a better interobserver agreement than the system of 2 points. For different modalities, tMIP or peak phase modalities had a better interobserver agreement than the source image modalities. CTP had a substantial to perfect interobserver agreement, a perfect intraobserver agreement, and good validity. The systems of 3 points or >4 points had a better interobserver agreement than the 2-point system. For difference modalities, TTP modalities showed a better interobserver agreement than other modalities. Multi-CT had a better diagnostic performance. FLAIR had a substantial to perfect interobserver agreement and good validity. Difference grade system had No significantly different interobserver agreement for FLAIR. MRA had a perfect interobserver agreement and ASL had a perfect intraobserver agreement. When used in patients receiving thrombolysis or endovascular treatment, CTA, CTP, and FLAIR all had a better reliability than used in commonly ischemic stroke patients.
DSA used to be a major method to evaluate LMF, but its clinical use has shrunk due to its invasiveness and complexity. Jansen et al [6] showed that DSA results were not associated with the patients' clinical outcome, indicating its limitation in clinical use. CT is more widely used for acute ischemic stroke patients because of its high efficiency and universality. Several studies showed that CTA results were positively correlated with ischemic stroke patients' outcome and prognosis. [6,23] Therefore, CTA may be a better tool for LMF assessment. CTP is another commonly used CT modality to assess LMF. One study concludes that LMF condition of CTP assessment had no significant relationship with ischemic stroke patient's outcome and prognosis. [28] Therefore, the diagnostic value of CTP in LMF assessment needs more evidence. Combined use of CTA and CTP may work. Liu et al and van Seeters T showed the high diagnostic value of multi-CT. [7,27] But more clinical studies are needed to verify the advantages of multi-CT. Although MRI is often used to evaluate the lesion in acute ischemic stroke patients, it is more frequently applied in ischemic stroke patients of stable stage because it is more time-consuming. FLAIR is a common MRI modality to evaluate LMF. Haussen et al and Karadeli et al showed that FLAIR-detected vascular hyperintensity was correlated with the ischemic stroke patient's LMF status and infarct core. [8,36] Therefore, FLAIR may be a preferable tool for LMF assessment. Although FLAIR and CTA are both preferable tools for LMF assessment, the 2 modalities only have a weak positive correlation. [36] These 2 modalities are based on different imaging mechanisms and applied in different stages, which may partly explain their weak association. [3,8] MRA and ASL are also MRI modalities to assess LMF, and both have a good reliability. But there lacked evidence proving their effect in the stroke patient's prognosis. More studies are needed to verify the advantage of the 2 modalities.
We found that different grading systems had different effects on the modality's reliability. For CTA, the 2-point grading system had a poorer reliability than the other grading system, suggesting that grading system of higher scores could improve the agreement of CTA. Meanwhile, we found the different grading systems did not exert significant effects on the reliability of CTP and FLAIR. The different grading system had difference effect above modalities partly because these modalities had each distinctive imaging mechanism. [3,7,8] When used to evaluate the LMF of patients receiving thrombolysis or endovascular treatment, CTA, CTP and FLAIR all showed a better reliability. The thrombolysis or endovascular treatment could partly recover the blood supply in patients. Hence, compared with other patients, these patients had a more stable blood supply for the lesion, [37] which increased the stability of LMF assessment. In addition, patients receiving thrombolysis or endovascular treatment often had more severe lesions, [37] so these patients' outcome depend more on larger brain collateral circulation, such as LMF. The above-mentioned 2 reasons indicate that CTA, CTP, and FLAIR are more suitable for patients receiving thrombolysis or endovascular treatment.
Our meta-analysis has several limitations. First, we could not pool data from studies on CTP and FLAIR, which might have affected the statistical accuracy of the results. Second, it is unavoidable that different bias could be found in different cohort studies. But the cohort studies reflecting a real-world condition had a better practical meaning for clinical. In addition, unpublished data and meeting abstracts were not included in this study, which could increase the bias of results.
In summary, CTA is a good tool for LMF assessment, and a grading system of >3 points could have a better reliability. CTP shows a good validity and reliability in evaluating LMF, but the diagnostic value needs more evidence. FLAIR is a good modality to assess LMF for ischemic stroke patients. The above 3 image methods had better validity and reliability to evaluate LMF of patients receiving thrombolysis or endovascular treatment than all ischemic stroke patients. Multi-CT, MRA, and ASL may serve as an optimum choice, but their advantages need more evidence.