Introduction

Subarachnoid haemorrhage (SAH) is a devastating event, which is marked by sudden onset of severe headache, causing substantial case-fatality. In 85% of the patients, the SAH is caused by rupture of an aneurysm (aSAH) [1, 2]. From those who survive the first month, approximately one-third remains dependent with respect to daily activities during their remaining lifetime [1]. Amongst patients who regain independency, quality of life remains reduced [3].

Early prediction of short term outcome in terms of case-fatality may support clinical decision making and may provide realistic and evidence based expectations to patients and relatives. Predictions may also be used to classify patients according to prognostic risk, which may be useful to compare outcome between different patient series, to study treatment results over time, or to stratify patients for randomised clinical trials (RCT).

Several other prognostic studies on outcome after aSAH have been performed, but most included relatively small numbers. Two included substantial numbers allowing analysis of the effects of multiple independent factors [4, 5]. However, these studies did not examine case-fatality, but arbitrarily dichotomized ordinal outcome scales (modified Rankin Scale or Glasgow Outcome Scale). Moreover, due to their design, these studies were unable to examine the effect of treatment on prediction of outcome.

Our aim was to develop a prognostic model for 60 day case-fatality, based on clinical features and neuro-imaging, regularly readily available on admission to a neurological or neurosurgical unit. These data were obtained from a large RCT conducted in mainly European countries.

Methods

Patients

Data were collected prospectively by the Medical Research Council funded International Subarachnoid Aneurysm Trial (ISAT) (International Standard Randomised Controlled Trial, number ISRCTN49866681). Full details of the ISAT study are available elsewhere [6]. The aim of the trial was to determine whether treatment using endovascular coiling reduced the risk of patients being dependent or dead at 1 year by 25 percent when compared with neurosurgical treatment (clipping).

Predictors and outcome

We considered all patient characteristics that can be collected easily and reliably within the first hours after hospital admission and that were also present in the ISAT database. These included age, gender, previous occurrence of SAH, CT scan Fisher grading, lumbar puncture, World Federation of Neurosurgical Societies (WFNS) grading, number of intracranial aneurysms, location of the aneurysm, maximum lumen size of the aneurysm, vasospasm on angiography, and intended treatment by randomization. Fisher grading of blood visible on a plain CT scan runs from grade 1 (“no blood visible”) up to grade 4 (“intraventricular or intraparenchymal blood”). Lumbar puncture was not performed in all participants. If it was performed it was graded 1 (“no blood in cerebrospinal fluid”) or 2 (“xanthochromia or blood”); 0 otherwise (“no lumbar puncture”). WFNS scale runs from grade 1 (“Glasgow Coma Scale (GCS) 15 and no motor deficit”) to grade 5 (“GCS 3-6 with or without motor deficit”). One category additional to the standard WFNS scale was created in ISAT for those in whom WFNS could not be assessed; ‘grade 6’. The number of aneurysms was categorized in 1, 2, and 3 or more intracranial aneurysms. We discerned four aneurysm locations: Anterior Cerebral Artery (ACA), Internal Carotid Artery (ICA), Middle Cerebral Artery (MCA), and Posterior Circulation (PC). The maximum lumen size of the aneurysm was expressed in millimetres. Vasospasm was examined on angiography and categorized as ‘none’, ‘mild’, ‘moderate’, or ‘severe’. Treatment was either neurosurgical clipping or endovascular coiling; we used treatment as allocated by the randomization procedure. We developed the model based on cases with a complete set of data. The outcome was 60 day case-fatality.

Model

We used univariate logistic regression analysis to estimate the association between single predictors and outcome, expressed as an odds ratio (OR). Predictors have a statistically significant effect if the 95 percent confidence interval (95% CI) does not include the value one. The prediction model was developed with multivariable logistic regression with backward stepwise selection. All potential predictors were entered into the model and those that met Akaike’s Information Criterion (AIC) were selected into the model. AIC compares models based on how well they fit the data, but penalizes for the complexity of the model. AIC requires that the increase in model χ2 when entering a new predictor has to be larger than two times the degrees of freedom: χ2 > 2 df. When considering a predictor with one df, such as gender, this implies that χ2 has to exceed two, equivalent to P < 0.157. When considering a predictor with two df, χ2 > 4, or P < 0.135; and in case of four df, χ2 > 8, or P < 0.092 [7].

Performance

The performance of the model was assessed with respect to calibration and discrimination. Calibration is the ability of the model to produce unbiased estimates of the probability of the outcome. Calibration was examined with a goodness of fit test, which assesses agreement between predicted and observed risks over the full range of predicted probabilities [8].

Discrimination is the model’s ability to separate patients with different outcomes. To quantify the discrimination, we used the concordance (c) statistic. For binary outcomes, c is identical to the area under the receiver operating characteristic curve [7]. The c statistic evaluates whether those with higher predicted risk are more likely to die within 60 days among all possible pairs of patients with different outcomes. A model with a c statistic of 0.5 has no discriminative power at all, for example a coin flip. A c statistic of 1.0 reflects perfect discrimination.

Model validation

The performance of a prediction model is generally worse in new patients than initially expected. This ‘optimism’ can be studied with internal validation techniques [7]. Internal validity of our model was assessed with standard bootstrapping procedures [7]. Bootstrapping involves drawing samples of patients with replacement from the study population. Each sample can be considered as if one is repeating the data collection with the same number of patients and under identical circumstances as the original. The multivariable logistic regression coefficients were re-estimated in 300 bootstrap samples. Each of these 300 models was evaluated on the original sample. The average difference in the c statistic was determined to indicate the optimism in the initially estimated discriminative ability [7]. A shrinkage factor was estimated from the bootstrap validation procedure and we shrunk the regression coefficients to provide better predictions for future patients [7].

All statistical analyses were performed using R software, version 2.8.1 (R foundation for statistical computing, Vienna, Austria).

Results

A total of 2,143 patients were recruited into the ISAT trial by 43 neurosurgical centres, mainly in Europe. CT scans of 14 patients were not performed or available, and in one patient no information on vasospasm was available. We excluded these cases from our analysis. Data on the outcome were available for all patients. Thus, we performed complete case analysis on 2,128 patients (99.3%) of whom 153 (7.2%) died within 60 days.

The distribution of patient characteristics of the study population is presented in Table 1. For reasons of small numbers in the ‘severe’ category of vasospasm, we aggregated data from the ‘moderate’ and ‘severe’ categories into one category. Univariate analysis showed a statistically significant relation with 60 day case-fatality for age, lumen size, Fisher grade, lumbar puncture, WFNS grade, and vasospasm. Sex, location and number of aneurysms and intended treatment were not significantly associated with 60 day case-fatality (Table 1). In the multivariable model with stepwise backward selection age, lumen size, Fisher grade, and WFNS grade met AIC and were included in the final model. In Table 2 the chi square statistics with corresponding P-values are presented as well as the point estimate of the OR. Age and WFNS grade were the most important predictors.

Table 1 Population characteristics and univariate association with 60 day case fatality
Full size table
Table 2 Statistical parameters of the final model
Full size table

The goodness of fit test yielded a P-value of 0.86, suggesting that the model fitted the data in which it was developed well. The c statistic of the original model was 0.73, meaning that the model discriminates reasonably between patients who die within 60 days from onset of the SAH and those who survive this period.

Validation by means of 300 bootstrap samples resulted in a shrinkage factor of 0.85, which was applied to the betas of the model. The c statistic of the internally validated model was 0.70. Details of the final prognostic model for 60 day case-fatality are described in the appendix.

Discussion

We developed a prognostic model to predict the risk of 60 day case-fatality in individual patients after aSAH. Predictions were based on characteristics that are regularly readily available on admission to a neurological or neurosurgical unit and which were collected in a large clinical trial. The full model yielded a c statistic of 0.73.

Previously, several models to estimate the probability of unfavourable outcome after aSAH have been developed. Our model was similar to those; we included roughly the same predictors: age, clinical status, and lumen size [4, 5, 9]. However, our study is of added value because of the substantial size and the inclusion of both clipped and coiled patients. The studies by Hoh et al. [4] (n = 515) and Mocco et al. [9] (n = 148) contained relatively few patients. The small numbers of coiled patients (79 and 35, respectively) and the design of the study did not allow for taking the effect of treatment in consideration. The study of Rosengart et al. [5] (n = 3,667) was not able to do that either, since patients treated with Guglielmi or other detachable coils were excluded. All three studies used a dichotomized ordinal scale as an outcome, for which the cut off can be (arbitrarily) chosen in different studies. In a sense, examining case-fatality is also a dichotomization of an ordinal scale, though less arbitrary. Therefore, we are convinced that logistic regression is well suited for an outcome that is by its nature dichotomous, whereas for an ordinal outcome we would prefer specific modelling techniques.

Several limitations of this study should be acknowledged. This study used data from one large trial on a selected population of patients who survived the early stage up till a treatment decision and who were in equipoise regarding that decision on treatment with either endovascular coiling or neurosurgical clipping, which may limit external validity. The model may perform well in this development sample, but worse when applied to other groups of patients, for example, a less strictly selected population. Nonetheless, according to a recently published paper, the ISAT population proved to be quite similar to the population admitted with an aSAH to neurosurgical units in the United Kingdom [10]. Although in ISAT, a lower proportion of poor grade patients were enrolled. Validation of a prognostic model in independent patient series is therefore considered an essential next step [11]. However, since large samples of systematically collected data on aSAH are sparse, assessment of external validity is difficult. For now the external validity of our model remains to be established. This will be a topic of future research.

Although our model represents knowledge obtained from 2,128 SAH patients in equipoise regarding treatment, statistical models can never replace the clinician with regard to decision making; they can only assist with this task. A prediction for an individual aSAH patient a particular situation is always subject to uncertainty. The model makes certain structural assumptions and statistical interaction terms were not included. It is hence possible that specific patterns of risk factors are inadequately reflected in the model predictions. Therefore, predictions should be regarded with care and not directly be applied for treatment limiting decisions.

Although the performance of the presented model was satisfactory, it might potentially be improved by including neuro-imaging biomarkers other than lumen size, location, Fisher grade on plain CT scan, and vasospasm on angiography. Biomarkers regarding anatomy and morphology might be considered, as well as aneurysm characteristics obtained from three and four dimensional angiography [12, 13]. Performance may also be improved by inclusion of subsequent information obtained after admission, including temporal course, neuro-imaging at later time points, eventual rebleeding of the aneurysm, delayed ischemic deficit, and other parameters such as hydrocephalus. The objective of the present study, however, was to investigate prognostic models that predict 60 day case-fatality with predictors available on admission.

Statistical testing for calibration has a number of drawbacks. First, the null hypothesis is of good calibration. Hence, if we test calibration in a small study, we have low power and will not reject the null hypothesis unless miscalibration is very severe. On the other hand, even a model with very good, though not perfect, calibration will fail the test in case of a sufficiently large sample. Moreover, reported goodness-of-fit tests are usually non-significant if they reflect apparent validation on the data that were also used to construct the model. Such non-significant results may contribute to the face validity of a model, but have no real scientific meaning [7].

In conclusion, we presented a prognostic model for predicting 60 day case-fatality after aneurysmal SAH. Our model contained age, lumen size, Fisher grade, and WFNS grade as predictors. After calibration and internal validation, our model showed reasonable performance, although external validity of our model remains to be established.