Evaluation of model performance to predict survival after transjugular intrahepatic portosystemic shunt placement

Background/Aims The MELD score was developed to predict survival after transjugular intrahepatic portosystemic shunt (TIPS) placement. Given changes in practice patterns and development of new prognostic tools in cirrhosis, we aimed to evaluate common models to predict mortality after TIPS placement. Methods Analysis of consecutive patients who underwent TIPS placement for ascites or bleeding. Performance to predict 90-day mortality was assessed by C statistic for six models (MELD, MELD-Na, CLIF-C ACLF, Child-Pugh, Platelet-Albumin-Bilirubin, and Emory score). Added predictive value to MELD score was assessed for univariate predictors of 90-day mortality. Stratified analysis by TIPS indication, emergent placement status, and TIPS stent type was performed. Results 413 patients were analyzed (248 with variceal bleeding, 165 with refractory ascites). 90-day mortality was 27% (113/413). Mean MELD score was 15 ± 7.9. MELD score best predicted mortality for all patients (c = 0.779), for variceal bleeding (c = 0.844), and for emergent TIPS (c = 0.817). CLIF-C ACLF score best predicted mortality for refractory ascites (c = 0.707). Addition of sodium to the MELD score did not improve predictive value across multiple strata. Addition of hemoglobin improved MELD score’s predictive value in variceal bleeding. Addition of age improved MELD score’s predictive value in refractory ascites. Conclusions MELD score best predicted 90-day mortality. Addition of sodium to the MELD score did not improve its performance, though mortality prediction was improved using Age-MELD for ascites and Hemoglobin-MELD for bleeding. An individualized risk stratification approach may be best when considering candidates for TIPS placement.

Introduction Transjugular intrahepatic portosystemic shunt (TIPS) placement has been an established therapy for complications of cirrhosis and portal hypertension dating back to 1989, when the procedure was first described for successful treatment of recurrent bleeding varices. [1] While randomized trials have demonstrated survival advantages for TIPS in patients with refractory ascites and variceal bleeding, [2][3][4] proper patient selection remains vital due to significant risks of post-procedure hepatic encephalopathy, liver failure, and the high overall morbidity/ mortality in this population. [5] To this end, prognostic scores have been developed and refined for multiple cohorts of patients with cirrhosis, including candidates for TIPS. The most widely used tool, the MELD score, [6] was originally developed to predict early death among those undergoing elective TIPS placement. [7] In more recent years, the MELD score has been adapted to include serum sodium and is used to model overall survival for transplant allocation in cirrhosis. [8] Over time, there have been updates to clinical guidelines and adoption of new procedural techniques that have impacted practice patterns and outcomes for TIPS recipients. Furthermore, there are key differences in patient characteristics depending on TIPS indication (e.g., refractory ascites vs. variceal bleeding) and acuity (e.g., emergent vs. elective TIPS referral) that may warrant real-time adjustments to prognostic models based on the relevant clinical scenario. Therefore, we aimed to evaluate six new and/or existing prognostic tools in cirrhosis (MELD score, [6] MELD-Na score, [8] Chronic Liver Failure Consortium Acute on Chronic Liver Failure [CLIF-C ACLF] score, [9] Child-Pugh [CP] score, [10] Platelet-Albumin-Bilirubin [PALBI] score, [11] and Emory score. [12]), stratified by these key clinical factors, in order to find the optimal models to predict 90-day survival after TIPS placement.

Patient population and data collection
We performed a retrospective cohort study of all patients who had a first-time TIPS successfully placed for refractory ascites or variceal bleeding between 1995 and 2016 within a network of acute care hospitals (Partners Healthcare, Massachusetts, USA), including one liver transplant center (Massachusetts General Hospital, Boston, MA) where the majority of TIPS cases were referred. Data were identified using a centralized clinical data warehouse designed for research and quality improvement purposes. [13,14] TIPS recipients were identified using the ICD-9 code "39.1 -intra-abdominal venous shunt" (or related ICD-10 codes) as well as review of TIPS procedure censuses. At least two authors manually identified all TIPS recipients to confirm the ICD-9/10 code corresponded to a new TIPS placement and the indication was consistent with guideline definitions. [15] Patients were excluded if they had a TIPS placed for other reasons (such as portal vein thrombosis or non-cirrhotic portal hypertension).

TIPS procedure and clinical care
TIPS placement was performed according to standard clinical practice, in interventional radiology suites under general anesthesia. [16][17][18] Bare metal stents (WALLSTENT, Boston Scientific, Natick, Massachusetts) were used starting in 1995, and were phased out by 2011. Covered stents (VIATORR, Gore, Flagstaff, Arizona) were used from 2003 to 2016. Patients were admitted for at least 24 hours observation following the procedure, and subsequent outpatient care was provided in interventional radiology and hepatology clinics. All data obtained for this study was taken in the context of routine care and was available in the longitudinal electronic medical record. Laboratory data were taken from the most recent values immediately prior to TIPS, up to 7 days prior to the procedure.

Prognostic scores and outcomes
Patients were assessed for the primary outcome of death by 90 days after TIPS placement. Stratified analysis was performed by (1) indication for TIPS (variceal bleeding vs. refractory ascites), (2) emergent vs. elective TIPS placement for variceal bleeding, and (3) use of covered vs. uncovered stent. Emergent TIPS was defined as requiring vasopressor support for hemorrhagic shock or use of an esophageal balloon tamponade. Post-TIPS hepatic encephalopathy noted if severe enough to require an unscheduled outpatient or emergency room visit. Patients whose vital status was unconfirmed at 90 days were excluded as being lost to follow-up (n = 6). Patients with hepatic hydrothorax (n = 20) were included in the refractory ascites group, and sensitivity analyses were performed by excluding these patients, as well as comparing them to those with ascites alone.
Six prognostic scores were assessed for their ability to predict 90-day mortality: MELD score, MELD-Na score, CLIF-C ACLF score, CP score, PALBI score, and Emory score. See S1 Table for individual score components. Two additional sensitivity analyses were performed for the performance of these six prognostic scores: evaluating one year mortality, and evaluating 90-day transplant-free survival. Individual variables were assessed to predict 90-day mortality in univariate analysis after stratification by TIPS indication. Variables represented in the MELD/MELD-Na score (total bilirubin, INR, serum creatinine, serum sodium) and the top univariate predictors of 90-day mortality were comparatively assessed for discrimination to predict 90-day mortality across multiple patient strata (bleeding vs. ascites, emergent vs. elective, covered vs. uncovered stent) using the methods described below.

Statistical analysis
For each prognostic score, area under the receiver operating characteristic (AUROC) curves/C statistics were calculated to evaluate model performance. These were compared within each stratum using Chi square testing and were displayed with 95% Wald confidence limits. The incremental improvement of adding individual variables to the MELD score was presented using three independent metrics: (a) direct change in C statistic (receiver operating characteristic contrast estimation using Chi square testing), (b) category-free net reclassification index (NRI) and (c) integrated discrimination index (IDI). [19] All model assumptions were checked and given a normal distribution, parametric testing was used for all univariate and multivariable analysis. Continuous variables were presented as means ± standard deviation. Descriptive tables were stratified by indication for TIPS and analyzed using Student's t-tests and Chi square tests (or Fisher's exact test when cells contained less than 5 values) for continuous and categorical variables, respectively. SAS version 9.4 (Cary, NC) was used for all analyses. Twotailed p values � 0.05 were considered statistically significant.

Ethics statement
The Partners Institutional Review Board approved this study. The need for informed consent was waived. All procedures and practices abide by the guidelines set forth by the Declarations of Helsinki and Istanbul.

Demographics and TIPS characteristics
456 patients underwent TIPS creation during the study period. 43 patients were excluded, including 37 first-time TIPS placed for other indications and 6 patients with unverifiable vital status at 90 days. 413 patients were included in the final analysis (see Fig 1). 248 TIPS were placed for variceal bleeding and 165 TIPS were placed for refractory ascites. Mean age was 56 ± 11.4 years, 123/413 (30%) were female, and the most common etiology of cirrhosis was alcoholic liver disease (see Table 1).
Hemodynamic and technical details of the TIPS procedure are described in Table 1. For all patients, the mean reduction in portosystemic gradient was 11 ± 6.5 mm Hg. Collateral veins were embolized during 57/411 (14%) of all procedures. A 10-mm diameter stent was used in 353/ 407 (87%) of cases. Covered stents were used in 254/410 (62%) of cases. Among those with TIPS placed for variceal bleeding, 61/248 (25%) were placed emergently. There were no major differences in procedural characteristics between patients with variceal bleeding and refractory ascites.  Table 2 describes mean prognostic scores for all patients, and compares them by TIPS indication. Four of the six scores were available for all 413 patients (n = 389 for CP and Emory scores due to missing pre-TIPS encephalopathy status). For all patients, mean MELD score was 15 ± 7.9, mean MELD-Na score was 18 ± 7.6, mean CLIF-C ACLF score was 40 ± 7.3, mean CP score was 9.3 ± 2.09, mean PALBI score was -2.4 ± 0.72, and mean Emory score was 1.2 ± 1.20. MELD score (p = 0.41) and CLIF-C ACLF score (p = 0.85) were similar between bleeding and ascites groups. Patients with ascites had higher MELD-Na, CP, and PALBI scores (p < 0.001 for all), and had lower Emory scores (p < 0.001).

Univariate predictors of 90-day mortality
Due to major baseline differences between patients with variceal bleeding and refractory ascites, analysis of univariate predictors of 90-day mortality were stratified by indication for TIPS placement (see Table 3). All 6 prognostic scores significantly predicted 90-day mortality in both bleeding and ascites subgroups (p � 0.01 for all).

Comparison of C statistics of prognostic scores
The AUROCs/C statistics for prediction of 90-day mortality for each prognostic score are presented in Table 4. Among all 413 patients, MELD score best predicted mortality (c = 0.779).  Its performance was similar to MELD-Na score (c = 0.767; p = 0.42). MELD score outperformed CLIF-C ACLF score (c = 0.695; p = 0.03), CP score (c = 0.673; p < 0.001), PALBI score (c = 0.712; p = 0.02), and Emory score (c = 0.667; p < 0.001). Among 248 patients with variceal bleeding, MELD score best predicted mortality (c = 0.844). Its performance was similar to MELD-Na score and it outperformed all other scores. Among 165 patients with refractory ascites, CLIF-C ACLF score best predicted mortality (c = 0.707), though its performance was statistically similar to all other scores. Among 61 patients who received emergent TIPS for bleeding, MELD score best predicted mortality (c = 0.817) and it outperformed all other scores. Among 187 patients who received elective TIPS for bleeding, MELD score best predicted mortality (c = 0.836). Its performance was similar to that of MELD-Na score and it outperformed all other scores. Among 156 patients where an uncovered stent was used, MELD score best predicted mortality (c = 0.732). Among 254 patients where a covered stent was used, MELD score best predicted mortality (c = 0.795). Added predictive value of sodium, hemoglobin, and age to the meld score With MELD score as the reference, the added predictive value of three variables was assessed: hemoglobin and age (the strongest non-MELD predictors of mortality for bleeding and ascites  for all patients (NRI 0.25; p = 0.02 and IDI 0.013853; p = 0.05) and refractory ascites (NRI 0.43; p = 0.01 and IDI 0.05274; p = 0.001; see Table 5).
For patients with refractory ascites, the risk of 90-day mortality can be modeled for MELD score and age as: 0:118 � ½MELD score� þ 0:063 � ½Age� À 6:668 For patients with variceal bleeding, the risk of 90-day mortality can be modeled for MELD score and hemoglobin as:  Model performance for post-TIPS survival death or liver transplant compared to death alone (see S2 Table). There was no difference in 90-day transplant-free survival between patients with bleeding and ascites (p = 0.73; see S1 Fig).
At one year, overall mortality was 113/413 (40%). Prognostic scores performed similarly when predicting mortality at one year compared to 90 days, though AUROC was lower across all scores (see S2 Table). Among those with available data, 150/380 (39%) experienced hepatic encephalopathy and 49/399 (12%) required a TIPS revision (for technical reasons or due to encephalopathy) within 90 days of TIPS placement. In sensitivity analysis, 20 patients with hepatic hydrothorax were similar to those with ascites alone (n = 145), including in age, sex, race, ethnicity, serum sodium, creatinine, hemoglobin, and pre/post-TIPS portosystemic gradient. Patients with hepatic hydrothorax were less likely to have CKD, had lower platelet counts, higher total bilirubin, and higher MELD scores (19 ± 8.4 vs. 15 ± 7.2; p = 0.02; see S3 Table). Mortality at 90 days was similar between those with hydrothorax and those with refractory ascites (30% vs. 26%; p = 0.72). Exclusion of the 20 patients with hydrothorax did not meaningfully change the C statistics for 90-day mortality of the six prognostic scores.

Discussion
In a large population of 413 patients with cirrhosis who received TIPS for variceal bleeding or refractory ascites, we provide a detailed analysis of six widely available prognostic models for 90-day mortality, including the recently adopted MELD-Na and CLIF-C ACLF scores. To our knowledge, this is the first time CLIF-C ACLF score has been assessed among TIPS recipients. We found that MELD score performed best in predicting post-TIPS mortality, and that the addition of sodium to the MELD score did not significantly improve its prognostic ability. However, in specific subgroups, the prognostic ability of the MELD score could still be improved by adding important clinical variables, such as "Hgb-MELD" in those with variceal bleeding and "Age-MELD" in those with refractory ascites.
Overall, all six prognostic scores significantly predicted 90-day mortality, as well as one year mortality and 90-day transplant-free survival in sensitivity analyses. This is not surprising, as there are many overlapping components to these scores. We presented three measures of model performance in our analysis: direct change in AUROC, NRI, and IDI. There is no universally accepted method for assessing added usefulness of new metrics to existing predictive models. While a change in AUROC is perhaps most concrete in describing model performance, NRI and IDI may be more sensitive to detect incremental improvement. [20] In most subgroups of our study, MELD score generally performed best, followed closely by MELD-Na score. In patients with refractory ascites, CLIF-C ACLF score had the best C statistic, though it was not statistically significantly better than MELD or MELD-Na scores. When comparing MELD and MELD-Na scores more directly, addition of sodium did not have added predictive value, and MELD-Na score did not perform better than MELD score alone in any stratum. There may be two reasons for this. First, unlike in overall survival in cirrhosis, where MELD-Na score demonstrates prognostic improvement compared to MELD score, [8] patients referred for TIPS may be uniquely selected such that the influence of serum sodium is not as apparent. Second, there may be specific demographic and geographic factors that lessen the impact of sodium. All hospitals in this study were in Organ Procurement and Transplant Network (OPTN) Region 1, which has a higher median MELD score at the time of liver transplantation than most other regions in the United States. [21][22][23] This may also affect timing or selection for TIPS referral in ways not easily captured from clinical data.
In their cornerstone study evaluating the predictive value of liver disease scores after TIPS placement, Gaba et al. examined prognostic scores in a similar group of 211 TIPS recipients out of a single center in OPTN Region 7. [24] In their thorough analysis, these authors came to similar conclusions: namely, that MELD and MELD-Na score performed best overall. There are several differences to highlight between our two studies. First, scores in their study had better performance overall compared to this one, including MELD score (c = 0.816 for Gaba et al. compared to c = 0.779 in our study), which highlights the variable performance of MELD based on geography and local referral patterns. Second, their models had better performance for uncovered stents, whereas performance was better for covered stents in our study. This observation is less well-explained, but may also be due to population differences. Finally, while they conclude that MELD and MELD-Na performed similarly, we extend their conclusions by comparing these two scores more directly, using NRI and IDI to assess sodium's added predictive value to MELD. Given the lack of statistical difference between direct comparisons of MELD and MELD-Na models in Chi square analysis, as well as the failure to reject the null hypothesis of NRI and IDI testing, we conclude that addition of sodium does not improve the performance of MELD score among TIPS recipients. While it would be ideal to have a single study comparing centers in multiple transplant regions to help address these differences in populations, it is reassuring that our results corroborate those of Gaba et al, adding to the generalizability of both results.
The original derivation and clinical application of each prognostic score may help better contextualize their statistical performances. CP score was the oldest score analyzed, as it was developed from 38 patients taken for emergent surgical ligation of bleeding varices in 1973. [10] Despite its vintage, it has routinely performed well in populations receiving TIPS. [12,25] Three scores were more recently designed specifically to predict post-TIPS mortality: MELD score, [7] Emory score, [12] and PALBI score. [11] Among these three, MELD clearly performed the best. CLIF-C ACLF score was validated to predict mortality in patients hospitalized with acute-on-chronic liver failure. [9] Perhaps expectedly, the CLIF-C ACLF did not outperform MELD, as those with acutely decompensated cirrhosis are often not considered for TIPS placement, except in cases of emergent bleeding.
One important observation was that model performance varied by patient subgroup. Small differences in model performance were present between covered and uncovered stent types, though overall conclusions remained the same. This likely reflects improvements in practice patterns and TIPS placement technique over time, as uncovered stents were largely phased out by the midpoint of the study. Results from those receiving covered stents can be interpreted as a sensitivity analysis restricted to a more recent patient cohort. In general, prognostic scores had higher C statistics in patients with variceal bleeding compared to those with refractory ascites, suggesting that there may be other factors contributing to the morbidity of refractory ascites that may be not captured with our current prognostic tools. The addition of age improved MELD score's performance among those with ascites, and may be another clinical consideration when making treatment decisions in this population. There are many factors to weigh when considering TIPS candidacy, including the overall survival benefit seen in early TIPS placement for both variceal bleeding and refractory ascites. [2,3] Overall, we would recommend clinicians continue to take a holistic and evidence-based approach when evaluating patients for TIPS placement, using scores like MELD as a guide, but not as the absolute arbiter of candidacy.
This paper should be viewed in the context of its limitations. This is a retrospective study, and as such, findings should be viewed as associations, rather than as causal relationships. All sites were located in OPTN Region 1, which colors the practice patterns of referring providers and may not reflect experiences in other transplant regions. This is a common limitation of the TIPS literature, as there is no detailed national clinical registry for TIPS outcomes (beyond the limited datasets available from payer networks, such as the National Inpatient Sample). [26] Our study is the largest of its kind, and we believe it provides an important reference point for other regions to assess and validate. Unavailable data limited the calculation of CP and Emory scores for 24 patients, but we believe this analysis was sufficiently large and comprehensive to convey its key messages. Cause of death was also not available, which may introduce bias in the performance of the scores.
Among six prognostic models, MELD score best predicted 90-day mortality. Addition of sodium to the MELD score did not improve its predictive value. Significant predictors in the subgroups of variceal bleeding (hemoglobin) and refractory ascites (age) did improve MELD's predictive ability in their respective strata. An individualized risk stratification approach, including modifications like an "Age-MELD" or "Hgb-MELD" score for appropriate subgroups, may be best when considering candidates for TIPS placement. A nationwide registry of post-TIPS outcomes is needed to better generalize these findings.