Estimating the effectiveness of a hospital’s interventions in India: impact of the choice of disability weights

Abstract Objective To calculate the effect of using two different sets of disability weights for estimates of disability-adjusted life-years (DALYs) averted by interventions delivered in one hospital in India. Methods DALYs averted by surgical and non-surgical interventions were estimated for 3445 patients who were admitted to a 106-bed private hospital in a semi-urban area of northern India in 2012–2013. Disability weights were taken from global burden of disease (GBD) studies. We used the GBD 1990 disability weights and then repeated all of our calculations using the corresponding GBD 2010 weights. DALYs averted were estimated for surgical and non-surgical interventions using disability weight, risk of death and/or disability, and effectiveness of treatment. Findings The disability weights assigned in the GBD 1990 study to the sequelae of conditions such as cataract, cancer and injuries were substantially different to those assigned in the GBD 2010 study. These differences in weights led to large differences in estimates of DALYs averted. For all surgical interventions delivered to this patient cohort, 11 517 DALYs were averted if we used the GDB 1990 weights and 9401 DALYs were averted if we used the GDB 2010 disability weights. For non-surgical interventions 5168 DALYs were averted using the GDB 1990 disability weights and 5537 DALYS were averted using the GDB 2010 disability weights. Conclusion Estimates of the effectiveness of hospital interventions depend upon the disability weighting used. Researchers and resource allocators need to be very cautious when comparing results from studies that have used different sets of disability weights.


Introduction
Comprehensive summary measures of population health were estimated in the global burden of disease (GBD) 1990, 2004 and 2010 studies. [1][2][3] The GBD 1990 study was commissioned by the World Bank and quantified the health effects of more than 100 diseases and injuries in each of eight regions of the world. 1 The disability-adjusted life-year (DALY) was used to facilitate comparisons of health outcomes and measures of the effectiveness and cost-effectiveness of various interventions. 1 Subsequently, there has been extensive debate on many of the variables that affect estimates of DALYs, such as the number of years lost on death, disability and age weights and time discounting. [4][5][6][7][8][9] In the GBD 1990 study, an expert panel arbitrarily assigned disability weights to a comprehensive set of disease conditions, by using the so-called person trade-off method. 1 After the results of the study were published, apparent inconsistencies in the derivation of these weights were noted. 10 The GBD 2004 study, 2 which focused mainly on injuries, was also criticized as the disability weights for several injuries appeared illogical. 10 Such inconsistencies led to the appropriateness and usefulness of many disability weights being questioned. 10 The GBD 2010 study 3 tried to address these criticisms using multinational community and webbased surveys. In these surveys, more than 30 000 respondents were asked to choose the healthier of two hypothetical health states. 11 Several researchers have pointed out that some of the disability weights estimated in the GBD 2010 study still do not make much sense. 10,12 In spite of the numerous criticisms that the GBD team have tried to address, the DALY has been widely used by re-searchers, policy-makers and several other stakeholders since its inception. Here we estimate the DALYs averted for several surgical and non-surgical interventions among patients admitted to a hospital in India. We investigate the effect of using alternative disability weighting on the results.

Methods
A 106-bed private hospital covering a semi-urban population in Uttar Pradesh, in northern India, was chosen for the study because its staff maintained a comprehensive computerized patient database and agreed to cooperate with the research team. As confidentiality issues prevented us from extracting data directly from the hospital's paper-based records, we only extracted data from the computerized database. To calculate DALYs, we gathered data on each surgical admission to the hospital between 1 April 2012 and 31 March 2013. Because the hospital only began digitizing the records of non-surgical admissions at the start of 2013, we included patients admitted for a non-surgical intervention between 1 January 2013 and 31 March 2013. At the time of our study, the hospital did not keep records for outpatient and emergency services. We collected data on age, sex, length of stay, diagnosis and/or procedure for 3865 inpatients, which represented 43% of the 8936 patients who were admitted in the year beginning 1 April 2012. After excluding the 420 inpatients who had only been admitted for pain management or childbirth, we assigned disability weights to the remaining 3445 inpatients.
For each patient, we estimated the DALYs associated with conditions for which they were admitted and the DALYs averted by the surgical and non-surgical interventions that were carried out. First, we used the GBD 1990 disability weights Objective To calculate the effect of using two different sets of disability weights for estimates of disability-adjusted life-years (DALYs) averted by interventions delivered in one hospital in India. Methods DALYs averted by surgical and non-surgical interventions were estimated for 3445 patients who were admitted to a 106-bed private hospital in a semi-urban area of northern India in 2012-2013. Disability weights were taken from global burden of disease (GBD) studies. We used the GBD 1990 disability weights and then repeated all of our calculations using the corresponding GBD 2010 weights. DALYs averted were estimated for surgical and non-surgical interventions using disability weight, risk of death and/or disability, and effectiveness of treatment. Findings The disability weights assigned in the GBD 1990 study to the sequelae of conditions such as cataract, cancer and injuries were substantially different to those assigned in the GBD 2010 study. These differences in weights led to large differences in estimates of DALYs averted. For all surgical interventions delivered to this patient cohort, 11 517 DALYs were averted if we used the GDB 1990 weights and 9401 DALYs were averted if we used the GDB 2010 disability weights. For non-surgical interventions 5168 DALYs were averted using the GDB 1990 disability weights and 5537 DALYS were averted using the GDB 2010 disability weights. Conclusion Estimates of the effectiveness of hospital interventions depend upon the disability weighting used. Researchers and resource allocators need to be very cautious when comparing results from studies that have used different sets of disability weights.

Research
Applying disability weights in an Indian hospital Susmita Chatterjee & Richard A Gosselin and then repeated the analyses using the GBD 2010 weights. For injuries only, we did another set of calculations using the disability weights from the GBD 2004 study -which, with a few exceptions, were essentially based on the GBD 1990 weights. 2 In each set of calculations we used identical scores for disease severity and the likelihood of treatment success.
We calculated DALYs averted using the method originally developed by Mc-Cord and Chowdhury 13 but with slightly simplified estimates of the risks of death and disability and the effectiveness of treatment. 14-16 Box 1 shows examples of our estimations of DALYs averted. These estimations were made without age weighting or discounting.

Results
Specific disability weights were available in both the GBD 1990 and 2010 studies for 12 of the conditions for which our study inpatients were admitted (Table 1). For another 10 conditions, we were able to find a disability weight in the GBD 1990 study that appeared to be a potential match to one in the GBD 2010 study -or vice versa (Table 1).
In the GBD 2010 study, disability weights for some surgical interventions differed markedly from those assigned in the GBD 1990 study. In consequence, our estimates of the total DALYs averted using GBD 1990 disability weights resulted in 11 517 DALYs, while using the GBD 2010 disability weights resulted in 9401 DALYs (Table 2). For example, our estimates of the numbers of DALYs averted by an abortion were 1649 when we used the disability weight given for abortion in the GBD 1990 study but 111 when we used the corresponding weight from the GBD 2010 study.
There were several conditions for which disability weights were not available in both the GBD 1990 and 2010 studies (e.g. hypertension). Further, in the GBD 2010 study, for example, no individual weights were given for peptic ulcer, kidney stone or appendicitis -although these conditions were loosely covered by the disability weights for abdominopelvic problems: mild, moderate or severe. Similarly, although the GBD 1990 study provided a specific disability weight for acute lower respiratory infection, no corresponding weight was included in the reported results of the GBD 2010 study. In our calculations based on the disability weights from the

Box 1. Examples of DALY-averted estimation
• A 30-year-old female with appendicitis has a disease severity score of 1 (i.e. more than 95% chance of being fatal or disabling without surgery) and effectiveness-of-treatment score of 1 (i.e. more than 95% chance of being cured after surgery) with 54 years of life-to-live (life expectancy as per 2010 life table). A successful appendectomy will avert 54 × 1 × 1 × 0.326 = 18 DALYs using the 2010 disability weights. • A one-year-old boy with septicaemia has more than 95% chance of death or disability without treatment and a chance of cure between 50% and 95% and 83.63 years of life-tolive. Successful medical treatment will avert 83.63 × 1 × 0.7 × 0.210 = 12 DALYs using the 2010 disability weights.  (Table 3). For a few non-surgical interventions, differences between the sets of disability weights that we used led to substantial differences in our estimates of the DALYs averted (Table 3). For example, our estimates of the numbers of DALYs averted by treating chronic nephritic syndrome with dialysis were 281 when we used the GBD 1990 disability weight but 1866 when we used the GBD 2010 weight.
Our estimates based on the GBD 1990 disability weights indicated that, among the 3445 inpatients included in our analyses, total DALYs were 23 829. The corresponding value based on the GBD 2010 weights -21 908 -was about 8% lower.
The GBD 2004 disability weights for fractures of the femur, radius or ulna, tibia and facial bones are the same as the corresponding GBD 1990 weights. For some procedures, however, the GBD

Discussion
We found that, for some conditions, our estimates of DALYs averted differed substantially according to which set of disability weights we used. It was not always possible to find perfect matches between the categories used in the GBD 1990 and 2010 studies. For example, cataract was given a GBD 1990 disability weight of 0.600 -under a cataract blindness category -but the most appropriate category in the GBD 2010 study appeared to be distance vision: moderate impairment, which had a much lower disability weight of 0.033. The GBD 2010 disability weights for more severe visual impairment, in the categories distance vision: severe impairment (0.191) or distance vision: blindness (0.195) were also much lower than the corresponding GBD 1990 values, as discussed elsewhere. 12 Our estimates of the numbers of DALYs averted by abscess drainage, among 50 inpatients, were 231 when we used the GBD 1990 disability weights but only 17 when we used the GBD 2010 weights. For both of these estimates we had to use the disability weight for open wound -i.e. the most appropriate category that was common to the GBD 1990 and 2010 studies -while acknowledging that not all open wounds are drained abscesses. The GBD 1990 disability weight for open wound (0.108) was 22-fold higher than the corresponding GBD 2010 weight (0.005). Surgical treatment of anal fissure, wound debridement and some non-surgical conditions -e.g. diarrhoea, septicaemia, hypothyroidism and neonatal respiratory distress -also have GBD 2010 disability weights that were very different from their GBD 1990 equivalents.
The findings raise two important questions. First, which set of disability weights is most accurate? Second, does the best set of weights vary depending on the intervention or condition be-ing investigated? As the method used to generate the GBD 1990 disability weights was completely different to that used to generate the GBD 2010 weights, it is perhaps not surprising that the two sets of weights show some differences. Although most studies on the cost-effectiveness of surgery and other conditions in low-and middle-income countries have used the GBD 1990 disability weights, future studies on the same topic are much more likely to use the GBD 2010 weights. As information on the cost of an intervention per DALY averted can be an important policy tool for resource allocation, researchers and resource allocators need to be very cautious when comparing results from studies that have used different sets of disability weights. Therefore, we are now evaluating whether the different sets of disability weights will affect the cost-effectiveness of the interventions available in the study hospital.
In the evaluation of disability weights, both the expert-panel approach of the GBD 1990 study and the survey approach of the GBD 2010 study led to some surprising and inconsistent results. We suspect that the respondents investigated in the GBD 2010 study were more biased towards acute pain and disability than to chronic impairment, and that  [13][14][15][16] our study has four major limitations. First, some of our inpatients' admission diagnoses were not covered by specific GBD 1990 or GBD 2010 disability weights. For most of these diagnoses, we used the closest possible weights. Second, whenever there were separate disability weights for mild, moderate and severe forms of an admission diagnosis, we tended to be conservative and chose the weight for the moderate form. In the Indian context, mild cases are rarely admitted to hospital. Third, the digitized records of the study hospital often indicated a fracture as humerus/tibia without specifying whether the fracture was of the humerus, the tibia or both. Without access to radiographs and the patient's charts, we had no way of distinguishing between arms and legs. In such cases, we were again conservative and used the disability weight for a fracture of the humerus -which, in both the GBD 1990 study and the GBD 2010 study, is lower than the disability weight for a fracture of the tibia. In consequence, our analyses included more fractures of the humerus than of the tibia -even though the latter are much more common in India. Whatever the scale of our misclassification bias, it remained unaltered by our choice of which set of disability weights to use. Finally, we had to assume that diagnoses were correct and that interventions were appropriate. Again, any related bias should not have been affected by our choice of which set of disability weights to use.
The evaluation of disability weights, which represent key components in the calculation of DALYs, remains very controversial. Though the GBD 2010 study attempted to respond to criticisms of the earlier GBD studies, many issues remain: the subjectivity in assigning disability weights to many given conditions, the many disability weights that make no medical sense, the non-inclusion of some conditions in the GBD studies and the difficulty in comparing studies that used different sets of disability weights. Perhaps some form of harmonization or consolidation of the GBD 1990 and GBD 2010 sets of disability weights should be considered. Although relatively few disability weights would require drastic adjustments, this would still lead to a third or, for some conditions, a fourth set of disability weights. While researchers, policy-makers and other stakeholders wait for the next set of disability weights, they need to keep in mind the limited comparability of studies based on the GBD 1990 disability weights and those based on the GBD 2010 weights. ■

Resumen
Estimación de la eficacia de las intervenciones de un hospital de la India. Impacto de la elección de los pesos de discapacidad Objetivo Calcular el efecto del uso de dos grupos diferentes de pesos de discapacidad para estimaciones de años de vida ajustados en función de la discapacidad (AVAD) evitados por intervenciones llevadas a cabo en un hospital de la India. Métodos los AVAD evitados por intervenciones quirúrgicas y no quirúrgicas se estimaron para 3445 pacientes que fueron ingresados en un hospital privado de 106 camas de un área semiurbana del norte de la India en 2012-2013.Los pesos de discapacidad se tomaron de estudios de carga mundial de morbilidad. Se utilizaron los pesos de discapacidad de 1990 de la carga mundial de morbilidad y a continuación se repitieron todos los cálculos utilizando los pesos de 2010 correspondientes. Los AVAD evitados se estimaron para intervenciones quirúrgicas y no quirúrgicas utilizando el peso de discapacidad, el riesgo de muerte o discapacidad y la eficacia del tratamiento. Resultados Los pesos de discapacidad asignados en el estudio de 1990 de carga mundial de morbilidad a las secuelas de enfermedades como cataratas, cáncer y lesiones eran sustancialmente diferentes a los asignados en el estudio de 2010. Estas diferencias en los pesos provocaron grandes diferencias en las estimaciones de los AVAD evitados. Para todas las intervenciones quirúrgicas realizadas a esta cohorte de pacientes, se evitaron 11517 AVAD si usamos los pesos de discapacidad de 1990 de carga mundial de morbilidad y 9401 si usamos los de 2010. Para las intervenciones no quirúrgicas, se evitaron 5168 AVAD si se utilizan los pesos de discapacidad de 1990 de carga mundial de morbilidad y 5537 si se utilizan los pesos de 2010. Conclusión Las estimaciones de la eficacia de las intervenciones hospitalarias dependen del peso de discapacidad utilizado. Los investigadores y los asignadores de recursos deben ser muy cautos al comparar los resultados de los estudios que han utilizado diferentes grupos de pesos de discapacidad.