Evaluation of the diagnostic accuracy and cost of different methods for the assessment of severe anaemia in hospitalised children in Eastern Uganda

Background: Severe anaemia in children requiring hospital admission is a major public health problem in malaria-endemic Africa. Affordable methods for the assessment of haemoglobin have not been validated against gold standard measures for identifying those with severe anaemia requiring a blood transfusion, despite this resource being in short supply. Methods: We conducted a prospective descriptive study of hospitalized children aged 2 months – 12 years at Mbale and Soroti Regional Referral Hospitals, assessed to have pallor at triage by a nurse and two clinicians. Haemoglobin levels were measured using the HemoCue ® Hb 301 system (gold standard); the Haemoglobin Colour Scale; Colorimetric and Sahli’s methods. We report clinical assessments of the degree of pallor, clinicians’ intention to transfuse, inter-observer agreement, limits of agreement using the Bland-Altman method, and the sensitivity and specificity of each method in comparison to HemoCue ® Results: We recruited 322 children, clinically-assessed by the admitting nurse (n=314) as having severe (166; 51.6%), moderate (97; 30.1%) or mild (51; 15.8%) pallor. Agreement between the clinicians and the nurse were good: Clinician A Kappa=0.68 (0.60–0.76) and Clinician B Kappa=0.62 (0.53–0.71) respectively ( P<0.0001 for both). The nurse, clinicians A and B indicated that of 94/116 (81.0%), 83/121 (68.6%) and 93/120 (77.5%) respectively required transfusion. HemoCue ® readings indicated anaemia as mild (Hb10.0–11.9g/dl) in 8/292 (2.7%), moderate (Hb5.0–9.9g/dl) in 132/292 (45.2%) and severe (Hb<5.0g/dl) in 152/292 (52.1%). Comparing to HemoCue® the Sahli’s method performed best in estimation of severe anaemia, with sensitivity 84.0% and specificity 87.9% and a Kappa score of  0.70 (0.64–0.80). Conclusions: Clinical assessment of severe pallor results has a low specificity for the diagnosis of severe anaemia. To target blood transfusion Hb measurement by either Hemocue® or Sahli’s method for the cost of USD 4 or and USD 0.25 per test, respectively would be more cost-effective.

report report report report

Introduction
Anaemia is a major public health problem in low income and malaria endemic areas 1,2 . Its consequences vary with severity, duration, age, maternal parity and underlying cause. The World Health Organization (WHO) defines severe anaemia as a haemoglobin (Hb) concentration of <5.0g/dl (in the presence of malaria) or < 6g/dl which equates to a haematocrit of <15% or 18% respectively. In sub-Saharan Africa (sSA) there is a need to balance the demand for blood for transfusion with the scarcity of blood supply 3 . Currently, there are insufficient data to be sure whether routinely giving blood to clinically stable children with a Hb of <5.0g/dl in malaria-endemic areas reduces immediate or long-term morbidity and mortality. As such, cost effective allocation of resources currently requires that only patients with symptomatic severe anaemia who have a high risk of a poor outcome should be transfused 1,4,5 . Against this background, not all children with a Hb <5.0g/dl receive a blood transfusion 6 , even though the WHO recommends it.
In resource-limited regions of sSA the choice of a diagnostic test is tagged to its sensitivity and specificity and weighed against its cost. Affordable and technologically appropriate methods are promoted for routine health care 7 . Highly specific and sensitive tests are desirable for both clinical care and research purposes. For the assessment of anaemia, Hb estimation is more specific and more readily measurable than packed cell volume (PCV), and is also more pragmatic as it is less hazardous for health personnel 8 . Despite being a common presentation and a leading cause of admissions in paediatric wards in Eastern Uganda, accurately testing for anaemia and its severity in the laboratory or at the bedside remains a challenge 9,10 . As a result, the diagnosis of anaemia and the decision to transfuse is often made clinically, through detection for the presence of pallor. Clinical assessment of pallor has been shown to be affected by wide inter-rater variability 11 and inaccuracy 12 . However, most previous studies have been conducted in out-patients or in communities that have been aimed at assessing less severe degrees of anaemia. As a result, many African children are transfused who do not meet the WHO criteria for transfusion 6 . In addition, laboratory estimation of Hb in East African district hospitals is among one of the least accurate tests 13 .
The stimulus to carry out this study was to identify an accurate and reliable method for haemoglobin assessment that could contribute to rational decision making in resource-limited malaria endemic areas 3 . We therefore aimed to compare the accuracy of four methods of haemoglobin estimation that are commonly available in Uganda in terms of their ability to correctly identify children who are clinically severely anaemic and require a blood transfusion: (i) Sahli's method; (ii) Haemoglobin Colour Scale (HCS); (iii) Colorimetric method; and (iv) clinical assessment. The performance of each of these methods was compared to results obtained from the HemoCue® method, which we used, pragmatically, as our 'gold standard' for the purposes of this study 14 .

Methods
We conducted an observational study among children aged 2 months -12 years who were admitted to the paediatric wards of Mbale and Soroti Regional Referral Hospitals (RRH) in Eastern Uganda. This is an area where the transmission of malaria is perennial, with peaks that correspond with the main rainy seasons. Clinical training and assessment Pre-study training was conducted through which the admitting nurses and clinician at each centre were taught how to classify pallor. Three levels were taught: (i) mild (suggested to be synonymous with Hb level 10.0-11.9); (ii) moderate (estimated Hb 5.0-9.9g/dl); and (iii) severe (estimated Hb<5g/dl). Refresher training on eligibility criteria for transfusion was also covered. Briefly, the Uganda National Paediatric Guidelines indicate that a child should be only be transfused if their haemoglobin is <4g/dl or <6g/dl in the presence of life-threatening complications such as hypoxia with cardiac compensation, acidosis, dyspnoea, impaired consciousness, septicaemia, meningitis, cerebral malaria or hyperparasitaemia (>20%) 18 .
At triage the admitting nurse, indicated the degree of clinical pallor (mild, moderate, severe) and whether, on the basis of this assessment, transfusion should be prescribed. A paediatric admission record (PAR) was completed by the admitting clinicians (Clinician A and B). All assessors were blind to the judgements of the others and whether the child should be transfused. Following these 3 clinical assessments, Hb levels were measured using the four laboratory methods 19 . The ultimate decision on whether or not children should be transfused was based on the result from Sahli's method, as this was the current practice in these hospitals. No specific delays to the receipt of treatment were introduced through the conduct of this study.

Amendments from Version 1
We have responded to the three reviewers' comments and made note of their observations on the inconsistencies of numbers presented and request for extra detail. These include correction of the numbers of children assessed (presented in the abstract); correction of the spelling of Colorimetric, addition of details on the ranges of ages; and a statement about the accuracy of the haemoglobin estimation and need for future validation of HemoCue in children with profound anaemia.

Haemoglobin assessment methods
The three different methods for haemoglobin assessment are shown in Figure 1. The HemoCue® Hb 301 system (Hemocue AB, Angelholm, Sweden) is a portable point of care Hb testing device that uses disposable microcuvettes and requires no calibration. A finger-prick drop of blood is sucked into a disposable cuvette via capillary action before inserting into the Hemocue® machine which reads out the Hb value on a digital display within 15-45 seconds. The measurement range is 0-25.6g/dl. The Haemoglobin Colour Scale (HbCS) was developed by Department of Essential Health Technologies, World Health Organization (WHO), Geneva as an inexpensive, simple tool to measure haemoglobin levels in resource-poor settings to detect and control anaemia 20 . It comprises of a card with six shades of red that represent Hb levels at 4, 6, 8, 10 12, and 14g/dl respectively. The test takes 30 seconds and the colour of the blood spot is matched against the hues on the kit's colour scale. The scale does not include a colour corresponding to Hb of 5g/dl so all colour values of 4 and below were regarded as having Hb <5g/dl. The Colorimetric method is an accurate method for the estimation of Hb concentration as a continuous variable that involves conversion of Hb to cyanmethaemoglobin by potassium ferricyanide 21 . The absorbance of the resultant solution is compared to the standard cyanmethaemoglobin solution by using a formula to obtain the amount of haemoglobin. The Colorimetric machine currently used is automated and gives the amount of Hb without manual calculations but takes about 10 minutes per test in addition to the time required to set up and calibrate the machine. Owing to the test being more time consuming, and reliant upon toxic cyanide reagents, it is less favoured for use in blood banks for Hb estimation. Finally, Sahli's method uses a drop of blood pipetted into a graduated tube containing hydrochloric acid. The blood is converted to acid haematin which is then diluted with drops of distilled water until its colour matches the colour of the tinted comparator glass. The haemoglobin is estimated by reading the value directly from the scale on Sahli's tube according to the height that the final solution has reached. This method must be read in an environment of natural light and takes only 2-3 minutes. The Hb level is read to one decimal point in grams per decilitre.

Statistical analysis
All statistical analyses were performed using R statistical software (version 3.5.1 for Windows) and MedCalc v18 for Windows (version 18.2.1, MedCalc Software, OstendMariakerke, Belgium). We used HemoCue® as the gold standard in this study because of its proven sensitivity and specificity across a wide range of Hb values 22 . The four other Hb estimation methods were tested against the HemoCue® Hb 301 system. We defined mild anaemia as Hb 10.0-11.9g/dl, moderate anaemia as Hb 5.0-9.9g/dl and severe anaemia as Hb <5.0g/dl. The diagnostic accuracy (specificity and sensitivity) of each method was calculated as previously described 23 . The Bland Altman 24 approach was used to analyse the agreement between the HemoCue® and other methods. Briefly, this approach assumes that if two methods are to agree then the mean of the difference between every paired determination will not be statistically different from zero. By using this approach, we calculated limits of agreement (within a given confidence interval) between the two methods and graphically visualized the dispersion of these differences across the Hb values. Cohen's kappa-statistic (κ) was used for assessing inter observer variation, through which agreement is scaled between zero and one, where 0 indicates no agreement (i.e. no better than would be observed by chance), and 1 signifies perfect agreement. Finally, we performed receiver-operating characteristic (ROC) curve analysis to examine the overall discriminatory power and calculability of the methods. Diagnostic performance for each method was ascertained from the area under the curve (AUC) for each method 25 .
The study was approved by the Mbale Regional Referral Hospital Research Ethics Committee (MRRH-REC) and the Uganda National Council of Science and Technology (UNCST).

Participants
All children aged over 2 months to 12 years were assessed at triage for pallor. Children were included, following parental consent, if assessed to have a pallor. Children whose guardian or parent declined consent were excluded from the study. We recruited a total 322 participants: 200 (62.3%) from Mbale and 122  Comparing different diagnostic methods of Hb assessment We first assessed the measurements of central tendency (mean and median) and variation (range and standard deviation, SD) for each of the Hb determining methods as shown in Table 2.
We found that the HemoCue® method showed the lowest mean (5.0g/dl; SD=2.0) and the lowest median (4.8g/dl) when compared to both the Colorimetric and Sahli's methods [mean=5.7g/dl (SD=2.5) and 5.1g/dl (SD=2.1)] respectively. The Bland-Altman concordance is shown in Figure 2: with the exception of outlying values, the majority of the measurement points were within 1.96 standard deviation of the mean   paired differences. The mean difference with limits of agreement between HemoCue® and the Colorimetric method and between HemoCue® and Sahli's method were 0.5 (95% CI -−12.3, 3.3) g/dl and 0.1 (95% CI-−2.3, 2.4) g/dl respectively (Figure 1), implying good agreement between these methods, particularly for Sahli's method.

Reliability and validity of the various methods
The sensitivities and specificities of each method in the diagnosis of mild, moderate and severe anaemia, compared to HemoCue® are summarised in Table 3 and Figure 3. Sahli's method performed best, with sensitivity 84.0%, specificity 87.9% and positive and negative predictive values of 88.3% (82.6-92.3) and 83.6 (77.7-88.1) respectively. The corresponding values for the other methods are summarised in Table 3.
The diagnostic performance of Sahli's method in the assessment of severe anaemia as determined by AUC was 84%, superior to that of any of the other three methods (Figure 2). The AUC values for the Colorimetric and HbCS methods were 75%, and 69%, respectively. The AUC values for the clinicians' interpretation were 70% (nurse), 71% (Clinician A), and 72% (Clinician B). The Cohen's Kappa statistic (κ) indicators comparing these three methods against the gold standard are summarised in Table 4. The HbCS rated very poorly for each of the three categories of anaemia: mild (κ=0.07), moderate (0.07) and severe (0.37). There was substantial agreement between the HemoCue® and Colorimetric method in the identification of mild anaemia, (κ=0.76, 0.50-1.00), but agreement was lower for severe anaemia (0.54; 0.41-0.66). Compared with HemoCue the Sahli's method performed best across all categories including the identification of severe anaemia cases (κ=0.70, 0.64-0.80).
Other methods, achieved fair-moderate reliability (Table 4) but the agreement of clinical assessment (nurse and clinician) rated as poor despite the inter-observer agreement between the nurse and each of the clinicians being fairly good κ>0.60 (Table 5).

Discussion
We investigated the diagnostic accuracy of a range of methods in the assessment of Hb values among children admitted to   hospital in a malaria endemic area in Eastern Uganda. We found a higher concordance between assessors in the identification of clinical pallor than observed in previous studies 26 . This may be explained by the joint training sessions that we held both during the FEAST trial and prior to this study 27 . Nevertheless, on clinical assessment of pallor alone, both clinicians and nurses recommended transfusion in 62-78% of children, considerably higher than the rate that would have been justified by HemoCue® (52%). In comparison to HemoCue®, HbCS and the Colorimetric method rated very poorly in the identification of severe anaemia. There was, however, substantial agreement between the HemoCue® and Sahli's method. Sahli's method detected 6/321 (1.9%) children with severe anaemia in comparison to 9/322 (2.8%) by HemoCue®, and was the test that was most specific and sensitive. For mild anaemia, the Colorimetric test was the most sensitive test and the most specific test was the Colorimetric method.
As expected, blood transfusion was recommended in a high proportion of children on the basis of clinical assessment. Firstly, many children 131/206 (63.6%) presented with fast breathing, and it is possible that these were classified as respiratory distress, a danger sign requiring transfusion 28 . In addition, a number of children had features of impaired circulation 129/322 (40.1%), one of the defining clinical characteristics for shock, coma 86/322 (26.7%) or prostration 43/300 (14.3%), all of which are indications for blood transfusion in anaemic patients in Uganda National Guidelines. Furthermore, in agreement with a study conducted in Pakistan 26 , we found that clinical assessment was inaccurate in the diagnosis of mild and moderate anaemia. Previous studies have largely been conducted in outpatient settings or conducted outside Africa 12,29 and, to the best of our knowledge, this is the first report conducted in African children presenting for admission with severe and life-threatening conditions.
HemoCue® has been shown to have reliable accuracy in determining Hb across the range of severity of anaemia 14 . Nevertheless, few studies have included groups of children with profound anaemia (haemoglobin < 4g/dl) and therefore future research should focus on validation in this group. In keeping with previous studies, we found that the method was simple to operate, and allowed for Hb estimation at the bedside to give results within <30 seconds 22 , allowing the attending clinicians to make rapid treatment decisions. Current costs for the two most efficient tests (in terms of positive and negative predictive values) approximately USD 4 per test for HemoCue® (after an initial cost of between USD 250-350 for the analyser) and USD 0.25 per test for Sahli's method (after an initial cost of USD 40-50 for the meter). Of note however, is that HemoCue® can be performed at the bedside whereas Sahli's method requires a functioning laboratory. Other limitations of Sahli's method include (i) its accuracy being dependent on natural light, (ii) difficulty in attaining a uniform colour in the test tube; (iii) the reagents are unstable and deteriorate with storage; (iv) dilutional discrepancies can occur. Nevertheless, if laboratory technicians are properly trained this could offer an affordable and reliable method for accurately assessing Hb level.
Our study had some limitations since sensitivity and specificity could not tell the probability of an individual patient having severe anaemia, a feature that would be useful for clinicians. We therefore included an analysis of positive and negative predictive that would give the clinician a better indication to rule in or rule out the need for a transfusion. The results of this study should, however, be interpreted with caution for children with severe anaemia in non-malaria endemic areas, where they should be validated in additional studies.
In conclusion, our data suggests that in situations where HemoCue® is either unaffordable or unavailable, among the commonly available methods, Sahli's method provided the next best alternative for the detecting severe anaemia, and could be used to guide treatment and avoid over-use of blood transfusion in resource-limited settings.

Data availability
The data underlying this study is available from Harvard Dataverse. Dataset 1: Evaluation of the diagnostic accuracy and cost of different methods for the assessment of severe anaemia in hospitalised children in Eastern Uganda-Dataset https://doi. org/10.7910/DVN/VMNDPO 30 This dataset is available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Ethics approval and consent to participate
The Mbale Regional Referral Hospital Research & Ethics Committee (MRRH-REC) approved the study (RECIRC 109/2010), and local permission to conduct the study was obtained from both Mbale and Soroti Regional Referral Hospitals.

Consent for publication
The Mbale Clinical Research Institute (MCRI, www.mcri.ac.ug), a research entity affiliated to the Uganda National Health Research Organization (UNHRO), permits the publication of this manuscript. This study was also funded by Imperial College London through support to KM.

Grant information
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. 1.
The manuscript by Olupot-Olupot reports a comparison of clinical and Hb assay estimation of anaemia severity in two hospitals in Uganda. It concludes that the Sahli's method is accurate and cheap to identify patients with Hb levels <5g/dl eligible for transfusion. Throughout the manuscript there is a persistent misspelling of the colorimetric method (properly spelled at beginning of page 4) for 'calorimetric'. The test having to do with colour, not heat, so colorimetric should be used unless the instrument and presumably the registered system is indeed labelled 'calorimetric'. The description of the 'Colour Scale' method against the presented Figure 1 is unclear. The scale bands in the Figure are presenting an empty circle that might suggest a space to deposit blood in order to directly compare with the printed scale.
If not, what is the use of this circle? If not, on what support is a drop of blood deposited to compare to the scale? In Table 1, there is a considerable deficit in age recording. Why?
The main objective of the Hb estimation is decision-making regarding indication of transfusion. In Table 3, it appears that the percentage of severe pallor, irrespective of the observer, is very close to the percentage scored by HemoCue. It would be interesting to provide a Table correlating the clinicians' decision to transfuse with their estimate of pallor. Taking the 152 cases of severe anaemia identified with the gold standard as <5g/dl and eligible for transfusion according to WHO, how many of those were correctly identified as severe pallor by clinicians and how many were considered for transfusion? In addition, it would be useful to know the reasons why X number of moderate or mild pallor patients was identified by clinicians eligible for transfusion.

If applicable, is the statistical analysis and its interpretation appropriate? Yes
Are all the source data underlying the results available to ensure full reproducibility? Partly

Are the conclusions drawn adequately supported by the results? Yes
No competing interests were disclosed.  The Figure is only a picture of all three devices for measurement side by side. We are sorry that this was a little confusing. As explained in the text under the methods section for the Colour Scale ' It (the Colour Scale) comprises of a card with six shades of red that represent Hb levels at 4, 6, 8, 10 12, and 14g/dl respectively. The test takes 30 seconds and the colour of the blood spot is matched against the hues on the kit's colour scale. The intention is not for the blood to be spotted onto the scale.
3. In Table 1, there is a considerable deficit in age recording. Why?
The purpose of the age groups presented < 5 years and > 5 years was to provide overarching summaries; since we do not stratify the results by age group. However we are happy to split down the age groups further: we have provided further breakdown for children < 1 year; children 1-4 years and those > 5 years. We have added these into the results section.
4. The main objective of the Hb estimation is decision-making regarding indication of transfusion. In Table 3, it appears that the percentage of severe pallor, irrespective of the observer, is very close to the percentage scored by HemoCue. It would be interesting to provide a Table correlating the clinicians' decision to transfuse with their estimate of pallor. Taking the 152 cases of severe anaemia identified with the gold standard as <5g/dl and eligible for transfusion according to WHO, how many of those were correctly identified as severe pallor by clinicians and how many were considered for transfusion? In addition, it would be useful to know the reasons why X number of moderate or mild pallor patients was identified by clinicians eligible for transfusion.C Children were all assessed clinically before blood was tested by each of the methods. The specificity of assessment for severe pallor for accurately predicting severe anaemia ranged from 70. 2% (61.9-77.6) to 71.6 (63.4-78.9) and the positive predictive value was 71.8 (66.0-77.0) to 73.0 (67.3-78.1); so the correlations are not as close as indicated. We agree that it is confusing that even after pre-study training doctors and nurses indicated their desire to transfuse. One reason is probably because the evidence supporting current guidelines are only based on expert opinion and there have been quite a number of reports (indicated in the manuscript) where doctors are not following national or international guidelines and transfusing at much higher thresholds.
No competing interests were disclosed.

If applicable, is the statistical analysis and its interpretation appropriate? Yes
Are all the source data underlying the results available to ensure full reproducibility? Yes Are the conclusions drawn adequately supported by the results? Yes No competing interests were disclosed.

Competing Interests:
I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Author Response 11 Mar 2019
, KEMRI-Wellcome Trust Research Programme, Kenya Kathryn Maitland 1. A few grammatical errors to correct (example: use of the wrong tense -"used " instead of "use "etc.) We have spelled checked and corrected these 2. The Abstract presents inconsistent results, and the arithmetics must be corrected: The total of 164 severely anaemic children, 99 moderate cases and 57 mild cases sums up to 320 children and not 322. Similarly, their corresponding percentages (51%, 30.7% and 17.7%) only add up to 99.4% and not 100%.
Thank you for noting this. We corrected these numbers.
3.The Introduction also talks of " We agree that this is not relevant in the methods section so we have removed the reference to Table 3 results at this point. We therefore do not require the production team to reorder the tables.
4. The Results: children under 12 years do not give direct consent. If guardians are providing consent, then, "assent " should be used. Furthermore, that information should be under METHODS and not under RESULTS.
We have added the word 'parental' into the consent sentence. But believe that the following sentence in the original manuscript did qualify this was parental consent. We are sorry this was misleading.
'Children were included, following consent, if assessed to have a pallor. Children whose parental guardian or parent declined consent were excluded from the study.' 5. As mentioned above, Table 1 should now read Table 2 and Table 2 should now read Table 3 etc.etc.
We request the production team to remove the reference to Table 3 in the methods therefore reordering the tables in not required.

The Conclusions respond to the research question.
We have reread the final conclusion and believe our summary reflects on the question we posed: which method would reliably assess haemoglobin.
No competing interests were disclosed. Competing Interests: