Errors generated by a point-of-care CD4+ T-lymphocyte analyser: a retrospective observational study in nine countries

Abstract Objective To estimate the proportion of invalid results generated by a CD4+ T-lymphocyte analyser used by Médecins Sans Frontières (MSF) in field projects and identify factors associated with invalid results. Methods We collated 25 616 CD4+ T-lymphocyte test results from 39 sites in nine countries for the years 2011 to 2013. Information about the setting, user, training, sampling technique and device repair history were obtained by questionnaire. The analyser performs a series of checks to ensure that all steps of the analysis are completed successfully; if not, an invalid result is reported. We calculated the proportion of invalid results by device and by operator. Regression analyses were used to investigate factors associated with invalid results. Findings There were 3354 invalid test results (13.1%) across 39 sites, for 58 Alere PimaTM devices and 180 operators. The median proportion of errors per device and operator was 12.7% (interquartile range, IQR: 10.3–19.9) and 12.1% (IQR: 7.1–19.2), respectively. The proportion of invalid results varied widely by country, setting, user and device. Errors were not associated with settings, user experience or the number of users per device. Tests performed on capillary blood samples were significantly less likely to generate errors compared to venous whole blood. Conclusion The Alere Pima CD4+ analyser generated a high proportion of invalid test results, across different countries, settings and users. Most error codes could be attributed to the operator, but the exact causes proved difficult to identify. Invalid results need to be factored into the implementation and operational costs of routine CD4+ T-lymphocyte testing.


Introduction
The CD4+ T-lymphocyte count is the method recommended by the World Health Organization (WHO) to assess eligibility for antiretroviral treatment (ART). 1,2 The CD4+ count also guides the clinical management of people living with human immunodeficiency virus (HIV). 2 WHO recommends testing of viral load to detect treatment failure, but the CD4+ count continues to be used for ART monitoring if viral load cannot be tested. 3 In settings where viral load can be determined, WHO suggests that routine CD4+ monitoring may be reduced or stopped for adults who are virologically stable. 4,5 Increased availability of ART has driven the establishment of laboratories analysing CD4+ count, but access to CD4+ testing requires adequate laboratory capacity and the means to transport specimens. 6,7 Point-of-care CD4+ testing provides rapid results without the need to transport specimens and is considered an important tool to improve patient retention in care before treatment initiation. [8][9][10][11] The Alere Pima CD4+ analyser (Alere Technologies, Jena, Germany) was commercially launched in 2010. It is an automated analyser intended for counting CD4+ cells in capillary or venous whole blood within 20 minutes. The analyser is portable; can be operated with a rechargeable battery; contains dried thermostable reagents and can be stored at room temperature. The analyser performs a series of checks to ensure that all steps of the analysis are completed successfully, if not, an invalid result is reported. It is recommended that internal Pima Bead cartridges are analysed daily for quality control purposes. The analyser fulfils most WHO criteria for point-ofcare tests 12 and is on the WHO list of prequalified diagnostics. 13 The analyser has been extensively evaluated with favourable diagnostic performance at laboratory level [14][15][16][17][18][19][20][21][22][23][24][25] and with acceptable results at clinic level, [26][27][28][29][30][31][32][33][34][35][36][37][38][39][40] in mobile clinics 41 and in communities. 42,43 Some studies showed higher variability of results at clinic level with capillary sampling. [44][45][46] A recent meta-analysis showed that the analyser was comparable in performance to laboratory-based methods. The analyser identifies patients for treatment at the 350 CD4+ cells/μL threshold with a sensitivity of 91.6% and specificity of 94.8%. 47 In 2011, Médecins Sans Frontières (MSF) introduced the analyser in various projects to make CD4+ counting more accessible and to reduce loss to follow-up. During the implementation phase, several MSF sites reported a relatively high proportion of errors. Previous studies have reported invalid results ranging from 2% to 15%. 29,32,38,39 However, these studies have been conducted under validation conditions and there is a lack of data describing errors under routine field conditions. Given the relatively high proportion of invalid results reported at MSF sites, we carried out a retrospective analysis of analyser data across a variety of settings. The objective of the study was to describe the proportion of CD4+ test errors and identify Objective To estimate the proportion of invalid results generated by a CD4+ T-lymphocyte analyser used by Médecins Sans Frontières (MSF) in field projects and identify factors associated with invalid results. Methods We collated 25 616 CD4+ T-lymphocyte test results from 39 sites in nine countries for the years 2011 to 2013. Information about the setting, user, training, sampling technique and device repair history were obtained by questionnaire. The analyser performs a series of checks to ensure that all steps of the analysis are completed successfully; if not, an invalid result is reported. We calculated the proportion of invalid results by device and by operator. Regression analyses were used to investigate factors associated with invalid results. Findings There were 3354 invalid test results (13.1%) across 39 sites, for 58 Alere Pima TM devices and 180 operators. The median proportion of errors per device and operator was 12.7% (interquartile range, IQR: 10.3-19.9) and 12.1% (IQR: 7.1-19.2), respectively. The proportion of invalid results varied widely by country, setting, user and device. Errors were not associated with settings, user experience or the number of users per device. Tests performed on capillary blood samples were significantly less likely to generate errors compared to venous whole blood. Conclusion The Alere Pima CD4+ analyser generated a high proportion of invalid test results, across different countries, settings and users. Most error codes could be attributed to the operator, but the exact causes proved difficult to identify. Invalid results need to be factored into the implementation and operational costs of routine CD4+ T-lymphocyte testing.
factors associated with these errors. We hypothesized that invalid results might be related to the experience of the operators or to the type of blood sample used.

Study population
We conducted a retrospective, observational, cross-sectional study using routine data from 39 MSF-supported sites using the analyser in the Central African Republic, the Democratic Republic of the Congo, Guinea, India, Kenya, Lesotho, Malawi, Mozambique and South Africa, between 1 January 2011 and 30 June 2013. Analysers were introduced in laboratories, primary health-care clinics, mobile clinics and in communities. HIV-positive individuals requiring ART eligibility assessment, and in some instances, ART monitoring, had CD4+ counts done on capillary or venous blood samples. The study protocol was submitted to the MSF ethics review board and exempted from ethics review because it complied with the standards for routinely collected data analyses.

Training
Laboratory technicians, clinicians and lay workers, were trained either by MSFtrained personnel or by the local Alere representative before use of the analyser. On-site training was conducted in half a day and focused on device operation, sampling technique, quality control and data management. In Lesotho and South Africa, a three-day centralized course in Alere's training department in Cape Town was also provided to key staff who, in turn, trained multiple users on-site.

Sampling technique
Venous blood samples were used at all sites in the Central African Republic, the Democratic Republic of the Congo, Guinea, India, Kenya, Malawi and two sites in South Africa; capillary sampling was used in Lesotho, Mozambique and the remaining nine sites in South Africa. All sites in the Central African Republic, the Democratic Republic of the Congo, Guinea, India, Kenya and Malawi used a fixed volume micropipette to transfer 25 µL of venous blood to the cartridge. Two sites in South Africa used a plain capillary tube for cartridge filling. Capillary sampling was performed using the Alere recommended safety lancet (Sarstedt, Nümbrecht, Germany) at remaining sites in South Africa and all sites in Mozambique and Lesotho.

Quality control
Daily quality control was done at all sites using the manufacturer-supplied bead standards (with normal and low CD4+ cell counts) before testing patient samples. Eleven sites in South Africa were also enrolled in a regional proficiency testing programme and tested stabilized whole blood samples with normal or low CD4+ counts every two months. 48 At most sites, dedicated paper-based registers were implemented to capture test results. None of the sites made use of the Alere online data portal during the study period. Test results and quality control data were analysed periodically in each country by a laboratory coordinator.

Data collection
Archived computer files containing CD4+ test results and quality control data were collated from each device. Information about the setting, user, training, sampling technique and device repair history were obtained by questionnaire, completed by the laboratory coordinator in each country. To ensure a representative number of tests per device and to eliminate errors due to inexperienced operators, we excluded data from devices with less than 50 tests, users performing less than 50 tests or unknown operators.

Analysis
We calculated the proportion of invalid CD4+ test results by device and by operator. Factors associated with invalid results were analysed using the binreg command in Stata version 13.0 (Stata-Corp. LP, College Station, United States of America). To determine the effects of user type and setting, we performed a series of analyses, restricted to those countries with comparable user type and setting.
We carried out the following comparisons: (i) tests done at clinics by lay workers versus tests done by clinicians in South Africa; (ii) tests done by laboratory technicians at clinics versus tests done at laboratories in the Central African Republic and the Democratic Republic of the Congo, and (iii) tests done by lay health workers in mobile For each of these three comparisons, we calculated the frequency of errors stratified by country and tested for effect modification by country using the Mantel-Haenszel χ 2 test. If there was no effect modification by country, we used generalized linear regression models to assess associations between error frequency and user type or setting. The χ 2 test for categorical variables and the two-sample test of proportions were used to assess the significance of differences; a P-value of less than 0.05 was considered statistically significant.

Associated factors
The factors associated with invalid tests are shown in Table 2 In clinics in South Africa, errors were significantly less frequent for lay health workers than clinicians (9.9% versus 19.8%; P = 0.001). Among laboratory technicians, the association between setting and the frequency of invalid tests differed by country. In the Central African Republic, invalid tests among laboratory technicians were more frequent at the clinic than at the laboratory (22.4% versus 16.3%; P < 0.001), while in the Democratic Republic of the Congo, invalid tests among laboratory technicians were less frequent at the clinic than at the laboratory (13.2% versus 18.7%; P < 0.001). Among lay health workers in Lesotho, Mozambique and South Africa, the association between setting and the frequency of invalid tests differed by country: in South Africa and Lesotho, the frequency of invalid tests was comparable in mobile clinic and fixed clinic settings (9.0% versus 9.9% in South Africa; P = 0.35; and 15.5% versus 16.1% in Lesotho; P = 0.66). However, in Mozambique, the frequency of invalid tests in mobile clinics was considerably lower than in fixed clinics (4.1% versus 13.6%; P < 0.001).

Blood sample type
There were fewer invalid results when capillary blood samples were used (12.0%, versus 14.0% for venous blood samples; P < 0.001). In a subgroup multivariate analysis restricted to South Africa, (the only country in which both sample types were used) this association was strengthened (RR: 0.46; 95% CI: 0.40-0.53).

Discussion
Under routine field conditions, the proportion of invalid CD4+ test results ranged from 5% in India to 24% in Malawi. Previous studies have reported comparable ratios from 5% in Thailand to 19% in South Africa. 15,33 In our study, invalid results were slightly more frequent in venous blood samples than capillary blood samples. The use of a pipette to fill the device cartridge may generate air bubbles; since the CD4+ test is based on image detection, the presence of air bubbles may affect the results. To decrease the generation of air bubbles, the manufacturer recommends that capillary tubes instead of pipettes be used to transfer venous blood to the device. Studies in Senegal and Uganda reported more errors using capillary blood specimens (14% and 18%) compared to venous blood specimens (5% and 8%). 31,44 However, other studies using venous blood have also found high ratios of invalid results: 10% in Ethiopia, 11% in USA and 15% in South Africa. 16,18,45 Some studies have shown an increased variability of Alere Pima CD4+ results when capillary blood is used. 20,44,45 However, a recent systematic review and meta-analysis found that capillary blood yielded more accurate results than venous blood. 47 The fact that errors were more likely to occur using venous blood in our study may be related to the pipetting skills of laboratory staff. Therefore, training of laboratory staff in pipetting skills is as important as training clinic staff to perform capillary bleeds required for field testing.
The proportion of invalid samples for calibration tests was low (2%); however, in some instances, operators did not know how to interpret the results, There were several limitations resulting from the retrospective nature of our study. We could not analyse errors according to cartridge lot number because this information was not contained in the cartridge barcode and sites did not document lot numbers systematically. It proved impossible to identify lot numbers via orders, as they were placed through various procurement centres in Europe as well as purchased from local distributors. We were unable to link results of repeat testing due to errors, so it was not possible to estimate the proportion of valid or invalid results after a second test. During the study period, none of the sites made use of online data systems for the analyser. Remote analysers can be connected to a centralised data warehouse via the internet, which could improve the timeliness of monitoring.
The high proportion of invalid CD4+ test results, necessitating the use of a second cartridge, raises the cost of testing per patient. 49 Overall, 3354 tests had to be repeated; at a current price of 6 United States dollars (US$) per test, this translates to a cost of US$ 20 124. Repeat tests take time to process, decrease throughput, and oblige the patient to provide another blood sample if capillary sampling is used.
Training, monitoring, quality assessment and troubleshooting are essential aspects of point-of-care testing. 50 It is important that post-market surveillance is done regularly and manufacturers continue to make the necessary improvements to their technologies and to supply service, maintenance and training for the full lifespan of the device.
In conclusion, 13.1% of results from the Alere Pima CD4+ analyser were invalid in this study. Most errors could be attributed to the operator but elucidating the exact cause proved to be difficult. Analyser errors are frequent and need to be factored into the implementation and operational cost for routine CD4+ testing. ■