A comprehensive review of the SLMTA literature part 2: Measuring success

Background Since its introduction in 2009, the Strengthening Laboratory Management Toward Accreditation (SLMTA) programme has been implemented in 617 laboratories in 47 countries. Objective We completed a systematic review of the published literature on SLMTA. The review consists of two companion papers; this article examines quantitative evidence presented in the publications along with a meta-analysis of selected results. Methods We identified 28 published articles with data from SLMTA implementation. The SLMTA programme was evaluated through audits based on a standard checklist, which is divided into 12 sections corresponding to the 12 Quality System Essentials (QSEs). Several basic service delivery indicators reported by programmes were also examined. Results for various components of the programme were reviewed and summarised; a meta-analysis of QSE results grouped by the three stages of the quality cycle was conducted for 126 laboratories in 12 countries. Results Global programme data show improved quality in SLMTA laboratories in every country, with average improvements on audit scores of 25 percentage points. Meta-analysis identified Improvement Management as the weakest stage, with internal audit (8%) and occurrence management (16%) showing the lowest scores. Studies documented 19% – 95% reductions in turn-around times, 69% – 93% reductions in specimen rejection rates, 76% – 81% increases in clinician satisfaction rates, 67% – 85% improvements in external quality assessment results, 50% – 66% decreases in nonconformities and 67% increases in staff punctuality. Conclusions The wide array of results reported provides a comprehensive picture of the SLMTA programme overall, suggesting a substantive impact on provision of quality laboratory services and patient care. These comprehensive results establish a solid data-driven foundation for program improvement and further expansion.


Introduction
Quality laboratory services are critical for ensuring optimal patient care and comprehensive public health response; however, laboratories in resource-poor countries have been one of the most neglected components of health systems. 1 The Strengthening Laboratory Management Toward Accreditation (SLMTA) programme was developed in an effort to improve the quality of laboratories throughout the developing world. It is a competency-based training programme designed to enable laboratories to implement practical quality management systems (QMS) and encourage continuous quality improvement.

Research methods and design
A comprehensive search of electronic bibliographic databases was performed, as described in Part 1. 3 We included all published and in-press studies that discussed the SLMTA programme.
The standard SLMTA implementation model includes three workshops, each of which is followed by a period of several months for laboratories to implement improvement projects, usually with onsite support and mentorship. 2 Laboratories implementing the SLMTA programme are evaluated through audits based on the Stepwise Laboratory Quality Improvement Process Towards Accreditation (SLIPTA) checklist. 4 Audit scores are categorised into star ratings, with zero stars corresponding to a score of 0% -54%, one star 55% -64%, two stars 65% -74%, three stars 75% -84%, four stars 85% -94%, and five stars 95% -100%. The checklist items are divided into 12 sections that represent the 12 Quality System Essentials (QSEs) as defined by the Clinical and Laboratory Standards Institute (CLSI). 5 These QSEs can be grouped by stages of the quality cycle: Resource Management (equipment; facilities and safety; organisation and personnel; purchasing and inventory), Process Management (client management; documents and records; information management; process control and internal/ external quality assessment) and Improvement Management (corrective action; internal audit; management reviews; occurrence management). 6 To assess progress, baseline and exit audits are conducted before and after SLMTA implementation, respectively, using the SLIPTA checklist. 'Surveillance' audits are also often conducted after the exit audit in order to monitor continued improvement and assess sustainability.
Several studies provided scores by individual QSEs. We combined these data and conducted a meta-analysis in Microsoft ® Excel 2013 so as to determine common areas of strength, weakness and improvement. For studies reporting only median or mean QSE data for multiple laboratories, laboratory-level data were solicited from authors to further enhance the analysis. All cost estimates reported in local currency in published articles were converted into US dollars, based on the official exchange rate as of August 1, 2014. Percent changes in indicator results were calculated from published results if not reported directly in the papers.

Global programme results
Data from all laboratories implementing the SLMTA programme were collated and summarised in a single paper describing the global results of the programme to date. 33 In total, 617 laboratories in 47 countries on four continents have implemented SLMTA in 65 training cohorts, with nearly 2000 laboratory staff trained in the programme. Most of the laboratories were at the district (38%), regional (27%) or national (18%) levels. The authors report that the starting level of laboratory quality in developing countries was very low, with 84% of SLMTA laboratories scoring below the one-star level at baseline. The 302 laboratories that had completed the programme had an average improvement of 25 percentage points; 70% achieved at least one star at exit audit and 22% of laboratories increased three or more star levels.
Estimates of the number of laboratory tests conducted by SLMTA laboratories suggested that the 617 laboratories enrolled in SLMTA conduct more than 100 million tests annually and that whilst only 16% of these tests were conducted by laboratories with at least one quality star before SLMTA, 68% were done by laboratories with at least one star after SLMTA implementation. That translates to approximately 58 million tests conducted by laboratories with little to no QMS prior to SLMTA which now have at least a basic quality system in place. 33

Quality System Essentials meta-analysis
Examining individual SLIPTA checklist scores for each of the 12 QSEs enables laboratories to pinpoint strengths, weaknesses and areas of improvement. QSE data have not been compiled systematically on a global scale. From the published papers, QSE data were presented for 126 laboratories in 12 countries. 8,11,12,14,15,18,20,21,22,25,26 Individual studies reported substantial variability in high-and lowscoring QSEs. For example, some laboratories scored 0% for five of the 12 QSEs at exit audit, whereas others scored 100% for the same five QSEs.
At baseline, the weakest areas overall were in the Improvement Management stage of the quality cycle, including internal audit (5%), occurrence management (16%), corrective action (25%) and management reviews (29%) (Figure 1). At an average of 20%, this stage scored less than half of the other two stages, namely, Resource Management (42%) and Process Management (40%). None of the 12 QSEs had mean baseline scores above 55%; the highest scores were in information management (51%), facilities and safety (47%), purchasing and inventory (42%) and process control and internal/external quality assessment (41%).
At the exit audit, the four Improvement Management QSEs still showed the lowest scores, ranging from 32% -50% (average 42%) ( Figure 1). The Resource Management and Process Management stages had higher scores ranging from 58% -74% (average 65% for Resource Management and 63% for Process Management). The greatest improvements were in documents and records (34 percentage points), client management (29 percentage points), and facilities and safety (27 percentage points). Each of the three stages had the same average improvement of 23 percentage points.
Based on results from five laboratories, Maina et al. found that the laboratories with the greatest overall score increases had focused on internal audit and corrective action; they then hypothesised that an improvement in these areas may be a catalyst for overall improvement in other areas. 14 Metaanalysis results suggest that the corrective action QSE may be the most predictive of overall improvement; laboratories in the top quartile of overall improvement outperformed those in the bottom quartile by 62 percentage points for the corrective action QSE, compared to a median of 40 percentage points for the other QSEs. CLSI defines corrective action as an 'action to eliminate the (root) cause of a detected nonconformity or other undesirable situation'. 34 In the SLIPTA checklist, corrective action is assessed through four questions about how the laboratory deals with occurrence reports, nonconformities and discordant results. 4 The International Organization for Standardization (ISO) confirms the importance of corrective action, saying that 'the corrective and preventive actions system is the most critical element for an efficient quality system'. 35 Additional work is needed to verify priority areas of improvement, as well as to delineate the set of essential improvement projects that will result in meaningful laboratory quality improvement.

Official WHO AFRO SLIPTA audits and accreditation
A July 2009 survey of accrediting body registers identified 340 accredited laboratories in sub-Saharan Africa; only 28 (8%) of these laboratories were located outside of South Africa and nearly all were private, parastatal or donor-supported research facilities. 36 By early 2013, little progress had been made, with 380 laboratories accredited in the region; only 35 (9%) laboratories outside of South Africa were accredited and three quarters of the 49 countries in the region had no accredited laboratories. 37 However, the impact of SLMTA is beginning to show; as of September 2014, six laboratories enrolled in SLMTA in Kenya, the Bahamas, Vietnam and Zimbabwe have been accredited, at a median of 31.5 months after starting the SLMTA programme. 10,11,33 Several laboratories have been recommended for accreditation or are in the process of application. 11,18,20,28 Ninety-seven SLMTA laboratories have received official WHO AFRO SLIPTA audits conducted by representatives from the African Society for Laboratory Medicine, 33 including 11 laboratories in published reports included in this review. 7,18,25,26,29

Service delivery indicators
In addition to audit scores, many of the studies reported improvements for indicators reflecting testing and customer and clinician satisfaction ( Table 2). Three studies reported reductions in turnaround time for testing, 10,20,28 with times decreasing by 19% -95%. Patient and clinician satisfaction were commonly measured using surveys. Four studies showed relative improvements in patient satisfaction ranging from 30% to > 100%, 9,10,25,28 although in one laboratory complaints from patients increased, possibly as a result of staff attrition. 17 Two studies reporting on clinician satisfaction found improvements of approximately 80%. 17,28 Indicators for laboratory management and overall functioning also showed improvements (Table 2). One laboratory reported a 65% decrease in corrective actions, 10 five laboratories in the Caribbean Region reported decreases in nonconformities of 50% -66% 11 and two laboratories showed improvements in external quality assessment results of 67% -85%. 10,17 In a Kenyan laboratory, staff punctuality increased 67% and the need for equipment repairs decreased 63%. 17 A Botswana laboratory successfully reduced losses resulting from expired reagents from $18 000 in 2010 to $40 in 2013; 28 and three studies showed reductions in specimen rejection rates of 69% -93%. 10,17,25 When SLMTA was adapted and implemented at a hospital in Cameroon, patient wait times decreased 67% -83%, infection rates and stillborn rates decreased (83% and 80%, respectively) and the number of patients and hospital revenue increased. 9

Cost
The reported costs per laboratory of implementing various components of SLMTA have varied widely ( Table 3). Much of this variability is because of differences in what was included in the cost estimates, as well as location-specific factors, such the price of fuel, salary levels and distances to participating laboratories. The estimated cost of conducting the three-workshop SLMTA series has ranged from $1482 per laboratory in Zimbabwe using local facilitators in a central location 31 to $21 480 in Cameroon using decentralised training. 22 Mentorship cost per laboratory has ranged from $5689 in Zimbabwe 30 to $24 000 in Ghana. 25 The cost of implementing improvement projects has ranged from $10 000 in Ghana 25 to $36 500 in a Kenyan laboratory seeking accreditation. 10 Three studies have compared the cost of various SLMTA implementation models. One study of 19 laboratories in Zimbabwe found that mentorship and supervision costs for four different models were similar ($5689-$9601 per laboratory), recommending that 'countries should carefully consider which mentorship model or models would be best suited to their individual situation'. 30 Another study in Zimbabwe found that implementing SLMTA using local (in-country) facilitators is more expensive than external facilitators for the first SLMTA cohort because of the costs associated with conducting an in-country training-oftrainers; however, over the course of national scale-up in 120 laboratories, use of local facilitators would save the country nearly 50% ($580 000 vs. $322 000). 31 A Cameroonian study found that the cost per laboratory of centralised training was approximately the same as decentralised training ($21 122 vs. $21 480, respectively); centralised training required less trainer time, whilst decentralised training allowed more staff to participate. 22 No published studies to date have reported a thorough examination of the cost of implementing the entire SLMTA programme, including each of the major components (training of mentors, trainers and auditors; conducting SLMTA workshops; mentorship, supervisory visits and implementation of improvement projects; and conducting audits). In addition, a more extensive cost-benefit analysis taking into consideration the value of laboratorians' time (i.e., opportunity cost) to participate in the programme and

Limitations to the study
This review is subject to several limitations. Firstly, whilst 28 studies on SLMTA were identified and summarised, these reflect only 18 (38%) of the 47 countries and 211 (34%) of the 617 laboratories that have implemented the programme. Their results may not be representative of the programme as a whole, or a comprehensive account of all laboratories' experiences. Secondly, whilst audit results were available for all laboratories because of the use of the SLIPTA checklist, the other indicators presented here were available in few of the published studies; in addition, methodologies varied between the studies, limiting the ability to combine and compare results directly.
Authors of the studies published thus far also point out several limitations. Firstly, the SLMTA programme as a whole is too young to allow an assessment of the long-term sustainability of results. 14,33 Secondly, all of the published studies were observational; several studies examining the effect of mentorship or training methodologies note that laboratories were not assigned randomly, but were rather selected purposively based on convenience or other programmatic considerations. Thus there may have been other factors that could account for some of the differences. 8,15,20,30 Similarly, none of the studies included control laboratories upon which to base a comparison. 22 Thirdly, there is a lack of consistency in the qualifications of auditors; whilst the SLIPTA checklist is designed to help standardise the audit process, some variability between auditors may remain. 8,29 Finally, several authors noted that their published studies are based on a small number of laboratories 14,15,20,30 and some indicators were either not measured systematically 9 or not measured at baseline. 9,28

Conclusion
In their summary of global-level findings, Yao et al. point out that 'few [other] management and leadership development programmes have been implemented on a such a large scale with results-oriented outcome measures'. 33 The wide array of results reported provides a comprehensive picture of the SLMTA programme overall, suggesting a substantive impact on provision of quality laboratory services and patient care. The full potential of the programme can be realised only if the lessons learned lead to informed action among laboratory workers, healthcare providers and policy makers toward the ultimate goal of providing quality patient care.