Towards harmonization of external quality assessment/proficiency testing in hemostasis

Introduction

Quality in diagnostic testing represents a key target of laboratory medicine. An assurance around the quality of testing is expected from all involved in the process, including the ordering clinicians and the patient from whom samples are collected and tested. Laboratories attempt to assure the quality of their testing by various processes, but key to assurance is the successful performance of internal quality control (IQC) and external quality assessment (EQA). This is especially true for tests of hemostasis and coagulation [1], [2], [3]. IQC can be readily and easily applied to most diagnostic hemostasis assays, although there are challenges for some tests, for example, platelet function. EQA describes a different and supplementary process to IQC, being a peer group assessment process that permits laboratories to assess individual analytes against those of other laboratories, which may use the same or even different reagents or instrumentation. IQC, in general, provides information on the precision of the assay, whereas EQA in general provides information on the accuracy. IQC represents a part of process control and determines the release of results (laboratories should not release results if QC fails). EQA also provides an evaluation of long-term laboratory performance. For hemostasis testing, the analytes assessed by EQA programs are those used in the diagnosis and treatment of bleeding or thrombotic disorders. EQA achieves some of its functions by distribution of material for testing of analytes, and this describes the “proficiency testing” side of EQA. Here, individual laboratory bias and drift can be monitored along with test accuracy and precision. However, most EQA providers offer far more than just proficiency testing, including educational support, either through expert advice or through distribution of publications or case series (e.g. [4]). Also, most EQA providers distribute and collate questionnaires related to current practice, which thus informs participants on various topics of interest, and provides additional information on reagents, instrumentation and laboratory practices (e.g. [5]). EQA providers may also be involved in providing assessment and/or education in the pre- and postanalytical area.

EQA providers or advising committees also produce guidance documents for laboratory testing. For example, North American Specialized Coagulation Laboratory Association (NASCOLA) has achieved this in collaboration with participants [6], [7], and the College of American Pathologists (CAP) by organizing expert consensus conferences [8], [9]. As another example, External quality Control of diagnostic Assays and Tests (ECAT) has published an algorithm for laboratories to help identifying the cause of an unexpected prolonged activated partial thromboplastin time (APTT) [10]. Additionally, UK National External Quality Assessment Service (NEQAS) has produced interpretative programs, such as via anticoagulant dosing exercises [11], and the Royal College of Pathologists of Australasia Quality Assurance Programs (RCPAQAP), via an Australasian working party, has produced consensus guidelines on anti-cardiolipin antibody testing and reporting [12], [13] that were eventually incorporated into an international consensus guideline [14].

Participation in EQA is often a laboratory accreditation requirement [2]. For example, within Australia, medical laboratories are required to meet International Standards Organization (ISO) 15189 standards, and accreditation is overseen by National Association of Testing Authorities (NATA). During such accreditation assessments, EQA results are examined and analyte data falling outside predefined “limits of performance” need to have been acknowledged, and/or evaluated/investigated/actioned or by the laboratory, and explanations recorded of any discrepancies. If the problem continues over several surveys, laboratories then need to be more proactive in solving the issue. In most European countries, laboratories are also required to meet ISO 15189 standards. In turn, EQA providers are also accredited, in this case to ISO 17043 standard, with National Accreditation bodies overseeing their continued accreditation.

The situation in the US is perhaps more complex, as there are several regulatory agencies involved for diagnostic testing of human samples. The US Food and Drug Administration (FDA) creates the rules and guidance related to laboratory test complexity. The Center for Medicare and Medicaid Services (CMS [through the Clinical Laboratory Improvement Act (CLIA)]) requires all laboratories in the US to be licensed, but also is responsible for publishing the rules and regulations, as well as monitoring laboratory performance of EQA. The Centers for Disease Control (CDC) may also develop guidelines, provide technical assistance and is the managing organization for Clinical Laboratory Improvement Advisory Committee. In addition to CMS accreditation, state licensure may also be required.

CMS has granted several accrediting organizations deemed status. Therefore, accreditation with one of those organizations (an example and the largest being CAP, which also oversees some EQA in the US) also qualifies to meet the CMS licensing regulation. Of course, the accreditation requirements of the accrediting organization must fulfill all of the CLIA requirements but may have requirements that exceed those of the CLIA regulation (see below). CMS grants deemed status based on a separate set of rules for both EQA providers and accrediting organizations. CMS strictly regulates only 83 analytes, imposing strict requirements regarding IQC and EQA for each of them, with only three hemostasis tests: APTT, prothrombin time (PT; but not INR, international normalized ratio) and fibrinogen.

For the CMS regulated tests, the EQA program and analysis is vaguely described in CFR §493.941, which applies to coagulation testing. “Agreement”, or successful EQA, is being within 80% of 10 of more “referee” laboratories, or within 80% or more of all participating laboratories. CMS determination for acceptable performance is defined as “target” (±15% for APTT and PT; ±20% for fibrinogen). What is less clear, is the defining or calculated “target” value, but CMS requires the EQA program to have a “scientifically defensible process for determining the correct result for each challenge” and “must determine the correct response for each analyte by the distance of the response from the target value”. In addition, the CMS regulation does not describe peer groups, methods of analysis for non-regulated analytes, or limits of acceptability for non-regulated analytes. As such, these decisions are often left to the accrediting agency, significantly limiting the “harmonization” that is desirable to optimize laboratory quality.

For regulated analytes, CMS requires at least five samples per testing event, and three testing events per year, with “samples that cover the full range of patient values that would be expected in patient specimens”. Under CAP accreditation, there are other non-regulated coagulation tests for which EQA is required, “graded” and thus requiring successful EQA participation (e.g. D-dimer, factor VIII [FVIII] and anti-cardiolipin antibodies). For every non-regulated analyte, the laboratory must perform “Alternative Performance Assessment (APA)” at least twice a year, to show the test is accurate. In most cases, this is done via EQA (e.g. CAP offers EQA for more than 1000 analytes and NASCOLA, which provides a specific EQA program for non-regulated analytes). However, some analytes are not available, and the laboratory must develop an alternative EQA of its own. There is no guidance or regulatory documents to aid laboratories in assessing EQA acceptability for these alternative (e.g. non-commercial) EQA programs. Therefore, there is no harmonization among providers regarding the large number of non-regulated coagulation analytes.

Regulated analytes are reported to CMS within 60 days of event completion and are assessed on a continuous basis by the providing EQA program, whereas alternative, non-commercial EQA should be assessed and checked at time of accreditation inspection, which is every 2 years. Penalties for failure may be onerous. Repeated failing of CMS-regulated EQA may have significant financial impacts as the laboratory may be required to cease testing the analyte for a minimum of 6 months. Laboratory accreditation may also be at risk for continued EQA failures.

Accordingly, although it is clear that EQA providers, and the laboratories they service, are all required to be accredited or otherwise their procedures assessed, the processes employed for oversight of practice may differ according to geography and, thus, will act to dampen the ability to fully harmonize EQA activities. Moreover, different EQA providers may offer different proficiency testing services, utilize different assessment criteria and approach EQA in different ways. This review aims to identify some of the strengths and weaknesses of EQA and target harmonization of EQA practice in order to achieve the best outcome for participant laboratories and, thus, for patients and their clinical care providers.

External quality assurance in thrombosis and hemostasis (EQATH): an international perspective

External quality assurance in thrombosis and hemostasis (EQATH) is a group that was formed in 2005 as an international collaboration of EQA programs and organizations, with a common interest in improving the quality of hemostasis testing. The goals of EQATH include exchanging of information regarding program operations, exchanging split specimens among programs to determine if there are differences in practice, participating in value setting of standards, providing outreach to locations in the world without EQA support of hemostasis testing in laboratories and targeting improvement in EQA practice by harmonizing towards best practice. Collaborative studies can also provide a greater amount of information on between-laboratory agreement for tests where there are few centers in any one country performing that test. In 2007, at the time of the first report, the organization comprised 11 EQA programs from 10 different countries [15]. A survey of program structure and function revealed much variation in size and structure of programs, with staffing levels paralleling EQA size and complexity. The number of laboratory participants in the EQA programs surveyed ranged from 58 to 1700. The presentation of testing covered (“modules”) ranged from one EQA program with a single module of a single test, to programs with single modules of many different test types, to one program with 13 modules, each of which contained a limited scope of related tests. Participating laboratories were often graded, but this comprised differences in approach (e.g. pass/fail, in-with/out-with consensus by six EQA programs), or alternatively, results were reported to the laboratory for self-evaluation (five EQA programs). Of the 11 survey respondents, seven identified “deemed status” from an accrediting or licensing agency and that successful laboratory participation satisfied requirements for continued accreditation. This type of benchmarking activity and cooperative activity among EQA programs was believed to lead to improvement in EQA, and ultimately perhaps also to harmonizing towards the best EQA practice.

In 2017, EQATH has evolved to having a website [16] and lists 13 participating EQA organizations (as listed in Table 1). These EQA programs may provide national, regional or international EQA services. The initial goals of EQATH (according to the website) are listed in Table 2. Meetings of the group are held annually in association with the International Society on Thrombosis and Haemostasis (ISTH) meetings, and minutes of these meetings are also available online [16].

One notable achievement of this group is the recent co-joint EQA publication of a study reflecting on the lack of agreement among international hemostasis EQA programs in terms of “pass” or “fail”, with the intention of highlighting one of the problems related to lack of EQA harmonization in this area [17]. It was highlighted that different EQA programs analyze the data returned to them using different statistical tools. Eleven EQA providers, members of EQATH, took part in this study, which compared data and performance analysis using the same sampling data set of laboratory results. Eight different analytical approaches were employed by the 11 participant EQA providers. Datasets of 218 results for a normal and prolonged APTT challenge, and a normal and reduced FVIII challenge, were sent to the EQA providers, who then analyzed the data as if it came from their participants, and returned an evaluation of performance for each record in the data set for collation and comparison. For the normal APTT data set, 79.8% of records were graded as passing by all programs and 2.3% were graded as failing by all programs. Thus, 17.9% of records had discordant grading, where the laboratory data would have been passed by some EQA programs but failed by others. For the prolonged APTT sample data set, 88.5% of records were passed by all programs, 0.5% failed by all programs and 11% had “disputed” grades. For the normal FVIII sample data set, 78.9% of results were passed by all programs, 0.9% failed by all programs and 20.2% had disputed grades. For the reduced FVIII data set, these figures were, respectively, 79.8%, 2.8% and 17.4%. Importantly, all programs in this study employed methods compliant with ISO 13528. All EQA programs were thus considered to be complying with international standards for proficiency testing provision, including appropriate statistical analysis. Here it should be recognized that the ISO 13528 standard (Statistical methods for use in proficiency testing by interlaboratory comparison; 2015) allows a number of different statistical approaches for assessment of performance. These results are important because EQA participation is required for laboratory accreditation in many countries, and in some cases, ongoing EQA success is also required for accreditation or licensure. However, differences in peer group analysis and choices of statistical analytical approaches, together with criteria for passing or failing laboratory performance, led to notable discrepancies between EQA programs in terms of data considered to pass or fail in this exercise. It was concluded that further evaluation of EQA methodology and agreement on appropriate clinical and statistical approaches might enable more uniform assignment of performance grading in international proficiency testing challenges.

Several other recent collaborations have been conducted among EQA members of EQATH. For example, one recent publication explored proficiency testing for hemophilia, including acquired inhibitors of coagulation factors among three international EQA providers [18]. A clinical diagnosis of hemophilia A or B in any patient begins with clinical assessment and is then confirmed by laboratory testing. Indeed, a diagnosis cannot be made without laboratory confirmation of a deficiency of FVIII or FIX, respectively. Moreover, the degree of hemophilia severity is specifically characterized by the test result, and future patient management, including choice and application of therapies, is influenced by both diagnosis and identification of disease severity. Alternately, an incorrect diagnosis may lead to inappropriate and/or unnecessary therapy/management, and thus to adverse outcomes. Furthermore, identification of factor inhibitors in hemophilia will lead to additional and differential treatments, and incorrect identification of inhibitors, or inhibitor levels, might similarly lead to inappropriate management. This report outlined the various problems in laboratory testing for hemophilia and provided various strategies or solutions to overcome these challenges [18]. Of relevance to EQA, the report outlined several key findings, including assay variation related to both factor and factor inhibitor proficiency testing. For factor testing, coefficient of variation (CV) approximated 15%–25% for most samples and geographies. EQA data also showed that although most laboratories provided similar results when testing the same sample, there were occasional outliers mirroring inaccurate test data reflected in each EQA program. Although reasons for these outlier (inaccurate) results are not always clear, they may include test failures, sample reconstitution issues, sample mismatches or transcription errors. The report also highlights lower limit of detection issues for some assays and laboratories. Inaccuracies in factor inhibitor testing were also evident from the EQA data. Each EQA group reported significant interlaboratory variation, as reflected by high interlaboratory CVs, which were often greater than 30%. In general, less variability was seen with Nijmegen method-based inhibitor assays compared with standard Bethesda assays. It was concluded that key to improvement in this area of laboratory medicine is the adoption of best practice by all involved, including clinicians, phlebotomists and laboratories.

Another more recent collaboration of the same three EQA programs has involved proficiency testing for von Willebrand disease (VWD), considered the most common inherited bleeding disorder [19]. The data for this collaboration is currently submitted for publication. In brief, different assays for von Willebrand factor (VWF), which is deficient or defective in VWD, showed different sensitivity to high molecular weight forms of VWF in tested samples. This has implications for diagnosis and management of not only VWD but also prothrombotic condition thrombotic thrombocytopenic purpura, caused by excessive presence of very large VWF multimer forms. The current EQA collaboration confirms and extends the findings of a previous smaller study undertaken by one of the EQA programs [20].

Finally, representatives of the same three EQA programs have joined other concerned colleagues, including members of the fibrinolysis and disseminated intravascular coagulation (DIC) standardization subcommittee of the ISTH in a call to action on D-dimer testing [21]. D-dimer is a stable, terminal product of fibrin degradation in plasma and now recognized as a biomarker of coagulation activation and fibrin formation. D-dimer is now used in the diagnostic process of various conditions, including the exclusion of venous thromboembolism, diagnostic scores for DIC, thromboembolic risk in patients with atrial fibrillation and, more recently, to help investigate acute-onset chest pain (e.g. aortic aneurysm). Although the D-dimer assay is a significant advancement to the preexisting measurement of fibrin and fibrinogen degradation products, several issues remain with its routine use, thus compromising its clinical utility. This particular communication was a call for further efforts to harmonize D-dimer measurement, which may increase in importance as laboratories opt in for age-adjusting cut-offs for venous thromboembolism (VTE) exclusion [22]. In particular, there is a current lack of uniformity in the type and wide array of units of measurement used for reporting results, and a lack of a calibrator that can be used to standardize the many assays currently in use [23]. Based on a review of recent proficiency testing, data revealed that laboratories are reporting fibrinogen equivalent units and D-dimer units with about equal frequency. In addition, the reported units of measurement (ng/mL, μg/mL, μg/L, etc.) are also widely variable, and on occasion due to conflicting recommendations. This variance was further highlighted by a recent global survey that identified the use of 28 different combinations of units of measurement currently used to report D-dimer results worldwide [23]. This confusion, or lack of awareness, even extends to the literature (peer reviewed and books chapters), in which authors (and editors) provide recommendations regarding concentration of D-dimer for clinical action (e.g. concentration for exclusion of VTE) but fail to indicate the type of units of measurement being employed (e.g. [24]). This occurred in more than 50% of randomly sampled articles [25]. Sometimes, the laboratories also report the wrong units of measurement (i.e. different to what they are actually using).

Activities of individual EQA providers that may lead to further EQA harmonization

Recognizing that EQA organizations largely operate independent of each other brings to focus certain initiatives from individual EQA providers that have the capacity to teach other EQA providers and, indeed provide incentives regarding future harmonization, and driving all EQA organizations towards best practice. With regard to individual EQA organizations, Table 3 and the text below provides a summary of some of the more interesting initiatives from individual EQA groups with such potential.

For example, ECAT recently performed a very interesting sequential study that attempted to harmonize laboratory test practice for factor inhibitors [26]. The impetus for this study was the recognition that such assays show wide variation in interlaboratory test results (often >40%), and also high potential for false low and false high inhibitor detection – findings consistent across different EQA providers [27], [28], [29], but which can result in undesired effects on treatment. The Bethesda and the Nijmegen versions of the inhibitor assay are most commonly used for the measurement of inhibitor levels in hemophilia A patients. Fifteen laboratories showing a high variation in regular EQA surveys, and using a variety of methods, participated in this wet workshop, which included four sequential sessions where variables probably contributing to high variation were investigated (e.g. use of [non-]buffered plasma, FVIII-deficient plasma, sample dilution and APTT reagents). The CV varied from 30% to 70% in the first workshop session where participants used their own test settings/reagents. Use of buffered normal pooled plasma and FVIII-deficient plasma as a reference sample by all participants did not significantly alter the CV (35%–50%) but did decrease the number of false positives. However, use of buffered pooled plasma in combination with standardized sample dilution procedures by all participants showed a significant improvement (CV, 10%–20%). This study showed factors that could contribute to harmonization and improvement of FVIII inhibitor testing. ECAT later performed another study, extending these findings to investigate more broadly ‘acquired inhibitors” [10]. The aim of this workshop was to investigate, within groups of experts from dispersed professional laboratories, the quality of inhibitor detection and the difficulties encountered during the analytical process. Eight samples representing varying milieu were tested by 10 groups of participants from 20 different countries. Workshop participants were asked to report the results of all investigations performed and to provide a likely diagnosis and/or a conclusion of the hemostasis abnormality represented by the test samples. Generally, the sensitivity of inhibitor detection was high, but the differential diagnosis of the type of inhibitors identified was generally less satisfactory, as many false-positive and false-negative results were reported. The most remarkable observation was the lack of a clear step-by-step analysis of the nature of an inhibitor once a positive mixing screening test had been identified.

The RCPAQAP hematology has similarly published many reports identifying problems in hemostasis testing (Table 3), and also providing suggestions for their mitigation. One of the more novel EQA “proficiency testing” modules to have developed is that related to platelet function analyzer (PFA)-100/PFA-200 (Siemens, Marburg, Germany) testing. The PFA represents a screening test system for platelet abnormalities, including identification/exclusion of VWD [30]. EQA for such a test system would normally be “impossible”, given the need for production and delivery of large volumes of stabilized whole blood representing various hemostatic defects to laboratories in order for them to undertake testing [31]. Originally manufactured as the PFA-100, the instrument has recently been “upgraded” by the manufacturer to the PFA-200. In this novel EQA/proficiency testing approach, laboratories are instead provided with stabilized “antagonists”, to which normal whole blood (collected locally) is then added and tested by EQA participants. This novel approach has now been in use for the past 10 years and has consistently shown similar results, verifying the feasibility of such an approach for evaluation of “platelet function” and thus potential applicability for other platelet function test systems [32], [33], [34], [35]. In terms of the PFA-100/200, different challenges have been distributed representing “mild”, “moderate” or “severe” defects. The CVs for test challenges (10%–25%) are similar to, and in some cases better than, those of baseline blood test results. The “proficiency test” material may also have additional IQC utility [34], [35].

Analyses of EQA data can be used to characterize the effects of anticoagulant therapies on laboratory tests in thrombosis and hemostasis. For example, a study by CAP”s EQA committee found that fondaparinux, even in prophylactic doses, slightly prolonged the PT and APTT and can falsely decrease factor VIII, but it had no clinically relevant effect on fibrinogen (even PT-based methods) or antithrombin (even factor Xa-based methods) [36]. A study from NASCOLA’s EQA committee found that homemade dabigatran calibrators differed from commercially available calibrators, and there was a statistically significant difference among some of the rivaroxaban reagents [37]. EQA can also uncover problems with laboratory performance, leading to improved laboratory practices and reduced laboratory errors on a national level. For example, an analysis of EQA data showed that up to 4% of laboratories were incorrectly calculating the INR, a dangerous error that can lead to hemorrhagic or thrombotic complications. After the overseeing EQA committee intervened, the error rate improved to <1% [38]. EQA data from NASCOLA and CAP have been analyzed to assess quality of testing in thrombosis and hemostasis in clinical laboratories across North America. These studies show important differences among coagulation test methods, including the following:

1. fibrinogen (about 1300 laboratories and over 50 methods assessed, revealing highly variable quality of results) [39],

2. factor XIII (clot solubility tests demonstrated poor sensitivity and a quantitative activity assay overestimated low factor XIII levels) [40],

3. factor VII (some methods overestimate factor VII compared to other methods) [41],

4. protein S (some methods were unable to detect type I deficiency) [42], [43],

5. VWF (methods were accurate but not all were precise) [44], and only a minority of laboratories were following National Heart Lung Blood Institute guidelines) related to testing [45], [46],

6. PFA-100 (precision was poor and many laboratories had difficulty identifying abnormal specimens as abnormal) [47],

7. heparin-induced thrombocytopenia (laboratories had difficulty detecting weak positive samples) [48],

8. thrombophilia tests for protein C, protein S and antithrombin deficiencies (methods varied in accuracy and precision) [49] and

9. platelet aggregation interpretation (laboratories had difficulty diagnosing some commonly encountered findings [50].

Other analyses showed good performance by laboratories and reagents. For example, laboratories successfully distinguished between low titer and no FVIII inhibitors [51], laboratories successfully identified dense granule deficiency by platelet electron microscopy [50], [52] and few differences were identified among protein C reagents [53] or among factor XI reagents [54]. These analyses of the quality of laboratory testing help determine which tests need improvements and greater standardization, facilitate the selection of high-quality methods by laboratories and educate laboratories regarding assay interferences and limitations.

UK NEQAS (blood coagulation) has also reported on issues with laboratory testing and practice, including failures in the diagnosis of lupus anticoagulant (LA) [55] protein S deficiency [56], FXIII deficiency [57], as well as errors in classification of hemophilia A [58] and VWD [59], and assay variability [60], [61]. EQA has been able to demonstrate improvements through harmonization procedures [62] or adherence to guideline recommendations [4], and to evaluate different diagnostic approaches employed to investigate a bleeding disorder [63].

Future perspectives on EQA harmonization

This overview has provided several examples whereby individual EQA programs have combined to assess a more broadened EQA assessment, as well as individually highlighting areas in which individual EQA programs have used novel approaches to investigate and improve particular areas of hemostasis testing. All the EQA programs participating in EQATH (Table 1) will learn from such activities, which in turn will also drive harmonization towards best test practice in hemostasis, as well as best EQA practice, especially as the learned experiences devolve to other EQA providers. Certain projects are also under consideration for potential future harmonization, including those based on PFA-100/200 testing [32], [33], [34], [35], VWF testing [20] and factor testing. The latter is perhaps most critical, given imminent release of a plethora of extended-life factor replacement agents [64], [65], as well as agents not based on factor replacement. Thus, a cross-EQA study involving postinfusion monitoring of clotting factor replacement therapy is in current planned development by UK NEQAS. Nevertheless, future cross-EQA evaluations for VWF are also important, given ongoing replacement of certain methodologies and increasing use of advanced technologies [66].

A key element in performance assessment of participants in EQA programs is the use of proper performance criteria and specifications. Currently, the majority of EQA programs in the field of hemostasis use the state-of-the-art approach for setting their acceptance limits (e.g. a particular % deviation from the consensus value, Z-scores, etc.). However, there is no consensus yet about which acceptance criteria are most appropriate for performance assessment. See for example the before mentioned study on APTT and FVIII [15]. Recently, a revised consensus statement of defining analytical performance specifications was published [66]. This includes three different models: clinical outcome studies, biological variation and state of- the art. Within the framework of EQATH, EQA organizations should work on the harmonization of performance criteria in use. This will support harmonization of performance assessment and as such harmonization of laboratory test results.

Additional cross-EQA discussions, both within EQATH and separately among several EQAs, are planned around the use and promotion of national and international guidelines on laboratory practice as a means to attempt to standardize methodological approaches, and also the role of EQA programs in carrying out surveys of practice to assess conformity with these guidelines (e.g. as already performed by UK NEQAS for LA testing) [4].

Alternatively, the requirement for programs in many countries to adhere to certain ISO standards introduces a level of harmonization, given EQA programs have to work to meet the same standards, including for statistical evaluation. Here, the US stands out as being different to Europe and Australia, thus challenging harmonization.