SoGAT—25 years of improving the measurement of nucleic acids in infectious disease diagnostics (a review)

The meeting forum called Standardisation of Genome Amplification Technologies (SoGAT) was established 25 years ago, with the purpose of improving the quality of infectious disease diagnostic data generated by polymerase chain reaction (PCR) and related techniques. This meeting has provided an expert forum to identify and prioritise the needs of the diagnostic community for reference materials established by the World Health Organisation Expert Committee for Biological Standardisation. Reference materials for 30 different infectious agents have been established and are available upon request. The improvement in the measurement of specific nucleic acid sequences of pathogens arising from the availability of their cognate standard and the challenges going forward in this field are discussed.


Introduction
The development of methods such as polymerase chain reaction [1] and isothermal amplification [2] of specific nucleic acid sequences in the 1980s and early 1990s led to a seachange in developments for the infectious disease diag nostics industry. These technologies provided a means for highly specific and exquisitely sensitive methods for the direct detection of a pathogen in complex biological samples, without having to wait for weeks either for the pathogen to grow in culture or for the appearance of detectable levels of specific antibodies. The initial beneficiary was the blood transfusion and plasma fractionation industries, reeling from the realisation that they had contributed to the spread of incurable, lethal diseases HIV and Hepatitis C.
In principle, nucleic acid amplification technology (NAT) assays have the capacity to detect the presence of a single molecular copy of the specific target sequence in a sample. However, it became apparent that if a common sample was shared between a number of expert laboratories, then they did not always come up with concordant results: on some occasions with a range of 10 × 10 4 difference in value for identical samples [3].
It was recognised, by some, that the challenges being faced with molecular amplification based diagnostic assays in the 1990s were strikingly similar with those of the measurement of insulin activity in the 1920s: how can one measure reproducibly a specific target present within a biologically complex sample, when the performance of apparently identical assays appear to be highly variable over time or between laboratories. There was a need to establish methods to harmonise measurement by NAT assays. As a result the international meeting forum-Standardisation of Genome Amplification Technologies or SoGAT was born in 1994.
From the discussions at these meetings, it was clear that at a time of break-neck speed of assay development, it was not only not possible to establish a reference method against which all other assays must perform, but also that such an approach could hinder the progress of this diagnostics technology from achieving its ultimate capability. As a result SoGAT adopted the principles of biological standardisation through the production of higher order reference materials to provide greater traceability. Since then the WHO have established more than 40 International Standards or Reference Reagents/Panels (reviewed in [4]).
By working with the WHO's Expert Committee on Biological Standardisation, SoGAT has provided a forum for scientific and clinical experts, diagnostic kit manufacturers, and regulatory laboratories to meet and provide expert oversight of the programmes to prepare these higher order reference materials and help to prioritise the programmes based on public health need. This paper analyses the data to establish whether this programme has been successful and identifies the future challenges for SoGAT.

Harmonisation of data using reference standards
The WHO establishes International Standards based upon evidence from international multi-centre collaborative trials that investigates the performance of the candidate standard. The design of these reference materials has been to mimic the infectious organism in a material that replicates the biological samples used for diagnosis. Frequently this involves freezedrying a whole pathogen in a matrix, incorporating excipients that have been demonstrated to enhance the stability of nucleic acid. As a result following reconstitution, the reference material goes through both extraction and amplification of nucleic acid, thus providing a control for the whole process.
The first test of a lyophilised candidate standard is whether its use harmonises data from a closely related material, frequently chosen is the bulk liquid preparation used to prepare the lyophilised candidate standard. This is undertaken to demonstrate whether the freeze drying process has altered the amount of 'activity' present in the preparation. As demonstrated for the BK virus candidate standard in 2015 (figure 1), when laboratories were asked to return estimates for the liquid bulk (sample E, figure 1(a)), the range of values returns as copies/ml varied by almost 2700 fold. Whereas when these same data were calculated relative to the candidate BKV IS, also supplied and tested contemporaneously by the laboratories, then the range was markedly reduced to less than ten fold for the majority of laboratories and less than 500 fold for all laboratories ( figure 1(b)). Harmonising results has a clinical impact. When this measurement relates to a cut-off used for 'intention for clinical treatment', then the variability of this diagnostic measurement must be minimised, in order to make it possible to compare the relative benefit of treatment plans undertaken at different clinical centres.
There has been a long term benefit from harmonising the quantification of nucleic acid of infectious agents. In the absence of a reference method the availability of a primary order standard provides the 'common benchmark' to calibrate secondary and working standards used by assay developers. Furthermore, results from studies using this common standard provide evidence of progress by the field. One of the earliest International Standards of this type prepared was for Hepatitis C virus RNA [3]. Because of its popularity to underpin the quality of assays that protect the blood and blood product industry, it has needed to be replaced on four subsequent occasions since 1998. Once again the absence of a reference method for calibration of these materials results in the need for a collaborative study engaging a range of laboratories with demonstrable expertise in measuring the specific analyte. This group of laboratories then work together to provide estimates of the quantity present in a panel of samples supplied, including the candidate International Standard. Moreover the data from all the participating laboratories are analysed to determine whether using the candidate material as a reference to determine the relative potency of the other panel members results in data becoming harmonised. When the 'raw data' submitted from the first collaborative study is compared with the fifth collaborative study (figure 2), it is clear that the performance of assays in the most recent study are less variable. At the same time these assays have become more sensitive. Current commercial diagnostic kits are able to detect with confidence fewer than 10 IU ml −1 HCV, whereas a twenty years ago the figure was about ten fold higher. This progress provides greater confidence in tests that protect the blood supply.
Nevertheless, the value and demand for the International Standards of blood viruses has meant that stocks of these physical standards have become exhausted and, thus, needed to be replaced. Replacing an International Standard requires as much effort as establishing the first standard, particularly in the cases where the virus is not amenable to culture in vitro. For example, for HCV it has been necessary to source starting materials for replacement standards from HCV containing blood donations that have been detected and removed from the blood supply (table 1). In the absence of a reference method, ensuring that the calibration of the new standard closely matches that of the previous one, when the data is derived from an international collaborative study is a significant challenge. Especially, when from a manufacturer's perspective, it is now possible to detect, with confidence, variation in the calibration of standards of as little as two fold. Such replicability of assays was unthinkable at the first SoGAT meeting 25 years ago and a measure of the improvement of the quality of NAT assays where higher order calibrants have existed for the longest time.

Commutability of standards with patient samples
The initial important target for SoGAT was to improve assays that protected the blood and blood product industry. This focus on diagnostic assays that used plasma as the source of the clinical sample, simplified the preparation of reference standards. They were prepared by spiking in a high titre virus preparation into a plasma or plasma substitute matrix (table 1). Ensuring that a candidate reference standard was commutable ie performed like a clinical sample when co processed with a patient derived specimen was relatively straightforward, since there was ready access to appropriate specimens from virus positive blood packs. However, over the past 10 years, SoGAT has turned its attention to measurement standards across a broader range of clinical virology targets. In this situation, a key challenge for the development of suitable standards is the wide variety of clinical specimen that may be taken for diagnosis. A standard may be required to calibrate assays performed on blood, urine, cerebrospinal fluid, amniotic fluid or sputum, or indeed, all of these simultaneously in one diagnostic laboratory. Creating a candidate material that may harmonise measurements is relatively straightforward, as a pathogen preparation may be lyophilised in relatively simple stabilising universal buffer. Proving that this candidate performs like a patient sample once it has been diluted in pathogen free clinical matrix, represents an enormous challenge. For example, for the 1st International Standard for BK virus, it was possible to show that whilst this material was effective at harmonising data from samples presented in blood (figure 3(a)), it was less effective when dealing with urine samples In progress (2019) Ab positive diluted in negative human plasma a Calibrated against the 2nd, all others calibrated against the previous standard. ( figure 3(b)). Likewise, subsequent to the establishment of the 1st International Standard for CMV (Cytomegalovirus) DNA [5], it was suggested that this standard was not commutable when used as a calibrant for clinical samples collected as whole blood on one diagnostic platform [6]. As a result, we undertook additional studies demonstrating that whilst the CMV IS was commutable in two assays performed simultaneously on two different diagnostic platforms using plasma samples, this was not the case when whole blood samples were analysed on the same platforms (unpublished observations). Nevertheless, these data could not necessarily discriminate between non-commutability of the standard or issues regarding an assay: for example the extraction from different matrices would impact on measurements of relative potency even when there were only minor difference in efficiency of nucleic acid recovery. Nevertheless, we have gone on and performed a broader collaborative study and observed that whilst the 1st IS for CMV DNA helps to harmonise assay results in plasm and amniotic fluid (figure 4(a)), this was not the case for whole blood ( figure 4(b)). Once again, these data cannot discriminate between the reference material or the assay, but it identifies a challenge for assay and reference standard development going forward.

Sequence variability
A feature of molecular detection of an Infectious Disease is the difficulty for assays to deal with sequence variability in the pathogen and ensuring that the target sequence amplified by diagnostic assays matches both the sequence in incident clinical infections and reference materials. Some of the earliest work by SoGAT wrestled with this issue pertaining with HIV. On the one hand the support of SoGAT to develop a HIV sub-type panel, established in 2001, did little more than demonstrate some of the limitations of assays at that time (figure In the LH pair of panels these data are presented directly, whereas in the RH pair of panels the data are presented as a relative potency with the 1st International Standard of CMV DNA. The numbers within each box represents data from a different laboratory. Each laboratory received three replicate samples resulting in up to three boxes in each panel containing the same number, Presenting data as a relative potency harmonised the data of amniotic fluid samples. This was not the case for whole blood samples.

5(a)
). However, the availability of this panel was able to track the marked improvement in the performance of assays for detecting 'unusual' clinical samples by the time that the original panel was replaced in 2013 ( figure 5(b)). The molecular make-up of any reference material is critical to its performance. Technical developments such as next generation sequencing have enabled the nucleic acid makeup of reference materials to be interrogated in detail that was not possible previously. This technology has identified uncomfortable information about the recently established International Standard for the polyoma virus BK [7]. NGS based in depth sequence analysis demonstrated that this standard comprises a number of different molecular populations, which vary around partial deletions across the T antigen gene [8]. We have applied complementary techniques and confirmed these changes. Nevertheless, at the present time it should be noted that the 1st IS for BK DNA does act as an effective biological standard, helping to harmonise results from the participants of the international collaborative study, when these data are presented as relative potency [9]. In addition, the observation around the genetic makeup of a stock of polyoma virus prepared on a cell line which does not express the large T antigen, raises the question of what is the sequence makeup of clinical isolates of polyoma viruses. Further work is currently ongoing to try and establish what is the variability to be found in the sequence of the polyoma virus present in patients. Clearly the ability of a standard to harmonise data depends upon the state of maturity of assay development in the field of NAT based diagnostics for that particular pathogen and the nature of viral sequences in clinical samples. For the time being the IS for BK DNA appears to improve the comparability of measurements, but soon, hopefully, an improved reference standard may be needed. By then, we should understand better the complexity of BK viral species in clinical samples and thus a replacement standard should be better by design. The availability of these improved technologies for characterising the sequence of pathogens will help us to produce better standards.

Current achievements, future challenges and opportunities
Over the past 25 years, the work of SoGAT and the WHO Collaborating Centres, that prepare the candidate reference materials and lead the collaborative studies, have transformed molecular diagnostics. The availability of these primary standards has improved the accuracy and quality of measurement of nucleic acids of disease causing agents in clinical diagnostic samples. This transformation enabled the blood and blood product industry to identify and remove contaminated donations with confidence. As similar improvements impact for more infectious agents and likewise for equivalent assays performed for non-infectious diseases eg genomics then it will lead to health benefits more broadly across clinical practice.
The development of new techniques which include a step of gene amplification, like NGS, and the potential of other new techniques, such as digital PCR, to improve the quality of diagnostics need to be considered by SoGAT. In its present form, digital PCR is unlikely to become a routine platform for infectious disease diagnosis. Nevertheless, for quantification of specific nucleic acid sequences, then digital PCR has the potential for absolute quantification as an SI traceable unit. As such it may overcome the issue of differential amplification efficiency of materials extracted from a variety of matrices [10]. Digital PCR has been applied in infectious disease diagnoses only to a limited degree. By contrast, in the field of genomics, NIBSC has recognised the application of this technology along with next generation sequencing based techniques for the detection and quantification of specific nucleic acid sequences. This has resulted in the establishment already of an IS for detection of KRAS amino acid 12 and 13 Figure 5. Histograms comparing the ability of participants in the international collaborative studies to establish the 1st (panel (a)) and 2nd (panel (b)) International Reference Panels for HIV-1 subtype RNA to detect the HIV-1 O subtype sample, that was derived from the sample virus isolate. The number within the box represents data from a different laboratory. The letters within each box represents data from a different assay. IH refers to in-house assay and other letters refer to commercial assays available at the time of the collaborative assay. Shaded boxes represent data derived from qualitative assays and open boxes from quantitative assays. mutations, where the quantification of the materials was based primarily using data from digital PCR [11]. Since increasingly reproducible assay platforms are available for markers like HCV and HIV, then minimising drift in the quantification of replacement standards is critical and digital based assays must be investigated to establish whether they provide a solution. Going forward, closer working between SoGAT and CCQM-NAWG as well as others may facilitate this.
At the other end of the spectrum of NAT diagnostics is the development of point of care (PoC) NAT assays. These self-contained and enclosed platforms, designed to be used by a nurse or clinician, offer the potential for quicker diagnoses and therefore more timely and effective clinical intervention. However, PoC tests are not always amenable to real time external quality assurance and this may compromise the potential benefits. Pragmatic ways of bridging this problem need to be identified.
Improving accuracy in the detection and quantification of specific nucleic acid sequences in biological samples will lead to better patient management and, as a result, improved clinical outcomes following infection. SoGAT continues to look for ways to accelerate this process.