Assessment of eight nucleic acid amplification technologies for potential use to detect infectious agents in low-resource settings

Nucleic acid amplification technologies (NAATs) are high-performance tools for rapidly and accurately detecting infectious agents. They are widely used in high-income countries to diagnose disease and improve patient care. The complexities associated with test methods, reagents, equipment, quality control and assurance require dedicated laboratories with trained staff, which can exclude their use in low-resource and decentralized healthcare settings. For certain diseases, fully integrated NAAT devices and assays are available for use in environmentally-controlled clinics or emergency rooms where relatively untrained staff can perform testing. However, decentralized settings in many low- and middle-income countries with large burdens of infectious disease are challenged by extreme environments, poor infrastructure, few trained staff and limited financial resources. Therefore, there is an urgent need for low-cost, integrated NAAT tools specifically designed for use in low-resource settings (LRS). Two essential components of integrated NAAT tools are: 1) efficient nucleic acid extraction technologies for diverse and complex sample types; and 2) robust and sensitive nucleic acid amplification and detection technologies. In prior work we reported the performance and workflow capacity for the nucleic acid extraction component. In the current study we evaluated performance of eight novel nucleic acid amplification and detection technologies from seven developers using blinded panels of RNA and/or DNA from three pathogens to assess both diagnostic accuracy and suitability as an essential component for low-cost NAAT in LRS. In this exercise, we noted significant differences in performance among these technologies and identified those most promising for potential further development.


Introduction
The accurate and rapid diagnosis of infectious disease can lead to improved patient care, appropriate use of antimicrobial drugs, and ultimately greater control of communicable diseases within populations [1]. The advent of molecular tests to accurately identify pathogen PLOS [18]. By comparison, the EUA clearances to rapidly develop diagnostic assays for Ebola virus include three fully integrated assays hosted on the BioFire (bioMérieux, France, GeneXpert (Cepheid, USA) and Idylla platforms (Biocartis and Janssen Diagnostics, both Belgium); this is likely due to the very significant biohazard risks associated with handling Ebola infected specimens and the lack of appropriate laboratory infrastructure in Ebola affected regions [19].
Cost is another critical issue-the global donor supported cost of US$9.98 per GeneXpert MTB/RIF cartridge is seen as unaffordable for many LMIC TB programs [9]. The MTB/RIF cartridge for use with the Omni is projected to increase to $11.40 due to adding a near field communication chip to the test cartridge [20]. The HIV-1 assays from Cepheid and Abbott are estimated to cost $17.95 and $25.00, respectively [20,21]. By contrast, a consensus meeting of TB experts suggested that the cost for a replacement test for sputum smear microscopy should be in the range of $4 to $6 [22]. Therefore, while it is exciting that new diagnostic tools are becoming available to improve clinical care in austere environments, the cost per test is a key barrier to uptake and sustained use [22].
The goal of this project was to investigate inexpensive nucleic acid amplification and detection technologies that can potentially be integrated into "sample to answer" NAAT platforms that are suitable for low-resource settings (LRS). In a companion paper we compared the performance and workflow characteristics of six nucleic acid extraction technologies for suitability in LRS [23]. These technologies are to be designed for primary use by minimally skilled operators, and so will require limited training and as few manual steps as possible. In the present study, we assessed the sensitivity, specificity, time to result, and complexity of eight novel nucleic acid amplification and detection technologies for potential use in LRS. Based on these criteria, we were able to identify technologies that have the potential to meet the required specifications for use in LRS [19].

Identification of technology developers
An extensive search of peer-reviewed literature, published technical reports and white papers, commercial literature, company websites, conference proceedings, and letters of interest to a donor-the Bill & Melinda Gates Foundation (BMGF) was conducted to identify multiple technology developers with innovative nucleic acid (NA) amplification and detection methods that have the potential for use in challenging environments by minimally skilled users, or capable of being engineered to require minimum hands on time by the user Seven developers agreed to participate in this study based upon the current maturity of their technology and the potential to meet the cost goal of approximately $3. The institutions were comprised of a variety of companies or university groups; Akonni Biosystems, Inc. (Frederick, MD, USA), Alveo Technologies (Alameda, CA, USA), Integrated Nano-technologies (Henrietta, NY, USA), Friz Biochem (Neuried, Germany), Tangen Biosciences, (Branford, CT, USA), Two Pore Guys (Santa Cruz, CA, USA) and University of Washington (Seattle, WA, USA). All of these groups are small in size and most are relatively new companies. Although these institutions had no role in designing the study, the authors agreed that the identities of each would be coded for this report if the groups wished to be anonymous. While there is a lot of studies reflecting the performance of different commercial assays or the comparison of laboratory designed assays versus a gold standard test, this study is relatively unique in that not only was the performance of multiple amplification assays assessed but also the detection platforms unique to these companies. The institutions have developed NA amplification and detection methods and devices incorporating a variety of innovative technologies, including fluorescence, immunocapture, and electrochemical methods [24]. The operating platforms ranged from fully integrated devices to prototype systems that still require some development to remove multiple manual steps prior to and/or during processing. Each of the developers received blinded test panels containing microbial DNA and/or RNA derived from three pathogens to test in their facilities. Developer F offered two different amplification methods using either PCR or loop mediated amplification (LAMP).

Materials for the construction of nucleic acid panels
We chose influenza A virus (a RNA-based pathogen), Mycobacterium tuberculosis (MTB), and Salmonella enterica serovar Typhimurium (both DNA-based pathogens) to prepare our test panels. Cultured influenza A virus (H3N1) supernatant was a gift from the Washington State Public Health Laboratories (Shoreline, WA, USA). DNA extracted from M. tuberculosis H37Rv strain Johannesburg was procured from the laboratory of Dr. B. Kana (University of Witwatersrand, Johannesburg, South Africa). S. Typhimurium LT2, a gift from Dr. S. Meschke (University of Washington, Seattle, WA, USA), was cultured in tryptic soy broth (Sigma-Aldrich, St. Louis, MO, USA) with overnight incubation at 37˚C with shaking. The cells were pelleted by centrifugation, washed once with phosphate-buffered saline (PBS), resuspended in 5 mL PBS, and stored at 4˚C until DNA extraction. With the exception of MTB, whose DNA was received as already purified, the extraction of the cultured bacterial and viral nucleic acids was performed using kits from Qiagen (Hilden, Germany) following the kit instructions. Influenza A samples were processed using the QIAamp Viral RNA Mini Kit. The input sample was 140 μL volume of culture supernatant, with RNA eluted in a final volume of 60 μL. DNA extraction from S. Typhimurium stocks used the Qiagen DNeasy kit. A 500 μL volume of washed cells was processed per column and eluted in a final volume of 200 μL. All nucleic acid extracts were stored at -20˚C until use.

Real-time RT PCR quality control of extracted nucleic acids
Nucleic acid extracts were qualified prior to construction of blinded panels via real-time reverse transcription polymerase chain reaction (RT PCR) for influenza A, or PCR for the DNA-based pathogens. Oligonucleotide primers and probes were purchased from Integrated DNA Technologies, Inc. (Coralville, IA, USA). The core amplification reagent used in all realtime assays was qScript XLT 1-Step RT-qPCR ToughMix (Quanta Bio, Gaithersburg, MD, USA), regardless whether a RT step was required. The real time RT PCR assays used in the PATH laboratory for each pathogen NA were as follows: for influenza A, the universal influenza A RT PCR assay [25]; for MTB, an assay that targets the insertion sequence IS6110 [26]; and for S. Typhimurium, an assay targeting the hns gene was used [27]. All reactions were prepared as 20 μL volumes to which 5 μL of nucleic acid extracts were added to make a final reaction volume of 25 μL. Amplification and detection in real time was carried out using an Mx3005P qPCR system (Agilent Technologies, Santa Clara, CA, USA) using the FAM detector channel during the 60˚C annealing stage. The Centers for Disease Control and Prevention (CDC) PCR protocols were different for MTB and Salmonella [28,29], but the core reagents were the same as described earlier. Each reaction used 5 μL of sample in a final reaction volume of 25 μL. The amplification protocol for reverse transcription used an incubation step of 45˚C for 10 minutes followed by a step of 95˚C for 10 min. PCR amplification then entailed 45 cycles of 30 seconds at 90˚C and 1 minute at 60˚C. Fluorescence was measured in real time using the FAM channel during the 60˚C stage. The CDC laboratory used the 7500 Fast Dx system (Applied Biosystems, Foster City, CA, USA).

Construction of NA test panels
Three test panels representing MTB (panel MTB), influenza A (panel INF) and S. Typhimurium (panel SAL) were made. Each panel consisted of six members, five containing different amounts of target NAs and one unspiked for use as a specificity control. The diluent used was 10 mM Trizma base (pH 8.0; Sigma-Aldrich, St. Louis, MO, USA) spiked with 5 μg/mL of human DNA (Promega Corp, Madison, WI, USA) to mimic expected human DNA co-extraction from clinical samples, and to serve as an internal amplification control if elected by the developers.
Because the intent of the study was to assess the performance of NA amplification and detection methods, sensitivity of the candidate technologies was the primary performance metric for comparison. Therefore, the panel designs were biased toward samples that had limited target NA for amplification to identify assays that were capable of detecting very low levels of target. Initially, we diluted the NA extracts until each stock solution had a C t value of 19 (± 0.5). Assuming the PCR assays had a theoretical efficiency of 100%, ten-fold dilutions of each NA extract should result in an increase in the C t value by 3.3 cycles per log dilution. Ten-fold dilutions of the stocks create a series of samples where the C t values derived from five such tenfold dilutions are calculated to be approximately 22.5, 25.8, 29.1, 32.4, and 35.7.
To prevent cross-contamination, the negative control stocks of 10 mM Tris with 5 μg/mL human DNA were aliquoted in multiple 10 μL aliquots before the target NAs were prepared. The five consecutive ten-fold dilutions were then prepared from the DNA or RNA stock solutions each having an initial C t value of~19.0, wherein 100 μL of material was added to 900 μL of Tris buffer (containing human DNA as background) at each dilution step. Mixing was critical in this process, especially for the DNA-based targets at the lowest dilutions, to ensure even distribution of the genomic DNAs. After a NA solution was transferred to its diluent, each sample was then vortexed six times in 10-second intervals. The tube was then briefly centrifuged prior to making the next dilution. Once each panel's dilution series was completed, multiple 10 μL aliquots were immediately stored at -20˚C. Subsets of these NA samples were then tested for quality control by real-time RT PCR to confirm that each dilution contained an appropriate amount of challenge material. Samples from the three panels were shipped on dry ice overnight to the CDC to independently replicate the initial test data and confirm the composition of the panels.

Shipping and use of the panels
To ensure developers had the opportunity to optimize the performance of their assays before the blinded panels were shipped, each group received aliquots of the purified NA extracts for each of the three pathogens used in this study. After this phase, the participating developers then received six replicates for each of the five titered levels across the three pathogen panels (total 90 panel members). Each panel member was blinded via four-digit numbering and the sets shipped on dry ice via an overnight courier with confirmation of integrity of cold chain upon receipt. To ensure that each developer processed the same relative amount of sample, they were asked to use the equivalent of 2.5 μL from each blinded sample for their respective NA amplification/detection technologies. For example, if an assay requires a 10 μL sample input, then 10 μL test sample would first be diluted with 30 μL nuclease-free water to a 40 μL final volume, mixed, and then 10 μL of the diluted sample would be used in the assay (equivalent to 2.5 μL of sample and 7.5 μL of nuclease free water). This requirement ensured that developers used the same amount of input material regardless of the volume required by their assays (which we assumed would be variable). This approach allowed for a direct comparison of sensitivity between the methods as dictated by the volumes required for each developer. The data were sent to PATH upon completion of testing. The technologies were ranked based on assay performance; however, assay run time and number of user steps were also evaluated for each technology. When all testing and analysis was complete, the data were unblinded to each developer.

Panel quality confirmation in-house and by a reference laboratory
After the three test panels were constructed and prepared, PATH screened all samples for each dilution in triplicate RT PCR reactions (Table 1). Each sample tested consistently gave the anticipated result, although the C t values from the replicate tests from each tenfold dilution varied slightly (typically +/-0.3 C t ) from the predicted values, which was likely due to potential uneven distribution of NAs during mixing and less than 100% efficiency of the PCR assays [30]. Because the NAs at all dilutions were consistently detected, the panels were deemed as suitable to assess and verify the performance of the developer NAATs. PATH shipped three blinded aliquots of each sample from all panels to the CDC laboratory for independent confirmation of the panels test results via the same or similar real-time PCR methods. All samples from the MTB, INF, and SAL panels were concordant for negative and positive samples ( Table 1). The mean C t values generated from either laboratory showed minimal variability for each panel (generally within 1-2 C t values). The CDC used different assays and a different real-time PCR platform to detect the MTB and Salmonella DNAs, which are likely reasons for some variation in C t values. Nonetheless, each sample tested at the CDC provided the expected semi-quantitative result when compared to the original PATH laboratory data. With this confirmatory analysis, the blinded panels were subsequently shared with the participating developers for their independent analyses.

Test data from the participating developers
Each of the seven developers completed panel testing within the allotted time and sent datasets to PATH. Prior to testing, developers were asked to use the normalized equivalent of 2.5 μL from each blinded sample, regardless of the volume specified for amplification on their platform. This directive ensured that each developer applied the same amount of RNA or DNA, allowing for a direct comparison across technologies. As the goal of the evaluation was to determine the performance of each technology's ability to detect the target at different concentrations, each dataset was evaluated based on correctly identifying the presence or absence of the various targets. Upon receipt of all datasets, the panel compositions were unblinded to each developer (Table 1). Developer E did not have a functional assay for influenza A and did not provide data for this panel. The sensitivities and specificities for each developer were graphically plotted for each of the three test agents (Fig 1). While most developers achieved high specificity, there was considerable variation observed in sensitivity. Annotated and complete PCR data sets and developer test results are publicly available (https://dataverse.harvard. edu/dataverse/SSB).
With the INF panel, each of the developers correctly identified the negative samples, as well as the higher-spiked samples (samples 1-3), except for developer E. Developer G incorrectly identified 3/5 and 2/5 negative samples as positive for MTB and AL, respectively; these were the only false positive results obtained by any developer for any panel. Developers C and F (when F used an RT PCR-based assay-F1) correctly identified all of the influenza A-positive samples. Developer A failed to detect one replicate from sample 5, the lowest RNA concentration, and developer F also failed to detect one replicate from sample 5 when using RT LAMP. Developers B and D failed to detect RNA within all replicates containing the lowest concentration of this nucleic acid (sample 5). Developers D also failed to detect influenza A RNA in all replicates of samples 3 and 4, and developer G only detected influenza A RNA in one replicate of the highest concentration sample.
All developers correctly identified the MTB-negative samples, except for developer G. Developer C correctly identified MTB DNA in all spiked replicates, and developer F missed only one of five replicates at the lower range (sample 11) when using PCR amplification ( Table 1). The other developers demonstrated limited detection of replicates for samples 10 and 11, those with the lowest levels of DNA; D and G did not detect any MTB DNA in sample 10. The lowest levels of performance were observed with Developers D and E which scored the lowest sensitivities for MTB DNA and developer G scoring lowest for specificity for MTB. For the SAL panel, Developers C and F (when using PCR amplification, F1) correctly scored all positive sample replicates, including the lowest spiked member, sample 17. Interestingly, the low level of SAL DNA in sample 17 was not detected by the other developers, and the LAMP assay from developer F (F2) produced only one positive replicate (1/5) for this sample. The moderate amounts of SAL DNA in samples 15 and 16 demonstrated improved rates of positivity for most developers. The specificity of tests for the SAL panel was typically excellent (5/5), with all but one developer correctly identifying the negative panels. Developer G had two false positive results.
When the test scoring data from the three panels were pooled, developer C had the best performance based on correctly identifying every positive or negative replicate member in each of the three panels. The performance of developer F was similar, with only a single false-negative result at the lowest concentration of influenza A when using RT PCR-based detection (F1, Table 1). The pooled performance scores for other developers were more variable, with developers A and F2 (using the LAMP method) tying for third-best performance. Developer G had the poorest performance (excluding developer E with no submitted results).
Despite efforts to optimize assays before testing, developer E encountered persistent inhibition with the human genomic DNA in the panel samples for both the influenza A and The evaluation of eight nucleic amplification methods salmonella assays. To limit the negative effects of the human DNA, developer E elected to use only 0.5 μL of each sample as opposed to the recommended 2.5 μL, effectively using only 20% of the test material as compared with the other developers. This may explain the limited sensitivity of developer E's assays, especially at the lower concentration ranges.
In addition to assessing the analytical diagnostic accuracy, we investigated operational aspects of testing for each technology, whereby the number of steps required for each assay's workflow was plotted against the overall turnaround time (TAT) to obtain a test result (Fig 2). The number of key steps listed by the developers was varied, with two to five operator steps required for assay preparation, amplification, and scoring the test result. However, there were more significant differences in TATs. Four assays took less than 1 hour, three required less than 2 hours, and the assay by developer C took almost 4.5 hours regardless of the nucleic acid tested (Fig 2).
The most rapid assays were the LAMP assay from developer F (F2) and the PCR assay from developer B; both methods had the same TAT irrespective of whether there was an RT step with subsequent DNA amplification (e.g., INF RNA) or not (e.g., MTB DNA). Isothermal methods can simultaneously employ RT and DNA amplification in the same incubation step because the RT and DNA polymerase use the same incubation temperature, reducing the time to amplify complementary DNA (cDNA) derived from RNA as opposed to traditional RT PCR methods [31,32]. RT PCR typically requires an initial reverse transcription step to generate cDNA before thermal cycling because the RT species used in molecular assays are denatured at elevated temperatures. Therefore, PCR cannot proceed until cDNAs are synthesized from the target RNA [33]. Despite this, developer B achieved detection of RNA in less than 1 hour using RT PCR. Assays from the next three developers (A, D, and G) required 1-2 hours. Developer C had the longest TAT of over 5 hours due to their methodology for interrogation of amplicons. They use a probe-based array that requires a hybridization stage from which to interrogate PCR amplicons. All developers used RT PCR with the exception of F2 and developer E, both of which used isothermal amplification methods.
The developers A, C, and F scored best overall in this exercise and are Friz Biochem, Akonni Biosystems and Two Pore Guys, respectively. In summary, the assays from Akonni Biosystems (C) were the most accurate but required a significantly longer TAT in comparison to the other developers. This was due to the Akonni test platform requiring a 3-hour DNA microarray hybridization step after amplification by asymmetric PCR [34,35]. The assays from Two Pore Guys (F) were next best in terms of sensitivity when using either PCR (F1) or LAMP (F2) and provided a very short TAT (20 and 35 minutes for LAMP and PCR, respectively). Friz (A) scored the same as F2 for test performance and had a moderate TAT (<1.5 hrs.) in terms of the assays tested.

Discussion
The overarching intent of this work and its companion study was to identify suitable component technologies for the development of affordable and effective fully integrated NAAT devices to diagnose globally-relevant infectious diseases in LRS [36]. In this report, the key components focus on the timely and effective amplification and detection of pathogen nucleic acids. We worked with a series of innovative developers to explore their proprietary platforms and NA amplification assays for influenza A, MTB, and salmonella. The assays and platforms were challenged with blinded replicate series of dilution panels to assess their performance with an emphasis on sensitivity, a critical parameter for the success of any NAAT-based technology. Although the developers' identities were coded to encourage participation in this head-to-head challenge, further details can be requested from each. The assays and devices screened in this study were in varying phases of the development pipeline, and any limited performance observed in this study may be related simply to early prototypes still in the development, as compared to mature devices in production.
The use of low concentrations of nucleic acids for testing was a key feature as it enabled insight into the limits of detection for each technology, and the study clearly identified the most sensitive technologies. An unexpected outcome was observed when adding human DNA as an optional internal control to qualify reagents and equipment performance. Although no developer chose to present data for this, the addition of human DNA highlighted one technology whose performance was adversely affected. A further point is that the addition of a small uniform amount of DNA does not necessarily mimic the actual dynamics of total nucleic acid recovered from clinical specimens due to the specimen type, heterogeneity and the method of nucleic acid extraction used. As it is expected that some human DNA will be present in most samples, ideally the technologies should be able to accommodate some degree of non-specific DNA/RNA from the host and in some instances commensal microflora (e.g. stool).
Interestingly, while there are many publications that advocate use of isothermal amplification methods in LRS based on rapid TAT, reduced power requirements, and simplicity of The evaluation of eight nucleic amplification methods device design, only two isothermal methods were offered by developers in the current study [24]. The LAMP assays (F2) from Two Pore Guys performed reasonably well but the isothermal method from developer E demonstrated relatively poor performance. We suspect that device developers prefer to use PCR-based methods as a result of familiarity with the technique and experience with assay design in addition to a variety of technical reasons. The ubiquity of PCR means that many vendors offer high-quality core reagents with exceptional performance in terms of sensitivity, specificity (e.g. proof reading and hot start), fidelity, processivity, and multiplexing capability. These reagents are widely available with competitive prices given that intellectual property for PCR is now largely off-patent. By contrast, most isothermal technologies are still under patent, and the holders of these patents may have their own devices in development, offer only reagents, or lack an integrated platform with which to perform test reactions.
The detection modalities used by these tools included real-time and endpoint detection of amplification via fluorescence, optical imaging system of hapten labeling, immunochromatography, and electrochemistry. Of these methods, the fluorescence detection modalities gave the most rapid TAT, as seen by developers B and Two Pore Guys (both F1 and F2). The other methods were slower, giving results in 1 to 2 hours. The requirement to hybridize test reactions to an array for detection slowed the TAT for Akonni, but this also gave maximal sensitivity and specificity which were the primary performance attributes sought in this exercise. Akonni have worked to overcome the inherent complexity of microarray-based diagnostics for routine use by combining asymmetric PCR amplification, microarray hybridization and waste fluid retention to within a single microfluidic chamber. The detection of array bound Cy3 labelled PCR products is enabled via a low-cost, field-portable microarray imager that Akonni is developing [35].
Interestingly the detection modalities from Two Pore Guys and Friz use electrical-based detection of target amplicons. Such systems are of great interest in austere environments as the technical design is much simpler than optical detection systems [37]. There may be further advantages to lower cost of production with solid state devices. As their name suggests, the Two Pore Guys technology uses a silicon chip that separates two fluidic volumes with amplicons pulled through a 20 nm nanopore by voltage. The amplicons are first labelled with a sequence specific molecular payload (probe) that cause a predefined shift in the trans-pore ionic charge as they are pulled through the pore [38]. Different sized probes enables multiplexed detection and the developers claim single molecule sensitivity. The Friz approach is different, measuring conduction via an integrated gold plated micro-electrode-array (MEA). Pathogen specific labelled probes are immobilized on the MEA and create a high electrochemical signal prior to amplification. The increasing presence of PCR amplicons promotes the displacement of the labelled probe from the test site, thus reducing electrochemical signal and indicating the amplification of specific target DNAs. If a target is not amplified then the high signal remains uniform as the probe remains bound to the MEA. While Two Pore Guys and Friz use PCR, any amplification method that creates a complementary DNA for competitive binding to the detector probe modalities may be used, (e.g. the LAMP format with F2).
We identified two technologies from Akonni and Two Pore Guys with nearly identical performance across all panels. These and other technologies such as Friz are also under review for potential further development, especially considering other key factors including manufacturing scale, access to core intellectual property, and ability to achieve an estimated COGs of approximately US$3 (targeting US$5 per test when combined with a suitable NA extraction method). Cost is a critical issue. Although several integrated NAATs with the potential for use in LRS currently exist, the cost per test is generally judged unaffordable for infectious disease diagnosis in most low-and middle-income countries. Together, this study and the companion assessment of nucleic acid extraction technologies have identified several promising candidate technologies for integration into low-cost integrated sample-to-answer platforms for infectious disease diagnosis. Further development will aim to integrate technologies and reach manufacturing scale at low cost with the potential to offer rapid and accurate diagnosis of the highest priority pathogens in LMICs.