Assessment of the clinical and analytical performance of three Seegene Allplex SARS-CoV-2 assays within the VALCOR framework

ABSTRACT The coronavirus disease 2019 (COVID-19) pandemic demonstrated the need for accurate diagnostic testing for the early detection of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Although the pandemic has ended, accurate assays are still needed to monitor viral spread at national levels and beyond through population and wastewater surveillance. To enhance early detection, SARS-CoV-2 assays should have high diagnostic accuracy and should be validated to assure accurate results. Three distinct SARS-CoV-2 assays were evaluated with clinical samples using the VALCOR (VALidation of SARS-CORona Virus-2 assays) framework, with the TaqPath COVID-19 assay (ThermoFisher Scientific, USA) as a comparator. We evaluated clinical sensitivity, specificity, limit of detection (LOD), and overall concordance between comparator and three index Allplex SARS-CoV-2 assays (Seegene, South Korea): Allplex-SC2, Allplex-SC2Fast (Fast PCR), and Allplex-SC2FabR (SARS-CoV-2/FluA/FluB/respiratory syncytial virus). Analytical performance and LOD of index assays were assessed using a dilution series of three synthetic SARS-CoV-2 sequence reference materials (RMs). Ninety SARS-CoV-2 positives and 90 SARS-CoV-2 negatives were tested. All Allplex assays had 100.0% sensitivity (95%CI = 95.9%–100.0%). Allplex-SC2 and Allplex-SC2Fast assays had 97.8% specificity (95%CI = 92.3%–99.7%) and 98.9% overall concordance [κ = 0.978 (95%CI = 0.947–1.000)]. Allplex-SC2FabR assay showed 100.0% specificity (95%CI = 95.9%–100.0%) and 100.0% overall concordance [κ = 1.000 (95%CI = 1.000–1.000)]. LOD assessment of index assays revealed detection down to 2.61 × 102 copies/mL in clinical samples, while the analytical LOD was 9.00 × 102 copies/mL. In conclusion, the evaluation of the three Seegene Allplex SARS-CoV-2 assays showed high sensitivity and specificity and an overall good assay concordance with the comparator. The assays showed low analytical LOD using RM and even a slightly lower LOD in clinical samples. Non-overlapping target gene sequences between SARS-CoV-2 assays and RMs emphasize the need for aligning targeted sequences of diagnostic assays and RMs. IMPORTANCE The coronavirus disease 2019 pandemic has a significant impact on global public health, economies, and societies. As shown through the first phases of the pandemic, accurate and timely diagnosis is crucial for disease control, prevention, and monitoring. Though the pandemic phase of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has concluded, diagnostic assays remain in demand to monitor SARS-CoV-2 at the individual patient level, regionally, and nationally, as well as to remain an infectious disease preparedness instrument to monitor any new SARS-CoV-2 dissemination across borders using population and wastewater surveillance. The anticipation by WHO and central health care policy entities such as the Center for Disease Control, EMA, and multiple national health authorities is that SARS-CoV-2 will reside as an endemic respiratory disease for years to come. The key strategic consideration is hence shifting from combating a pandemic situation with a high number of patients to instead allowing precise diagnostics of suspected patients with the intention of correct management in a low-prevalence setting.


MATERIALS AND METHODS
The full protocol of VALCOR and the composition of the VALCOR samples' panel were described previously (11).In brief, three types of samples were investigated.(i) Clinical accuracy of index assay was assessed with 180 undiluted clinical samples (90 SARS-CoV-2 positives and 90 SARS-CoV-2 negatives) collected in the routine testing from the National Reference Laboratory for Respiratory Pathogens, Department of Laboratory Medicine (UZ Leuven, Belgium).The samples were collected while the Wuhan-Hu-1 strain was dominant and before the emergence of the Alpha SARS-CoV-2 variant.(ii) Forty dilutions of clinical SARS-CoV-2-positive samples (with a dilution matrix of 1:2, 1:10, 1:20, and 1:50) were used to assess the clinical LOD of the index assays and the comparator by constructing a standard curve with an RM.(iii) The analytical LOD of the index assays and the comparator was assessed by constructing a standard curve with synthetic SARS-CoV-2 RMs that were manufactured by RM manufacturers in Europe [Joint Research Centre (JRC), European Commission, Geel, Belgium] and in the USA [National Institute of Standards and Technology (NIST), Gaithersburg, USA).Additionally, sequences of target genes in the index assays, the comparator assay, and the RMs were assessed.

Testing of samples with the index assays
Testing of the VALCOR panel with the three Seegene Allplex SARS-CoV-2 assays was carried out at the Laboratory of Molecular Diagnostics (GLMD-ZOL, Genk, Belgium).Nucleic acid extraction, based on magnetic beads, was performed on the Seegene STARlet extraction platform using 300 µL of input material (i.e., sample), resulting in a 100 µL eluate.Reverse transcriptase PCR (RT-PCR) and PCR were performed according to the manufacturer's specifications for the Allplex-SC2, Allplex-SC2Fast, and Allplex-SC2FabR assays, respectively.Seegene Viewer analysis software was used for auto-inter pretation of the results from the three Allplex assays, providing a binary outcome (i.e., presence or absence of SARS-CoV-2) along with Ct values.Viral load (VL), expressed in copies/mL, was estimated using a standard curve.Retesting of samples was carried out in case a sample tested positive for SARS-CoV-2 while its internal control was negative.
All three index assays are multiplex RT-PCR assays detecting the common SARS-CoV-2 target E, N, RdRP, and S genes (Table 1) (13): the Allplex-SC2 assay targets sequences within the E, RdRP/S, and N genes with the RdRP and S genes detected in the same fluorophore channel and using an exogenous internal control; the Allplex-SC2Fast assay targets the E, N, and RdRP genes and has an endogenous internal control; and the Allplex-SC2FabR assay detects N, RdRP, and S genes, has one exogenous and one endogenous internal control, and detects three additional pathogens, besides SARS-CoV-2.In the Allplex assays, a sample is considered positive if it detects at least one of the target genes of SARS-CoV-2 and the internal control is positive.Conversely, a sample is considered negative if all the target genes of SARS-CoV-2 are not detected, while the internal control is positive.According to the manufacturer's specifications, all three Allplex assays can detect SARS-CoV-2 particles from nasopharyngeal swabs, saliva swabs, and combo swabs (nasal + oral swabs).

Testing of samples with the initial panel testing and a comparator assay
VALCOR samples were tested with the initial panel at UZ Leuven using the TaqPath COVID-19 assay as a comparator (11,12).The initial panel testing was performed, between 1 April 2020 and 15 January 2021.Nucleic acids were extracted using the MagMax Viral Pathogen II extraction kit (ThermoFisher Scientific) on the KingFisher High Throughput platform.The target genes of Taqpath COVID-19 are N gene, ORF1ab gene, and S gene.The results were reported as log copies/mL based on the quantification cycle (Cq, also called Ct) value of the N gene.Detection of two or more target genes was considered a positive result.Samples were reported negative when all viral gene targets were Cq > 37 and the MS2 internal control was Cq < 27.5.VL was estimated using a standard curve and semi-quantitatively reported (14).In case of discordance between the initial panel testing and the TaqPath COVID-19 assay, a third assay, either Abbott Alinity m system (IL, USA), Aptima SARS-CoV-2 assay (Hologic Panther system, MA, USA) (12), or Xpert Xpress SARS-CoV-2 assay (Cepheid GeneXpert system, CA, USA), was deployed as an adjudicator.

Statistical analyses
Concordance matrices were made with the paired data of the initial panel testing versus comparator, and comparator versus any index assay.The sensitivity of an index assay was defined as the proportion of true positives (TP) by the comparator and the index assay.
The specificity of an index assay was defined as the proportion of true negatives (TN) by the comparator and the index assay to the number of SARS-CoV-2 negatives detected by the comparator.The overall concordance between the comparator and an index assay was defined as the proportion of the sum of TP and TN to the total number of SARS-CoV-2 positives and negatives in the study (n = 180).Exact binomial 95% confidence interval (CI) was calculated.Cohen's kappa values (κ) were calculated ( 15 16).Nonparametric Wilcoxon Signed Rank test was performed to determine if there was a statistically significant difference in the median-estimated VLs between the comparator and an index assay.Interquartile range 1 (Q1) and 3 (Q3) of estimated VLs were described.Statistical significance was set at 0.05.The Bland-Altman plots were visualized to describe a quantitative agreement between two VL measurements determined with two assays.An average VL difference substantially higher than zero indicates that the VL TaqPath COVID-19 tends to be higher than the VL Seegene Allplex assay and vice versa.Statistical analyses were performed with STATA version 16 (College Station, TX, USA).Similarly, 89/91 undiluted negative panel samples were negative with the Allplex-SC2 and Allplex-SC2Fast assays, leading to a specificity of 97.8% (95% CI = 92.3%-99.7%).

Assay performance
In this study, we validated the assay performance of three Seegene Allplex SARS-CoV-2 assays using clinical SARS-CoV-2 samples within the VALCOR framework.The Allplex-SC2 and Allplex-SC2Fast assays had a sensitivity of 100.0% and a specificity of 97.8%, while the Allplex-SC2FabR assay had 100% sensitivity and specificity to the defined compara tor, the TaqPath COVID-19 assay.The overall concordance for the three Allplex assays and the comparator was 98.9% for the Allplex-SC2 and Allplex-SC2Fast assays, and 100.0%for the Allplex-SC2FabR assay, having an excellent Cohen's kappa of 0.978 and 1.000, respectively.A comparison with the performance of other validated SARS-CoV-2 assays, as reported in the literature, indicates that the Seegene Allplex SARS-CoV-2 assays perform at a level similar to the Aptima SARS-CoV-2 assay.The Aptima assay had also been validated using the VALCOR panel (12).In terms of sensitivity, the Seegene Allplex assays outperform the Abbott RealTime SARS-CoV-2 assay, achieving a sensitivity of 93% (17).It is worth noting that the Abbott RealTime assay, although achieving a lower sensitivity, was validated using a different set of clinical samples.The Abbott RealTime SARS-CoV-2 assay showed 100% specificity, which is similar to the Seegene Allplex-SC2FabR assay, while the Seegene Allplex-SC2 and Allplex-SC2Fast assays have a slightly lower specificity followed by the Aptima SARS-CoV-2 assay (96.7%) (12).The three Allplex assays were also compared with the TaqPath COVID-19 assay at the level of target genes.The N gene was common in the three Allplex assays and the TaqPath COVID-19 assay.The median estimated VL of the N and S genes in each Allplex assay is significantly different from the comparator, having a lower median estimated VL than the comparator.However, while the Allplex-SC2Fast and the Allplex-SC2FabR assay have a higher median estimated VL than the comparator, the direction of their z scores was opposite.Time-to-result is crucial for any respiratory patient management, and the runtime of Allplex-SC2 and Allplex-SC2FabR assays was approximately 2 hours, while 1 hour for the Allplex-SC2Fast assay.

Limit of detection
The clinical LOD was evaluated using clinical samples by the three Allplex SARS-CoV-2 assays, revealing a clinical LOD lower than the comparator assay.Similar findings were observed during the analytical LOD assessment, which utilized three SARS-CoV-2 RMs, and the three Allplex assays exhibited a slightly higher sensitivity than the TaqPath COVID-19 assay.Notably, the clinical LODs with the three Allplex assays were even marginally lower than the analytical LODs.

Target gene sequences in RMs
Target genes of the three Seegene Allplex SARS-CoV-2 assays were detected in the SARS-CoV-2 RMs, fulfilling the minimum requirement of viral genes as included in the RMs.Nonetheless, the sequence of target genes was not consistent between these Allplex assays and the RMs since not all target genes could be detected at once in the RMs (Table 4).The TaqPath COVID-19 assay could detect fewer target genes of the three synthetic RNA fragments than expected (Table 4-"after sequence assessment" versus "before sequence assessment").Possible explanations for this could include the (partial) absence of the targeted gene sequences in the RMs or an improper sequence alignment of the primers and probes used in the assays with the RM.As a secondary finding, RGTM 10169-2 contained fragments of the RdRP gene, which were not stated in the reference sequence.However, the detection of the RdRP and S genes was in the same fluorophore channel of the Allplex-SC2 assay, and hence specific detection of the S gene cannot be determined.

Limitations
Throughout the pandemic, clinical samples were tested using available but not necessarily the same assays on different platforms, and assays were used in parallel to address the limited diagnostic capacity experienced across the globe.Also, no SARS-CoV-2 standard comparator was available at that time of study.The choice of the SARS-CoV-2 assay of UZ Leuven as a comparator also represents a limitation because the VALCOR panel relies on residual clinical samples, which is a challenge to scale up.The initial sample volume was relatively low, approximately ranging from 2 to 3 mL, again limiting the number of panels available.Given that the VALCOR panel of this study was collated during the first peak of the COVID-19 epidemic in Belgium, at a time when the Wuhan-Hu-1 strain was dominant and prior to the emergence of the Alpha SARS-CoV-2 variant, it is possible that there may be a bias in terms of SARS-CoV-2 variants.Furthermore, a limitation of the VALCOR panel is the absence of other viruses within its composition.This omission missed the opportunity for the assessment of cross-reactivity, for instance, FluA/FluB/RSV, in terms of specificity.Incorporation of a variety of SARS-CoV-2 variants into future VALCOR panels will enhance the generalization of SARS-CoV-2 assays' validation, similar to the built-in heterogeneity in the four VALGENT [VALidation of Human Papilloma Virus (HPV) GENotyping Test] panels (18).VALCOR is a framework for validating SARS-CoV-2 virus assays and is inspired by the principles of VALGENT, which is a globally recognized forum for HPV assay comparison and validation (19).Also, due to the lack of primer sequences from commercially available SARS-CoV-2 assays, sequence alignment analysis (i.e., TaqPath COVID-19 assay) could not be performed to inspect the compatibility between the assays and the circulating variants.When looking at the results of the in silico alignment prediction with the EURM-019 RM and the recent SARS-CoV-2 variants against the reference strain Wuhan-Hu-1, one amplicon [i.e., WH-NIC N amplicon (Fig. 1)] is not predicted to amplify the concerned sequence in the recent SARS-CoV-2 variants.Out of the nine amplicons in the EURM-019, four amplicons contain at least one position in the sequence where the nucleotide is not aligned with the Wuhan-Hu-1 strain.However, the actual alignment and amplification depend on the forward primer, the reverse primer, the probe, and their physical properties as a whole (such as sequence and GC content).The correlation to clinical outcome is a general limitation when characterizing the analytical LOD of any diagnostic assays, and especially so for SARS-CoV-2 given the number of co-morbidities influencing whether a person becomes severely sick or not by SARS-CoV-2.

Perspectives of SARS-CoV-2 assay validations
In the pandemic phase, all available molecular technologies were brought into play to address the immediate demands of healthcare services worldwide.In this context, real-time reverse transcriptase polymerase chain reaction has served as the workhorse and gold standard technology for SARS-CoV-2 assays, enabling the quantification of SARS-CoV-2 copy numbers (20,21).In comparison, ddPCR has a higher analytical sensitivity than the real-time RT-PCR technology ( 22) also with quantification and absolute quantification of SARS-CoV-2 copy numbers.Another assay technology that is used in rapid or point-of-care detection assay is the CRISPR (clustered regularly interspaced short palindromic repeats) technology, which can be combined with reverse transcriptase loop-mediated amplification or recombinase polymerase amplification (23)(24)(25)(26).The turnaround time of such a CRISPR-based assay is around 50 min, which is nearly similar to the Seegene Allplex SARS-CoV-2 Fast PCR assay (27).High sensitivity and a fast turnaround time of SARS-CoV-2 assays are crucial in the early detection of outbreaks in the population and their surrounding environment.Wastewater surveillance is gaining significance in the early detection of outbreaks, offering passive surveillance that has the potential to mitigate population incentives and mobility activities (28)(29)(30).A duplex real-time RT-PCR SARS-CoV-2 assay based on two ultra-conserved SARS-CoV-2 sequences has been reported to detect all variants (including the Omicron variant) in clinical and wastewater samples (31,32).This duplex assay has demonstrated similar assay perform ance to that of the Seegene Allplex SARS-CoV-2 assay.
As the COVID-19 pandemic has passed, the anticipation by WHO and central health care policy entities such as Centers for Disease Control, EMA, and multiple national health authorities is that SARS-CoV-2 will reside as an endemic respiratory disease, and thus a return to standard operating practice with respect to evidence-based medicine is called for.
For clinical diagnostic purposes, combined assays detecting SARS-CoV-2, influenza virus, and/or RSV (6,7), such as the Allplex-SC2FabR assay can allow precise diagnostics of suspected patients with the intention of correct management in a low-prevalence setting.Confounding the clinical presentation of COVID-19 are other respiratory virus infections such as influenza and RSV, which play a role in screening and diagnosing primary and co-infections, including identification of the pathogen in question (33,34).

Conclusion
All three Seegene Allplex SARS-CoV-2 assays demonstrated very high clinical accuracy, with 100% sensitivity and specificity ranging from 98% to 100%.The Allplex assays remained positive up to dilution 1:50 of clinical samples, which was not the case with the TaqPath COVID-19 assay.The analytical LODs (measured by ddPCR and using reference materials) were lower for the three Allplex assays compared to the TaqPath COVID-19 assay.The clinical LOD was slightly lower when compared to the analytical LOD using the Allplex assays.The target gene assessment revealed that target gene sequences of the Seegene Allplex SARS-CoV-2 assays and the SARS-CoV-2 reference materials were not completely overlapping.

TABLE 1
SARS-CoV-2 genes of the targeted sequence of comparator, index assays, and three SARS-CoV-2

TABLE 3
Analytical LOD (measured by ddPCR b ) on the diluted synthetic RNA RMs for each target gene a -indicates that the respective assay did not have the primers and probes of the target genes.ND, not detected, although the primers and probes of the assays were present in the RT-PCR.
a b ddPCR: droplet digital PCR.c The ORF1ab region includes the RdRP gene.d Primers and probes for RdRP/S gene of the Allplex-SC2 assay were in the same detection channel.e The detection of the RdRP and S gene were in the same channel; therefore, either the detection of the S gene might not be true.f Only one dilution (VL = 7.42 × 10 7 copies N3/mL, measured by ddPCR) was positive.

TABLE 4
Target genes of the TaqPath COVID-19 assay, the three Allplex assays, and the three synthetic RNA RMs: before and after sequence assessment of target genes Undetermined, as only one dilution (VL = 7.42 x 10 7 copies N3/mL, measured by ddPCR) was positive.
a b The detection of the RdRP and S genes was in the same fluorophore channel, which implies that the detection of the S gene might not be true.c Target gene(s) not detected.