Evaluation of four laboratory-based high-throughput SARS-CoV-2 automated antigen tests compared to RT-PCR on nasal and oropharyngeal samples

Background The demand for RT-PCR testing has been unprecedented during the SARS-CoV-2 pandemic. Fully automated antigen tests (AAT) are less cumbersome than RT-PCR, but data on performance compared to RT-PCR are scarce. Methods The study consists of two parts. A retrospective analytical part, comparing the performance of four different AAT on 100 negative and 204 RT-PCR positive deep oropharyngeal samples divided into four groups based on RT-PCR cycle of quantification levels. In the prospective clinical part, 206 individuals positive for and 199 individuals negative for SARS-CoV-2 were sampled from either the anterior nasal cavity (mid-turbinate) or by deep oropharyngeal swabs or both. The performance of AATs was compared to RT-PCR. Results The overall analytical sensitivity of the AATs differed significantly from 42% (95% CI 35–49) to 60% (95% CI 53–67) with 100% analytical specificity. Clinical sensitivity of the AATs differed significantly from 26% (95% CI 20–32) to 88% (95% CI 84–93) with significant higher sensitivity for mid-turbinate nasal swabs compared to deep oropharyngeal swabs. Clinical specificity varied from 97% to 100%. Conclusion All AATs were highly specific for detection of SARS-CoV-2. Three of the four AATs were significantly more sensitive than the fourth AAT both in terms of analytical and clinical sensitivity. Anatomical test location significantly influenced the clinical sensitivity of AATs.


Introduction
Since the beginning of the SARS-CoV-2 pandemic in early 2020, the need for testing has been immense all over the world. Quick and precise diagnostics were established as a cornerstone in controlling the pandemic and to prevent spreading. Reverse Transcription Polymerase chain reaction (RT-PCR) is considered as the gold standard of testing, but since it is time consuming and requires highly trained personnel, the need for faster and less complex diagnostic tools rapidly emerged. Worldwide, the attention quickly turned towards rapid antigen tests (RAT), which provide rapid results with no training necessary for testing personnel, but yield significantly lower sensitivity compared to RT-PCR [1]. In contrast to RAT, automated antigen tests (AAT) still require laboratory facilities and trained personnel.
AATs detect one or more antigens by the use of chemiluminescence immunoassays (CLIA) with antibodies targeting specific antigens. The antibodies are marked with molecules that emit chemiluminescent light when binding to antigen sites. The emitted light is detected by electrodes and converted in to a positive or negative test result [2].
Prior to the SARS-CoV-2 pandemic, RAT/AAT have been used as a diagnostic tool in other viral infections including influenza A + B and respiratory syncytial virus (RSV) infections [3][4][5]. Previous studies have found varying clinical sensitivities for RSV (29% in adults and 81% in children [5]) and Influenza A + B (Inf A 54% and Inf B 53% [3]) for AATs.
Like most commercial RT-PCR assays, AATs provide automated evaluation and interpretation of the test results, which greatly reduces the risk of human error. In contrast to RT-PCR, they are simple to use thus requiring less personnel training.
However, studies comparing analytical and clinical sensitivity and specificity of different AATs in comparison with RT-PCR for SARS-CoV-2 are lacking. This study compares the analytical and clinical sensitivity and specificity of four different AATs to RT-PCR.

Automated antigen tests
Manufacturer of AATs were invited to participate by the Danish Regions through relevant trade organizations in Denmark. Each AAT was enrolled by the manufacturer, who decided how the test should be taken and if multiple sampling sites should be included for each test. Manufacturers were allowed to supply other sampling swabs than included in the CE-approval to increase the performance of the test e.g. larger flocked swabs for oropharyngeal testing than the standard thinner flocked swabs intended for anterior nasal cavity testing (mid-turbinate testing). The use of oral swabs is outside the regulatory claim for the Atellica and Liaison tests. The four AAT included in this study were the Liaison SARS-CoV-2 Ag test from Diasorin S.p.A (Saluggia, Italy), Vitros Immunodiagnostic SARS-CoV-2 Ag test from Ortho-Clinical Diagnostics (Pencoed, United Kingdom), Elecsys SARS-CoV-2 Ag test from Roche Diagnostics GmbH (Mannheim, Germany) and Atellica Solution CoV2Ag chemiluminescent immunoassay test kit from Siemens Healthcare Diagnostics (Tarrytown, NY, USA).

Study design
The study consists of two parts. The first part is a retrospective analytical sensitivity and specificity study based on 304 patient samples stored in universal transport medium (UTM, Copan Diagnostics Inc, Brescia, Italy). This panel of samples have previously been used for evaluation of RATs [6,7]. The second part is a prospective accuracy observational study on 206 SARS-CoV-2 positive and 199 SARS-CoV-2 negative individuals tested simultaneously for all four ATTs.

Participants
Individuals, who tested positive for SARS-CoV-2 by routine RT-PCR performed by a public test provider, either the Danish National Test Center (national screening) or a regional department of clinical microbiology (health care workers, out-patients or individuals having appointments with a healthcare professional) were invited by secure mail to contact a test-coordinator to plan a new test at home performed by an out-patient testing team within 72 h after the first initial positive RT-PCR test for SARS-CoV-2. Individuals presumed negative for SARS-CoV-2 were recruited directly at the regional test-centers.
Participating individuals had to be at least 18 years of age and had to sign an informed consent form prior to inclusion in the study. Each person was tested from both anatomic test locations including a deep oropharyngeal swab for a second RT-PCR as control for SARS-CoV-2 status at the time of inclusion in the study.

Retrospective analytical testing
UTM samples from SARS-CoV-2 RT-PCR positive individuals participating in the prospective arm of the study were collected by the out-patient testing team between March 10th and May 28th, 2021. All samples were collected as deep oropharyngeal swabs and immediately stored at − 80 • C until further processing. The positive samples were thawed on ice and adjusted to specific cycle of quantification (Cq) ranges by adding UTM at 0 • C. In total, 50 samples with Cq <25; 54 samples with Cq between 25 and 30; 50 samples with Cq between 30 and 35 and 50 samples with Cq between 35 and 40 were prepared. One hundred SARS-CoV-2 RT-PCR negative samples were prepared by pooling 1000 UTM samples that had tested negative for SARS-CoV-2. All 304 samples were checked by a two-target Laboratory developed test (LDT) RT-PCR targeting the E and N gene of SARS-CoV-2 with human RNAse P as process control, aliquoted and stored at − 80 • C until testing by the included AAT. Aliquots were thawed on ice and 200 µL sample material was transferred to the lysis buffer of each AAT and processed according to the instruction for use (IFU). AAT results were recorded as reported. For each sample, one aliquot was Cq verified after storage by the same LDT RT-PCR and the highest Cq level of the two targets was used to stratify the sample into a Cq range group.

Prospective AAT testing
From each included individual, twin swabs from anterior nasal cavity (mid-turbinate) and oropharynx were collected in 3.5 mL UTM for AAT testing. 500 µL UTM was added to the lysis buffer of each AAT and analyzed according to the IFU using utensils provided by the manufacturer. Furthermore, an additional oropharyngeal swab was collected from each individual for RT-PCR testing. Samples collected from other anatomical sites than CE-marked were collected and handled as instructed by each manufacturer.

Prospective RT-PCR
The deep oropharyngeal swabs for RT-PCR were collected in a NEST disposable sampler inactivation transport medium with an oropharyngeal specimen collection swab (Wuxi NEST Biotechnology Co., Ltd, Wuxi City, China) and sent to RT-PCR testing at the Technical University of Denmark, Lyngby, Denmark (DTU). At DTU, samples were analyzed with the CoviDetect -COVID-19 multiplex RT-qPCR assay from Penta-Base (Odense, Denmark). As the fluorescence level is not recorded during the first 7 cycles of the RT-PCR protocol, samples were stratified as strongly positive at Cq <15, medium positive at Cq 15 to 20 or weakly positive at Cq 20 to 35. Results were automatically uploaded and inconclusive samples (negative for all targets) were double checked by a laboratory scientist at DTU. Result registration was validated by a molecular biologist or a clinical microbiologist

Analysis and raw data
RT-PCR for two SARS-CoV-2 targets was used as gold standard. In the retrospective part of the study RT-PCR was performed targeting the SARS-CoV-2 E-gene and N-gene, as well as a human control target to verify the quality of the sampling procedure. Cq values in the retrospective study were recorded as the highest Cq level of the N-and Egenes targets. In the prospective study, average Cq-values between two N-genes targets were calculated and stratified in three Cq ranges. If one target was negative, stratification was based on the Cq level for the other positive target.
Individuals were excluded from further analysis, if RT-PCR results were missing, the sample for RT-PCR had been lost or the sample was negative for human target.
Sensitivity and specificity were calculated for each individual AAT in relation to RT-PCR. Confidence intervals were estimated by the nonparametric bootstrap method, with 1000 re-samplings for each estimate. Additional calculation was done stratifying strong, medium and weak positive samples according to RT-PCR Cq level. Comparisons of sensitivity estimates between any two tests were done paired (McNemar test for the true positive distribution within gold standard positive measurements). To adjust for multiple testing, Bonferroni correction of the p-values was used by multiplying the p-value by the number of tests performed. A p-value larger than 1 after correction was set to 1.
All analysis was done using R 3.6.1 (R Foundation for Statistical Computing, Vienna, Austria), all p-values less than 0.05 were considered statistically significant.

Role of the funding source
The participating commercial companies had no influence on study design, sample collection, analysis, interpretation or drafting the manuscript, nor decision on publishing.

Participants and SARS-CoV-2 variants
In the retrospective part of the study, a total of 304 samples (100 RT-PCR negative samples and 204 RT-PCR positive samples) were included. The 204 RT-PCR positive samples were divided into four Cq-ranges (Fig. 1A).
Between May 28th, 2021 and June 28th, 2021, 416 participants, of which, 231 were newly-identified SARS-CoV-2 positive individuals, were enrolled in the prospective part of the study. Among the newly positive individuals, eight were excluded, including one missing sample and seven samples negative for the RT-PCR human control target. Among the remaining 223 patients, 17 (8.3%) were negative for SARS-CoV-2 by the RT-PCR test carried out concurrently with the AAT. In the screening group, a total of 185 patients were included. Three samples were excluded due to negative human control target in the RT-PCR tests resulting in a total of 199 samples that were RT-PCR negative for SARS-CoV-2 (Fig. 1B). The

Analytical sensitivity and specificity of AAT
The overall analytical sensitivity was comparable among three of the four included AATs from 56% (95% CI 49-63) for the Siemens Atellica test to 60% (95% CI 53-67) for the Roche Elecsys AAT. The AAT from Diasorin had a significantly lower analytical sensitivity (42%, 95% CI 35-49) compared to the other three AATs (Fig. 2). All AATs had an analytical specificity of 100%. Subgroup analysis reveal that all four tests performed similar at RT-PCR Cq < 25, but the Liaison AAT from Diasorin performed significantly worse compared to the other three AATs at Cq between 25 and 35. All four AATs were not able to detect RT-PCR samples with a Cq > 35 (Supplementary data).

Clinical sensitivity and specificity of AAT
The overall clinical sensitivity was significantly higher for AATs performed on samples from the anterior nasal cavity compared with samples from the oropharyngeal cavity (p < 0.001) (Fig. 3). The three AATs tested on anterior nasal cavity samples had similar sensitivities, with the Vitros AAT from Ortho-Clinical having the highest clinical sensitivity of 88% (95% CI 84-93). Three AATs were tested on deep oropharyngeal swabs with similar sensitivities for the Siemens Atellica and Roche Elecsys AAT 39% (95% CI 33-46) and 37% (95% CI 31-44) respectively, whereas the Liaison AAT from Diasorin had a significantly lower clinical sensitivity 26% (95% CI 20-32). The clinical specificity was not significantly different between tests and varied between 97% (95% CI: 94-99) and 100% for all tests. The difference in sensitivity for the Liaison AAT from Diasorin compared to the Atellica AAT from Siemens and the Elecsys AAT from Roche was primarily caused by lower sensitivity for moderate positive clinical samples and to a lesser extend for highly positive clinical samples, whereas weaker positive deep pharyngeal samples was almost all missed by all three AAT (Supplementary data).

Discussion
In this study, we compared the analytical and clinical sensitivity and specificity of four different AATs for detection of SARS-CoV-2. Three of the four AATs had similar analytical and clinical sensitivities, whereas the Liaison AAT from Diasorin had a significantly lower analytical and clinical sensitivity. This was unexpected, as previous studies evaluating each of the AATs have shown similar sensitivities [8][9][10][11][12]. Our findings highlight the need for comparison studies to allow direct comparison of different diagnostic tests.
The present study is a continuation of our previous study comparing a large number of RATs [7]. We have in the analytical sensitivity and specificity part of the study used aliquots of the same samples as in our RAT study. In this study, we find that AATs were able to detect SARS-CoV-2 in a number of samples with Cq values between 30 and 35. This is interesting as the RAT in general were not able to detect SARS-CoV-2 in these samples. This indicates that the analytical sensitivity of AAT may be higher than the analytical sensitivity of RAT. Still AAT are less sensitive than RT-PCR, which limits their use for clinical diagnostics, infection control measurements and infectious disease preparedness purposes as high sensitivity testing is requested [13]. But AATs are cheaper, workflow is shorter and instrumentation allow high capacity testing compared to RT-PCR, which make them useful for surveillance purposes in which the use of the same assay and sampling method is more important than the performance of the test [14].
In the prospective part of the study, all data were paired between the participating AATs as aliquots of each sample from each individual was tested in parallel by all AATs. As seen in Fig. 3, we find that AAT performed on deep oropharyngeal swabs were less sensitive compared to AAT performed on anterior nasal cavity (mid-turbinate) testing from same individuals. For the Atellica and the Elecsys AAT the overall sensitivity raised with 48 (95% CI 36-58) and 49 (95% CI 37-59) percentage points respectively by changing the sampling site from a deep oropharyngeal swab to an anterior nasal cavity (mid-turbinate) swab. This is in agreement with previous findings by RT-PCR showing a significantly higher sensitivity for various respiratory viruses, when comparing nasopharyngeal to oropharyngeal swabs [15].
This study has two main limitations. Firstly, the study is a comparison study and cannot be used to predict the clinical sensitivity of an AAT in a specific population or a specific clinical setting. Secondly, the alpha and delta variants dominated at the time of the study and the sensitivity of AATs may, as for all antibody-based testing, be reduced by structural mutations in future variants of SARS-CoV-2.
In conclusion, this study compares analytical and clinical sensitivities to RT-PCR for four different AATs. We show that three of the four tests had a significantly higher analytical sensitivity than the fourth. Furthermore, we find that the clinical sensitivity of anterior nasal cavity swabs for AATs outperformed the clinical sensitivity of deep oropharyngeal swabs for AATs, making studies of sampling site very important to include as the sampling site may impact clinical sensitivity more than the selection of AAT.

Funding
The study was funded by a participant fee for each test and by the Danish Regions.

Authors' contributions
UVS and JGL conceptuated the study and were responsible for the funding acquisition. Acquisition of data was conducted by all TDL, KG, LFH, SA, SEE, CMGM, AC, RLJ, MBH, PRK, HL; NS, CBJ, MWF, NSK, UVS. Local supervision of research activity was conducted by LFH, SA, SEE, AC, PRK, HL, NS, MWF, NSK. Overall project administration was conducted by UVS and JGL. Data was curated by CBJ, JGL and UVS. Data was validated by or under supervision of MWF, CBJ, JGL, and UVS. TK and UVS performed the formal analysis of data. TDL, JGL and UVS drafted the manuscript. All authors read and commented on the manuscript and all authors approved the final manuscript prior to submission. All authors agree to be held accountable for all aspects of the work.

Ethics committee approval
The study was evaluated by the National Committee on Health Research Ethics in the Danish Capital Region to be a method validation study without the need of approval of the committee (decision H-20,068,579). Access to test results for research was granted by the Capital Region of Denmark -Research and Innovation (R-20,083,753) and contact to participants without prior consent from the individual was granted by the board of directors at the Hospitals at which the participating DCM are situated.

Declaration of Competing Interest
None of the authors have any personal conflict of interests to report. The project received a participation fee from each of the participating companies to cover the cost of the project. Fig. 3. Prospective study part. Overall clinical sensitivity and specificity of four automated laboratory antigen tests for SARS-CoV-2. Sensitivity and specificity are reported as mean with 95% CI. N is an anterior nasal cavity test (mid-turbinate) and OP is a deep oropharyngeal test.