Post-market surveillance of six COVID-19 point-of-care tests using pre-Omicron and Omicron SARS-CoV-2 variants

ABSTRACT Post-market surveillance of test performance is a critical function of public health agencies and clinical researchers that ensures tests maintaining diagnostic characteristics following their regulatory approval. Changes in product quality, manufacturing processes over time, or the evolution of new variants may impact product performance. During the COVID-19 pandemic, a plethora of point-of-care tests (POCTs) was released onto the Canadian market. This study evaluated the performance characteristics of several of the most widely distributed POCTs in Canada, including four rapid antigen tests (Abbott Panbio, BTNX Rapid Response, SD Biosensor, and Quidel QuickVue) and two molecular tests (Abbott ID NOW and Lucira Check IT). All tests were challenged with 149 SARS-CoV-2 clinical positives, including multiple variants up to and including Omicron XBB.1.5, as well as 29 clinical negatives. Results were stratified based on whether the isolate was Omicron or pre-Omicron as well as by reverse transcriptase quantitative PCR Ct value. The test performance of each POCT was consistent with the manufacturers' claims and showed no significant decline in clinical performance against any of the variants tested. These findings provide continued confidence in the results of these POCTs as they continue to be used to support decentralized COVID-19 testing. This work demonstrates the essential role of post-market surveillance in ensuring reliability in diagnostic tools. IMPORTANCE Post-market surveillance of diagnostic test performance is critical to ensure their reliability after regulatory approval. This is especially critical in the context of the COVID-19 pandemic as the use of point-of-care tests (POCTs) became widespread. Our study focused on four rapid antigen tests (Abbott Panbio, BTNX Rapid Response, SD Biosensor, and Quidel QuickVue) and two molecular tests (Abbott ID NOW and Lucira Check IT) that were widely distributed across Canada, assessing their performance using many SARS-CoV-2 variants, including up to Omicron subvariant XBB.1.5. Overall, we found no significant difference in performance against any variant, reinforcing confidence in their use. As concerns in test efficacy have been raised by news outlets, particularly regarding the BTNX Rapid Response, this work is even more timely and crucial. Our research offers insights into the performance of widely used COVID-19 POCTs but also highlights the necessity for post-market surveillance.

through the use of point-of-care tests (POCTs).This shift was accelerated following the emergence of the highly transmissible Omicron variant (Pango lineage B. 1.1.529)in December 2021 (3)(4)(5) causing unprecedented testing demand that overwhelmed centralized testing facilities (4)(5)(6).As of August 2023, 55 POCTs have been authorized for use in Canada through the "Interim order respecting the importation and sale of medical devices for use in relation to COVID-19" (7).These devices can be broadly categorized into two groups: molecular tests and rapid antigen detection tests (RADTs).Molecular POCTs detect the presence of SARS-CoV-2-specific nucleic acid sequences through PCR or isothermic amplification techniques such as loop-mediated isothermal amplification (LAMP) (1,8,9).Molecular tests generally provide a high level of sensitivity and may detect viral RNA weeks after infection (9)(10)(11).In comparison, RADTs detect SARS-CoV-2 surface proteins with decreased sensitivity, particularly when considering asymptomatic cases with lower viral loads (12)(13)(14).
Post-market surveillance monitors the performance of POCTs following commerci alization and distribution as well as provides independent investigations that assess the manufacturers' claims regarding test efficacy in real-world settings (15).Ongoing monitoring becomes exceedingly important for POCTs as few quality controls are in place compared to clinical testing performed in centralized test facilities.Test perform ance may also deviate over time due to changes in manufacturing materials or processes, or through viral evolution of test targets (16,17).For molecular POCTs, test performance may be impacted by base substitutions at the primer binding sites, which can often be predicted in silico (18).RADT performance may also be impacted by mutations within target proteins; however, predicting this computationally is more challenging as it is influenced by protein 3D structure.Therefore, post-market surveillance of RADTs relies more heavily on experimentation.
Initial studies evaluating the performance of various RADTs found reduced impaired detection of Omicron variants compared to previously circulating strains (19)(20)(21).For example, Osterman et al. reported that multiple RADTs tested had 101-fold lower sensitivity against Omicron BA.1 compared to Delta (21).Although this study may raise concerns surrounding the continued use of these RADTs, multiple subsequent studies demonstrated equivalent sensitivity regardless of the variant (22)(23)(24).Together, this emphasizes the need to disseminate negative test results in post-market surveillance studies.
Due to the variety of POCTs available globally, tests evaluated in published research often have little to no overlap with the POCTs widely distributed in Canada.Although manufacturers may provide post-market surveillance data regarding performance, these may be based heavily on in silico analyses and experimental results may differ (25).Therefore, there is a need for regulatory bodies, clinicians, and public health officials to conduct ongoing aftermarket evaluations and to effectively communicate the results to POCT users.Accordingly, the aim of this post-market surveillance study was to assess the analytical performance of six widely distributed POCTs and to determine if there were differences in the detection of pre-Omicron and Omicron variants.

MATERIALS AND METHODS
A total of 149 SARS-CoV-2 positive and 29 negative clinical samples were used to determine the sensitivity, specificity, and experimental limit of detection (LoD) of each COVID-19 POCT.Clinical samples, reference cycle thresholds (Ct), and lineage data were provided by Cadham Provincial Laboratory (CPL; Manitoba, Canada).Samples were anonymized and collected from majority symptomatic individuals who visited healthcare providers for the purpose of diagnosis with the minority collected from asymptomatic individuals as a part of routine surveillance activities.The remnant universal transport media (UTM) used to store each nasopharyngeal swab sample was tested using the Roche 6800 Cobas SARS-CoV-2 test targeting the Open Reading Frame 1a and Envelope genes to determine positivity and the reference Ct value.Samples were considered positive if a Ct value was generated.Positive samples were amplified using the primer scheme described by Freed et.al (26) and sequenced on the Oxford Nanopore Minion.Lineages were then assigned using Pangolin (v3.1.16or v4.0.6) (27).Samples collected before routine sequencing lacked lineage data; however, given their collection between March 2021 and July 2021, they were presumed to be likely Alpha due to its prevalence during that time.For each POCT, 10 µL of remnant UTM was transferred directly into the assay buffer and subsequently mixed with the manufacturer-provided swab.Each test was then performed according to the manufacturer's instructions.For RADT results that required visual interpretation, test strips were read independently by three technicians and the consensus was reported.If necessary, invalid tests were repeated until a positive or negative result was encountered.
Data were analyzed using R version 4.2.1 software.Sample Ct values were adjusted to account for the quantity of virus input into the device corresponding to the concen tration when eluted into a 3 mL volume.Continuous variables such as Ct are represen ted as medians and interquartile ranges (IQR).Categorical variables, such as sensitivity and specificity, are reported as percentages with 95% confidence intervals (95% CI) calculated using the Wilson score method.Sensitivity was also determined for each POCT when samples were stratified based on Ct values of ≤30 and >30, i.e., the acceptable LoD outlined by the World Health Organization (WHO) for RADTs (29) and the cut-off typically used in clinical settings and other evaluations (2,16,30).For each POCT, the overall and stratified sensitivity values were compared between the pre-Omicron and Omicron groups using a two-tailed Fisher's Exact Test with an α of 0.05.
Next, based on the guidelines recommended by the Clinical & Laboratory Standards Institute (31), a probit model was constructed for each assay to determine the 95% LoD, representing the estimated concentration at which 95% of positive samples are successfully identified.The 95% CI for each LoD was calculated using the Wald method, then the LoDs were compared between pre-Omicron and Omicron to determine if the analytical sensitivity of each POCT differed by variant.

Performance characteristics of POCTs with pre-Omicron and Omicron variants
A  2).For all tests, sensitivity was similar between Omicron compared to pre-Omicron samples when analyzing all Ct values together.Similarly, when stratify ing test results by Ct for all tests, no significant difference in sensitivity was observed between pre-Omicron and Omicron samples with high viral loads (i.e., Ct ≤ 30; Table 2).At low viral loads (i.e., Ct > 30), significant differences in ID NOW (P = 0.021) and QuickVue (P = 0.028) sensitivity were observed between pre-Omicron and Omicron variants.

Limit of detection analysis of POCTs with all variants
Probit regression analyses were performed using data from all 149 SARS-CoV-2 clinical specimens to determine the 95% LoD of each test (Fig. 1).The molecular POCTs, the ID NOW and Check IT, had 95% LoDs of 34.7 and 39.3 Ct values, respectively, highlighting their ability to detect low quantities of virus.RADTs had 95% LoDs ranging between 25.0 and 28.4 (Table S1).

Limit of detection analysis of POCTs for pre-Omicron and Omicron variants
A probit regression model was used to assess if the limit of detection of each test differed when stratified by variant (pre-Omicron vs Omicron).The 95% LoDs of all the POCTs were similar, regardless of the variant, as demonstrated by the sizeable overlap between their confidence intervals (Fig. 2).Due to a poor model fit from lack of false negative results, it was not possible to determine the 95% LoD of the Check IT for the pre-Omicron and Omicron variant samples, suggesting that performance of this molecular test is on par with the reference test.

DISCUSSION
POCTs have been central to the COVID-19 pandemic response, particularly following the appearance of the Omicron variant in late 2021 (2,3).As new variants emerge, it is important to monitor the performance of POCTs to ensure that test results continue to be reliable.Consequently, the purpose of this study was to evaluate six widely distrib uted Health Canada-approved POCTs using COVID-19 clinical samples incorporating pre-Omicron and Omicron variants (Alpha to XBB.1.5)to determine if test characteristics differed over time.Overall, our assessment revealed consistent and reliable performance across the variants tested, affirming manufacturer's claims that there is no notable decrease in clinical efficacy.The molecular tests evaluated, the ID NOW and Check IT, demonstrated the highest sensitivity in this study, making them particularly useful for high consequence testing where laboratory-based PCR is unavailable (32).
To our knowledge, only two studies, conducted by Stokes et al. (33,34), have assessed the performance characteristics of the ID NOW in detecting both pre-Omi cron and Omicron variants.Per Stokes et al. (33,34), for symptomatic patients, the ID NOW demonstrated improved sensitivity against Omicron (91.6-96.0%)compared to pre-Omicron (90.0-92.5%).This aligns with the significant increase in test sensitivity against high Ct clinical Omicron samples noted in this study.Potential explanations for this finding may be attributed to various factors including changes in testing procedures over time, patient infection stage, or sample storage and quality.Studies have also reported decreased ID NOW performance with higher Ct samples (35)(36)(37).In contrast to traditional RT-qPCR, the ID NOW employs LAMP-based technology that targets a large RNA fragment (>1 kb).Therefore, the device may exhibit reduced sensitivity with fragmented samples from long-term storage, which may be a confounding factor in clinical LAMP validation studies.Additionally, the ID NOW targets RNA-dependent RNA polymerase (RdRp), an uncommon target among COVID-19 POCTs and RT-qPCR assays.This feature may cause the ID NOW to be affected uniquely by variant mutations as compared to other tests, and the Ct value from the reference assay which does not target RdRp may not accurately reflect the abundance of target RNA in each sample.
The Check IT has only been independently evaluated in one other publication to date.In a study by Zahavi et al. (8), the device exhibited a sensitivity of 91.1% using samples collected before the emergence of Omicron.This is consistent with the high sensitivity of the Check IT when using both pre-Omicron and Omicron variants observed in this study, with the LoD comparable to that of laboratory-developed qRT-PCR tests.
Using the WHO guidelines for RADT evaluation and clinical relevance (29), at high viral loads (Ct ≤ 30), all RADTs met the criteria for acceptable clinical sensitivity (>80%).In agreement with previous reports and manufacturer claims, test sensitivity of RADTs decreased markedly at low viral loads (Ct > 30) regardless of the variant.It is important to note that comparing LoDs and sensitivity across studies, especially for RADTs, is often challenging due to differences in experimental design.For example, reference Ct values can differ greatly based on methodology and gene target.Recently, media reported on possible issues in the design of the BTNX validation studies used for regulatory approval (38); however, this study corroborates manufacturer claims and has identified no appreciable differences in analytical test performance between the BTNX and the other RADTs investigated in this study.
A meta-analysis by Mohammadie et al. (39) on RADT performance revealed a pooled sensitivity of 67% and a non-significant reduction in overall RADT sensitivity against Omicron.Collectively, we found that RADTs were compatible with emerging variants.While these overarching trends can offer insight into RADT efficacy as a whole, it is crucial to acknowledge the diversity among the many available tests.Despite most tests, including all evaluated in our study, targeting the viral nucleocapsid protein, we observed slight variation between tests in overall performance.These differences may be due to factors such as test design, reagent composition, and variability in specific antigen epitopes.
QuickVue showed significantly higher sensitivity against low viral load Omicron samples, consistent with findings by Sugiharto et al. (24); however, work by Krenn et al. (40) showed no change in QuickVue sensitivity against Omicron variants.This finding may be due to differences in the epitope of the Omicron nucleocapsid protein targe ted by the test antibody; however, this hypothesis cannot be confirmed unless shown experimentally with proprietary test antibodies.Although all rapid tests validated target the nucleocapsid, their target epitopes may differ, as each test likely utilizes unique antibody complements.
All other RADTs evaluated in this study showed no significant change in test performance against Omicron variants.Previous studies evaluating RADTs have consistently reported similar or reduced clinical sensitivity with Omicron as compared to early pandemic variants (21,39,40).For example, multiple studies investigating the Abbott Panbio using cultured virus (41) and clinical samples (23,42) have shown no sensitivity loss against Omicron.Alternatively, work by Bekliz et al. (19) reported a significant reduction in Panbio sensitivity with Omicron BA.1 compared to Delta.Biosensor demonstrated comparable performance with cultured Omicron variants (41,43) but, in other studies, exhibited reduced clinical sensitivity (19,20).To our knowledge, this is the first study investigating the sensitivity of the BTNX Rapid Response against multiple SARS-CoV-2 variants including Omicron.
A limitation of this study is the use of anonymized remnant clinical samples instead of clinical patient swabs.The use of clinical swabs was not feasible as multiple variants investigated in this study are no longer circulating and this study required a large number of paired samples to evaluate multiple technologies.Due to the anonymization of samples used in this study, we were not able to incorporate information that may affect test sensitivity or specificity, such as vaccination status or patient clinical presenta tion, into our analyses (13,14,40).
Unlike many studies that primarily focused on comparing BA.1 exclusively with the Delta variant, our research included a representative range of both pre-Omicron and Omicron subvariants (Alpha to XBB.1.5).Considering both the overall sensitivity and limit of detection, we observed no decline in test performance against Omicron variants; however, it is important to continue to evaluate these parameters as new variants emerge.Sustained post-market surveillance efforts are required to proactively identify issues that could impact decentralized testing for COVID-19.

FIG 1 FIG 2
FIG 1 Probit regression analysis showing the 95% limit of detection of six SARS-CoV-2 point-of-care tests based on testing outcomes of 149 positive SARS-CoV-2 clinical samples.The cycle threshold of the sample, determined using the RT-qPCR reference standard test result, is plotted against a positive (1) or negative (0) test outcome.The horizontal dashed line indicates the 95% probability of a positive test result while the vertical dashed lines show the 95% limit of detection of each test with shaded areas representing their 95% CI.

Table 1 )
. No ethics were required under Article 2.4 and Article 2.5 of the TriCouncil Policy Statement Ethical Conduct for Research Involving Humans (28) as these were remnant clinical specimens used for assay validation.This exemption was confirmed by our institutional review board.To minimize sample degradation, each sample was limited to a maximum of two freeze-thaw cycles.Six Health Canada-approved COVID-19 POCTs were evaluated in this study: four RADTs including the BTNX Rapid Response COVID-19 Antigen Rapid Test (BTNX Inc., Canada), the Abbott Panbio COVID-19 Ag Rapid Test Device (Abbott Rapid Diagnostics Jena Gmbh, Germany), the SD Biosensor COVID-19 Ag Test (SD Biosensor Inc., South Korea), and the Quidel QuickVue At-Home OTC COVID-19 Test (Quidel Corporation, United States) as well as two LAMP-based molecular tests [the Abbott ID NOW COVID-19 Assay (Abbott Diagnostics Scarborough, Inc., United States) and the Lucira Check IT COVID-19 Test Kit (Lucira Health, Inc., United States)].

TABLE 1
Lineage of SARS-CoV-2 positive clinical samples used in the study total of 149 SARS-CoV-2 clinical positives with a median Ct value of 30.2 (IQR, 26.8-33.8)and29 SARS-CoV-2 clinical negatives were tested using all six POCTs.Of the positive samples, 94 were determined to be Omicron subvariants (median Ct value, 29.7; IQR, 27.1-33.5)andthe remaining 55 were identified as pre-Omicron (median Ct value, 30.0;IQR, 26.0-34.4).No false positives were recorded resulting in a specificity of 100% (95% CI, 88-100%) across all tests.Six invalid results were obtained across all testing: three with the ID NOW and three with the Check IT.All testing results can be found in Supplementary Data 1.Overall sensitivity of RADTs ranged from 46-65% compared to 86-98.9% for molecular tests (Table

TABLE 2
Performance characteristics of six COVID-19 point-of-care tests with pre-Omicron and Omicron samples b a Significant at P < 0.05 for difference in POCT sensitivity between pre-Omicron and Omicron variants using Fisher's exact test.b CI: Confidence Interval; Ct: Cycle Threshold; n: number of SARS-CoV-2 positive samples tested.