Multicenter Diagnostic Evaluation of OnSite COVID-19 Rapid Test (CTK Biotech) among Symptomatic Individuals in Brazil and the United Kingdom

ABSTRACT The COVID-19 pandemic has given rise to numerous commercially available antigen rapid diagnostic tests (Ag-RDTs). To generate and to share accurate and independent data with the global community requires multisite prospective diagnostic evaluations of Ag-RDTs. This report describes the clinical evaluation of the OnSite COVID-19 rapid test (CTK Biotech, CA, USA) in Brazil and the United Kingdom. A total of 496 paired nasopharyngeal (NP) swabs were collected from symptomatic health care workers at Hospital das Clínicas in São Paulo, Brazil, and 211 NP swabs were collected from symptomatic participants at a COVID-19 drive-through testing site in Liverpool, United Kingdom. Swabs were analyzed by Ag-RDT, and results were compared to quantitative reverse transcriptase PCR (RT-qPCR). The clinical sensitivity of the OnSite COVID-19 rapid test in Brazil was 90.3% (95% confidence interval [CI], 75.1 to 96.7%) and in the United Kingdom was 75.3% (95% CI, 64.6 to 83.6%). The clinical specificity in Brazil was 99.4% (95% CI, 98.1 to 99.8%) and in the United Kingdom was 95.5% (95% CI, 90.6 to 97.9%). Concurrently, analytical evaluation of the Ag-RDT was assessed using direct culture supernatant of SARS-CoV-2 strains from wild-type (WT), Alpha, Delta, Gamma, and Omicron lineages. This study provides comparative performance of an Ag-RDT across two different settings, geographical areas, and populations. Overall, the OnSite Ag-RDT demonstrated a lower clinical sensitivity than claimed by the manufacturer. The sensitivity and specificity from the Brazil study fulfilled the performance criteria determined by the World Health Organization, but the performance obtained from the UK study failed to do. Further evaluation of Ag-RDTs should include harmonized protocols between laboratories to facilitate comparison between settings. IMPORTANCE Evaluating rapid diagnostic tests in diverse populations is essential to improving diagnostic responses as it gives an indication of the accuracy in real-world scenarios. In the case of rapid diagnostic testing within this pandemic, lateral flow tests that meet the minimum requirements for sensitivity and specificity can play a key role in increasing testing capacity, allowing timely clinical management of those infected, and protecting health care systems. This is particularly valuable in settings where access to the test gold standard is often restricted.

3.3. There are different formats of data presentation in the Table 1, e.g., "7, 1%" and "6.5%, (32/496)". Some percentages have one decimal place (e.g., 99.6%) while others have no decimals (e.g., 4%). The format in the Table 1 should be consistent among different rows.   Table 2: the unexpected lower sensitivity of the OnSite test may be due to a limited number of PCR-confirmed positive samples enrolled in this study, e.g., 31 PCR positive samples in Brazil and 77 PCR positive samples in UK. The authors should test more PCR-confirmed positive samples by the OnSite test to further examine the clinical sensitivity. 5.1. Table 3: There are different formats of data presentation in the Table 3, e.g., "1, 3%" and "90.5%, 42". Some percentages have one decimal place (e.g., 95.0%) while others have no decimals (e.g., 1%). The format in the Table 3 should be consistent among different rows. 5.2. Ct > 25 (n, %) : 7, 22% should be 22.6% or 23% (7/31) depending on how many decimal places the author prefer. Definitely 22% indicated in the Table 3 is inaccurate. 6. Statistical analysis: please clarify the statistical analysis were two-tailed or one-tailed.
Minor comments for authors 7. Line 46 "clinical sensitivity than claimed by the manufacturer..." Please change "..." to "." 8. Line 58-60 "However, since April 2022 the UK government has ceased free Ag-RDT testing, now requiring the responsibility of the acquisition, and performance of tests to be placed on the individual." What does it mean by "now requiring the responsibility of the acquisition, and performance of tests to be placed on the individual"? Please clarify. 9. Line 123-125: "Swabs were taken systematically, NP swab samples in UTM (Copan Diagnostics Inc, Italy) were collected for the reference RT-qPCR test, this was followed by an NP swab to perform the Ag-RDTs." This sentence has grammatical issue so please re-write it. In addition, what does it mean by "swabs were taken systematically"? Please clarify. 10. Line 126-127: "All samples were transported in cooler boxes to the Liverpool School of Tropical Medicine (LSTM) and processed upon..." What is the shipment conditions and temperature in the cooler boxes?
Reviewer #2 (Comments for the Author): This is an important research and met analysis of Multicentre diagnostic evaluation of OnSite COVID-19 Rapid Test (CTK Biotech) among symptomatic individuals in Brazil and The United Kingdom, it pulls the data together in a way that is likely to be highly impactful and provides a state of the art overview of the current state of knowledge on analytical evaluation of the Ag-RDT. It carries key information about the Ag-RDT assay using direct culture supernatant of SARS-CoV-2 strains from Wild-Type (WT), Alpha, Delta, Gamma, and Omicron lineages, and is likely to be highly cited in the future. However, the manuscript needs to be improved for certain necessary changes and spelling/grammar mistakes throughout and I support its publication after some minor changes. I would like to advise the author to strictly follow the journal's guidelines such as reference style etc.
1-Give full names of all the abbreviations used first, then you can use abbreviations. The English language of the article also needs to be improved Italicize all the "et al., or et al." and scientific words (line 105). The discussion part needs a more detailed analysis of the results.

Preparing Revision Guidelines
To submit your modified manuscript, log onto the eJP submission site at https://spectrum.msubmit.net/cgi-bin/main.plex. Go to Author Tasks and click the appropriate manuscript title to begin the revision process. The information that you entered when you first submitted the paper will be displayed. Please update the information as necessary. Here are a few examples of required updates that authors must address: • Point-by-point responses to the issues raised by the reviewers in a file named "Response to Reviewers," NOT IN YOUR COVER LETTER. • Upload a compare copy of the manuscript (without figures) as a "Marked-Up Manuscript" file. • Each figure must be uploaded as a separate file, and any multipanel figures must be assembled into one file. For complete guidelines on revision requirements, please see the journal Submission and Review Process requirements at https://journals.asm.org/journal/Spectrum/submission-review-process. Submissions of a paper that does not conform to Microbiology Spectrum guidelines will delay acceptance of your manuscript. " Please return the manuscript within 60 days; if you cannot complete the modification within this time period, please contact me. If you do not wish to modify the manuscript and prefer to submit it to another journal, please notify me of your decision immediately so that the manuscript may be formally withdrawn from consideration by Microbiology Spectrum.
If your manuscript is accepted for publication, you will be contacted separately about payment when the proofs are issued; please follow the instructions in that e-mail. Arrangements for payment must be made before your article is published. For a complete list of Publication Fees, including supplemental material costs, please visit ourwebsite.

Introduction
To meet the immense diagnostic demand of the COVID-19 pandemic, the use of rapid diagnostic tests for the detection of SARS-CoV-2 antigens (Ag-RDTs) has become a priority.
To date, there are currently 321 SARS-CoV-2 Ag-RDTs on the market or in development according to the foundation for new innovative diagnostics (FIND) (date accessed March 2022) [1]. However, clinical evaluation of these Ag-RDTs has been relatively limited and performance results differ greatly between studies [2,3]. In the UK, the use of Ag-RDTs has been integral to reducing the spread of COVID-19 [4]. However, since April 2022 the UK government has ceased free Ag-RDT testing, now requiring the responsibility of the acquisition, and performance of tests to be placed on the individual.
In Brazil, the national SARS-CoV-2 testing approach has been insufficient in its use of this diagnostic tool in the efforts to contain this pandemic [5]. Many initiatives such as recruiting capacity in university research laboratories and biotechnological enterprises, investments in new laboratory infrastructure, and fast-track regulatory measures were launched to scale up SARS-CoV-2 RT-qPCR testing in Brazil. However, the expansion of the quantitative reverse transcriptase polymerase chain reaction (RT-qPCR) capacity has not been sufficient to control the progress of the pandemic within this country [5].
Despite the commercialization of several vaccines for SARS-CoV-2, the COVID-19 pandemic is still ongoing due to vaccine inequity [6], uneven vaccine uptake between populations [7], and the emergence of new SARS-CoV-2 highly transmissible variants [8].
The gold standard for diagnosis of COVID-19 remains the detection of SARS-CoV-2 ribonucleic acid (RNA). However, RT-qPCR requires skilled laboratory scientists, installed capacity, and expensive consumables and reagents which can be challenging to implement in low and middle-income countries (LMIC), where the burden of COVID-19 is disproportionately felt. Additionally, the turnaround of results of RT-qPCR can take up to one week [9].
To continue to meet the challenges of testing capacity, prospective diagnostic evaluation studies across multiple, independent sites are required to determine the accuracy of COVID-19 Ag-RDTs available for purchase to the public.
In this study, OnSite COVID-19 Rapid Test (CTK Biotech) was evaluated against the SARS-CoV-2 diagnostic gold standard RT-qPCR. Testing was undertaken in Brazil and the UK across different settings: on healthcare workers (HCWS) at Hospital das Clínicas, a tertiary-care hospital affiliated with the University of São Paulo (Brazil), and at a National Health Service COVID-19 drive-through community testing center in Liverpool, UK.

Clinical evaluation
This was a prospective evaluation of consecutive participants enrolled in two different settings:

Brazil
Healthcare workers (HCW) with suspected COVID-19 symptoms (fever, cough, shortness of breath, tight chest, runny nose, sore throat, anosmia, ageusia, headache, and diarrhea) were enrolled at the HCW service of Hospital das Clínicas in São Paulo from July to October 2021. Ethical approval was obtained from the Hospital´s Ethics Committee with the CAAE number 35246720.0.0000.0068. Informed consent was obtained from all study participants for respiratory samples and clinical data collection.
Participants were clinically evaluated and RT-qPCR for SARS-CoV-2 was performed from

United Kingdom (UK)
In the UK, adults presenting with symptoms of COVID-19 (fever, cough, shortness of breath, tight chest, runny nose, sore throat, anosmia, ageusia, headache, diarrhea, and tiredness) at a national community testing facility, the Liverpool John Lennon Airport drive-through COVID-19 test center, were asked to participate in the study. Participants were recruited between July and August of 2021 under the Facilitating Accelerated COVID-19 Diagnostics

Analytical Sensitivity (UK only)
Viral culture methods to propagate SARS-CoV-2 isolates and to calculate plaque forming units per milliliter (PFU/mL) followed that previously described [11]. Briefly, isolates of SARS-CoV-2 from the wild type (Pango, B1) (REMRQ0001/Human/2020/Liverpool, GISAID ID lineages were used to evaluate the limit of detection (LOD) of the OnSite Ag-RDT. For the determination of the LOD, a fresh aliquot was serially diluted from 1.0x 10 5 plaque-forming units (pfu)/mL to 1.0 x 10 2 pfu/mL. Each dilution was tested in triplicate. Two-fold dilutions were made below the ten-fold LOD dilution to confirm the lowest LOD (LLOD).
Viral RNA was extracted from each dilution using QIAmp Viral RNA mini kit (Qiagen, Germany) according to the manufacturer's instructions, and quantified using Genesig RT-qPCR (Primer Design, UK). Genome copy number/mL (gcn/mL) was calculated as previously described [12].

Statistical Analysis
The sensitivity and specificity, with 95% confidence intervals (CIs) were calculated based on the results of the reference method by RT-qPCR assay. Statistical analyses were performed using R scripts, Epi Info, and GraphPad Prism 9.1.0 (GraphPad Software, Inc, California). The 95% confidence interval (CI) for the sensitivity and specificity was calculated using Wilson's method. Fisher's exact and chi-squared tests were used to determine non-random associations between categorical variables. Statistical significance was set at < 0.05.

Clinical Evaluation
The demographics of both the Brazilian and UK study cohorts are shown in Table 1  In Brazil, of the 496 participants included, 32 were SARS-CoV-2 RT-qPCR positive (6.5%) (see Table 2). Twenty-eight of the RT-qPCR positive samples (90.3%) were Ag-RDT positive, while 3 (9.7%) were Ag-RDT negative and one was invalid (3.1%). Invalid results were removed for further analysis. Of the 464 RT-qPCR negative samples, 3 were Ag-RDT positive (0.6%  Table 3). No statistical significance was found in sensitivity between different Ct value groups.
In the UK, of the 211 participants recruited, 77 (36.5%) were SARS-CoV-2 RT-qPCR positive (see Table 2). Fifty-Eight (75.3%) of the 77 RT-qPCR positive samples were also Ag-RDT positive, while 19 (24.7%) were Ag-RDT negative. Of the 134 RT-qPCR negative samples, 128 (95.5%) were also Ag-RDT negative and 6 (4.5%) were Ag-RDT positive. For the UK evaluation, the sensitivity and specificity were 75.3% [95% Cl 64. 6  Subgroup analyses of the Brazilian and UK evaluation cohorts (Table 4) were performed to determine any associated differences in sensitivity compared to vaccination status and days from symptom onset. In the Brazilian cohort, the sensitivity of the OnSite Ag-RDT was significantly lower on samples from patients with symptoms onset >7 days compared to samples with 0-3 symptoms onset (P = 0.02924) and samples with 0-7 days of onset (P = 0.03115) but no differences in sensitivity were found between groups of different vaccination status. In the UK, no difference in sensitivity was observed between groups of different symptoms onset and vaccination status (all P values >0.05). In Brazil, 52% of the positive samples were classified as Delta and 39% Gamma. In the UK, the variant determination was not performed but at the time of enrollment, 100% of genome     Table 4 Ag-RDT results by the onset of symptoms, and vaccinated individuals in Brazil and the UK

Discussion
The study aimed to evaluate the diagnostic performance of the OnSite COVID-19 Ag Rapid Test (CTK Biotech) in two different settings. Evaluating rapid diagnostic tests in diverse populations is vital to improving diagnostic responses as it indicates the diagnostic accuracy in real-world scenarios. In the case of rapid diagnostic testing within this pandemic, lateral flow tests which meet the minimum requirements for sensitivity and specificity can play a key role in increasing testing capacity, allowing timely clinical management of those infected and protecting healthcare systems [14]. This is particularly valuable in settings where access to the gold-standard RT-qPCR is often not available. Ag-RDTs are low-cost, easy to use, and do not require specialized skills or equipment which is essential to promote universal access.
The sensitivity and specificity of the OnSite Ag-RDT in a hospital setting in Brazil fulfilled the performance criteria determined by the World Health Organisation (WHO). However, sensitivity obtained in a community setting at a drive-through testing site in the UK missed the minimum recommendations [15] for both sensitivity and specificity. In guidance published by the WHO, minimum performance requirements for an Ag-RDT include a sensitivity of >80% and specificity of >97% [15]. RNA copies/mL [16] with the Gamma variant, slightly outside this threshold. In the Brazilian cohort, the Gamma variant was responsible for 39% of infections and the Delta variant was responsible for 52%. This is an interesting finding as it does not reflect the wider variant circulation in Brazil during this period as the Gamma variant was responsible for over 93% of infections in July 2021 and 70% of infections in August 2021 followed by Delta at 5% rising to 29% respectively [17]. In the UK, positive RT-qPCR results were not sequenced but it is assumed that all infections were Delta (B.1.617.2) due to the >99% circulation of this variant in the UK during the time of collection [18].
This study has several strengths, it is a multicentre and multinational evaluation across two different settings with differing testing capacities, the prevalence of SARS-CoV-2, and population characteristics. In Brazil, samples were taken from a very exclusive population, healthcare workers in a healthcare setting with a high vaccination uptake compared to the rest of the population [19]. In the UK, data was collected from a diverse population, any person over the age of 18 presenting with COVID-19 symptoms at a government-run, drivethrough COVID-19 testing facility. It is important to evaluate Ag-RDTs in a heterogeneous population and setting to obtain meaningful diagnostic accuracy data.
The main limitation of the study is that the drive-through testing setting in the UK did not allow for Ag-RDT testing to be performed at the point-of-care just after sample collection as recommended by the IFU. Guidance in the UK restricted testing of suspected COVID-19positive individuals to high containment laboratories. Currently, there are limited studies on the stability of Ag-RDT's. AA systematic review of Ag-RDTs did not find a significant difference between Ninety-six data sets that involved fresh specimens for antigen testing, and 23 data sets that included freeze/thawed specimens for antigen testing [20] Although it is not stated whether the swabs were frozen dried or using transport buffer. However, one review of Ag-RDT performance in sub-Saharan Africa suggested that a delay in performing the test (CORIS COVID-19 Ag Respi-strip) may impact its stability if stored at 4 o C rather than frozen at -20 o C immediately [21]. Conversely, studies have shown that SARS-CoV-2 RNA remains stable for up to 9 days in dry swabs at the ambient temperature of 20°C [22] and proteins are shown to be more stable than RNA [23]. Therefore, further investigation must take place to determine whether the time from sample collection to Ag-RDT testing has a significant impact on the sensitivity.
Two other limitations of this study are that the RT-qPCR methodologies varied between both cohorts and the differences in SARS-CoV-2 prevalence. These factors have been attributed to a major cause of index case diagnostic accuracy [24]. For future evaluations, quantification of the viral copy numbers rather than Ct values is recommended to mitigate differences in RT-qPCR assay performances. This Ct variability has been estimated to be > 1000-fold in viral copy numbers/mL [24], as the RT-qPCR used in the UK has a LOD 10-fold more sensitive (10 genome copies/mL) than the RT-qPCR used in Brazil (100 genome copies/mL) [25]. The higher sensitivity of the RT-qPCR assay used in the UK, together with the higher cut-off used (Ct 40 versus Ct 32-33 in Brazil) could have contributed to higher numbers of false negatives in the index test compared to the Brazilian cohort. Additionally, there is a significant difference in sample size and in confirmed RT-qPCR positives (SARS-CoV-2 prevalence) between the two cohorts, with a low number of positive samples found in the Brazilian evaluation (6.3%) compared to the UK (36.5%). It has been reported that differences in prevalence can affect the sensitivity and specificity of index tests [26,27].
In conclusion, the data indicate that OnSite Ag-RDT had lower performance quality than published by the manufacturers for the detection of SARS-CoV-2 in clinical samples and varied greatly between the two settings in this study. We appreciate this observation and we acknowledge that this is a limitation of the study. However, the low number of positives reflected the low prevalence of COVID-19 at the time of recruitment in the UK and Brazil. We enrolled participants over a defined time period; in the UK between July and August 2021, in Brazil between July and October 2021. In future studies, we will ensure minimum positivity rates are required to examine clinical sensitivity.
Thank you for highlighting this discrepancy, the data in Table 3 has been updated to a one decimal place format.
Thank you for highlighting this error, the manuscript has been updated to the correct value and a one decimal place percentage in line with the rest of the data.
6. Statistical analysis: please clarify the statistical analysis were two-tailed or one-tailed.
Thank you for this comment. This should have been clarified in that the fishers exact and Chi Squared data analysis was two-tailed. This has been updated in the manuscript to reflect this clarification (LINE 162) Minor comments for authors 7. Line 46 "clinical sensitivity than claimed by the manufacturer..." Please change "..." to "." Thank you for highlighting this error. This has been amended.
8. Line 58-60 "However, since April 2022 the UK government has ceased free Ag-RDT testing, now requiring the responsibility of the acquisition, and performance of tests to be placed on the individual." What does it mean by "now requiring the responsibility of the acquisition, and performance of tests to be placed on the individual"? Please clarify.
Thank you for highlighting this potential issue, as this sentence is not clear in its meaning, we have revised the wording to make this clearer for the reader to understand. This sentence now reads "However, since April 2022 the UK government has ceased free Ag-RDT testing, now requiring the responsibility of the purchase and use of the test to be placed on the individual." (LINE 62) 9. Line 123-125: "Swabs were taken systematically, NP swab samples in UTM (Copan Diagnostics Inc, Italy) were collected for the reference RT-qPCR test, this was followed by an NP swab to perform the Ag-RDTs." This sentence has grammatical issue so please re-write it.
In addition, what does it mean by "swabs were taken systematically"? Please clarify.
Thank you for your suggestion, this sentence has been re-written to correct the grammatical issues. "Swabs were taken systematically; first an NP swab sample in UTM was collected from the patient for the reference RT-qPCR test, then an NP swab sample was taken to perform the Ag-RDTs." (LINES 130-132) To clarify, the word "systematically" was used to describe the process of swab taken to clarify a process that is consistent and methodical.
10. Line 126-127: "All samples were transported in cooler boxes to the Liverpool School of Tropical Medicine (LSTM) and processed upon..." What is the shipment conditions and temperature in the cooler boxes?
Thank you for this comment, the samples were transported in cooler boxes with cool blocks for cold chain transport (+2C to +8C) during the transit to the laboratory (within 2h). The temperature in the cooler boxes was not monitored during the course of this study so we cannot comment on the exact temperature.
Reviewer #2 (Comments for the Author): ¬ 1-Give full names of all the abbreviations used first, then you can use abbreviations.
Thank you for this feedback. The manuscript has now been amended to include all full names before the use of abbreviations.
¬ The English language of the article also needs to be improved Thank you so much for your feedback, the manuscript has now been reviewed by a native English speaker and we have addressed the grammar mistakes and sentences that were unclear.
¬ Italicize all the "et al., or et al." and scientific words (line 105).
Thank you for highlighting this error. This has been amended.
¬ The discussion part needs a more detailed analysis of the results.
Thank you for this feedback, the discussion has been expanded to give a more detailed analysis of the results.