Prospective study of three saliva qualitative antigen testing kits for the detection of SARS-CoV-2 among mainly symptomatic patients in Japan

Introduction Rapid qualitative antigen testing has been widely used for the laboratory diagnosis of COVID-19 with nasopharyngeal samples. Saliva samples have been used as alternative samples, but the analytical performance of those samples for qualitative antigen testing has not been sufficiently evaluated. Methods A prospective observational study evaluated the analytical performance of three In Vitro Diagnostics (IVD) approved COVID-19 rapid antigen detection kits for saliva between June 2022 and July 2022 in Japan using real-time reverse transcription polymerase chain reaction (RT-qPCR) as a reference. A nasopharyngeal sample and a saliva sample were simultaneously obtained, and RT-qPCR was performed. Results In total, saliva samples and nasopharyngeal samples were collected from 471 individuals (RT-qPCR-positive, n = 145) for the analysis. Of these, 96.6% were symptomatic. The median copy numbers were 1.7 × 106 copies/mL for saliva samples and 1.2 × 108 copies/mL for nasopharyngeal samples (p < 0.001). Compared with the reference, the sensitivity and specificity were 44.8% and 99.7% for ImunoAce SARS-CoV-2 Saliva, 57.2% and 99.1% for Espline SARS-CoV-2 N, and 60.0% and 99.1% for QuickChaser Auto SARS-CoV-2, respectively. The sensitivities of all antigen testing kit were 100% for saliva samples with a high viral load (>107 copies/mL), whereas the sensitivities were <70% for high-viral-load nasopharyngeal samples (>107 copies/mL). Conclusion COVID-19 rapid antigen detection kits with saliva showed high specificity, but the sensitivity varied among kits, and were also insufficient for the detection of symptomatic COVID-19 patients.


Introduction
The proper diagnosis of COVID-19 is critical for infection control, and the gold-standard test for such a diagnosis is a nucleic acid amplification test with reverse transcription polymerase chain reaction (RT-qPCR) using nasopharyngeal (NP) specimens [1]. However, it can take hours to receive results from RT-qPCR after sample submission, and the specimen collection procedure of NP samples requires special handling by healthcare professionals and induces significant discomfort in the patient as well as coughing and sneezing [2]. This limits its application in household and community settings [3].
Qualitative antigen tests, which have an easy-to-perform specimen-handling procedure, wide availability, short performance time, have been developed as an alternative to RT-qPCR [4], and NP samples and anterior nasal samples have been used for testing in both symptomatic and asymptomatic patients [5]. Saliva samples have also been widely used as samples for RT-qPCR and quantitative antigen tests [6], but their diagnostic performance has been considered insufficient for qualitative antigen tests [7]. In 2022, several saliva qualitative antigen detection kits were newly developed, and In Vitro Diagnostics (IVD) approval was given in Japan. However, their analytical performance was evaluated only by the manufacturers. We therefore conducted prospective evaluations of three IVD-approved saliva antigen qualitative testing kits.

Method
This study was performed with samples submitted by both symptomatic and asymptomatic patients between June 8, 2022, and July 12, 2022, at a drive-through PCR center at Tsukuba Medical Center Hospital (TMCH), which intensively performed COVID-19 PCR evaluations with NP samples or saliva samples in the Tsukuba district of Tsukuba, Ibaraki Prefecture, Japan. People with and without symptoms were referred from 49 clinics and a local public health center during the study period. Asymptomatic individuals had a history of contact with confirmed or suspected COVID-19 cases.
All evaluations were performed after informed consent was obtained. The informed consent process was performed verbally with documentation in the patient's electronic medical record in order to prevent infection transmission. The ethics board of TMCH (approval number:2021-055) approved the protocol.

Study process
NP and saliva samples were simultaneously obtained from individuals. The sample collection was performed as previously described [2,[8][9][10][11][12][13][14][15][16][17][18][19]. All antigen tests for saliva were immediately performed on site after sample collection. Each antigen test was performed based on the manufacturer's instructions in the package insert ( Fig. 1). In cases with a poor control response, a re-test was performed. After the antigen evaluation, all saliva samples were preserved at − 80 • C until reference RT-qPCR.
Each NP swab was diluted in 3 mL of Universal Transport Medium (Copan Italia S.p.A., Brescia, Italy) on site, and the sample was transferred to the TMCH microbiology department for in-house RT-qPCR. After in-house RT-qPCR, each sample was preserved at − 80 • C along with saliva samples.
Reference RT-qPCR was performed using a method developed by the National Institute of Infectious Diseases (NIID), Japan, for SARS-CoV-2 [20,21] with purified samples with magLEAD (Precision System Science Co., Ltd., Chiba, Japan). A 200-μL aliquot of each sample was extracted, and 100 μL of purified sample was eluted. For saliva samples, samples were diluted 1:2 with phosphate-buffered saline (PBS) 1x with vortex mixing and then centrifuged for 3 min at 13,000×g, and the supernatant was used as the sample. For RT-qPCR, 5 μL of the extracted RNA was used for one-step quantitative RT-qPCR with the THUNDER-BIRD® Probe One-step RT-qPCR kit (TOYOBO Co., Ltd.) and the Light-Cycler® 96 Real-time PCR System (Roche Diagnostics KK, Basel, Switzerland). A duplicate analysis for N2 genes was performed for the evaluation of SARS-CoV-2. EDX SARS-CoV-2 Standard (Bio-Rad Laboratories, Inc., Hercules, CA, USA) and sterile purified water (Merck & Co., Inc., Kenilworth, NJ, USA) were used as positive and negative controls, respectively. The calibration curves were generated with 5, 50, and 500 copies/reaction of EDX SARS-CoV-2 Standard.

Statistical analyses of the rapid antigen tests
Patient with SARS-CoV-2 detected by RT-qPCR in either nasopharyngeal swab or saliva was defined as COVID-19 patient. The sensitivity and specificity of the antigen tests were calculated with 95% confidence interval (CI). The positive concordance rates stratified by the copy number based on the N2 set of the NIID method were also evaluated. The copy numbers, according to sample type were compared by Wilcoxon's signed-rank test.

Results
In total, saliva samples and NP samples were collected from 471 individuals during the study period; 455 were from symptomatic individuals, and 16 were from asymptomatic individuals. In this study, both nasopharyngeal samples and saliva samples were successfully obtained from all individuals. No cases were excluded due to a sample volume that was insufficient for evaluation. Of the simultaneously obtained saliva samples and NP samples, 140 saliva samples and 143 NP samples were SARS-CoV-2-positive by RT-qPCR with the NIID method. The median copy number was 1.7 × 10 6 copies/mL (interquartile range [IQR]: 1.4 × 10 5 -1.8 × 10 7 copies/mL) for saliva samples and 1.2 × 10 8 copies/mL (IQR: 3.9 × 10 7 -3.2 × 10 8 copies/mL) for NP samples (p < 0.001). The copy number of the saliva samples and NP samples are shown in Fig. 2. Both the saliva and NP samples were positive in 138 individuals, whereas saliva samples were positive and NP samples were negative in 2 individuals, while saliva samples were negative and NP samples were positive in 5 individuals. Finally, 145 SARS-CoV-2positive patients (30.8%) were identified in this study.
The analytical performance of the three antigen testing kits for the detection of SARS-CoV-2 in saliva among all individuals and symptomatic patients is described in Table 1-A and Table 1-B. For all individuals (Table 1-A), the sensitivity and specificity were 44.8% and 99.7%, respectively, for ImunoAce SARS-CoV-2 Saliva, 57.2% and 99.1% for Espline SARS-CoV-2 N, and 60.0% and 99.1% for QuickChaser Auto SARS-CoV-2. Espline SARS-CoV-2 N required re-tests for 3 samples due to non-reactivities for the positive control line after 20 min. There were no other re-tests performed during the study. The results did not differ from those among symptomatic individuals (Table 1-B). The comparison of the antigen testing kits with each RT-qPCR for nasopharyngeal samples and saliva samples is summarized in Supplementary Table 2 and Table 3.
The sensitivity stratified by the time (in days) after the onset of symptoms is shown in Supplementary Fig. 1. For day 0-2 samples, the sensitivity was 48.7% for ImunoAce SARS-CoV-2 Saliva, 60.5% for Espline SARS-CoV-2 N, and 63.9% for QuickChaser Auto SARS-CoV-2. Meanwhile, the sensitivities of all antigen testing kits were 0% for day 6-10 samples.
The positive concordance rates of the antigen testing kits for the detection of SARS-CoV-2 in saliva stratified by the copy numbers of saliva samples are shown in Table 2. For copy numbers of >10 7 copies/ mL, 10 6 -10 7 copies/mL, 10 5 -10 6 copies/mL, and <10 5 copies/mL, the positive concordance rates for ImunoAce SARS-CoV-2 Saliva were 100%, 50%, 13.2%, and 3.7%, respectively; the positive concordance rates for Espline SARS-CoV-2 N were 100%, 78.1%, 36.8%, and 3.7%, respectively; and the positive concordance rates for QuickChaser Auto SARS-CoV-2 were 100%, 81.3%, 42.1%, and 3.7%. The positive concordance rates of the antigen testing kits for the detection of SARS-CoV-2 in saliva stratified by the copy numbers of nasopharyngeal samples are also shown in Table 3. For a copy number of >10 7 , the positive concordance rates were 50.8% for ImunoAce SARS-CoV-2 Saliva, 64.9% for Espline SARS-CoV-2 N and 66.4% for QuickChaser Auto SARS-CoV-2 while the positive concordance rates were 0% for ImunoAce SARS-CoV-2 Saliva, 16.7% for Espline SARS-CoV-2 N and 16.7% for QuickChaser Auto SARS-CoV-2 for a copy number of 10 6 -10 7 . The positive concordance rates of the antigen testing kits for the detection of SARS-CoV-2 in saliva stratified by the Ct values of saliva samples and nasopharyngeal samples are listed in Supplementary Tables 4 and 5 4

. Discussion
The current study revealed that each COVID-19 antigen detection kit for saliva had good specificity with infrequent false-positive findings; however, the sensitivity varied among the kits and was insufficient for the detection of symptomatic COVID-19 patients, probably due to the lower viral loads in saliva samples than in NP samples. In this study, high viral loads of SARS-CoV-2 were detected in NP samples in most COVID-19-positive patients, but there were many false-negative results with the antigen detection kits for saliva samples. The sensitivity is >100% for saliva samples with a high viral load (>10 7 copies/mL), whereas the sensitivity is <70% for high-viral-load nasopharyngeal samples (>10 7 copies/mL).
While the sensitivity of RT-qPCR for the laboratory diagnosis of COVID-19 is similar between NP samples and saliva samples [3,22] including for omicron variants [23], the results of evaluations of saliva qualitative antigen test have not been favorable. Chen et al. performed a systematic review of qualitative antigen tests with saliva in 2022 and found that the pooled sensitivity was 27.4% (95% CI: 8.1%-61.9%), and the pooled specificity was 100% (95% CI: 93.8%-100%) (n = 1536), which were significantly lower than those values for quantitative antigen tests (sensitivity: 85.6% [95% CI: 69.2%-94%], specificity 98.9% [95% CI: 94.5%-99.8%]). Yokota et al. evaluated the first-generation Espline kit with 34 frozen positive saliva samples and reported that only 14 samples (41%) were positive [24]. Ishii et al. reported that the sensitivity was 33% (3/9) and the specificity 100% (84/84) [25]. The SD Biosensor saliva antigen rapid test, which is not approved in Japan, was reported to have a better diagnostic performance than other antigen tests in a study of 789 saliva samples. Indeed, Igloi et al. reported that the sensitivity and specificity of the SD Biosensor saliva antigen rapid test were 66.1% and 99.6%, respectively, and the sensitivity increased to 88.6% with a Ct ≤ 30 cut-off [26].
In the current study, all three evaluated antigen detection kits were newly released and had been adjusted for detection with saliva samples. All of the kits were able to detect SARS-CoV-2 in high-viral-load saliva samples (>10 7 copies/mL), and Espline SARS-CoV-2 N and QuickChaser Auto SARS-CoV-2 showed better sensitivity in moderate-viral-load samples (10 6 -10 7 copies/mL) than ImunoAce SARS-CoV-2 Saliva. According to the copy number-stratified results, the diagnostic performance of Espline SARS-CoV-2 N and QuickChaser Auto SARS-CoV-2 appears similar to that of the SD Biosensor saliva antigen rapid test [26].
Of note, false-negative results were frequently observed despite the high viral loads (>10 7 copies/mL) in nasopharyngeal samples. In this study, ImunoAce SARS-CoV-2 Saliva missed positive results in approximately half of those high viral load samples. The other kits showed false-negative results in one-third of the cases. Our recent study showed that the sensitivity of qualitative antigen testing with nasopharyngeal samples was >90% in a similar situation, which was performed several months earlier than the current study for omicron-variant SARS-CoV-2 [27]. Saliva samples for qualitative antigen detection kits that can be used at home as initial screening because they can be obtained less invasively and self-collected. However, saliva samples may not be suitable for situations requiring high diagnostic performance, such as acute care facilities.
For Espline SARS-CoV-2 N, Murakami et al. [28] reported the analytical performance with 60 positive samples and 60 negative samples obtained in 2022, and the sensitivity and specificity were 58.8% (95% CI: 44.2%-72.4%) and 100.0% (95% CI: 94.0%-100.0%). They reported that the sensitivity was 69.8% (95% CI: 53.9%-82.8%) for Ct < 30, 92.9% (95% CI: 76.5%-99.1%) for Ct < 27, and 100% (95% CI: 80.5%-100%) for Ct < 25, which was similar to the current results. The deterioration of sensitivity of qualitative antigen detection kits using saliva samples was considered to be due to the difference in viral loads and sample characteristics between NP samples and saliva samples [23]. As another factor, the superior performance of nasopharyngeal RT-qPCR, reverse transcription-polymerase chain reaction; Ag, antigen testing. Data in parentheses indicate 95% confidence interval. a RT-qPCR of SARS-CoV-2 developed by the National Institute of Infectious Diseases, Japan(21)with nasopharyngeal samples and saliva samples were used as the reference standard. An RT-qPCR result was considered positive if SARS-CoV-2 was detected in either a nasopharyngeal sample or a saliva sample.  Data in parentheses indicate 95% confidence interval. The copy number was determined using a duplicated analysis of the National Institute of Infectious Diseases (N2 gene), Japan method (21). The calibration curves were generated with 5, 50, and 500 copies/reaction of EDX SARS-CoV-2 Standard. Data in parentheses indicate 95% confidence interval. The copy number was determined using a duplicated analysis of the National Institute of Infectious Diseases (N2 gene), Japan method (21). The calibration curves were generated with 5, 50, and 500 copies/reaction of EDX SARS-CoV-2 Standard.
qualitative antigen testing can be attributed to the effect of sample dilution. While saliva samples are only diluted two or three folds by solution buffer for RT-qPCR, the nasopharyngeal samples obtained for RT-qPCR are diluted with a large volume of solution buffer (1-3 mL). Nasopharyngeal swabs for qualitative antigen testing can directly use the sample after the suspension of the swab in antigen solution buffer, which results in a difference of viral loads, with the viral load in the same volume of sample being approximately 1 log copy number higher in comparison to that in a nasopharyngeal swab for RT-qPCR. Several limitations associated with the present study warrant mention. First, the samples were collected at one site in Japan, and most samples were collected from symptomatic patients soon after the symptom onset. Second, the sample size for asymptomatic individuals was insufficient in this study. While there is debate regarding differences in viral shedding in symptomatic and asymptomatic patients [29], the sensitivity of qualitative antigen testing is clearly lower in asymptomatic patients than in symptomatic patients [5]; thus, the sensitivity of COVID-19 rapid antigen detection kits using saliva samples may be worse in populations with high rates of asymptomatic individuals. Third, most of the positive samples were obtained from individuals with high viral loads in the NP, so the sensitivity might be lower at other PCR centers.
In conclusion, the current study showed that COVID-19 rapid antigen detection kits with saliva showed high specificities, but the sensitivity varied among kits, and was insufficient for the detection of symptomatic COVID-19 patients.

Declaration of competing interest
FUJIREBIO, Inc., and MIZUHO MEDY Co., Ltd., provided funds for research expenses and antigen kits without charge. Hiromichi Suzuki received advisory fees from MIZUHO MEDY Co., Ltd. The other authors have no conflicts of interest.