A multi-country phase 2 study to evaluate the suitcase lab for rapid detection of SARS-CoV-2 in seven Sub-Saharan African countries: Lessons from the field

Background The COVID-19 pandemic led to severe health systems collapse, as well as logistics and supply delivery shortages across sectors. Delivery of PCR related healthcare supplies continue to be hindered. There is the need for a rapid and accessible SARS-CoV-2 molecular detection method in low resource settings. Objectives To validate a novel isothermal amplification method for rapid detection of SARS-CoV-2 across seven sub-Sharan African countries. Study design In this multi-country phase 2 diagnostic study, 3,231 clinical samples in seven African sites were tested with two reverse transcription Recombinase-Aided Amplification (RT-RAA) assays (based on SARS-CoV-2 Nucleocapsid (N) gene and RNA-dependent RNA polymerase (RdRP) gene). The test was performed in a mobile suitcase laboratory within 15 min. All results were compared to a real-time RT-PCR assay. Extraction kits based on silica gel or magnetic beads were applied. Results Four sites demonstrated good to excellent agreement, while three sites showed fair to moderate results. The RdRP gene assay exhibited an overall PPV of 0.92 and a NPV of 0.88. The N gene assay exhibited an overall PPV of 0.93 and a NPV 0.88. The sensitivity of both RT-RAA assays varied depending on the sample Ct values. When comparing sensitivity between sites, values differed considerably. For high viral load samples, the RT-RAA assay sensitivity ranges were between 60.5 and 100% (RdRP assay) and 25 and 98.6 (N assay). Conclusion Overall, the RdRP based RT-RAA test showed the best assay accuracy. This study highlights the challenges of implementing rapid molecular assays in field conditions. Factors that are important for successful deployment across countries include the implementation of standardized operation procedures, in-person continuous training for staff, and enhanced quality control measures.


Keywords: SARS-CoV-2 Diagnostics-in-a-suitcase Recombinase polymerase amplification assay A B S T R A C T
Background: The COVID-19 pandemic led to severe health systems collapse, as well as logistics and supply delivery shortages across sectors.Delivery of PCR related healthcare supplies continue to be hindered.There is the need for a rapid and accessible SARS-CoV-2 molecular detection method in low resource settings.Objectives: To validate a novel isothermal amplification method for rapid detection of SARS-CoV-2 across seven sub-Sharan African countries.Study design: In this multi-country phase 2 diagnostic study, 3,231 clinical samples in seven African sites were tested with two reverse transcription Recombinase-Aided Amplification (RT-RAA) assays (based on SARS-CoV-2 Nucleocapsid (N) gene and RNA-dependent RNA polymerase (RdRP) gene).The test was performed in a mobile suitcase laboratory within 15 min.All results were compared to a real-time RT-PCR assay.Extraction kits based on silica gel or magnetic beads were applied.Results: Four sites demonstrated good to excellent agreement, while three sites showed fair to moderate results.The RdRP gene assay exhibited an overall PPV of 0.92 and a NPV of 0.88.The N gene assay exhibited an overall PPV of 0.93 and a NPV 0.88.The sensitivity of both RT-RAA assays varied depending on the sample Ct values.When comparing sensitivity between sites, values differed considerably.For high viral load samples, the RT-RAA assay sensitivity ranges were between 60.5 and 100% (RdRP assay) and 25 and 98.6 (N assay).Conclusion: Overall, the RdRP based RT-RAA test showed the best assay accuracy.This study highlights the challenges of implementing rapid molecular assays in field conditions.Factors that are important for successful deployment across countries include the implementation of standardized operation procedures, in-person continuous training for staff, and enhanced quality control measures.

Background
In March 2020, the World Health Organization (WHO) declared COVID-19 caused by severe acute respiratory syndrome coronavirus type 2 (SARS-CoV-2) as a global pandemic [1].Early detection of infected cases is still regarded as essential to reduce the disease burden [2].
Real-time reverse transcription polymerase chain reaction (RT-PCR) is the standard approach in terms of sensitivity and accuracy.However, this technique requires well-established laboratory.Additionally, supply and delivery shortages have been reported across various sectors [3,4].Antigen lateral flow tests were quickly developed and deployed, allowing carriers with high viral load to be diagnosed more easily.However, these tests often do not reach the WHO recommended minimum of >80% sensitivity and >97% specificity for new diagnostics tests [5].Deployment of a simple, rapid molecular test method could offer marked advantages.
Recombinase-Polymerase/aided Amplification (RPA/RAA) assays have been described as a rapid and effective nucleic acid amplification technique, due to its simplicity and fast sample-to-result test time [6].RPA/RAA is an isothermal probe-based nucleic acid detection method that neither requires template denaturation nor primer annealing steps [7].The proper selection of a polymerase working at low temperature (39-42 • C), the robustness of the assay, and compactness of the fluorescence detection device make RPA/RAA an optimal technique for molecular diagnosis at the point of need.To enable the widespread use of the technology, mobile suitcase laboratories were deployed to many sub-Saharan African countries [8].

Objectives
To further clinically evaluate the system in real-life settings, a multicountry single blinded phase 2 study was conducted in seven sub-Saharan African countries.The aim was not only to determine the accuracy of RAA assay for detection of SARS-CoV-2 in local African settings, but also to assess performance differences between research institutions.

Sample size calculation
The sample size was calculated using the formula for comparing two independent proportions, which is used to estimate sample size for studies comparing sensitivity and/or specificity of two tests of unpaired design [9].A minimum of 300 samples per site were required to achieve 95% confidence.

Study site and population
A multi-country, single blinded, phase 2 diagnostic evaluation study was conducted in seven sites: Institut Pasteur de Dakar (IPD), Senegal; Institut Pasteur de Madagascar (IPM), Madagascar; Kumasi Centre for Collaborative Research (KCCR), Ghana; University of Ibadan (UI), Nigeria; Institut National de Recherche Biomédicale (INRB), Democratic Republic of the Congo; University of Khartoum (UofK), Sudan; and Makerere University (MAK), Uganda.For the purpose of this study, a total of 3231 archived samples were used to evaluate the assay.Archived samples included in this study were from patients who tested positive for SARS-CoV-2 by the reference laboratory and patients who were suspected COVID-19 and tested negative by the reference laboratory.All samples were tested irrespective of age, sex, and race.During the study, all samples were handled anonymously.Samples included nasal, midturbinate swab or saliva in viral transport media (VTM), PBS or stored dry and maintained in − 80 • C. Additional information regarding the samples was included in supplementary file #1.

Study design
This study was conducted according to the guidelines for diagnostic kit evaluation (Fig. 1) [9].
Each country initially tested an inactivated test panel of 20 samples using RT-RAA and real-time RT-PCR reagents to assure their preparedness to perform the tests (Fig. 2).Once successful (at least 90% accurate detection rate), each country started clinical sample testing as follows: all samples were labelled with random numbers by the site principal investigator, and the laboratory personnel were divided into two teams: one team performed the real-time RT-PCR and the other performed the RT-RAA.Both teams were blinded and did not know whether the sample was from a positive or negative subject.Simultaneously, the three parts A. Ceruti et al. carbonless copy paper-based laboratory report forms (LRFs) were made available in two modes, one by the unblinded staff and the other by the blinded staff (supplementary file #2).All laboratory data were reported in LRFs by both teams separately.The data was then decoded by the data management team, where it was merged and statistically analysed.Patient information was not shared between study sites, only positivity and negativity rates were recorded.To ensure the quality of the study activities, periodic monitoring was performed.

Mobile suitcase laboratory set up
The total set up consisted of a Glove Box (Bodo Koennecke, Berlin, Germany) and a mobile suitcase laboratory (Fig. 3).The Glove Box protects the technician while handling and inactivating the sample before nucleic acid extraction.The isothermal amplification test was performed in the Mobile Suitcase Lab (Fig. 4).

Laboratory analysis 3.5.1. Real-time RT-PCR assay for detection of SARS-CoV-2 RNA
RNA was isolated from clinical samples of subjects, who were recently suspected of contracting COVID-19.At KCCR, UofK, UI and INRB, the QIAamp Viral RNA Mini Kit (Qiagen, Hilden, Germany) was used.At MAK, Liferiver Viral DNA/RNA isolation kit (Shanghai ZJ Biotech co. let, Shanghai, China) was used.At IPD, the Veri-Q PREP M16 Automatic Nucleic Acid Extraction System (MiCo BioMed, South Korea) was used.IPM used the NucleoSpin Dx Virus Mini kit (Macherey-Nagel GmbH, Düren, Germany).
All samples were originally tested as part of routine diagnostics and divided into positive and negative according to the locally used realtime RT-PCR results (supplementary file #2).To assure integrity of RNA on the day of testing, an additional real-time RT-PCR was performed.This result was considered as reference method (Fig. 2

RT-RAA assay for detection of SARS-COV-2 RNA
Two RT-RAA assays were evaluated: one based on the RdRP gene and one on the Nucleocapsid (N) gene.The primers, probe and reaction conditions for SARS-CoV-2 genes N and RdRP RT-RAA assays were  based on a previous study [8] Primer and probe were synthesized by TIB MOLBIOL (Berlin, Germany).The RT-RAA nucleic acid amplification kit (Fluorescent RT-RAA) from Jiangsu Qitian Gene Biotechnology Co. (Wuxi, China) was used.The kit comprises lyophilized enzymes, including the reverse transcriptase necessary for RNA amplification.The RT-RAA reaction total volume was 50 μL including 21.5 μL of the oligonucleotide mix (21 pMol for forward primer, 42 pMol for reverse primer and 6 pMol for exo-probe), 25 μL rehydration buffer, 2.5 μL Magnesium Acetate and 1 μL template or control.The mix was added into the lid of the reaction tube containing the freeze-dried pellet.The tube was closed, centrifuged, mixed, centrifuged, and placed immediately into the isothermal device: UofK and IPM used the TwistDx TS1 device (Cambridge, UK).UI used the Qiagen ESEquant TS2 model (Hilden, Germany), while all other countries used the Axxin T8 Isothermal instrument (Fairfield, Australia).The reaction was incubated at 42 • C for 15 min.A mixing step was conducted after 320 s for the N gene assay and after 230 s for the RdRP gene assay.For signal interpretation, a combined threshold time (TT) and first derivative analysis was used with the corresponding software of each device.

Data analysis
Standard formulas were used with MedCalc [10] to determine the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) [11].Cohen's kappa coefficient (k) and McNemar's test were performed to determine the concordance and discordance between the RT-RAA assays and real-time RT-PCR-based method.The values of Cohen's kappa coefficients were interpreted according to Landis and Koch [12] and calculated using the online Graphpad version.The McNemar test was calculated via https://epitools.ausvet.com.au/mcnemar.A P-value <0.05 was considered to be statistically significant.

Mutational analysis of the RdRP amplicon
Over 10,000 SARS-CoV-2 RNA sequences from local strains in the involved countries were screened for potential mutations in the targeted RdRP region.A previously published pipeline was used [13].

Performance of the deployed isothermal amplification assays
In total, 3231 samples were identified at study sites (Fig. 1) to validate the assays.After data curation, the N gene assay results were evaluated with a total of 1890 negative samples and 580 positive samples and the RdRP gene assay with a total of 2326 negative samples and 868 positive samples (supplementary file #3 displaying sample flowchart and raw data).The RdRP gene assay showed an overall PPV of 0.92 and a NPV of 0.88.The N gene assay showed an overall PPV of 0.93 and a NPV 0.88.
The sensitivity of both RT-RAA assays varied depending on the sample Ct values (Table 1).Real-time RT-PCR positive samples with high viral load (Ct <30) showed the best results with 90.8% and 81.8% overall sensitivities for the RdRP and N gene RT-RAA assays, respectively.When comparing sensitivity between sites, values differed considerably.For high viral load samples, the RdRP assay sensitivity ranged between 60.5 and 100%.The N gene did not perform as well as the RdRP gene, with a sensitivity between 25 and 98.6%.The specificity among all study sites ranged from 91.1 to 100% for the RdRP gene RT-RAA assay and from 94.3 to 100% for the N gene RT-RAA assay (Table 2).The overall sensitivity and specificity with both targets Fig. 3. Example of the suitcase lab, which is fully equipped to perform molecular tests in the field.combined was 58.2% and 99.2%, respectively.The agreement between test methods varied across sites (Table 3).Sites IPD, KCCR, UI and INRB showed good to excellent agreement, while sites IPM, UofK and MAK showed fair to moderate results.

Mutational analysis of the amplicon
No significant mutations were found in the RdRP amplicon across countries (supplementary files #4 and #5).

Discussion
The worldwide spread of SARS-CoV-2 has taken viral diagnostics to a new level of importance and publicity.Alternative methods to real-time RT-PCR with equal sensitivity and specificity are urgently needed to overcome shortage in supply chains [14].The WHO recommended the ASSURED criteria (affordable, sensitive, specific, user-friendly, robust, deliverable to end-users) for future diagnostics [15].Isothermal amplification assays address most of these criteria as shown in many outbreak situations [16][17][18].Although clinical research is rapidly progressing in the field of new diagnostic tests, multi-country approaches for SARS-CoV-2 are lacking, especially in sub-Saharan Africa [40].Only one multicenter study was reported using RT-loop mediated amplification in four African countries in both east and west Africa with very promising sensitivity of 87% [19].
Using the RT-RAA technology helped to circumvent the worldwide supply shortages of real-time RT-PCR test kits [20,21].The specificity for the two SARS-CoV-2 genome targets across sites ranged from 91.1 to 100% (RdRP gene RT-RAA) and 94.3 -100% (N gene RT-RAA).In contrast the SARS-CoV-2 an E gene target RT-RAA, showed a high number of false positive results in a previous study and was not included in the current screening [22].Detecting true negative samples accurately at high specificity avoids unnecessary clinical implications and social upset [23][24][25].SARS-CoV-2 Rapid antigen tests have shown a higher false positive rate (96-99.7%)[26], and especially in low prevalence settings a molecular confirmatory test is needed [27].
When deploying both RT-RAA assays in this study, sensitivity  showed a large range of intercountry variations.For high viral load samples, promising overall 90.8% (RdRP RT-RAA) and 81.8% (N RT-RAA) sensitivities were determined.For samples with Ct 31 -40, values were inconsistent between sites.While higher assay accuracy was identified at three sites, two sites did not produce the expected outcomes despite the success during the preparatory phase.Compared to the RdRP gene, the N-gene RT-RAA assay demonstrated lower sensitivity.The insufficient performance of N gene RT-RAA assay led to its exclusion for further testing to maximize usage of laboratory materials and resources, further underlining the importance of to unanticipated events during a large diagnostic study.Overall, the performance of the RT-RAA assays is much better than the commercially available rapid antigen tests, whose sensitivity values differed considerably with sensitivities ranging from only 28-86% (Ct 17 -36) [28,29].Rapid antigen tests are suited best for detection of symptomatic carriers with high viral load [30,31].Low viral load samples are often undetected as well as certain SARS-CoV-2 mutations [32,33].The RdRP gene RT-RAA assay showed promising but very variable sensitivity values across sites.Some sites showed more than 90% sensitivity and others under 50% even with high viral load samples.A potential mutation in the target region of RdRP primers, was excluded by screening over 10,000 SARS-CoV-2 RNA sequences from local strains in the involved countries (supplementary file #4 and #5) using a recently published screening method [13], identifying no significant changes.Reagents deterioration during transportation was unlikely, since RAA reagents are lyophilized, cold-chain independent, robust, and stable over long periods of time [6].In addition, a quality control check was conducted upon delivery of the kits to exclude this possibility.Clinical sample integrity is one of the factors for decreasing assay sensitivity especially since RNA is unstable [8].Degraded RNA or samples contaminated with RNases can lead to poor assay performance [20].In our study, all samples were tested with real-time RT-PCR and RT-RAA in a very short time window to assure sample integrity.Thus, it can be assumed that each sample had a similar viral load when tested with both methods.A limitation represents the use of different extraction kits and amplification devices across sites.This adds on the variability of the clinical settings of the study.It is difficult to standardize protocols and equipment across healthcare laboratories.Nonetheless, all devices and equipment used were approved to be used for in vitro diagnostics.Nucleic acid extraction remains the bottleneck of molecular diagnostics.In this study, standardized kits were used to ensure RNA quality to validate the isothermal amplification assays.However, to implement point-of-need molecular tests, extraction protocols need to be simplified and user-friendly.Different rapid methods have been described to extract SARS-CoV-2 RNA.Combinations of detergent, heat and magnetic beads can be used for quick extraction [34,35].RPA/RAA has been shown to be more tolerant against inhibitors in clinical samples [36][37][38].Thus, rapid extraction methods are feasible with this technology, as shown in various studies [39,40].However, further refinement is needed to enhance RNA purity and yield.In contrast, the RT-PCR is more intolerant to inhibitors from different matrices and requires highly purified RNA [41][42][43].Further large clinical studies are needed to combine both rapid extraction and SARS-CoV-2 molecular assays on site.
Surprisingly, one test site discovered an unusual cluster of discordance on certain days.After retesting those samples, better results were achieved.Deviations in performance of diagnostic tests have been attributed to sample quality/quantity, settings, and operators [22].The latter could hinder the homogeneity of sensitivity values between sites.Thus, correct sample handling is an essential factor to be considered.Although the influence of individual operators on the results of a diagnostic test cannot be fully avoided, certain actions could help to reduce such events.In this study, continuous and in-person training was not possible due to the COVID-19 pandemic and travel restrictions.Thus, the variability of the assay's results could be partially explained due to this difference in quality of training.Furthermore, attention to details regarding workflow should not be underestimated.For example, after careful troubleshooting, inaccuracies while transcribing records and test

Table 2
Clinical sensitivity and specificity at each study site.NA is not applicable as the site did not perform the assay.The 95% confidence interval is showed in parenthesis.results, or the influence of reduced concentration while working after a certain hour of the day were reported.As a consequence, standardized operations and in-person training for staff are of utmost importance before operating diagnostic samples, in addition to quality control checks [44].
The findings of this study provide evidence for the importance of the suitcase lab as a deployable and feasible setup for accurate, sensitive and specific pathogen detection, particularly in low-resource settings.Furthermore, the RT-RAA method, especially based on the RdRP gene, is a promising on-site detection method for SARS-CoV-2 infection, overall showing higher accuracy than commercially available rapid antigen test and bypassing supply shortages.However, variations in assay sensitivity between sites revealed the importance of quality control and face to face training for the staff.The influence of global and regional disruptions should not be underestimated in large multi-country diagnostic trials.Additionally, sample handling by staff was regarded as a bottleneck for test performance.Continuous in-person training is an essential tool for successful diagnostic testing, in case of excellent quality and the quantity of kits and devices.These lessons learned should be considered when planning and performing large multi-country diagnostic clinical trials in poor resource settings.

Fig. 2 .
Fig. 2. Workflow of the performed tests at the study sites.The new real-time RT-PCR performed was used as the reference method to validate the RT-RAA assays.

Table 1
Overall clinical sensitivity of all samples across sites categorized according to the Ct values of real-time RT-PCR.RT-PCR Ct range