https://orcid.org/0000-0002-6646-1868
Figures
Abstract
Background
COVID-19 is an ongoing public health pandemic regardless of the countless efforts made by various actors. Quality diagnostic tests are important for early detection and control. Notably, several commercially available one step RT-PCR based assays have been recommended by the WHO. Yet, their analytic and diagnostic performances have not been well documented in resource-limited settings. Hence, this study aimed to evaluate the diagnostic sensitivities and specificities of three commercially available one step reverse transcriptase-polymerase chain reaction (RT-PCR) assays in Ethiopia in clinical setting.
Methods
A cross-sectional study was conducted from April to June, 2021 on 279 respiratory swabs originating from community surveillance, contact cases and suspect cases. RNA was extracted using manual extraction method. Master-mix preparation, amplification and result interpretation was done as per the respective manufacturer. Agreements between RT-PCRs were analyzed using kappa values. Bayesian latent class models (BLCM) were fitted to obtain reliable estimates of diagnostic sensitivities, specificities of the three assays and prevalence in the absence of a true gold standard.
Results
Among the 279 respiratory samples, 50(18%), 59(21.2%), and 69(24.7%) were tested positive by TIB, Da An, and BGI assays, respectively. Moderate to substantial level of agreement was reported among the three assays with kappa value between 0 .55 and 0.72. Based on the BLCM relatively high specificities (95% CI) of 0.991(0.973–1.000), 0.961(0.930–0.991) and 0.916(0.875–0.952) and considerably lower sensitivities with 0.813(0.658–0.938), 0.836(0.712–0.940) and 0.810(0.687–0.920) for TIB MOLBIOL, Da An and BGI respectively were found.
Conclusions
While all the three RT-PCR assays displayed comparable sensitivities, the specificities of TIB MOLBIOL and Da An were considerably higher than BGI. These results help adjust the apparent prevalence determined by the three RT-PCRs and thus support public health decisions in resource limited settings and consider alternatives as per their prioritization matrix.
Citation: Sisay A, Abera A, Dufera B, Endrias T, Tasew G, Tesfaye A, et al. (2022) Diagnostic accuracy of three commercially available one step RT-PCR assays for the detection of SARS-CoV-2 in resource limited settings. PLoS ONE 17(1): e0262178. https://doi.org/10.1371/journal.pone.0262178
Editor: Basant Giri, Kathmandu Institute of Applied Sciences, NEPAL
Received: November 4, 2021; Accepted: December 17, 2021; Published: January 20, 2022
Copyright: © 2022 Sisay et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files
Funding: This work was financed by Addis Ababa University through adaptive research and problem solving project, Ref #-PR/5.15/590/12/20. However, the funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The contents are purely the responsibilities of the authors and did not represent and reflect the view of the funder. There was no additional external funding received for this study.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: COVID-19, Corona virus disease; Ct-value, Cycle threshold; Da An, Da An Gene Co., Ltd. of Sun Yat-sen University; DNA, Deoxyribonucleic acid; NNAF, Nucleic Acid Amplification Test; ORF, Open reading frame; PCR, Polymerase chain reaction; RdRp, RNA-dependent RNA polymerase; RNA, Ribonucleic Acid; RT-PCR, Reverse transcriptase polymerase chain reaction; SARS-CoV-2, Sever acute respiratory syndrome corona virus; VTM, Viral transport Medium; WHO, World Health Organization
Introduction
Coronavirus disease 2019 (COVID-19) is still an ongoing global pandemic and a public health emergency resulting in more than 4,891,942 deaths from 240,061,637 documented cases as of 13 October, 2021 the first reported case in China, Wuhan in December 2019 [1,2].
The global community had three epidemic surges of Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) causing COVID-19. This has imposed an immediate increasing demand on daily diagnostic screening, with the aim to shorten the epidemic curve and to prevent the occurrence of new variants of concern and interests, which could lead the next wave as has, occurred in some parts of the globe [3,4]. It is a pushing factor and time for designing and a critical need for accurate and rapid diagnostic assays to prompt clinical and public health interventions. This can help to perpetuate over the upcoming time, due to the recurrence of outbreaks [5]. One step reverse transcriptase PCR is a popular diagnostic technique for novel Corona virus ever since the first case of COVID-19 reported elsewhere [6,7].
In Africa, including Ethiopia, COVID-19 laboratory testing facilities are centralized, that have been managed in a referral linkage of a limited access to laboratory facilities, by which there is a possibility of overloading and flooding of the already overwhelmed reference laboratories during emergency conditions. Availability, low cost and user friendliness are highly desirable in Ethiopia. While high sensitivity is of paramount importance not to miss any COVID-19 case and stop virus transmission [8], diagnostic specificity is highly relevant when tests are used as confirmatory and to avoid the costs of false positive results. In most COVID-19 diagnostic tests accuracy studies, RT-PCR is considered as the gold standard, i.e. having 100% diagnostic sensitivity and specificity. Despite the tremendous worldwide research activities related to COVID-19 diagnostic tests, very few diagnostic test accuracy studies were conducted to adjust imperfect tests [9–13]. So far, very few attempts were made to assess the diagnostic performance of RT-PCRs in Ethiopia in a clinical setting.
COVID-19 pandemic in Ethiopia
Ethiopia is the second most populous country in Africa. It is one of the 220 COVID-19 affected countries and territories, since the first confirmed COVID-19 was reported on March 13, 2020 [14]. Presently case numbers are increasing rapidly. There were two peaks of kurtosis of the epidemic; first wave from August to September 2020, and second wave occurred during February to March 2021. There are reports of Delta variants in parts of Africa including Ethiopia which might have resulted in the third wave [15,16]. As of 13 October 2021 there are 356,772 confirmed cases which make Ethiopia the 4th most affected country in Africa [15] with 6,103 documented deaths. A total of 3329 health professionals have contracted the disease and 36 of them have died [15,17].
PCR-based laboratory testing is indispensable in identifying the infected cases and also curbs the transmission epidemic. In Ethiopia, so far there are 3,574,218 total tests using RT-PCR, which equates to 30,136 tests per million population, since the first COVID-19 case was reported and the subsequent declaration of global pandemic [15,17]. Even though the country has come a long way in increasing the testing capacity through redirecting and mobilizing resources and organizing testing campaigns, the tests per the total population are still very low and need due attention [18,19].
Available diagnostic platforms for the diagnosis of COVID-19
Since the occurrence of the current pandemic: Severe Acute Respiratory Syndrome Coronavirus-2 (SARS-CoV-2) infection on December, 2019 in Wuhan, China, Scientist have been developing diagnostic tools with different diagnostic tests, relying on different biological principles [20].
Detection methods based on nucleic acid amplification tests (NAAT) are popular and preferable for novel Coronavirus in many countries. The test demonstrated highest sensitivity when it performed at the earliest time point in the acute phase of infection. The current reference or gold standard for diagnosing COVID-19 depends up on detection of the viral genetic material (RNA) in a nasopharyngeal swab or sputum sample., it requires PCR, a technology that amplifies the amount of genetic material to detectable levels and takes several hours to perform [21].
One-step reverse transcriptase-polymerase chain reaction (RT-PCR) is the established reference standard test for the diagnosis of COVID-19 because of its performance in identifying SARS-CoV-2 infection. This technique is sensitive and can detect the virus earlier in the infection [21]. Currently, there are different RT-PCR assays with quite different analytical and diagnostic sensitivity and specificity have been developed and recommended by the WHO and used for the diagnoses of SARS-CoV-2 [22,23].
Evidence indicated that, in many countries all available test kits have been used including rapid diagnostic tests that detects the virus either an antibody or an antigen based, whereas Ethiopia uses both RT-PCR and Ag RDT for the diagnosis of SARS CoV- 2 [8,18,24]. Indeed, evidence based information on the diagnostic performance of the commercially available test assays in the country and currently used by most COVID-19 testing laboratories are critical and timely for the program managers and have its impact in the containment strategy of the virus through quality test kit selection on their prioritization and continuous quality improvements on its development as part of the control of the pandemic [9].
In view of this, many pharmaceutical companies have accomplished different activities across the world as part of the pandemic response, which can benefit synergetic modality. However, so far because of its analytical and diagnostic presentation, RT-PCR is the reference method for the detection of SARS-CoV-2, irrespective of their variability in performance [25]. As of September 11, 2021 there are 1129 SARS-CoV-2 tests that are commercially available or in development for the diagnosis of COVID-19 and evaluated through FIND in collaboration with WHO and partners. FIND has been conducting independent evaluation of the available assays and summarized data for emergency use authorization (EUA) [23].
Testing for SARS–CoV-2 is becoming increasingly available in high-income countries and though still scant in resource limited countries, and even when available, it is highly centralized and limited to more specialized institutions with greater supply chain bottlenecks [26].
Reverse transcriptase-PCR test is usually done to identify positive cases in a population to prevent transmission to others and monitoring patient outcome. Likewise, it is critical importance for the mitigation of the pandemic. This is very much pondering if and only if the laboratory used better comparable quality diagnostic assays [27–30].
In Ethiopia, there are considerable numbers of one step RT-PCR assays commercially available for the diagnosis of COVID-19 either by donation or direct procurement by government. Among the available assays, three of them Sarbeco virus SARS-CoV-2 TIB MOLBIOL (hereafter named as TIB), Da An and BGI are widely used for the diagnosis of SARS CoV-2 at different time. TIB assay has two targets, E and RdRp involving two steps. The first step is screening by the E gene and the second step is confirming all positive cases by the RdRp gene. Da An assay works by multiplexing and targets two viral genes (nucleocapsid (N) and the ORF 1ab). The BGI assay targets only one viral gene the ORF 1ab. Each assay has its own procedural and result interpretation steps. However, their performance in a clinical setting in Ethiopia has not been assessed.
As PCR is not considered as gold standard our main scientific hypothesis is the assessment of comparing the ct levels and then the demographic data, and finally assessing agreement between the PCRs and obtaining robust estimates for diagnostic sensitivities and specificities in the less developed countries.Therefore, the current study was aimed to evaluate the diagnostic sensitivities and specificities of three commercially available one step reverse transcriptase assays for the detection of SARS-CoV-2 in resource limited settings, Ethiopia using Bayesian latent class model (BLCM).
Materials and methods
Study design, period and settings
A health facility based cross sectional study was conducted at Parasitology COVID-19 testing laboratory of Ethiopian Public Health Institute (EPHI) from April to June 2021. Nasopharyngeal swabs were collected from 279 individuals referred for COVID-19 testing from health facilities of Addis Ababa. The laboratory is amongst the first established national COVID-19 testing laboratory with qualified and competent performance which has been serving COVID -19 testing for samples come from referring health facilities of Addis Ababa through the national emergency operation center (EOC) [18].
Flow of study participants.
In this study a total of 468 eligible participants who fulfill the WHO criteria for COVID-19 suspected cases were screened [2]. Of these a total of 279 nasopharyngeal swabs collected by trained health professionals who encompass from community surveillance consider them as population one (32.62%), contacts of confirmed cases, consider as population two) (17.2%), and suspects, consider them as population 3 (50.18%). The detail is depicted at diagram 1 in S1 Annex.
Test kits selection.
Reverse transcriptase-PCR test has been designed based by targeting some parts of SARS-COV-2: spike (S), open reading frame1a/b (ORF1a/b), ORF1b-nuclear shuttle protein14 (ORF1b–nsp14)), envelope (E), RNA-dependent RNA polymerase (RdRp) or nucleocapsid (N) genes from the structural and nonstructural part of the gene [27]. Accordingly, the current study evaluated test kits employed E gene, RdRp, N gene and ORF as illustrated in Table 1. However, their respective pharmaceutical companies had not had any involvement with the research methodology design, analysis and write up of the research manuscript.
Laboratory procedures.
RNA purification. Respiratory specimens (throat/nasopharyngeal) collected in 2 mL VTM (China, Miraclean Technology Co.,Ltd., www.mantacc.com) by well trained professionals were used for RNA purification. Briefly, 200 μL of respiratory samples were transferred into 1.5 ml eppendorf tube. Then, 50 μl of proteinase K and 200 μl of lysis buffer were added after brief centrifugation and incubation of the tube at 72°C for 10 minutes. 250 μl of absolute alcohol was added and the whole mixture was transferred to spine column. The remaining procedures were performed as per the Da An RNA/DNA purification kit insert (Da An Gene Co., Ltd. of Sun Yat-sen University). In all extraction procedures, positive and negative controls were included.
Detection of SARS-CoV-2 by RT-PCR.
Following the Viral RNA extraction the extracted samples were detected by the three (TIB, Da An and BGI) commercially available RT-PCR assays against their respective manufacturer recommendations and instructions on a Quant Studio 5 DX real-time PCR system (catalog number A34322, Thermo Fisher Scientific), and data were analyzed and computed. Master Mix preparation and cycling conditions for respective assays presented in Table 2.
Determination of results.
For the TIB assay, a 76 bp long fragment from the E gene is amplified with specific primers and detected with a FAM label hydrolysis probe. The assay detects SARS and 2019-nCoV pneumonia virus (bat-associated SARS related Sarbeco virus has its own control reaction of 70 bp fragment from Equine Arteritis Virus detected with an Atto647 labeled probe. It’s RNA positive control contains all diagnostic targets E gene, N gene and RdRP. Mainly it had FAM Color Module 1 and Cy5: Color module 5 (IC) by RdRp also FAM: Color Module 1 and Cy5: Color module 5 (IC) by E gene. It has two steps to declare positive: screening test done by E gene and if any clinical sample positive by the E-gene assays are subject to repeat and confirmatory tests done by the RdRp gene-specific to SAR-CoV-2 [35].
For Da An assay, if the test sample has no amplification curve in the FAM and VIC channels, but amplification curve in the Cy5 channel, the result can be judged as there is no 2019 Novel Coronavirus (2019-nCoV) RNA in the sample; If the test sample has obvious amplification curve in the FAM and VIC channels and Ct values ≤40, the result can be judged as the sample is positive for 2019 novel coronavirus; if the test sample only has the ct value ≤40 in a single channel of FAM or VIC and there is amplification curve in Cy5 channel, the results need to be re-tested. If the re-test results are consistent, the sample can be judged as positive for 2019 Novel Coronavirus (2019-nCoV). If the re-test results are negative, it can be judged that no 2019 Novel Coronavirus (2019-nCoV) RNA has been detected. The ROC curve method is used to determine both the reference Ct value of the kit and the internal standard reference values are 40 for positive value judgment [36].
Interpretation of test results. in each test procedure, negative, positive and no template controls were included. Only when the controls meet the requirements can the test results be determined. When the FAM and VIC detection channels are positive, the result from the Cy5 channel (internal standard channel) may be negative due to the competition of the system and when the internal standard result is negative, if the FAM and VC detection channels of the test tube are also negative, it indicates that the system is inhibited or the operation is wrong, the test is invalid. Therefore, the sample needs to be re-tested [36].
In case of BGI, the controls for the Real-Time Fluorescent RT-PCR Kit for Detecting SARS-2019-nCoV are evaluated using the nucleic acid amplification curve and Ct values generated by the RT-PCR system. Ct cut-off values were determined using the receiver operator characteristic curves of the tested clinical samples. The Ct value in the FAM channel for a valid no template (negative) control should be “0” and there should be no sigmoidal amplification curve. Experimental analysis found that Ct value for positive SARS-CoV-2 samples should be no higher than 38. Thus, the Ct value in the FAM channel for a valid positive control should be no higher than 38 and there should be a sigmoidal amplification curve. Experimental analysis found that Ct value for the internal positive control samples should be no higher than 35. Thus, the Ct value in the VIC/HEX channel for a valid internal positive control should be no higher than 35 and there should be a sigmoidal amplification curve [37].
Data quality assurance.
Sample and data was collected by well trained professionals. All laboratory procedures were performed as per the documented standard operating procedures and according to specific manufacturing recommendations. The quality of each reagent like lot number, expire date, storage conditions, physical leak proof and not breakages, etc were cheeked before the actual laboratory analysis. Samples and reagents were stored at appropriate temperature as indicted on the manufacturer inserts. Internal and external quality controls were run as required during laboratory analysis.
Data was double checked manually for completeness and consistency before data entry. We have used a respective recommended manufacturer protocols. The above requirements must be met at the same time in each experiment; otherwise, the experiment is invalid and needs to be carried out again.
Data analysis.
For the best narrative, only results findings by three RT-PCR tests were analyzed and considered for our further analysis and interpretation, the rest were trimmed from the final analysis. Results were interpreted as positive and negative based on the cut-off Ct values of each manufacturer. The binomial 95% confidence intervals (95% CI) were calculated using MedCalc (MedCalc Software Ltd.) and P-values <0.05 considered statistical significant using Vassarstats and SPSS v.23 statistical software Cohen’s kappa statistic among the three assays. A value of 1 implies almost perfect agreement and values less than 1 imply less than perfect agreement, with a range of values between 0 and 1 [38].
Diagnostic sensitivity, specificity, predictive and kappa values were computed. Positive and negative predictive values were computed following.
Bayesian latent class model (BLCM).
With the aim to obtain diagnostic tests accuracies in the absence of a perfect gold standard, a Bayesian latent class model (BLCM) was fit to the data following the approach from Hui and Walter for three tests and three populations using MCMC (Markov chain Monte Carlo) simulation to construct posteriors in JAGS version 4.3.0 [39] using the runjags package [40].
The frequencies of the eight combinations of dichotomized RT-PCR results (+++; ++-; +-+;+—;-++,-+-,—+,—) in the three populations, respectively, were modeled with a multinomial distribution. To allow for potential conditional dependencies, pair wise covariances between sensitivities and specificities of all RT-PCRs were included in separate models. Model selection, i.e. in- or exclusion of conditional dependencies was based on the 95% credibility intervals (including 0 or not) and on Bayesian p-values. The model code (S1 File) was obtained with the function “auto huiwalter” of the runjags package [41], with three chains of 50 000 iterations each, a burn-in of 5000 iterations, and a thinning of 10 iterations. Minimally informative priors (beta (1,1)) were used for the sensitivities and prevalence’s. In contrast, for the specificities informative priors (beta (34.17, 1.33)) were included. The shape parameters were obtained with beta buster assuming “to be 95% sure that the specificity is greater than 90% with a mode at 99%”. Convergence was assessed by visual inspection of the trace plots and the potential scale reduction factor (Gelman Rubin statistic) being below 1.1. A sensitivity analysis was performed by using different combinations of minimally (dbeta(1,1)) or weakly informative priors (dbeta(2,1)).
Ethical consideration.
This research has been accomplished after getting ethical approval from EPHI and AAU (EPHI–IRB-259-2020 and 029/20/Lab (AAU, CHS) respectively). All participants’ identifiers were removed and only codes were used throughout the study to keep confidentiality.
During data and sample collection the data collectors inform each study participant about the purpose and anticipate benefits of the research project and also informed on their full right to refuse, withdraw or completely reject partly or all of their part in the study. Then, we obtained written informed consent from adult study participants and parents or legal guardians of study participants under the age of 18 years to participate in the study and to use their files and records for the purpose of this study.
Furthermore, this work was done accordance with the principles of Helsinki declaration.
Results
Laboratory results of study participants by the three assays
A total of 279 respiratory samples were analyzed using by three assays. The age distribution of the participants ranges from 10 to 85 years with majorities (40%) of the participants were 21–30 years and 144(51.6%) of the participants were female. We have three different populations, of which half of the participants samples 140(50.18%) were from suspect group followed by community surveillance 91(33%) and the remaining from close contacts, Table 3.
Results with threshold cycle (Ct) values >40 or those that remained undetected during the 45 cycles of the experiment work and a probability of false negatives using these three test assays with a positivity rate of from 18 to 25%. There were 2 more cases were positive by E gene (the screening test of TIB) while become negative by RdRp (the confirmatory one). The results were evaluated in comparison with those of the TIB™, Daan™ COVID-19 Direct assay and the BGI COVID-19 assay as indicated Fig 1 and Table 4.
The performance of each assay was evaluated and Ct values of 2019-nCoV positive samples ranged from 12.4 to 37.6 and 13.3–38.7 for RdRP and E genes of TIB, 13.3–39.22,16.65–38.91 for N genes and ORF of DaAn and from 10.9–35.5 of ORF of BGI. The value for the ORF gene of BGI (28.78±5.7), which makes the lowest ct value among the average of the positive samples.
The raw data, comprising the frequencies of the eight possible combinations of dichotomized test results of the three RT-PCRS in the three populations are displayed in Table 5A.
b. Measure of Cohen’s kappa coefficient (κ) Value among the three tests kits, Addis Ababa, Ethiopia, 2021.
We perform a Cohen’s kappa coefficient(k) to see the inter test agreement beyond chance of the three assay using 279 COVID-19 suspected individuals and we have a moderate to substantial level of agreement finding report among the three RT-PCR test kits with a kappa value of 0 .55 to 0.72 as the detail illustrated in Table 5B.
Thus, since none of the currently available COVID-19 tests can be considered as a perfect gold standard with both a perfect sensitivity and specificity, we employed a Bayesian latent class model (BLCM) using Markov Chain Monte Carlo sampling with Just another Gibbs sampler (JAGS) to the clients’ data using three tests in three populations.
Posteriors means and corresponding 95% credible intervals (CrI) of models without any covariance or one covariance between either two sensitivities or two specificities are displayed in Table 6. There was no evidence, based on visual inspection of the histograms and on 95% credible intervals (not including 0), that adding a covariance term led to a better model fit. Convergence was checked by visual inspection of the trace plots and the Gelman-Rubin statistic with all potential scale reduction factors (psrf) being below 1.05. Adding a covariance term between the sensitivities if Tib Molbiol and Da An led to a slight decrease in the sensitivities of all RT-PCRs, whereas adding covariance’s between Tib Molbiol and BGI or between Da An and BGI did not substantially alter the posteriors. Adding covariance’s between the specificities, mildly affected the sensitivities, but the specificities were not affected considerably.
Based on the obtained posteriors for the sensitivities, specificities and prevalence, positive and negative predictive values were determined for each test and in each population as depicted at Table 7.
Discussion
High-throughput quality SARS-CoV-2 diagnostic testing is the pivotal factor and has an indispensable role in the control of the current global pandemic [42]. The pandemic is not over. Thus, accurate, reliable and timely laboratory supported testing and management of the pandemic is very much important to curve the infection through swiftly to test and care for patients and to trace their contacts for brining to care. However, this efforts might be hampered by a lack or limited information on the test kit quality diagnostic performance by which have great impact on the testing and containment of the virus [9,10,17].
Accordingly, the current study aimed to evaluate three commercially available one step RT-PCR assays in Ethiopia with an overall diagnostic sensitivity of 0.813(0.658–0.938), 0.836(0.712–0.940) & 0.810(0.687–0.920), and a specificity of 0.991(0.973–1.000), 0.961(0.930–0.991) & 0.916(0.875–0.952) of TIB MOLBIOL, Da An and BGI respectively, which is slightly lower performance than a study done by public health England using BGI test kits of sensitivity 97.4% (86.5 to 99.9%; 95% CI and by Corman et al., as sensitivity of 97.9% [92.8–99.7 by TIB MOLBIOL, Berlin, Germany using E-gene [42,44] and its superior findings than SARS-COV-2 R-GENE (BioMérieux) with 60.2% [49.8–70.0] to 66.3% [56.1–75.6]) and combined with those for N or the RdRp genes (71.4% [61.4–80.1] and 69.4% [59.3–78.3], respectively) reported by Alcoba et al. and it’s concordance finding with in positive case of BGI of SARS-CoV-2 in Australia showed high diagnostic power in detecting SARS-CoV-2 with a positive percent agreement of 88.89% (95% CI: 83.4%, 94.3%). the main difference might be study time and sampling of the presumptive cases too. More over all of these studies try to present their finding against the classical “gold standard” approach. However, we used an innovative approach with the Bayesian latent class model (BLCM), which is recently recommended to see the diagnostic tests accuracies in the absence of a perfect gold standard [43–45].
The current results showed a moderate to substantial level of agreement of the inter test agreement beyond chance of the three assays with an average Cohen’s kappa value of 0 .55 to 0.72. Which is an inter performance agreement a measure of 0.72(95% CI(0.61–0.82)) among TIB and DaAn assays, 0.61 (95% CI(0.49-.0.72)) b/n TIB and BGI with substantial agreement where as a moderate agreement were reported among DaAn and BGI assays with a kappa value of 0.55(95% CI (0.44–0.67)) via 279 COVID-19 suspected individuals, which was consistent with similar studies(27,51,52) and lower finding compared with Altamimi, Asmaa M., et. al., the difference might be the defined study population [46].
From the current study, we learnt that, TIB and Da An assays correctly identified all negative samples while two assays each failed to correctly identify one different negative sample. GI has been presented in low specificity. The consistent detection of positive samples at different Ct values gives an indication of when to repeat testing and/or establish more stringent in-house cut-off value. The varied performance of different diagnostic assays, mostly with emergency use approvals, for a novel virus is expected. This comparative assay’ performance findings may guide country laboratories to determine both their repeat testing Ct range and/or cut-off value [43,47].
Notably, all professional and pharmaceutical, diagnostic developer should take their maximum capacities to get the best quality diagnostic assay to flatten the curve of COVID-19. Likewise, RT-PCR is the current established gold standard approaches for the diagnosis of COVID-19, but it has its own limitation in high false negativity rate. This is more or less concordance with our study findings. Thus, in light of this, further action should be considered to get the most sensitive and specific test kits, that can minimize the false positive and false negative cases in to insignificant level [43,47,48].
We noted that the positivity proportion of these three assays were higher among the age group of 21-30years of the presumptive, and there is no gender specific difference among the participants, which is similar and concordance with prior studies [25,33]. Likewise, the evaluated assays sensitivity diagnostic performances have been improved its levels of sensitivity improved to those of all other kits when one primer-probe combined with other targeted genes, but it takes more time to repeat for confirming the screening test with more sensitive. Thus, one touch step assays of combining different targets should be considered to have more competitive with cost effectiveness and benefit from combining the results of more than one primer probe set [22,27,31].
In the current study, we are at most the highest level of care and rightfully controlled procedure from the pre analytical to post analytical, which will be the contributing factors for the quality performance difference. Given that, we used similar analysis, similar testing time, similar personnel, sample collection strategy.Moreover we try to use similar written standard procedure, which is all assay solutions were uniformly affected. Thus, the differences in diagnostic sensitivity among the evaluated test kits were purely related to their different components. This finding will give a highlight which target gen is more sensitive while selecting a gene target, quality control, primers and give a clue for pharmaceuticals how to choose alternative test kits for the diagnosis of COVID-19 and that minimizes misdiagnoses of an active SARS-CoV-2 infection [29,49].
The current global pandemic is in a way of highly genomic variation and changing its primary nature that might have a diagnostic variability performance and RT-PCR assay have not yet documented gold standards for their comparative analysis. Accordingly, we followed the STARD-BLCM guidelines by which specificity robust estimates independent of conditional dependencies, fewer data to estimate sensitivity which leads less robust. Prevalence estimates are in line with having a higher prevalence in contact cases and suspects as of the study populations, the predictive values were found influenced by prevalence’s. In case of TIB MOLBIOL; there were 2 discordant cases of the primary screening E gene based targeted test compared with a confirmatory of RdRp and also its time taking and resource intensive while repeating those discordant results. On top of this, there might be more false negative clients with Da An and TIB MOLBIOL comparing to BGI; which might be a source of infection in the community as false negative diagnosis [43,44,47].
We report a higher ct value in case of DaAn (31.95±5.6 (in case of N gen & 34.20±5.6 in case of ORF1) and lower ct value in BGI with a mean (± SD) Ct value of 28.78±5.7 by ORF1 of BGI among the positive cases. Higher ct value mean it needs more resources and time, while smaller ct value mean, positive result with in shorter turnaround time(TAT) and cycle of PCR reaction that can be more efficient than the others. In case of DaAn, in case of inconclusive test results and situations like ct value of the test sample is NMT 40 only in one channel b/n FAM and VIC, and no amplification curve in the other channel, the manufacturer recommend to repeat the test.if the result of the retest consist with the original, it can be determined as positive for SARS-CoV-2; if the result of the retest is negative, it can be determined as negative. This leads more strenuous and more tiring among the professional working in this lab that might be pushing looking for and searching more recommend user friendly and single touch/step test kits in the era of highly demanding laboratory results with in short around time [20,32,33,36].
Among the evaluated assays the laboratory try to perform the SARS-CoV-2 diagnosis based on the availability of the test kits. Among these TIB MOLBIOL urge the test to run two times as a manner of screening by using Egen and confirmatory based RdRp, the final verdict of the test judge based on the RdRp findings and in case of DaAn the final diagnostic result should be based on the agreement of Ngen and ORF. While in case of BGI the final decision of the presumptive to have positive or negative based on only and only one test using ORF based. From these we understand that the former two tests need more time and resources to finalize the result with much lever intensive and not user friendly, which makes significant difference and BGI be more upper weigh and be superior, that concur with different similar studies [35,36,46,47].
In the other hand, we get a chance of feedback about the supply chain management system from the professionals and try to test out the reagent supply in light of sustainable laboratory reagents and consumables have been a pillar in the global response to the COVID-19 pandemic. In this regard, we learnt that the testing laboratories were suffering a lot with erratic supply chain management system.However, this should be expatiated sustainable supplies will go on with to be a key tool for controlling COVID-19 and regress effort by supply chain managers, policy makers and laboratory professionals [31,43,46].
Limitation of the study
We noted that possibly not enough data to estimate conditional dependencies between sensitivities which is minor differences in posteriors. Due to high estimates for specificities, conditional dependencies between specificities are implausible. Thus, readers should take in to consideration while inferring our findings.
Conclusion and recommendation
Based on the current research findings, we wind up as Tib Mol Biol performs best, followed by DaAn: the sensitivity of all tests are rather similar, but there are differences with regard to specificity. So for issues and costs associated with false positive, one should go for TIb Mol Biol.; and we understand that these assays are helpful tools for the routine diagnosis of COVID-19 in resource limited settings and could be implemented accordingly considering their prioritization contexts.
Despite the incredible efforts have been made to control the pandemic a global crisis has been aggressively going on. To contain this calamity, based on the recent study finding, the following points should be considered. Large scale study should be conducted and there might be the genetic mutation of the circulating virus during our sampling that might be the source of the performance variation, possibly false negative. Thus, the research team highly recommends the whole genome sequencing to know the circulating variant/strain and to come up a conclusive justification.
Supporting information
S1 File. Supplementary material used for the BLCM code for analysis, Addis Ababa, Ethiopia, 2021.
https://doi.org/10.1371/journal.pone.0262178.s001
(R)
Acknowledgments
We are very much gratitude and acknowledge the Addis Ababa University, EPHI, and Yekatit 12 Hospital medical college and Mr. Zerihun Woldesenbet for their strong support and assistance in accessing diverse resources used in the study. Our indisputable gratitude goes to Professor Paul Torgerson (professor of Veterinary Epidemiology Vetsuisse Faculty, Zurich) for his commendable assistance in grammatical review and for proofreading the document. We are also acknowledge all the data collector and study participants.
References
- 1.
University JH. Johns Hopkins University, J.H., worldometers COVID 19 Daily update. worldometers. Available from: https://coronavirus.jhu.edu/. Accessed October 13, 2021.
- 2.
World health organization. Coronavirus disease (COVID-19) Weekly Epidemiological Update and Weekly Operational Update. Available from: https://www.who.int/emergencies/diseases/novel-coronavirus-2019/situation-reports. Accessed 13 October,2021.
- 3. Luigi C., Fulvio P., Caterina L. et al,. COVID-19 Pandemic: Prevention and Protection Measures to Be Adopted at the Workplace. Sustainability 2020, 12, 3603.;
- 4. Khanna RC, Cicinelli MV, Gilbert SS, Honavar SG, Murthy GS. COVID-19 pandemic: Lessons learned and future directions. Indian J Ophthalmol 2020; 68:703–10. pmid:32317432
- 5. Kumar A, Singh R, Kaur J, Pandey S, Sharma V, Thakur L, et al (2021) Wuhan to World: The COVID-19 Pandemic. Front. Cell. Infect. Microbiol. 11:596201. pmid:33859951
- 6. Luk HK, Li X, Fung J, Lau SK, Woo PC. Molecular epidemiology, evolution and phylogeny of SARS coronavirus. Infection, Genetics and Evolution 2019; 71:21–30. pmid:30844511
- 7. Patel B.V., Haar S., Handslip R. et al. Natural history, trajectory, and management of mechanically ventilated COVID-19 patients in the United Kingdom. Intensive Care Med 47, 549–565 (2021). pmid:33974106
- 8. Andargachew M. et al. The challenges of COVID-19 testing in Africa: the Ethiopian experience. Pan African Medical Journal. 2021; 38(6). pmid:33520075
- 9. Saeed S , O’Brien SF, Abe K, Yi Q-L, Rathod B, Wang J, et al. (2021) severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) seroprevalence: Navigating the absence of a gold standard. PLoS ONE 16(9): e0257743. pmid:34555095
- 10. Alex Perkins T., Melissa Stephens, Wendy Alvarez Barrios, Sean Cavany, Liz Rulli, Michael E. Pfrender. Performance of three molecular tests for SARS-CoV-2 on a university campus estimated jointly with Bayesian latent class modeling. medRxiv 2021.07.31.21261425; https://doi.org/10.1101/2021.07.31.21261425.
- 11. Hag-Ali M., AlShamsi A.S., Boeijen L. et al. The detection dogs test is more sensitive than real-time PCR in screening for SARS-CoV-2. Commun Biol 4, 686 (2021). pmid:34083749
- 12. Hartnack S, Eusebi P, Kostoulas P. Bayesian latent class models to estimate diagnostic test accuracies of COVID-19 tests. J Med Virol. 2021 Feb;93(2):639–640. Epub 2020 Aug 25. pmid:32770741; PMCID: PMC7436623.
- 13. Kostoulas P, Eusebi P, Hartnack S. Diagnostic Accuracy Estimates for COVID-19 Real-Time Polymerase Chain Reaction and Lateral Flow Immunoassay Tests With Bayesian Latent-Class Models. Am J Epidemiol. 2021 Aug 1;190(8):1689–1695. pmid:33823529; PMCID: PMC8083455.
- 14. Zikargae MH. COVID-19 in Ethiopia: Assessment of How the Ethiopian Government has Executed Administrative Actions and Managed Risk Communications and Community Engagement. Risk Manag Healthc Policy. 2020;13:2803–2810 pmid:33299368
- 15.
Worldo meters COVID 19 Ethiopian Daily update. worldometers. Available from: https://coronavirus.jhu.edu/. Accessed 13 October, 2021. https://www.worldometers.info/coronavirus/country/ethiopia/).
- 16.
Public presentation presented by DDG-EPHI@ national COVID-19 research summit a@SKYLight Hotel, Addis Ababa.#COVID19Ethiopia.https://t.me/EthPHI.
- 17.
Medical Express. Delta variant driving COVID surge in South Africa. https://medicalxpress.com/news/2021-06-delta-variant-covid-surge-south.html.
- 18. Abera A, Belay H, Zewude A, Gidey B, Nega D, Dufera B, et al,. Establishment of COVID-19 testing laboratory in resource-limited settings: challenges and prospects reported from Ethiopia. Glob Health Action. 2020 Dec 31;13(1):1841963. pmid:33200686
- 19. Sisay A, Tesfaye A, Desale A, Ataro I, Woldesenbet Z, Nigusse B, et al. Diagnostic Performance of SARS-CoV-2 IgM/IgG Rapid Test Kits for the Detection of the Novel Coronavirus in Ethiopia. J Multidiscip Healthc. 2021;14:171–180 pmid:33536760
- 20. Falzone L, Gattuso G, Tsatsakis A, Spandidos DA and Libra M: Current and innovative methods for the diagnosis of COVID-19 infection (Review). Int J Mol Med 47: 100, 2021. pmid:33846767
- 21. Gorbalenya A.E., Baker S.C., Baric R.S. et al. The species Severe acute respiratory syndrome-related coronavirus: classifying 2019-nCoV and naming it SARS-CoV-2. Nat Microbiol 5, 536–544 (2020). pmid:32123347
- 22. Waidi F. and Daniel O. Real-time RT-PCR for COVID-19 diagnosis: challenges and prospects. Pan Afr Med J. 2020; 35(Suppl 2): 121. pmid:33282076
- 23.
FIND EVALUATION UPDATE: SARS-COV-2 MOLECULAR DIAGNOSTICS. Available from: https://www.finddx.org/covid-19/sarscov2-eval-molecular/.
- 24.
Africa CDC. New guidance to expand rapid antigen testing for COVID-19 response in Africa released. https://africacdc.org/news-item/new-guidance-to-expand-rapid-antigen-testing-for-covid-19-response-in-africa-released/.
- 25. Oishee MJ, Ali T, Jahan N, Khandker SS, Haq MA, et. al., COVID-19 Pandemic: Review of Contemporary and Forthcoming Detection Tools. Infect Drug Resist. 2021;14:1049–1082 pmid:33762831
- 26.
World Health Organization. Clinical management of severe acute respiratory infection when novel coronavirus (2019-nCoV) infection is suspected: interim guidance, 28 January 2020. https://apps.who.int/iris/handle/10665/331446).
- 27. Chan JF, Yip CC, To KK, et al. Improved Molecular Diagnosis of COVID-19 by the Novel, Highly Sensitive and Specific COVID-19-RdRp/Hel Real-Time Reverse Transcription-PCR Assay Validated In Vitro and with Clinical Specimens. J Clin Microbiol. 2020;58(5):e00310–20. pmid:32132196
- 28.
Health Information and Quality Authority. Rapid HTA of alternative diagnostic technologies for the detection of SARS-CoV-2. https://www.hiqa.ie/sites/default/files/2020-05/Rapid_HTA_COVID-19_tests.pdf.
- 29. Julia A., Helena G., Diego G., Rafaela G., Agustín V., Laura C. et al. Sensitivity of different RT-qPCR solutions for SARS-CoV-2 detection,International Journal of Infectious Diseases, 99,:2020,190–192, pmid:32745627
- 30. Russo A, Minichini C, Starace M, Astorri R, Calò F, Coppola N; Vanvitelli COVID-19 group. Current status of laboratory diagnosis for COVID-19: a narrative review. Infect Drug Resist. 2020; 13:2657–2665. pmid:32801804
- 31. Younes N, Al-Sadeq DW, Al-Jighefee H, Younes S, Al-Jamal O, Daas HI, et al. Challenges in laboratory diagnosis of the novel coronavirus SARS-CoV-2. Viruses. 2020 Jun;12(6):582. pmid:32466458
- 32. Mollaei HR, Afshar AA, Kalantar-Neyestanaki D, Fazlalipour M, Aflatoonian B. Comparison five primer sets from different genome region of COVID-19 for detection of virus infection by conventional RT-PCR. Iran J Microbiol. 2020 Jun;12(3):185–193. pmid:32685113; PMCID: PMC7340604.
- 33. Daniel K W Chu, Yang Pan, Samuel M S Cheng, Kenrie P Y Hui, Pavithra Krishnan, Yingzhi Liu, et al., “Molecular Diagnosis of a Novel Coronavirus (2019-nCoV) Causing an Outbreak of Pneumonia,” Clin. Chem., vol.66, pp. 549–555, 2020, pmid:32031583
- 34. Lu R, Zhao X, Li J, Niu P, Yang B, Wu H, et al. Genomic characterisation and epidemiology of 2019 novel coronavirus: implications for virus origins and receptor binding. The lancet. 2020 Feb 22;395(10224):565–74. pmid:32007145
- 35.
TIB MOLBIOL, Germany. Instructions for use https://www.tib-molbiol.de/.
- 36.
https://www.who.int/diagnostics_laboratory/eual/eul_0513_200_00_novel_ncov_nucleic_acid_detection_kit.pdf.
- 37.
Manufacturer’s Instructions for Use (IFU), Real-time fluorescent RT-PCR kit for detecting 2019-nCoV, MFG030010. BGI Biotechnology (Wuhan) Co., Ltd, Building B2, Zone B/C/D, Wuhan National Bioindustry Base, NO.666 Ga0xin Avenue, East Lake High-tech Development Zone, Wuhan.
- 38. Cohen’s kappa Kappa value interpretation (see Landis JR & Koch GG (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174). pmid:843571
- 39. Just another Gibbs Sampler. https://mcmc-jags.sourceforge.io/.
- 40. Denwood Matthew J. (2016). runjags: An R Package Providing Interface Utilities, Model Templates, Parallel Computing Methods and Additional Distributions for MCMC Models in JAGS.Journal of Statistical Software, 71(9), 1–25. ).
- 41.
Team, R. Core. "R: A language and environment for statistical computing." (2013): 201. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/.).
- 42. Mercer T.R., Salit M. Testing at scale during the COVID-19 pandemic. Nat Rev Genet 22, 415–426 (2021). pmid:33948037
- 43. Julia Alcoba-Florez, Helena Gil-Campesino, Diego García-Martínez de Artola, et al., Increasing SARS-CoV-2 RT-qPCR testing capacity by sample pooling. International Journal of Infectious Diseases,.103,2021, 19–22, pmid:33220439
- 44.
Manufacturer’s Instructions for Use (IFU), Real-time fluorescent RT-PCR kit for detecting 2019-nCoV, MFG030010. BGI Biotechnology (Wuhan) Co., Ltd, Building B2, Zone B/C/D, Wuhan National Bioindustry Base, NO.666 Ga0xin Avenue, East Lake High-tech Development Zone, Wuhan.
- 45.
Public Health England. Rapid assessment of the BGI Real-Time Fluorescent RT-PCR kit for detecting 2019-nCoV .https://www.gov.uk/government/publications/covid-19-phe-laboratory-assessments-of-molecular-tests.
- 46. Altamimi AM, Obeid DA, Alaifan TA, et al. Assessment of 12 qualitative RT-PCR commercial kits for the detection of SARS-CoV-2. J Med Virol. 2021; 93:3219–3226. pmid:33629747
- 47.
Thomas T., Gina P., Charlene M., Julian D., Mike C., Post-market validation of the Beijing Genomics Institute (BGI) SARS-CoV-2 Real Time PCR platform. https://www.health.gov.au/sites/default/files/documents/2020/06/post-market-validation-of-the-beijing-genomics-institute-bgi-sars-cov-2-real-time-pcr-platform.pdf.
- 48. Chan JF, Yip CC, To KK, Tang TH, Wong SC, Leung KH, et al. Improved Molecular Diagnosis of COVID-19 by the Novel, Highly Sensitive and Specific COVID-19-RdRp/Hel Real-Time Reverse Transcription-PCR Assay Validated In Vitro and with Clinical Specimens. J Clin Microbiol. 2020 Apr 23; 58(5):e00310–20. pmid:32132196; PMCID: PMC7180250.
- 49. Vogels CBF, Brito AF, Wyllie AL, Fauver JR, Ott IM, Kalinich CC, et al., Analytical sensitivity and efficiency comparisons of SARS-CoV-2 RT-qPCR primer-probe sets. Nat Microbiol. 2020 Oct;5(10):1299–1305. pmid:32651556