https://orcid.org/0000-0002-0376-0377
Figures
Abstract
Background
Lateral flow tests (LFTs) have been used to screen for SARS-CoV2 in Wales since January 2021. Between May and August 2021, adult care home staff policy was for biweekly Innova LFT and weekly Polymerase Chain Reaction (PCR) testing while asymptomatic. We estimated test performance of LFTs conducted in adult care home staff using PCR tests as a reference standard.
Methods
Test results from surveillance data were matched by individual where both LFT and PCR were taken on the same day. We calculated sensitivity, specificity, positive and negative predictive values, and agreement using Matthew’s correlation coefficient. Ct values of positive PCR results were compared by matched LFT result. Analysis was conducted using R v4.1.3.
Results
We analysed 115,593 test pairs, 499 (0.43%) of which were PCR positive. Median age was 48 (IQR: 22) and 85.00% of the study population were female. Test result agreement was 99.59% (95%CI 99.55–99.63; MCC: 0.38, p<0.001). Sensitivity and specificity were 25.65% (95%CI 22.02–29.67) and 99.91% (95%CI 99.89–99.93), respectively. PPV was 55.90% (95%CI 49.42–62.17) and NPV was 99.68% (95%CI 99.64–99.71). Crude Ct values were significantly lower in positive PCR tests matched to a positive LFT compared to a negative LFT.
Conclusions
Specificity and negative predictive value were high in an asymptomatic population of care home staff indicating this test is an effective tool for identifying cases of SARS-CoV-2 infection during periods of high prevalence where transmission is likely, due to the presence of high viral loads. Positive predictive value results are lower than existing literature yet should be considered in light of the asymptomatic study population and low prevalence (under 1%) at the time most of these tests were conducted. Performance improved at times of higher prevalence during the study. These results suggest that whilst lateral flow tests are effective for identifying SARS-COV-2 infections with high viral loads, they are not effective at identifying cases with a low viral load. When an LFT provides a negative result, false negatives should be considered and additional diagnostic tests performed.
Citation: Hogg C, Boots S, Howorth D, Williams C, Heginbothom M, Salmon J, et al. (2023) Test performance of lateral flow rapid antigen tests for COVID-19 in Welsh adult care home staff using routine surveillance data. PLoS ONE 18(8): e0290406. https://doi.org/10.1371/journal.pone.0290406
Editor: André Ricardo Ribas Freitas, Faculdade Sao Leopoldo Mandic, BRAZIL
Received: January 27, 2023; Accepted: August 7, 2023; Published: August 23, 2023
Copyright: © 2023 Hogg et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting information files.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Background
On the 11th March 2020 the World Health Organisation declared a coronavirus disease-19 (COVID-19) pandemic as a result of a newly identified Severe Acute Respiratory Syndrome-related coronavirus (SARS-CoV-2) [1, 2]. Worldwide epidemiological surveillance systems to monitor COVID-19 have developed rapidly as has the development of rapid non-invasive antigen lateral flow tests (LFTs) designed to detect SARS-COV-2. Whilst polymerase chain reaction tests (PCRs) are able to pick up lower levels of SARS-COV-2 than LFTs, rapid antigen testing can provide results quickly and cost effectively [3, 4]. They can therefore be a useful early detection tool for COVID-19 case clusters in asymptomatic individuals [5]. However, there has been much debate as to whether LFTs exhibit adequate performance to meet the needs of a mass-surveillance and screening system [6].
In January 2021, the Welsh Government launched a COVID-19 testing strategy that included the use of LFTs to aid in preventing the transmission of COVID-19 across Wales [7]. A part of this strategy involved the use of LFTs in closed settings such as care homes to reduce the risk of transmission from staff to care home residents. In these closed settings, asymptomatic staff members were advised to undertake Innova LFTs twice or more per week along with a weekly Polymerase Chain Reaction (PCR) test [8].
In January 2022, policy change removed the requirement to confirm a positive LFT results with a PCR test [9]. In April 2022, policy identified LFTs as the primary testing route for the symptomatic diagnosis of COVID-19. These testing policies and their subsequent changes (as is applicable for all testing policies) require careful evaluation, particularly so with the rapid changes to testing policies implemented throughout the pandemic. Interpretation of this evaluation will be affected by restrictions and management of cases deemed to be ‘false positive’ [10], and the use of LFTs as a routine screening tool [11].
With the data generated as a result of the implementation of the Testing Strategy for Wales, we aim in this paper to measure the sensitivity and specificity of LFTs against PCRs for those tests taken in Adult Care Home settings in Wales during the period 1st May 2021 – 31st August 2021.
Methods
Study population
Individuals were included in the study population where their postcode of residence was in Wales, and they were of working age (between 18 and 67 years old at time of first test within the timeframe). Both the LFT result and PCR result must be positive or negative (not void), and conducted between 1st May 2021 and 31st August 2021. From this study population, we then only included individuals whose LFT record was associated with an adult care home, and who identified as a member of staff in the care home. Following guidance, symptomatic individuals should book a PCR test in the first instance, therefore this study population is presumed to be primarily asymptomatic.
Data sources
Lateral flow testing activity data for Wales was sourced from the UK Government Lateral Flow Testing Portal, collated by NHS Digital, and supplied to Public Health Wales (PHW) via a data feed and a set of SQL views from Digital Health and Care Wales (DHCW).
PCR testing data was sourced from NHS and lighthouse laboratories across Wales and supplied to PHW through a data feed from DHCW.
A list of adult care home unique organisation numbers (UONs) and UON groups in line with the Testing Strategy for Wales was provided to PHW and DHCW by the Welsh Government.
All data was collected as part of routine surveillance and management of COVID-19. Patient identifiable information was anonymised when necessary and held securely on the NHS Wales Network.
Test matching
Lateral flow test records and PCR test records were matched on a 1:1 basis based on a unique reference code where both LFT and PCR were taken on the same day. Where an individual conducted multiple tests of the same type on the same day, the earliest lateral flow with a valid result was matched with the earliest PCR with a valid result conducted on that day; all other tests on that day were excluded. Subsequent LFTs and PCRs were matched in the same way in order to capture repeat testing on different days.
Lateral flow assay
Lateral flow assay tests were taken by the individual using Innova test kits [12]. Presuming guidance was followed correctly, self-administered combined nose and throat swabs were sampled and interpreted at 30 minutes. Results were then self-reported in conjunction with contact and demographic information to an online portal hosted by the UK Government (https://www.gov.uk/report-covid19-result).
PCR assay
RT-PCR was used as the ‘reference standard’ in this analysis. No classification was recorded regarding whether the swab was conducted by the individual or a trained professional. Combined nose and throat swabs or dry throat swabs were sampled and sent to a variety of NHS Wales laboratories and lighthouse laboratories. RT-PCR was then conducted. Results were graded and authorised by trained laboratory staff according to manufacturer definitions and standard operating procedures [13]. Sample processing equipment specification was likely to vary by time and test processing location due to the use of multiple PCR platforms across NHS Wales and lighthouse laboratories, and therefore was not identical in every test. Ct values for positive PCR results were not available for stratified analysis.
Statistical analysis
Specificity, sensitivity, and predictive values were calculated using the RT-PCR results as reference standard. Where applicable, 95% Confidence Intervals for proportions were calculated using the Wilson method. Matthew’s correlation coefficient (MCC) was used to evaluate the level of agreement between tests. Ct values were extracted where available for PCR results and were compared between LFT positive and LFT negative groups, using Mann-Whitney to test for significance. Statistical analyses were performed using R v4.1.0.
Ethics
Public Health Wales was established under The Public Health Wales National Health Service Trust (Establishment) Order 2009. Its functions include the provision and management of a range of public health, health protection, healthcare improvement, health advisory, child protection and microbiological laboratory services and services relating to the surveillance, prevention and control of communicable diseases. This work falls under the establishment order and therefore no further NHS research permissions were required. Data were held and processed under Public Health Wales information governance arrangements in compliance with the Data Protection Act, Caldicott Principles and Public Health Wales guidance on the release of small numbers. All methods were performed in accordance with the relevant guidelines and regulations. No data identifying protected characteristics of an individual were released outside Public Health Wales. Data used in the study was anonymised prior to analysis. By choosing to take a test for COVID-19 via the NHS Test and Trace programme, of which all data in this study falls under, implied consent was obtained for the use of data collected for surveillance and research purposes [14].
Results
Characteristics of the study population
Characteristics of the selected study population (n = 115,593) are shown in Table 1. These demographics are shown at test level and include repeated testing by individuals.
Test results
In these 115,593 linked test records, 0.43% of PCR results were positive (n = 499) and 99.57% were negative (n = 115,094), whilst 0.20% of LFT results were Positive (n = 229) and 99.80% were negative (n = 115,364) (Table 2).
Agreement between groups was calculated to be 99.59% (MCC: 0.38; 95%CI 99.55, 99.63). Of the 115,593 tests in the study population, 472 matched tests did not agree. A total of 78.60% (n = 371) of these cases had a negative LFT followed by a Positive PCR (“False Negatives”) whilst 21.40% (n = 101) had a positive LFT followed by a negative PCR (“False Positives”).
Test performance
Using this data, overall sensitivity of the lateral flow devices was 25.65% (95%CI 22.02–29.67) and specificity was 99.91% (95%CI 99.89, 99.93). Across the entire study period, Negative Predictive Value (NPV) was estimated to be 99.68% (95%CI 99.64, 99.71), whilst Positive Predictive Value (PPV) was estimated to be 55.90% (95%CI 49.42, 62.17) (Table 3). PPV however varied over time due to changing prevalence during the study period, ranging from a minimum (non-zero) estimate of 22.22% to a maximum of 85.19% (Fig 1) [15].
Data excluded for weeks 17 and 35 due to incomplete weekly data.
PCR Ct values
Ct values were available for 51 (10.2%) of positive PCR results included in the study. We tested the distribution of Ct results and found them not to be normally distributed (Shapiro-Wilk: W = 0.904, p < 0.001***). Control target values were available for less than half of these results (45.1%) so we did not proceed to normalisation of minimum Ct values. Wilcoxon rank sum test yielded a p value < 0.001, suggesting a significant difference in crude minimum Ct values between matched LFT result groups (Fig 2).
Labels show median of each group and statistical difference between groups.
Discussion
We believe our study at time of publication to be the first study of lateral flow test performance using self-reported results from the UK Government online portal demonstrating real-world test performance estimates. The study has identified that Innova lateral flow testing kits supplied as part of the Testing Strategy for Wales yielded a high specificity reaching above 99%, indicating that lateral flow tests could be an effective tool to rule in SARS-CoV-2 infection where the test result is positive due to the very low false positive rate during a time of low prevalence. We also believe that the use of LFTs during this time period promoted testing in an asymptomatic population, and therefore likely identified many cases which would otherwise have gone unidentified. Our study’s low PPV results can be attributed to the asymptomatic study population and low prevalence (under 1%) of SARS-COV-2 during the study’s analytical time period. PPV increases notably with increased prevalence, however sensitivity analysis, which is unaffected by prevalence, yielded a lower sensitivity than expected with an overall estimate of 25.65%. The low sensitivity of lateral flow tests identified in this study therefore does not support the use of lateral flow tests as a reliable diagnosis platform as the test cannot reliably rule out infection, particularly during times of low prevalence where NPV will be reduced. Our study’s sensitivity results contrasts with the results of a systematic review by Mistry et al. [16], which calculated a pooled sensitivity estimate of 78.68% in Innova test kits.
Previous studies have demonstrated lower sensitivity estimates in self-swabbing samples compared to those in studies where the test was undertaken by a trained professional [17]. However, a study by Amoo et al. [17] found that self-sampling for qPCR analysis was reliable and yielded a high agreement percentage compared with samples collected by healthcare workers. A likely major factor contributing to the low results found in our study is that the testing programme is targeted towards asymptomatic individuals, which has been shown to yield lower sensitivity estimates than a symptomatic population [18]. We also hope that the large sample size of this study will mitigate bias introduced by the effects of possible poor sampling technique.
Viral load is also a possible contributing factor to the likelihood of detection in lateral flow devices [19], however evidence is limited to support a difference in viral load between symptomatic and asymptomatic cases [20, 21]. A further limitation is that less than half of available Ct values had control values which were able to be used to analyse normalised Ct results. Whilst preliminary analysis of the available Ct values suggested a high proportion of LFT negatives had lower Ct values indicative of higher viral load, a more robust approach would be required to draw any meaningful conclusion from such analysis.
Research has shown a person may shed SARS-COV-2 for up 17 days, however live virus was only found within 9 days [22, 23]. Therefore, as a result of the high sensitivity of PCRs, they could yield a positive result for a longer period of time compared to a LFT which only detect higher viral loads, creating a narrow time period where both results would give a positive result [23]. There is also variability in sensitivity and specificity of PCR tests which may differ depending on the manufacturer of the assay or machine used. Differing factors may therefore influence test performance characteristics due to variance in the PCR as ‘reference standard’ test.
Due to the repeated testing in our study population, it is possible that previously PCR positive individuals may continue to test positive by PCR where public testing guidance would advise against repeated testing in this post-infectious period [24, 25]. As Lateral Flow Tests would be unlikely to identify infection in this period [23], this aspect of the study population’s testing regime may negatively impact the sensitivity estimates in this analysis. There may also be implications to the calculation of prevalence in this scenario, and therefore may partially account for the Positive Predictive Value of 55.90%. However, comparison with ONS Coronavirus Infection Survey [13, 26] showed similar prevalence to the overall Welsh population at the time the tests of our study were conducted (Fig 1), further supporting the reliability of the PPV estimate given in this paper.
Further validation of occupational data would be beneficial on a wider scale, as initial data exploration showed inconsistencies in reported job roles and reasonable demographic information. Regardless, the linkage of data to a care-home via unique organisation number suggests that the tests were likely to have been taken in a domestic care setting and would therefore be conducted in accordance with guidance.
We propose the study contributes further to the justification of the use of lateral flow testing at times of high prevalence and pre-test probability due to the high testing specificity and PPV at times of increased prevalence. It is also reassuring to consider the large number of asymptomatic cases which would have otherwise gone unidentified in this population. The low sensitivity identified is concerning, however the sensitivity results should be considered in light of prevalence at the time the study’s testing results were reported, alongside the bias toward asymptomatic cases in the study population leading to low numbers in some subgroups.
The time period in which this study was conducted was chosen to reflect a time where regular asymptomatic testing was promoted, however the wide scope of testing at the time will undoubtably include pauci-symptomatic individuals and those with atypical symptoms. Whilst this limits the generalisability of these findings to a strictly asymptomatic population, we believe this study remains reflective of testing behaviour at the time, particularly with regard to social behaviour and individual decision making following results. In light of this, the results of this study highlight the potential scale of false positive results and the consequences this may introduce when applied to our study population of care home staff. There was notable impact on staffing levels in care homes during this period [27], not only impacting patient care but increasing burden of work for other healthy staff. A high number of false positives will therefore needlessly increase this burden. Future testing policy should therefore take these findings into account and it is hoped that the scale of this problem will allow policymakers to make more informed decisions when balancing this risk against the benefit of regular testing leading to reduced transmission where positive results are, in fact, reflective of SARS-CoV-2 infection. In addition to this, the impact of false negatives must be considered due to the risk of increased transmission if a false negative result is used to justify work or social attendance. The low sensitivity in this analysis indicates that this could be a substantial factor to consider when implementing similar testing policies in future.
Public messaging must continue to promote the reliability of lateral flow tests whilst making clear the limited ability of the test to rule out SARS-CoV-2 infection [6, 28]. The self-reporting mechanism in place throughout the first two years of the COVID-19 pandemic offered a unique and extensive system by which data from symptomatic and asymptomatic testing streams could be collected separately and collated with regard to the limitations of each system. Wider qualitative study is needed to understand attitudes towards rapid testing, and the reliability of reported results in light of policy change and individual decision making.
Acknowledgments
We would like to thank all staff involved in the curation and supply of data flows used in COVID-19 surveillance in Wales, particularly Gareth John from Digital Health Care Wales, and Helen Clayton and Laura Dexter from Public Health Wales for their work in the development and upkeep of COVID-19 testing data used in this analysis. Thanks also to Laia Fina for epidemiological advice in the early stages of this project. Finally, we would like to extend our gratitude to all laboratory staff working in both NHS Wales and Lighthouse labs for their hard work processing COVID-19 tests throughout the pandemic. The author(s) read and approved the final manuscript.
References
- 1.
WHO Director-General’s opening remarks at the media briefing on COVID-19. World Health Organisation. 2020. https://www.who.int/director-general/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19---11-march-2020
- 2. Coronaviridae Study Group of the International Committee on Taxonomy of Viruses. The species severe acute respiratory syndrome-related coronavirus: Classifying 2019-ncov and naming it SARS-COV-2. Nature Microbiology. 2020;5(4):536–44. pmid:32123347
- 3. Drakesmith M, Collins B, Jones A, Nnoaham K, Thomas DR. Cost-effectiveness of a whole-area testing pilot of asymptomatic SARS-COV-2 infections with lateral flow devices: A modelling and Economic Analysis Study. BMC Health Services Research. 2022;22. pmid:36138455
- 4. Du Z, Pandey A, Bai Y, Fitzpatrick MC, Chinazzi M, Pastore y Piontti A, et al. Comparative cost-effectiveness of SARS-COV-2 testing strategies in the USA: A modelling study. The Lancet Public Health. 2021;6. pmid:33549196
- 5. Hsieh WY, Lin CH, Lin TC, Lin CH, Chang HF, Tsai CH, et al. Development and Efficacy of Lateral Flow Point-of-Care Testing Devices for Rapid and Mass COVID-19 Diagnosis by the Detections of SARS-CoV-2 Antigen and Anti-SARS-CoV-2 Antibodies. Diagnostics. 2021 Sep 24;11(10):1760. pmid:34679458
- 6. Peto T, UK COVID-19 Lateral Flow Oversight Team. Covid-19: Rapid antigen detection for SARS-COV-2 by lateral flow assay: A national systematic evaluation of sensitivity and specificity for mass-testing. EClinicalMedicine. 2021;36:100924. pmid:34101770
- 7.
Welsh Government. Testing Strategy for Wales. 2021
- 8.
Welsh Government. Written Statement: Enhancing COVID-19 Testing in Care Homes. 2021
- 9.
Welsh Government. Written Statement: Prioritising PCR Testing. 2022
- 10. Healy B, Khan A, Metezai H, Blyth I, Asad H. The impact of false positive COVID-19 results in an area of low prevalence. Clinical Medicine. 2020;21. pmid:33243836
- 11. Larremore DB, Wilder B, Lester E, Shehata S, Burke JM, Hay JA, et al. Test sensitivity is secondary to frequency and turnaround time for covid-19 screening. Science Advances. 2021;7. pmid:33219112
- 12.
Welsh Government Technical Advisory Cell. Core principles for utilisation of RT-PCR tests for detection of SARS-CoV-2. 2020.
- 13.
Department for Health and Social Care. Your step-by-step guide for COVID-19 self-testing for adult social care staff (TC2465). 2021. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1028957/TC2465_Innova25T_Adult_Social_Care_Self_Test.pdf
- 14.
UK Health Security Agency. Testing for coronavirus: privacy information. 2022. https://www.gov.uk/government/publications/coronavirus-covid-19-testing-privacy-information/testing-for-coronavirus-privacy-information--2
- 15.
Office for National Statistics. Coronavirus (COVID-19) Infection Survey: Wales Dataset (2022 Edition). 2022. https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/conditionsanddiseases/datasets/covid19infectionsurveywales/2022
- 16. Mistry DA, Wang JY, Moeser M-E, Starkey T, Lee LY. A systematic review of the sensitivity and specificity of lateral flow devices in the detection of SARS-COV-2. BMC Infectious Diseases. 2021;21. pmid:34407759
- 17. Amoo OS, Adewara F, Tijani B, Onuigbo TI, Ikemefuna AS, Oraegbu JI, et al. The use of DIY (do it yourself) sampling and telemonitoring model for COVID-19 qPCR testing scale up. PLOS ONE. 2021;16. pmid:34735503
- 18. Mitchell SL, Orris S, Freeman T, Freeman MC, Adam M, Axe M, et al. Performance of SARS-COV-2 antigen testing in symptomatic and asymptomatic adults: A single-center evaluation. BMC Infectious Diseases. 2021;21. pmid:34663212
- 19.
UK Government. Asymptomatic testing for SARSCoV-2 using antigen-detecting lateral flow devices: Evidence from performance data October 2020 –May 2021. Department for Health & Social Care; 2021
- 20. Walsh KA, Jordan K, Clyne B, Rohde D, Drummond L, Byrne P, et al. SARS-COV-2 detection, viral load and infectivity over the course of an infection. Journal of Infection. 2020;81: 357–371. pmid:32615199
- 21. He X, Lau EHY, Wu P, Deng X, Wang J, Hao X, et al. Author Correction: Temporal dynamics in viral shedding and transmissibility of COVID-19. Nature Medicine. 2020 Aug 7;26(9):1491–3. pmid:32770170
- 22. Cevik M, Tate M, Lloyd O, Maraolo AE, Schafers J, Ho A. SARS-COV-2, SARS-COV, and MERS-COV viral load dynamics, duration of viral shedding, and infectiousness: A systematic review and meta-analysis. The Lancet Microbe. 2021;2. pmid:33521734
- 23. Crozier A, Rajan S, Buchan I, McKee M. Put to the test: Use of rapid testing technologies for covid-19. BMJ. 2021; pmid:33536228
- 24. Jefferson T, Spencer EA, Brassey J, Heneghan C. Viral cultures for coronavirus disease 2019 infectivity assessment: A systematic review. Clinical Infectious Diseases. 2020;73. pmid:33270107
- 25.
Welsh Government. Self-isolation Guidance. 2021.
- 26.
Office for National Statistics. Coronavirus (COVID-19) Infection Survey, UK: 18 June 2021. 2021.
- 27.
Social Care Wales. Social care workforce report, 2021. 2022.
- 28. Deeks JJ, Raffle AE. Lateral flow tests cannot rule out SARS- COV-2 infection. BMJ. 2020;: m4787. pmid:33310701