Clinical performance and analytical accuracy of a C6 peptide-based point-of-care lateral ﬂ ow immunoassay in Lyme borreliosis serology

We evaluated the analytical accuracy and the clinical performance of a ReaScan+ C6 LYME IgG point-of-care immunoassay (Reagena; index test). Analytical accuracy was evaluated in comparison to a C6 Lyme ELISA TM reference method (Oxford Immunotec) with retrospectively identi ﬁ ed serum and CSF samples. The clinical performance was evaluated by using Lyme borreliosis patient and control subject serum and CSF samples. The study was conducted by following the 2015 Standards for Reporting of Diagnostic Accuracy Studies procedure. The sensitivity and speci ﬁ city of the index test with serum samples were 83% and 91.6%, respectively, when C6 Lyme ELISA TM was used as a reference. The clinical sensitivity of the index test was 97.2%/96.8% for identifying Borrelia speci ﬁ c antibodies in de ﬁ nite/possible Lyme neuroborreliosis. With CSF samples, the clinical sensitivity was 97.2% for de ﬁ nite and 87.1% for possible Lyme neuroborreliosis. The clinical speci ﬁ city of the assay was 96.1% with serum and 100% with CSF samples


Introduction
Lyme borreliosis (LB) is a tick-borne infection caused by Borrelia burgdorferi sensu lato spirochetes (Borrelia).It is the most important tick-transmitted disease globally [1−3].Diagnosis of the disseminated manifestations of LB such as Lyme arthritis (LA), acrodermatitis chronica atrophicans, Lyme carditis and Lyme neuroborreliosis (LNB) is based on the presence of typical symptoms, objective signs, and the patient's medical history in combination with laboratory evidence of Borrelia infection.The main approach in laboratory testing is the detection of Borrelia-specific antibodies in patient serum samples [4].In cases where LNB is suspected, both serum and cerebrospinal fluid (CSF) should be analyzed [5].Traditionally, Borrelia-specific antibodies are analyzed by enzyme immune assays requiring laboratory equipment and often also batching of samples, both of which cause delay of test results.Therefore, rapidly available serology results would be valuable to guide the initial clinical decision-making concerning, for example, diagnostics of arthritis in children [6], and the prompt initiation of antibiotic treatment in LNB [7].
The C6-peptide is a molecule derived from the VlsE surface protein of Borrelia.C6 peptide-based assays appear to be well-performing tests [8−14] suitable for the diagnostics of early and disseminated LB.In this study, we evaluated the performance of a novel C6 peptide-based lateral-flow-based point-of-care (POC) assay (ReaScan+ C6 LYME IgG) head-to-head with C6 Lyme ELISA TM , and importantly, also using well-defined patient cohorts.The study was conducted using the 2015 Standards for Reporting of Diagnostic Accuracy Studies (STARD) [15,16].

Design and test methods of the analytical accuracy study
We used retrospectively collected anonymous left-over samples from the sample archives of the Clinical microbiology laboratory of Turku University Hospital, Turku, Finland.The sample material consisted of (1) 200 Borrelia antibody negative sera (C6 Lyme index ≤0.9;determined with the C6 Lyme ELISA TM (Oxford Immunotec, Oxford, UK)) (2) 200 Borrelia antibody positive sera (C6 Lyme index ≥3.0),(3) 100 sera with borderline or low positive (0.91−2.9)C6 Lyme index values, (4) 100 Borrelia antibody negative CSF samples, and (5) 23 Borrelia antibody positive CSF samples (Supplementary Table S1, and Figure S1).The negative serum samples were negative also with an in-house Borrelia whole-cell sonicate IgM/IgG ELISA using sonicate of B. burgdorferi sensu stricto B31 as the antigen [17,18], and with recomBead Borrelia IgM/IgG bead-immunoassays (Mikrogen, Neuried, Germany).Sera with C6 Lyme index ≥3.0 were positive also in the in-house ELISA and/or recomBead Borrelia assays.Ninety-eight out of 100 sera with borderline or low positive (0.91−2.9)C6 Lyme index values were borderline reactive or positive in the in-house and/or recomBead Borrelia assays.The one hundred Borrelia antibody negative CSF samples were negative in the in-house ELISA, while C6 Lyme ELISA TM was not performed on these samples.23 Borrelia antibody positive CSF samples were positive in the C6 Lyme ELISA TM assay and in the in-house ELISA.
ReaScan+ C6 LYME IgG (index test; Reagena, Toivala, Finland) analyzes were performed as instructed by the manufacturer.Briefly, 5 ml of serum was added to the Dilution buffer vial (containing 2 ml of dilution buffer resulting in 1:400 dilution) and mixed by vortexing.
Then, 100 ml of the diluted serum was added into the Conjugate vial and mixed by pipetting up and down.For the CSF samples, 10 ml of CSF and 90 ml of Dilution buffer were added to the Conjugate vial.Finally, 80 ml of the mixture was added to the test cassette, and the result was read with a ReaScan+ reader (Reagena) after 20 minutes of incubation.The cut-offs given by the manufacturer for the ReaScan+ C6 LYME IgG are lot specific.Analytical comparison was performed with lot XG21/1.Cut-offs provided by the manufacturer for this lot were arbitrary units <7.00 for negative, arbitrary units 7.00 to 11.99 for equivocal, and arbitrary units ≥12.00 for positive samples.In the analytical comparison, however, we also used cut-offs determined by receiver operating characteristic (ROC) analysis performed in this study.C6 Lyme ELISA TM was used as the reference standard test and was performed as instructed by the manufacturer and absorbances were measured with Multiskan GO (Thermo Scientific, Vantaa, Finland).CSF samples were analyzed using the same protocol with the exception that the samples were diluted 1:5 [19].C6 Lyme ELISA TM was chosen as a reference standard since it has the same antigen (C6 peptide) as the index test for detection of antibodies in human serum, and it is a widely used test in diagnostic laboratories.In the analyses, we used cut-offs given by the manufacturer (C6 Lyme index ≤0.9 for negative samples; ≥1.1 for positive samples) and also the cut-offs we have previously shown to be more optimal (2.4 when C6 Lyme ELI-SA TM is used as the first-tier test; 3.0 when C6 Lyme ELISA TM is used as the second-tier test) [20].
RecomBead Borrelia is a bead-immunoassay with antigens produced by recombinant techniques for the detection of IgG or IgM antibodies against Borrelia in human serum, plasma or CSF.Recom-Bead Borrelia IgM and IgG were used as reference tests when grouping borderline/low positive C6 Lyme index samples into positives/ negatives for the ROC analysis, and when evaluating the samples which gave contradictory results with ReaScan+ C6 LYME IgG and C6 Lyme ELISA TM .RecomBead Borrelia analyses were performed as instructed and with cut-offs given by the manufacturer.
Reference standard results were available for the performers of the ReaScan+ C6 LYME IgG.However, the ReaScan+ C6 LYME IgG result was read with a ReaScan+ reader and not evaluated by the performer of the analyses.The ReaScan+ C6 LYME IgG result, on the other hand, was not available for the performers of the reference tests.Clinical information was not available to the performers of either of the tests.

Analytical accuracy of ReaScan+ C6 LYME IgG
Normality of the data was assessed with Shapiro-Wilk test and, subsequently, correlation was analyzed with Spearman method (IBM SPSS Statistics 27, Armonk, NY).A receiver operating characteristic (ROC) analysis was performed to evaluate the optimal cut-off value in serum for the ReaScan+ C6 LYME IgG in SPSS.In the analysis, 200 Borrelia antibody negative sera (C6 Lyme index ≤0.9)were considered as negatives, 200 Borrelia antibody positive sera (C6 Lyme index ≥3.0)were considered as positives, and 100 sera with borderline or low positive (0.91−2.9)C6 Lyme index were considered as either negatives or positives based on the recomBead Borrelia IgM/IgG results.The sera that were positive or borderline in recomBead Borrelia were considered as positive samples, and sera with a negative recomBead Borrelia result were considered as negative samples.This resulted altogether in 254 positive samples and 239 negative samples.There were 7 samples with borderline or low positive (0.91 −2.9) C6 Lyme index results without recomBead Borrelia results available, and thus, they were excluded from the ROC analysis.Accuracy of ReaScan+ C6 LYME IgG was studied by determining positive percent agreement (PPA; positive samples with ReaScan+ C6 LYME IgG (index test) and C6 Lyme ELISA TM (reference standard)/positive samples with the reference standard test) and negative percent agreement (NPA; negative samples with ReaScan+ C6 LYME IgG (index test) and C6 Lyme ELISA TM (reference standard)/negative samples with the reference standard test) values with different cut-offs with Method Comparison study in Validation Manager TM software (Finbiosoft, Espoo, Finland).
Borrelia antibody positive CSF samples were analyzed in parallel with C6 Lyme ELISA TM and ReaScan+ C6 LYME IgG.Borrelia antibody negative CSF samples were analyzed only with ReaScan+ C6 LYME IgG.

Design and patient characteristics of the clinical performance study and the clinical performance of the ReaScan+ C6 LYME IgG
The clinical performance evaluation was a retrospective case-control study including selected negative and positive left-over serum and CSF samples from 242 patients investigated for suspected LNB in J€ onk€ oping County, Sweden, during the years 2012−2017.Furthermore, serum samples from 100 healthy blood donors and from 20 syphilis seropositive individuals were used in the evaluation (Supplementary Table S1 and Figure S1).The sample size was determined based on calculations in a previous study [14].
For classification of patients investigated for suspected LNB, the following assays were used according to the instructions from the manufacturers: IDEIA Lyme Neuroborreliosis (Oxoid, Hampshire, UK) for paired serum and CSF samples to determine IgM and IgG AI, and Enzygnost Lyme link VlsE/IgG and Enzygnost Borrelia Lyme IgM (Siemens Health Care Diagnostics Products GmbH, Germany) for serum samples.Serum and CSF from patients classified as Possible LNB were analyzed with recomBead Borrelia IgG, a more sensitive method for determination of intrathecal production of Borrelia-specific antibodies as previously shown by us [21].Intrathecal AI was calculated ad modum Reiber [22] and all patients in this group were found to have an elevated AI using this method.
Treponema pallidum antibodies were analyzed by Architect Syphilis TP (Abbott Laboratories, IL) as screening method and Serodia TP-PA (Fujirebio, Shanghai, China) as confirmatory method.Syphilis-positive sera that were reactive in the ReaScan+ C6 LYME IgG assay was analyzed with Liaison Borrelia IgG (Diasorin, Saluggia, Italy) to verify the presence of Borrelia-specific IgG.The blood donor sera originated from another study and had previously been analyzed with the Liaison Borrelia IgG and the Enzygnost Lyme link VlsE/IgG assays [14].
All assays were performed, and cut-off values applied as recommended by the manufacturers, with the exception of Enzygnost Borrelia Lyme IgM where the laboratory used 2.0 (instead of 1.0) as cutoff in clinical routine.Borderline results were considered as positive in the calculations regarding test performance.The ReaScan+ C6 LYME IgG analyses of serum and CSF were performed as instructed by the manufacturer and briefly described above.In the clinical performance analyses, lot XH20/1 was used with cut-offs given by the manufacturer (<6.00: negative; 6.00−9.99:borderline; >10.00: positive; equivalent to the cut-offs of the lot XG21/1 used in the analytical comparison study).Borderline results of ReaScan+ C6 LYME IgG were considered positive in the statistical calculations.
The patients investigated for LNB were classified according to the criteria established by the European Federation of Neurological Societies [5].The classification was done by 2 clinical microbiologists, 1 of which is also a specialist in infectious diseases.Importantly, classification was not solely based on results from another serological test, but also on retrospective assessment of clinical symptoms and other biochemical laboratory tests.Clinical data were collected through review of medical charts.Patients were classified as Definite LNB (n = 108), Possible LNB (n = 31) or Non-LNB (n = 103) (see Supplementary Table S2).The Non-LNB group used in this study was selected among the patients investigated for suspected LNB but who were found to not have elevated Borrelia-specific AI, CSF pleocytosis or detectable Borrelia-specific antibodies in serum.The distribution of diagnoses or main symptoms (in cases where specific diagnosis was not obtained) in the non-LNB group is presented in Supplementary Table S5.

Ethical considerations
The study was conducted in accordance with the Declaration of Helsinki.Samples used in the analytical accuracy study were identified retrospectively from the laboratory information management system of the Clinical microbiology laboratory of the Turku University Hospital.Permission for the study was obtained from the Health District of South-Western Finland (Permission No T183/2018).All samples were collected with informed consent from patients suspected to have LB/LNB as a part of routine clinical practice.All samples were coded, and strict anonymity maintained throughout the study.According to the Finnish Medical Research Act (No. 488/1999), Chapter 1, Sections 1, 2 and 3, the research in the analytical accuracy study is not medical research, and thus, it was not necessary to obtain a separate approval from the local Ethics Committee.
The clinical performance evaluation study was approved by the Regional Ethics Review Board in Link€ oping; 2018/525-31, 2013/238-31 and 2016/211-32.All samples were collected with informed biobank consent from patients/blood donors as part of routine clinical practice.The samples were pseudonymized by a study code prior to analysis.

Results
Samples of the analytical accuracy study originated both from women (54%) and men (46%).All age groups were represented among the samples with median age of 57 years, and the range from 1 to 94 years.Serum and CSF samples of the clinical performance evaluation originated from patients investigated for suspected LB/ LNB.Patient characteristics are comprehensively presented in supplementary material (see Supplementary Table S3).

Results of the analytical accuracy study with serum samples
There was a strong correlation between C6 Lyme ELISA TM and ReaScan+ C6 LYME IgG results (r s = 0.814, P < 0.001; Fig. 1A).In the ROC analysis, C6 Lyme ELISA TM serum sample results were categorized as explained in the Materials and methods section.The area under the curve (AUC) was 0.921 (P < 0.001; 95 % CI 0.896−0.946;Fig. 1B).Sensitivity and specificity with the lot-specific cut-offs for The results of these 2 methods correlated strongly with each other (r s = 0.814, P < 0.001).(B) ROC-curve for the ReaScan+ C6 LYME IgG results by using C6 Lyme ELI-SA TM as a reference method.C6 Lyme ELISA TM results ≥3.0 were considered as positives, results ≤0.9 were considered as negatives, and results in between these 2 cut-offs were determined as positives or negatives depending on the results of recomBead Borrelia IgM/IgG analyses.Cut-off 5.3 for the ReaScan+ C6 LYME IgG results was found to be optimal when sensitivity and specificity are considered equally important.Other turning points in the ROC-curve were 14.2 and 2.3.Also, the cut-offs given by the manufacturer for the lot (7.0 and 12.0) are presented.(C) Correlation between ReaScan+ C6 LYME IgG and C6 Lyme ELISA TM results of the CSF samples in the analytical accuracy study.Correlation was analyzed with Spearman method in SPSS.The results of these 2 methods correlated moderately with each other (r s = 0. 654, P < 0.001).
ReaScan+ C6 LYME IgG given by the manufacturer (7.0 and 12.0), and with additional cut-offs based on the ROC curve are presented in Table 1.The use of the cut-off 7.0 resulted in sensitivity and specificity of 83 % and 91.6%, respectively, when C6 Lyme ELISA TM was used as reference standard.However, according to the ROC curve when sensitivity and specificity are considered equally important, the optimal cut-off value for the used ReaScan+ C6 LYME IgG lot would be 5.3, which results in sensitivity of 85.8% and specificity of 90 %.Interestingly, the use of cut-off 2.4 for C6 Lyme ELISA TM [20] in the ROC curve analysis results in the same optimal cut-off 5.3 for ReaScan+ C6 LYME IgG (data not shown).
The results of PPA and NPA with different cut-offs for ReaScan+ C6 LYME IgG test and for the C6 Lyme ELISA TM are shown in Table 2.The cut-offs used in the analyses for ReaScan+ C6 LYME IgG are the lotspecific cut-offs provided by the manufacturer and the cut-offs derived from the ROC-analysis of this study.The cut-offs for C6 Lyme ELISA TM are the cut-offs given by the manufacturer and the cut-offs determined in our previous study [20].
There were 22 serum samples with the Lyme index ≥3.0 but remaining negative (<5.3) in the ReaScan+ C6 LYME IgG.We performed recomBead Borrelia IgM/IgG analyses to evaluate whether C6 reactivity originates from IgM antibodies which are not detected by the ReaScan+ C6 LYME IgG.As a result, 1 of these samples clearly had only IgM antibodies towards VlsE (the parental molecule of C6 peptide) while, in fact, 10 of the samples had IgG antibodies to VlsE.Moreover, 8 samples that had a Lyme index ≥3.0 but remained negative (<5.3) in ReaScan+ C6 LYME IgG were completely negative in the recomBead Borrelia IgM/IgG analysis.The remaining 3 samples with Lyme index ≥3.0 but negative (<5.3) in the ReaScan+ C6 LYME IgG were positive in recomBead analysis but the reactivity was towards OspC and not VlsE.

Results of the analytical accuracy study with CSF samples
ReaScan+ C6 LYME IgG results of the 23 Borrelia antibody positive CSF samples were normally distributed while C6 Lyme ELISA TM results of these samples were bimodally distributed.Thus, the correlation between the results of the 2 methods was analyzed with Spearman method.There was a moderate correlation between the results obtained with these 2 methods (r s = 0.654, P < 0.001; Fig. 1C).C6 Lyme ELISA TM result of the Borrelia antibody positive CSF samples ranged between 2.47 to 9.91.All of these samples were positive also with ReaScan+ C6 LYME IgG (range 38.24−139.01arbitrary units).Of the 100 Borrelia antibody negative CSF samples 99 were negative (<7.0) also in ReaScan+ C6 LYME IgG.One sample was reactive with ReaScan+ C6 LYME IgG (13.5 arbitrary units).

Results of the clinical performance study
The results of the ReaScan+ C6 LYME IgG analysis of serum samples are presented in Table 3.When the lot-specific cut-off values recommended by the manufacturer were used, the clinical sensitivity for Definite LNB was 97.2%, for Possible LNB 96.8% and for Definite + Possible LNB taken together 97.1%.The clinical specificity of the assay was 96.1%.For comparison, the results of the Enzygnost IgG and IgM analyses of the serum samples are presented in Supplementary Table S4, showing a sensitivity of 92.6% in Definite LNB, 87.1% in Possible LNB and 91.4% for Definite + Possible LNB taken together for the IgG assay.
The results of the ReaScan+ C6 LYME IgG analysis in CSF samples are presented also in Table 3.The clinical sensitivity for Definite LNB was 97.2%, for Possible LNB 87.1% and for Definite + Possible LNB taken together 95.0% when the cut-off recommended by the manufacturer was used.The clinical specificity of the assay was 100% in our study.In the sample material of healthy blood donors (n = 100), 17 tested positive in the ReaScan+ C6 LYME IgG assay.Two of these blood donors tested negative in both the Liaison Borrelia IgG and the Enzygnost Lyme link VlsE/IgG assays, while the remaining 15 blood donors tested positive in both assays.Four of the blood donor sera showed borderline results in the ReaScan+ C6 LYME IgG assay; 1 tested positive in the Liaison Borrelia IgG assay but negative in the Enzygnost Lyme link VlsE/IgG assay, while 1 tested positive in both assays and 2 tested negative in both.
Cross-reactivity was assessed by analyzing 20 serum samples positive in syphilis screening.Three of the samples were positive and 17 were negative with the ReaScan+ C6 LYME IgG assay.Two of the 3 positive samples were also positive with the Liaison Borrelia IgG test.

Discussion
To our knowledge, there has been only 1 POC test on the market for measuring Borrelia antibodies in human samples.However, no scientific literature on the performance of the test is available.In this study, we evaluated the analytical performance of the ReaScan+ C6 LYME IgG, a new rapid lateral flow test head-to-head with C6 Lyme ELISA TM , and importantly, also the clinical performance was defined using well-characterized LNB patient cohorts.The novel test appears to be a well-performing test.
In the analytical comparison, the use of the cut-off provided by the manufacturer (7.0) for the ReaScan+ C6 LYME IgG resulted in sensitivity and specificity of 83 % and 91.6%, respectively.With the more optimal cut-off 5.3 for the evaluated assay lot, the sensitivity and specificity were 85.8 % and 90 %, respectively, however, suggesting that the cut-off values by the manufacturer perform well.
In the clinical performance evaluation with the cut-offs recommended by the manufacturer, the clinical sensitivity of the ReaScan+ C6 LYME IgG test in serum was 97.1% and in CSF 95.0% (Definite and Possible LNB taken together).The clinical specificity in serum was 96.1% and in CSF 100%.Thus, the cut-off suggested by the manufacturer seems appropriate also based on this evaluation.The clinical performance study was performed as a case-control study using selected negative and positive serum and CSF samples, and therefore the prevalence of antibodies does not represent the seroprevalence in the general population.Thus, calculations of the assay's positive and negative predictive values could not be performed.Further, the specificity presented for the index test was calculated using non-LNB serum samples, which were pre-tested to be Borrelia antibody negative, which may lead to inflated specificity compared to a situation, where the non-LNB samples would originate from a non-preselected general population with a certain degree of background LB seropositivity.Using the ReaScan+ C6 LYME IgG assay, the seroprevalence among blood donors in J€ onk€ oping County, Sweden, was 21% when borderline results were interpreted as positive.This finding is in line with a previous study including blood donors from the same region and 5 different ELISA [14].
There are some limitations in our study.First, there is an imperfect reference standard bias in our analytical accuracy study since the C6 Lyme ELISA TM used as a reference standard cannot be considered as 100 % accurate.Second, in the clinical performance study, we only included patients with suspected LNB and no other manifestations of LB.However, the advantage of using LNB samples is that diagnostic criteria are established and explicit, and LNB can be assumed to be representative of disseminated LB.Third, the manufacturer of the ReaScan+ C6 LYME IgG uses lot-specific cut-offs for the test which hampers the comparison of different experiments and especially individual studies to each other.Furthermore, calculation of intrathecal AI requires analysis of reference molecules such as albumin and total-IgG in serum and CSF.The intended use of ReaScan+ C6 LYME IgG as a point-of-care test will therefore not allow for AI calculations, which is a limitation of the assay.Ambiguous cases may therefore require additional/confirmatory analyses in a clinical laboratory.
In conclusion, ReaScan+ C6 LYME IgG test is sensitive and specific as a POC test in LB serology.As for all serological assays, it is advisable to determine the test cut-offs locally considering the diagnostic application of the test.Also, the results must be interpreted in relation to the patient's medical history, as well as clinical signs and symptoms, for a correct diagnosis.
research agreement with Abbott Laboratories (IL).The other authors report no conflicts of interest.

Fig. 1 .
Fig. 1. (A) Correlation between ReaScan+ C6 LYME IgG and C6 Lyme ELISA TM results of the serum samples in the analytical accuracy study.Correlation was analyzed with Spearman method in SPSS.The results of these 2 methods correlated strongly with each other (r s = 0.814, P < 0.001).(B) ROC-curve for the ReaScan+ C6 LYME IgG results by using C6 Lyme ELI-SA TM as a reference method.C6 Lyme ELISA TM results ≥3.0 were considered as positives, results ≤0.9 were considered as negatives, and results in between these 2 cut-offs were determined as positives or negatives depending on the results of recomBead Borrelia IgM/IgG analyses.Cut-off 5.3 for the ReaScan+ C6 LYME IgG results was found to be optimal when sensitivity and specificity are considered equally important.Other turning points in the ROC-curve were 14.2 and 2.3.Also, the cut-offs given by the manufacturer for the lot (7.0 and 12.0) are presented.(C) Correlation between ReaScan+ C6 LYME IgG and C6 Lyme ELISA TM results of the CSF samples in the analytical accuracy study.Correlation was analyzed with Spearman method in SPSS.The results of these 2 methods correlated moderately with each other (r s = 0. 654, P < 0.001).

Table 1
Sensitivity and specificity in serum of ReaScan+ C6 LYME IgG with different cut-offs when C6 Lyme ELISA TM is used as a reference standard.

Table 2
Accuracy of ReaScan+ C6 LYME IgG with different cut-offs when using C6 Lyme ELISA TM as a reference.