Laboratory evaluation of two point-of-care test kits for the identification of infectious syphilis

Background: Syphilis is a sexually transmitted disease that can have atypical clinical presentations. Conventional laboratory tests to confirm the diagnosis are not rapid enough to affect clinical decision on treatment and contact tracing. Rapid point-of-care tests (POCT) can be useful for control of infectious diseases; however, no POCT for syphilis detection is currently available in Canada. The aim of this study is to evaluate two POCTs (RevealTM Rapid TP (Treponema pallidum) Antibody test and DPP® Syphilis Screen and Confirm test) for detection of infectious syphilis. Methods: One hundred serum samples with known syphilis serological status, based on treponemal and non-treponemal test results, were analysed in the laboratory with two POCTs by two independent operators in a blind fashion. Results were analysed to evaluate their ability to detect infectious syphilis. Results: The Reveal Rapid TP Antibody POCT showed an overall sensitivity of 95.0% and a specificity of 83.3%, while the DPP Syphilis Screen and Confirm POCT showed a sensitivity of 87.5% and a specificity of 98.3%. Both POCTs gave a sensitivity of 100% on active syphilis samples with Venereal Disease Research Laboratory (VDRL) titres of greater than 1:4, but their sensitivities decreased for samples with low VDRL titres. Both POCTs gave weakly or very weakly reactive results on 11.3%–25.0% of the treponemal antibody positive samples. Conclusion: This laboratory evaluation has shown promising results for both POCTs to detect infectious syphilis. Further evaluations in the field would be required to confirm this preliminary finding. This work is licensed under a Creative Commons Attribution 4.0 International License.


Introduction
Infectious syphilis has been on the rise in North America since the early 2000s (1,2). In recent years, the increase has been substantial: in Canada, the rate of infectious syphilis has increased about 3.4-fold within 10 years from just under five cases per 100,000 population in 2009 to about 17 cases per 100,000 population in 2018 (3). Also, a shift in increasing rates of infectious syphilis in females of reproductive age (15 to 39 years) has been observed in both the United States (US) and Canada (2,4), leading to an increase in congenital syphilis in both countries. The reasons behind the increase in syphilis cases in Canada have been discussed recently (5), and may include better access to testing, increased sensitivity of enzyme immunoassays employed in the screening of syphilis, lapse in the practice of safe sex and changes in the social behaviour coupled with availability of social media platforms to facilitate recreational sexual encounters. To combat this increase, education to increase better awareness of syphilis, more timely and in-depth surveillance data to allow interventions to be developed that can target at risk behaviours or different at-risk ethnic groups, and better testing methods that can provide immediate results to allow for earlier treatment and contact tracing, have also been suggested (6).
Syphilis has been described as a great imitator (7); clinical and laboratory diagnosis of infectious syphilis can be challenging. The most rapid and confirmatory diagnostic test for infectious syphilis is dark-field microscopy, which looks for motile spirochetes in clinical specimens, but this method is now almost non-existent in Canada due to the required technical expertise and the relative insensitivity of the method. Conventionally, syphilis is diagnosed serologically by measuring antibodies to both treponemal antigens (e.g. using chemiluminescent microparticle immunoassay, CMIA, which detects IgG and IgM antibodies) and non-treponemal antigens (e.g. using either rapid plasmin reagin, RPR, or Venereal Disease Research Laboratory, VDRL, tests). Patients with active or infectious syphilis show positive results with both the treponemal and non-treponemal tests while patients with past syphilis show positive results only with the treponemal test. Although these conventional serological tests and the newer molecular polymerase chain reaction diagnostic methods are accurate and sensitive, the results are not available rapidly enough to allow clinical decisions on both treatment of patients and contact tracing. Furthermore, there are currently no licensed commercial nucleic acid amplification tests for syphilis on the market.
While point-of-care tests (POCTs) for syphilis have been developed in the early 2000s and have been in use in a number of resource limited countries that lack the laboratory infrastructure, equipment and qualified personnel to carry out conventional laboratory tests, their use in resource rich countries are less well defined. Indeed, the US Centers for Disease Control and Prevention and the American Association of Public Health Laboratories, in consultation with other sexually transmitted disease experts, concluded in 2017 that there was insufficient data to recommend the use of POCT for routine syphilis testing in the US (8).
By October 2, 2020, the US Food and Drug Administration licensed two syphilis POCT kits (9): the Syphilis Health Check POC assay (Diagnostic Direct LLC, Stone Harbor, New Jersey or Trinity Biotech, Jamestown, New York) and the DPP POCT for HIV and Syphilis serology (Chembio Diagnostic System, Medford, New York). However, some evaluations of these test kits were based on small numbers of positive samples as well as serum specimens obtained by venipuncture rather than finger-prick whole blood, which is the specimen of choice for POCT (due to ease of obtaining specimen). Using finger-prick whole blood specimens, the Syphilis Health Check POCT had a sensitivity of 100% (with seven positive samples) when compared to consensus reference testing of RPR and treponemal enzyme immunoassay. This sensitivity decreased to 77.8% (with 18 positive samples) when compared with Syphilis Health Check POCT done with serum in the laboratory, and to only 50.0% sensitivity (with 16 positive samples) when compared with the treponemal enzyme immunoassay (10). Difficulty in reading the Syphilis Health Check POCT has also been reported for those specimens considered positive in the POCT but negative by consensus reference testing. A 2016 study that reported high sensitivity (94.7%) and specificity (100%) for the DPP POCT for HIV and Syphilis serology, the results for the syphilis component of the POCT were based on testing serum specimens and comparing the data to that obtained by the Treponema pallidum (T. pallidum) particle agglutination test (11). Currently, no syphilis POCT has been licensed in Canada.
In this study, our objective was to evaluate two syphilis POCTs (one has treponemal antigen and another that has both treponemal and non-treponemal antigens) to detect infectious syphilis and report on their preliminary performance based on studies done in the laboratory using serum samples with known syphilis serological test results.

Materials and methods
Patient serum samples and testing by conventional syphilis serology One hundred serum samples from individual subjects, identified as either syphilis positive or syphilis negative at the Cadham Provincial Public Health Laboratory (Winnipeg, Manitoba), were provided in a blind fashion (regarding the syphilis test results) to the National Microbiology Laboratory for evaluation by POCT. Samples were defined as syphilis positive if tested positive by CMIA for antibodies to treponemal antigens. Samples were defined as syphilis negative when CMIA testing gave negative results. Qualitative RPR was performed on CMIA positive samples, followed by a quantitative VDRL test at the Cadham Provincial Public Health Laboratory. Both RPR and VDRL are considered as non-treponemal tests that detect non-treponemal antibodies, and positive results of either the RPR or VDRL tests (and in the presence of positive CMIA results) are indicative of active or infectious syphilis. Also, a different non-treponemal (VDRL) test was used to confirm the RPR qualitative results to increase the specificity of the non-treponemal test data.

Point-of-care test
Since there are no POCT kits licensed for sale in Canada, we obtained permission from Health Canada Special Access Program before the DPP ® Syphilis Screen and Confirm Assay (ChemBio Diagnostic Systems, Inc., Medford, New York) and the Reveal TM Rapid TP Antibody test (MedMira Laboratories, Inc., Halifax, Nova Scotia) were purchased directly from the suppliers. The Reveal Rapid TP Antibody POCT detects antibodies to synthetic peptides that resemble T. pallidum recombinant antigens and can be used with serum, whole blood or plasma samples (12). The DPP Syphilis Screen and Confirm POCT detects antibodies to both treponemal and non-treponemal antigens in serum, whole blood or plasma specimens (13).
Two qualified laboratorians at the National Microbiology Laboratory carried out independent testing on each of the 100 serum samples with both POCT kits according to the manufacturers' instructions. Results were read manually and recorded as reactive, weakly reactive or very weakly reactive, even though any intensity of colour change in the test zone is considered reactive according to the package insert for both the Reveal Rapid TP Antibody POCT and the DPP Syphilis Screen SURVEILLANCE and Confirm POCT. We chose to score the test results in more detail since reading intensity of colour change is subjective and difficulties in reading some syphilis POCTs have been described by others. Weakly reactive and very weakly reactive results were determined by comparison with the reactions given by the controls as well as other reactive samples obtained in the same run. When discordant results were obtained by the two independent operators, the test was repeated a second time by both operators and if the results matched on the second test, they were then used. In no circumstances, were any major discordant results encountered, such as reactive by one operator and non-reactive by another operator.
After the 100 samples have been tested by both operators using the two POCTs, results were provided to the investigators at the Cadham Provincial Public Health Laboratory, who then disclosed the CMIA, RPR and VDRL results to the investigators at the National Microbiology Laboratory to perform the analysis. The sensitivity (percent of infectious syphilis samples tested positive), specificity (percent of samples without infectious syphilis tested negative), false positive rate (percent of samples without infectious syphilis tested positive) and false negative rate (percent of infectious syphilis samples tested negative) of the POCTs for detection of infectious syphilis were calculated.

Characteristics of clinical samples
Of the 100 samples provided for POCT evaluations, 80 were reactive or positive by CMIA (sample over cut-off, S/CO, values ranged from 1.11 to 27.11, mean=15.68, median=16.81) and 20 were non-reactive or negative (S/CO values ranged from 0.03 to 0.14, mean=0.61, median=0.05). The 20 negative CMIA samples did not undergo further testing and were regarded as syphilis negative for both active and past infections. The 80 CMIA positive samples were further divided into 40 that were non-reactive by RPR (which suggested absence of active syphilis infections but with treated past syphilis infections) and 40 that were RPR reactive (which suggested active syphilis infections). Among the 40 CMIA positive and RPR reactive samples, six were only weakly reactive by RPR. Of these six RPR weakly reactive samples, five were tested by VDRL to give reactive results with an undiluted sample. The remaining CMIA positive, RPR weakly reactive sample was VDRL reactive at 1:2 dilution. Of the 34 CMIA positive and RPR reactive samples, 14 had low or no VDRL titres (n=7 with VDRL titres of 1:2, n=6 with VDRL titres of 1:4, and n=1 was VDRL reactive at undiluted), while 20 samples were found to have VDRL titres of at least 1:8.

Performance of Reveal Rapid
Treponema pallidum Antibody and DPP Syphilis Screen and Confirm tests Table 1 shows the results of the Reveal Rapid TP Antibody POCT and the DPP Syphilis Screen and Confirm POCT on the three categories of serum samples (no syphilis infection, past syphilis infection and active syphilis infection). Sera from subjects with no evidence of current or past syphilis infection were found to be non-reactive by both POCTs. Only ten of the 40 samples categorised as having past syphilis infections were reactive in the Reveal Rapid TP Antibody POCT. The DPP Syphilis Screen and Confirm POCT detected antibodies to the treponemal antigen in 19 of the 40 samples from subjects with past syphilis infection; only one sample was reactive for both treponemal and non-treponemal antigens, while the remaining 20 were non-reactive for both antigens.
The percentage of samples from those with active syphilis and reactive by both Reveal Rapid TP Antibody and DPP Syphilis Screen and Confirm POCTs are correlated to the quantitative non-treponemal test results, and this is particularly evident with the DPP Syphilis Screen and Confirm POCT (Table 1).
If very weakly reactive reactions were read conservatively as non-reactive (shown in parentheses in Tables 1 and Table 2), fewer samples from those with active and past syphilis infections were found to be reactive in both POCT. Twenty samples when tested by the Reveal Rapid TP Antibody POCT test gave either weakly reactive (n=12; three were past syphilis samples and nine were active syphilis samples) or very weakly reactive (n=8; six were past syphilis samples and two were active syphilis samples) reactions. In the DPP Syphilis Screen and Confirm POCT, 12 samples gave weakly reactive (n=1; past syphilis) or very weakly reactive (n=11; six past syphilis and five active syphilis) reactions in the treponemal antigen, while nine samples (n=8 active syphilis and one past syphilis) gave very weakly reactive results in the non-treponemal antigen (data not shown). Most syphilis POCT kits on the market use treponemal antigen to detect anti-treponemal antibodies and only a handful of kits employ both treponemal and non-treponemal antigens that can simultaneously detect anti-treponemal as well as non-treponemal antibodies. Tests that detect anti-treponemal antibodies alone This study shows the two POCT evaluated have good sensitivity and specificity for the detection of infectious syphilis. However, in those with low non-treponemal antibody titres, their performance may be compromised, thus confirming the findings by others for this category of active syphilis patients (10,14,15). Of the two POCTs evaluated, the Reveal Rapid TP Antibody POCT may be affected less by the low non-treponemal antibody titres and was able to maintain reasonable sensitivity of 92.3% to 85.7% (to detect infectious syphilis) even in samples with VDRL antibody titres of 1:2 to 1:4 or only reactive in undiluted sera.
In the Reveal Rapid TP Antibody POCT, 25.0% of the CMIA positive samples were found to give either weakly or very weakly reactive results. In the DPP Syphilis Screen and Confirm POCT, 11.3% and 15.0% of the CMIA positive samples were found to give either weakly or very weakly reactive results to the non-treponemal and treponemal components, respectively. Both weakly reactive and especially very weakly reactive results may lead to difficulty in reading these tests in actual field use by non-trained operators. Also, the use of finger-prick whole blood specimens can potentially make reading the results even more difficult. Both training and clear guidelines on how to read POCT results may be required. Some POCT come with an electronic reader (11), and this may avoid inconsistencies in reading results between samples and operators. Another potential challenge in the implementation of POCT in sexually transmitted infection clinics or rural areas is related to the de-centralized syphilis testing and the resultant difficulties in the capturing of test results by public health for surveillance purpose and for the subsequent development of public health intervention policies.
One of the reasons that has been put forward to explain the recent increase in syphilis infection in Canada (in both urban and remote or rural locations) is the inequity of accessibility within the healthcare system, including the testing facilities for infections (5,6). Vulnerable populations, whether living in urban or rural areas, can be hard to reach due to homeless or unstable housing, mistrust in the healthcare system or lack of accessible testing facilities in remote localities. Alternate testing approaches, such as the use of POCT, may help to meet the need created by the circumstances of these at-risk populations. Current POCT provide only qualitative results of positive or negative findings, and the lack of quantitative result on antibody titres may prevent their use for monitoring response to treatment as well as differentiating repeated or reinfections from past infections. This contrasts with the current practice of using a quantitative non-treponemal test result (such as RPR or VDRL titres) to monitor for either a decrease in antibody titres as proof of positive response to treatment or a rising antibody titer in the case of no response to treatment or an occurrence of reinfection.

Limitations
There are several limitations in the interpretation of this study. First, serum samples collected for conventional syphilis serology were used instead of finger prick whole blood specimens, which will likely be the sample of choice when these tests are used in the field. Secondly, these assays were done in a controlled laboratory environment by trained laboratory staff with experience in carrying out clinical diagnostic tests; therefore, actual performance of these POCT in the field or in a real-world situation may differ. Another limitation is the small number of samples tested, especially in the subcategories of active syphilis cases with different VDRL titres and, as such, the sensitivity data may be inaccurate. Finally, the usefulness or performance of a POCT would also depend on the setting or prevalence of the disease where the POCT will be deployed.

Conclusion
The two POCT evaluated in this laboratory study appeared to show promising results for detection of infectious syphilis especially in those with non-treponemal antibody titres equal to or greater than 1:4, but not for detection of past syphilis. Further evaluations in the field will be required in order to confirm the findings in this preliminary study. Field evaluations and clinical studies will offer further experience in the use of syphilis POCT that may ultimately contribute towards better control of infectious and congenital syphilis. Data from this kind of study may also be useful for potential future licensure of such test kits in Canada.