Optimal Cutoff and Accuracy of an IgM Enzyme-Linked Immunosorbent Assay for Diagnosis of Acute Scrub Typhus in Northern Thailand: an Alternative Reference Method to the IgM Immunofluorescence Assay

The enzyme-linked immunosorbent assay (ELISA) has been proposed as an alternative serologic diagnostic test to the indirect immunofluorescence assay (IFA) for scrub typhus. Here, we systematically determine the optimal sample dilution and cutoff optical density (OD) and estimate the accuracy of IgM ELISA using Bayesian latent class models (LCMs). Data from 135 patients with undifferentiated fever were reevaluated using Bayesian LCMs. Every patient was evaluated for the presence of an eschar and tested with a blood culture for Orientia tsutsugamushi, three different PCR assays, and an IgM IFA. The IgM ELISA was performed for every sample at sample dilutions from 1:100 to 1:102,400 using crude whole-cell antigens of the Karp, Kato, and Gilliam strains of O. tsutsugamushi developed by the Naval Medical Research Center. We used Bayesian LCMs to generate unbiased receiver operating characteristic curves and found that the sample dilution of 1:400 was optimal for the IgM ELISA. With the optimal cutoff OD of 1.474 at a sample dilution of 1:400, the IgM ELISA had a sensitivity of 85.7% (95% credible interval [CrI], 77.4% to 86.7%) and a specificity of 98.1% (95% CrI, 97.2% to 100%) using paired samples. For the ELISA, the OD could be determined objectively and quickly, in contrast to the reading of IFA slides, which was both subjective and labor-intensive. The IgM ELISA for scrub typhus has high diagnostic accuracy and is less subjective than the IgM IFA. We suggest that the IgM ELISA may be used as an alternative reference test to the IgM IFA for the serological diagnosis of scrub typhus.

S crub typhus and murine typhus are important causes of acute febrile illness in Thailand and Laos (1)(2)(3)(4)(5)(6). Diagnosis of scrub typhus still relies on serology due to the convenience of sample collection; the relative simplicity of the assays; and the expense, complexities, and deficiencies in sensitivity of other modalities (5,7,8). The indirect immunofluorescence assay (IFA) is the serological reference standard despite being inconsistent in terms of antigen composition, diagnostic cutoff (9), and repeatability (10). Enzyme-linked immunosorbent assays (ELISAs) offer a number of advantages over IFA, particularly in terms of simplicity, reading objectivity, and sample throughput. Use of ELISA for acute scrub typhus diagnosis has been previously evaluated (11,12); however, its optimal sample dilution and cutoff optical density (OD) have not been determined.
Defining the cutoff value of a new serological test is conventionally done by comparing the results of the serological test of interest with the results of a reference test. However, this conventional approach assumes that the reference test has perfect sensitivity and specificity; if this is not the case, then estimates of sensitivity and specificity for the test under evaluation will be biased and the selected cutoff value suboptimal (13). Bayesian latent class models (LCMs) are increasingly used to determine the optimal cutoff value and to estimate the accuracy of a diagnostic test, since they do not need to assume that the accuracy of reference tests is perfect (14)(15)(16)(17). We recently used Bayesian LCMs to reanalyze our existing data set from a previously published prospective study (18) and showed that an IgM IFA with a cutoff titer of Ն1:12,800 in admission samples or a 4-fold rise to Ն1:200 in convalescent/ discharge samples had low sensitivity and specificity (70.0% and 83.8%, respectively) (19) and that optimal cutoff titers for IgM IFA of Ն1:3,200 in admission specimens or a Ն4-fold rise to Ն1: 3,200 in convalescent/discharge specimens provided higher sensitivity and specificity (81.6% and 100%, respectively) (20).
In this study, we used Bayesian LCMs and blood samples collected from our previously published prospective study (18) to systematically determine the optimal sample dilution and optimal cutoff OD and to examine the accuracy of the Naval Medical Research Center (NMRC) "in-house" scrub typhus IgM ELISA.

MATERIALS AND METHODS
Study patients and blood samples. The data set and blood samples used in this study were from patients recruited into a prospective study of acute febrile illness from August 2007 to August 2008 in Chiangrai, Thailand (18). In brief, 161 patients over 15 years old presenting with acute fever of less than 2 weeks, with three negative malaria blood smears and no evidence of a primary focus of infection, were recruited into the study (18). Every patient was examined for the presence of an eschar. Admission blood samples were collected and tested using in vitro culture for Orientia tsutsugamushi, a nested-PCR (nPCR) assay targeting the 56-kDa gene, a 47-kDa gene-based quantitative real-time PCR (qPCR) assay, a groELbased qPCR assay, and an IgM IFA. Convalescent-phase or hospital discharge (convalescent/discharge) blood samples were also collected and tested by IgM IFA. All the blood samples were kept at Ϫ80°C and tested using the IgM ELISA in 2014.
Ethical statement. Ethical approval for the prospective study was obtained from the ethical committee of Chiangrai Hospital; the Ministry of Public Health, Thailand; and the Oxford Tropical Research Ethics Committee, United Kingdom. Signed written informed consent was obtained from every patient before sample collection (18).
The IgM ELISA was essentially the same as that previously described by Suwanabun et al. (11). All ELISAs were performed at the Mahidol-Oxford Tropical Medicine Research Unit (MORU), Bangkok, Thailand. In brief, whole-cell antigen lysates of Karp, Kato, and Gilliam reference strains of O. tsutsugamushi and mock-infected cell lysate produced at the Viral and Rickettsial Diseases Department of the NMRC, Silver Spring, Maryland, USA (11,12), were used as ELISA antigens. The scrub typhus antigen quality was assessed for each strain of O. tsutsugamushi Karp, Kato, and Gilliam for identity (qPCR and sequencing) and adventitious contamination (blind culture to blood agar plates, thioglycolate broth, and qPCR for mycoplasma). To standardize the antigen preparations, each strain was assessed for reactivity with reference scrub typhus-positive and -negative human serum controls at the NMRC. Following assessment, the reactivity of each antigen strain was compared to those of previous antigen batches, and the concentration was adjusted to permit the use of the component antigens at a standard dilution of 1:1,000 in coating buffer. In this evaluation, a single batch of component antigens was used at the MORU. All ELISAs were performed by two experienced operators (A.T. and S.J.). Serum samples were serially 2-fold diluted from 1:100 to 1:102,400. Negative-and positive-control samples were used as a control for within-and between-day consistency and were included in four wells each on each plate (2 per control sample). Bound anti-O. tsutsugamushi IgM antibodies were detected with anti-human IgM peroxidase conjugate (Invitrogen Corporation, USA) and tetramethylbenzidine substrate (KPL Inc., Maryland, USA). The OD was read at a wavelength of 450 nm (minus a reference OD value read at 650 nm) with a microtiter plate reader (Multiskan FC; Thermo Scientific, Finland). The ODs from the mock-antigen wells were subtracted as background absorbance to give a final average total net absorbance (net OD, or OD at 450 nm).
Statistical analysis. The objective of the study was to determine an optimal sample dilution and a single optimal cutoff titer for IgM ELISA that provided the highest accuracy, not only when an admission sample was initially available, but also when a convalescent/discharge sample was available. Therefore, only patients with both admission and convalescent/ discharge samples were included in the study, and we performed two stages of statistical analysis.
The first stage represented the acute clinical situation where only an admission sample was available. We used Bayesian LCMs to generate unbiased receiver operating characteristic (ROC) curves for the sensitivities and specificities of all possible cutoff titers of IgM ELISA in the admission sample alone, without using the convalescent/discharge sample IgM IFA and IgM ELISA results. We performed this for all possible sample dilutions. In brief, Bayesian LCMs estimated prevalence and the sensitivity and specificity of each diagnostic test with their 95% credible intervals (CrIs) using the Markov Chain Monte Carlo (MCMC) method (17). Bayesian LCMs do not assume that any diagnostic test or combination of diagnostic tests is perfect. The true disease status of each patient was estimated by the model in each MCMC iteration and expressed as the overall disease prevalence. The diagnostic tests included in the model were culture, a combination of PCR assays, IgM IFA for the admission sample alone, ELISA IgM for the admission sample alone, and presence of an eschar. The combination of PCR assays was considered positive when at least two out of three PCR assays, targeting the 56-kDa, 47-kDa, and groEL genes, were positive, as previously described (18). Models that took account of correlation between the IgM IFA and the IgM ELISA were used (19,20,25). The IgM IFA in the first model was considered positive when the admission IgM IFA titer was Ն1:3,200 (20). Unbiased ROC curves were generated as previously described (20).
The second stage of analysis represented the situation when a convalescent/discharge sample was available. We used Bayesian LCMs to generate unbiased ROC curves for the sensitivities and specificities of all possible cutoff titers of IgM ELISA using paired samples and for all possible sample dilutions. The higher OD of either the admission sample or the convalescent/discharge sample was used. The IgM IFA in the second model was considered positive when the admission IgM IFA titer was Ն1:3,200 or there was a 4-fold rise to Ն1:3,200 in convalescent/discharge samples compared to admission samples (20).
The optimal sample dilution was selected by considering that it should provide a cutoff OD between 1.00 and 2.00, which was recommended by the microtiter plate reader manufacturer as the central and optimal regions of the linear detection range (0 to 3 OD units) (Thermoscientific). The optimal cutoff OD was selected by considering that it should provide the highest accuracy in both situations (i.e., an admission sample alone and with paired convalescent/discharge samples). We evaluated the sensitivity, specificity, and accuracy of all possible ODs between the recommended cutoff ODs suggested by both models side by side. The OD that had the highest overall accuracy was selected as the optimal cutoff OD.
All Bayesian LCMs assumed that no prior information (noninformative priors) about the unknown parameters (i.e., prevalence, sensitivities, and specificities) was available, except that the specificity of the culture was fixed at 100%. Bayesian LCMs were performed in WinBUGS 1.4 (26). Texts S1 and S2 in the supplemental material provide full data sets and the models used, respectively.
Post hoc model evaluation. To validate the sensitivity of the IgM ELISA estimated by the Bayesian LCMs, we estimated the naive sensitivity of the IgM ELISA in the patients who had a firm diagnosis of scrub typhus. The firm diagnosis of scrub typhus was made on the basis of either a positive blood culture, a combination of positive PCR assays, or the presence of an eschar in our data set. To validate the specificity of the IgM ELISA estimated by the Bayesian LCMs, we estimated the naive specificity of the IgM ELISA results in patients with a firm single diagnosis of other diseases, including murine typhus and dengue, in the data set described previously (19,20). The diagnosis of murine typhus was made if there was a Ն4-fold increase in the Rickettsia typhi IgM IFA titer between paired samples (using slides coated with R. typhi strain Wilmington; ARRL, Geelong, Australia) (19,20). The diagnosis of dengue was defined by the detection of NS1 antigen using Panbio ELISAs and IgM antibodies (Panbio, Brisbane, Australia) in paired samples (19,20).
Evaluation of ELISA repeatability. Between-and within-day variation was determined by calculating the mean and the standard deviation (SD) of the positive control on each plate to derive the between-plate, between-day, and within-day percent coefficient of variation (%CV) as follows: %CV ϭ (SD/mean) ϫ 100. Median values and interquartile ranges (IQR) were also calculated for the population.

RESULTS
Of 161 patients with acute undifferentiated fever evaluated, 135 (83.8%) had both acute and convalescent/discharge serum samples available for ELISA IgM and were included in the study. The median age was 42 years (IQR, 29 to 52 years; range, 15 to 84 years), and 83/135 (63%) were male. The median duration of fever before admission was 5 days (IQR, 4 to 7 days). The median duration between the admission and convalescent/discharge sample was 12 days (IQR, 3 to 14 days). Culture, a combination of PCR assays, IgM IFA, and the presence of an eschar were positive in 7, 22, 25, and 14 patients, respectively. There was a dominance of IFA titers at the lower end of the dilution scale for IgM titers in both admission and convalescent/discharge samples (Fig. 1).
The distribution of IgM ELISA ODs was skewed to the right for all sample dilutions and for both admission and convalescent/ discharge samples ( Fig. 2A and B). The IgM ELISA ODs in convalescent/discharge samples were generally higher than those in admission samples for all sample dilutions (all P values Ͻ 0.001; Wilcoxon paired signed-rank test). For example, at a sample dilution of 1:100, 86 (63.7%) patients had ODs in the convalescent/ discharge sample higher than that in the admission sample. Patients who had a high OD at a 1:100 sample dilution also had comparatively high ODs at every dilution for both admission and convalescent/discharge samples. For example, Fig. 2A shows that the patient who had an IgM ELISA OD in the 10th percentile in the admission sample at a 1:100 sample dilution always had IgM ELISA ODs higher than those of the patients who had IgM ELISA ODs in the 20th, and 30th to 90th percentiles in all sample dilu-tions. The OD difference between each 10th percentile was narrower at the higher sample dilution ( Fig. 2A and B).
Using the ROC curve to evaluate the performance of IgM ELISA with the admission sample alone, we found that the recommended cutoff ODs ranged from 1.854 to 0.028, decreasing with increasing dilution (see Fig. S1 in the supplemental material). For paired samples, the recommended cutoff ODs ranged from 2.427 to 0.053 for the dilution range 1:100 to 1:102,400 (see Fig. S2 in the supplemental material). A dilution of 1:400 was selected as the recommended sample dilution because it had recommended cutoff OD values for both scenarios between 1.00 and 2.00 (see Fig. S1 and S2 in the supplemental material), i.e., in the middle of the linear range of the microtiter plate reader.
To select a single optimal cutoff OD at a sample dilution of 1:400, the sensitivities and specificities of all possible ODs were evaluated between the recommended OD range when only the admission sample was available (OD ϭ 1.010) and that when the paired convalescent/discharge samples were available (OD ϭ 1.474) ( Table 1). An OD of 1.474 was selected as the optimal cutoff because it provided the highest overall accuracy. Using a cutoff OD of 1.474 at a sample dilution of 1:400, Bayesian LCMs estimated that the IgM ELISA had a sensitivity of 69.0% (95% CrI, 60.6% to 76.9%) and a specificity of 100% (95% CrI, 99.1% to 100%) for the admission sample alone and a sensitivity of 85.7% (95% CrI, 77.4 to 86.7%) and a specificity of 98.1% (95% CrI, 97.2 to 100%) with the addition of the paired samples ( Table 2). The Bayesian LCM estimated that there were 29 scrub typhus patients (95% CrI, 27 to 33 patients), representing a prevalence of scrub typhus at 21.8% (95% CrI, 15.1% to 29.5%) in the study participants. There was no significant difference between the sensitivities of the IgM ELISA and the IgM IFA for admission samples alone (69.0% versus 69.0%; Bayesian P value ϭ 0.50) or between the sensitivities of the two tests using paired samples (85.7% versus 86.2%; Bayesian P value ϭ 0.31).

Post hoc model evaluation.
We evaluated the robustness of the cutoff OD of 1.474 at a sample dilution of 1:400 and showed that the IgM ELISA had a naive sensitivity of 69.0% (20/29) using paired samples in patients with a firm diagnosis of scrub typhus made using other diagnostic tests ( Table 3). Eight of the nine patients with negative IgM ELISA results had very low IgM ELISA ODs (Ͻ0.5), even though the overall durations between the onset of fever and the convalescent/discharge sample date were 5 to 10 days (n ϭ 3) and more than 10 days (n ϭ 5) (see Table S1 in the supplemental material). The naive specificity of IgM ELISA in patients with a firm single diagnosis of either murine typhus or dengue virus infection was 96.3% (26/27), using paired samples in this group of patients (Table 3).

DISCUSSION
We have described an unbiased approach using Bayesian LCMs to define optimal sample dilution and optimal cutoff OD values for the use of an IgM ELISA for the diagnosis of acute scrub typhus. We demonstrated that the sensitivity and specificity of the IgM ELISA are comparable to those of the IgM IFA with a cutoff of Ն1:  3,200 in admission samples or a 4-fold rise to Ն1:3,200 in convalescent/discharge samples compared to admission samples and suggest that the IgM ELISA could be used as an alternative serological reference test instead of the IgM IFA. In addition to comparable accuracy, there are a number of other reasons why ELISA technologies could replace IFA as the scrub typhus reference serological test. In general, the IFA is a subjective semiquantitative test that requires highly trained operators to prepare the slides and determine the results (9, 10). Furthermore, the IFA does not lend itself to the testing of large numbers of samples, hindering throughput when performing clinical studies or in clinical settings with large numbers of patients. These problems could easily be overcome by the ELISA, which could be automated and is easier to standardize, as it provides fully quantitative and consistent results (11,12).
The sensitivity and specificity values estimated by our study are unbiased, because they are based on the true status of patients predicted by the Bayesian LCM. Previous studies reported that the NMRC IgM ELISA had higher sensitivity and lower specificity (11,12) than the immunoperoxidase (IIP) assay (the IIP assay is analogous to IFA with the exception that different substrates are employed) titer of 1:400, which was assumed to be "perfect." Suwanabun et al. reported that the NMRC IgM ELISA had a sensitivity of 92.9% and a specificity of 93.6% (11). Coleman et al. reported that the NMRC IgM ELISA at a sample dilution of 1:400 had a sensitivity of 94.2% and a specificity of 91.3% (12). However, the sensitivities reported by Suwannabun and Coleman were predicted to be overestimated, and this could be because low cutoff ODs were used and sensitivity was biased toward a group of patients who had IIP assay titers of Ն1:400. The specificities reported by Suwanabun and Coleman could be underestimated because low cutoff ODs were used and the IgM ELISA could be positive in scrub typhus patients who had an IIP assay titer of Ͻ1:400. These problems were due to disease misclassification by the imperfect gold standard (14)(15)(16)(17). In contrast, the sensitivity and specificity values estimated by our study were not biased toward an imperfect reference test. This is also supported by the post hoc model evaluation showing that the sensitivity and specificity estimated by the Bayesian LCMs are comparable to the naive sensitivity and specificity values estimated in a group of patients with firm diagnoses of scrub typhus and of other diseases, respectively.
The choice of an appropriate sample dilution and diagnostic cutoff is an important step in diagnostic test validation and one   (12). The rationale for the sample dilution selection was unclear (11,12). Bayesian LCMs and the generation of unbiased ROC curves have been increasingly used for many diseases (14,16,27,28). In this study, we also show that the selection of sample dilution for the IgM ELISA could be systematically performed. Although all sample dilutions were capable of providing comparable accuracies using the IgM ELISA if an optimal cutoff OD was chosen (see Fig. S1 and S2 in the supplemental material), an optimal sample dilution can be recommended that fits with the optimal OD reading range recommended by the ELISA reader manufacturers. Application of the methodology described here would lead to a broader understanding of the utility of reference tests in the evaluation of quantitative diagnostic tests. This could lead to changes in the diagnostic process and clinical practice for many infectious diseases. This study has some limitations. First, the sample size in our study was small. Second, the antigenic variation characteristic of O. tsutsugamushi strains can affect assay sensitivity and specificity. Previous studies have characterized the O. tsutsugamushi strains causing human disease (29)(30)(31)(32)(33)(34) and presenting in vectors (35)(36)(37) in Thailand, showing a dominance of Karp-and Gilliam-like genotype strains, as well as TA716 and TA763 genotype strains. However, the O. tsutsugamushi strains that cause disease in the Chiangrai locality have not been fully characterized, and the incorporation of contemporary strains into the antigenic mix used in the test may increase the sensitivity. This is supported by our finding that some patients who had a firm diagnosis of scrub typhus were IgM ELISA negative, despite the time between the onset of fever and convalescent/discharge sampling being more than 10 days (see Table S1 in the supplemental material). An IgM ELISA using well-characterized contemporary local strain recombinant O. tsutsugamushi 56-kDa outer membrane antigens (38,39) could be developed and evaluated in clinical settings using appropriate statistical models. Third, the accuracy of diagnostic tests varies based on prevalence, clinical variability, and availability and timing of convalescent-phase samples (40). Further studies to evaluate the optimal cutoff titers and accuracy of the IgM ELISA in different settings are still required.
In conclusion, we propose that IgM ELISA should be used as an alternative serological reference test for acute scrub typhus in the locality of Chiangrai, Thailand, with the recommendation that geographically specific diagnostic cutoffs be determined for other localities and employed using appropriate statistical models.