Head-to-head comparison of three stool calprotectin tests for home use

Objective Treatment decisions in inflammatory bowel diseases are increasingly based on longitudinal tracking of calprotectin results. Many hospital laboratories measure calprotectin levels in sent-in stool samples with an enzyme-linked immunosorbent assay (ELISA). Several manufacturers introduced a lateral flow–based test with software application that turns a smartphone camera into a reader for quantitative measurements. We compared three home tests (IBDoc, QuantonCal and CalproSmart) and companion ELISA tests (fCAL, IDK-Calprotectin and Calprotectin-ALP) to see if measurement pairs agreed sufficiently. Design A method comparison study was conducted with stool samples from patients with active or quiescent inflammatory bowel disease. Medical students without any specific laboratory training carried out the home tests with two iOS (iPhone 6 and 7) and two Android devices (Samsung Galaxy S6 and Motorola Moto G5 Plus). Two experienced laboratory technicians measured the calprotectin concentration with the ELISA method. Primary outcome was test agreement (defined as percentage of paired measurements within predefined limits of difference). Secondary outcome included reading error rate (RER) per smartphone type. Results We performed 1440 smartphone readings and 120 ELISA tests. In the low calprotectin range (≤500 μg/g) IBDoc, QuantOnCal and CalproSmart showed 87%, 82% and 76% agreement with their companion ELISAs. In the high range (>500 μg/g) the agreement was 37%, 19% and 37%, respectively. CalproSmart and QuantOnCal had significantly higher RERs than IBDoc (respectively 5.8% and 4.8%, versus 1.9%). Forty-three percent of reading errors was on the Motorola device, in particular with the QuantOnCal application. Conclusions All three calprotectin home tests and companion ELISAs agreed sufficiently when concentrations are ≤500 μg/g. To minimize wrongful interpretation of calprotectin changes over time it is essential to always use the home test and companion ELISA of one and the same manufacturer. Manufacturers should explicitly evaluate and report the suitability of commonly used smartphones for quantitative calprotectin readings.


Introduction
Treatment decisions in patients with inflammatory bowel disease (IBD) are increasingly based on longitudinal tracking of stool calprotectin test results. There are indications that calprotectin-guided escalation of therapy results in better short-term clinical and endoscopic outcomes than symptom-guided decisions alone. [1] Repeated calprotectin concentrations below 250 μg/ g are considered to be a useful treatment target and are synonymous with a low risk for disease flare. [2] Many hospital laboratories measure the calprotectin concentration in sent-in stool samples with an enzyme-linked immunosorbent assay (ELISA). It was generally believed that the protein was stable in stool samples for up to seven days, [3] but recent experiments have shown that calprotectin values gradually decline to approximately 65% of baseline levels after 6 days storage at room temperature. [4,5] To prevent underestimation of disease activity stool samples collected at home should be refrigerated until arrival at the hospital laboratory, but this complicates transport. A more practical solution would be to measure the calprotectin concentration immediately after stool sample collection at home.
A new measurement method has recently been introduced commercially. The method itself and convenience of use are similar to those of a pregnancy test. By taking a photo of the test strip with a smartphone, the patient can perform quantitative measurements at home. We compared three of these home tests (IBDoc, Quanton Cal and CalproSmart) with the ELISA method of the same manufacturer (fCAL, IDK-Calprotectin and Calprotectin-ALP) to see which of the pairs has the best agreement. In addition, we evaluated the reading error rate (RER) per smartphone type.

Study design and pre-analytical sample handling
This method comparison study was performed at the University Medical Center Groningen (UMCG, the Netherlands). Between March and May 2018 IBD patients treated at the Department of Paediatric Gastroenterology volunteered in donating stool samples for this study. They defecated onto a stool collection sheet held above the toilet water at home, and transferred a sample of stool into a classical screw top container with a spatula. They described the stool consistency by completing the Bristol Stool Form Scale (BSFS), [6] and send both the screw top container and BSFS in a resealable biomaterial envelope to the Department of Laboratory Medicine. For the purpose of this study the patients themselves were not involved in the measurement of stool calprotectin. Upon arrival at the laboratory, the stool sample was homogenized and then aliquoted for analysis with two separate methods. Both aliquots were stored at -20˚C until analysis.

Measurement of stool calprotectin with home test
Three third year medical students without any specific laboratory training carried out the home tests of three manufacturers (Table 1). Each manufacturer-specific test cassette was read with four different smartphones which were purchased especially for this project:: two iOS (iPhone 6 and 7) and 2 Android devices (Samsung Galaxy S6 and Motorola Moto G5 Plus) (Fig 1). The sequence of use of smartphone type and home test was randomized to prevent bias. An hour before the testing session the students defrosted one of two aliquots. They all dipped the assay-specific sampling pin in the sample and transferred the fecal material into the assay-specific tube with extraction buffer. After observing the advised extract processing time, they then applied a drop of extraction fluid in the measurement window of the test cassette as per manufacturer's instructions. After the appropriate incubation time the cassette was read with the smartphone by placing the camera above the cassette. The image was automatically analyzed by the smartphone application and the quantitative calprotectin result was directly shown on the smartphone screen.

Measurement of stool calprotectin with ELISA
Experienced laboratory technicians, who were blinded for the smartphone readings, thawed the second aliquot and carried out batch ELISA testing with three assays that corresponded with the home tests (see Table 1). The samples were manually weighted and the measurements were done on a Dynex DS2 automated ELISA system (Alpha Labs, Easleigh, UK).

Outcome measures
The agreement between the home test and the companion ELISA measurement was considered as the primary outcome measurement, and was analyzed with a Bland-Altman plot. As described previously [7], we reasoned that disagreement in the lower range of the tests (i.e., below 500 μg/g) could lead more easily to misclassification of disease activity than disagreement in the higher range (>500 μg/g). We therefore used predefined acceptable limits of difference, which were arbitrary set at ±100 ug/g for the lower range, and ±200 μg/g for the higher range. Secondly, concordance of home test and companion ELISA test were determined in each of 3 calprotectin ranges commonly used in our clinical practice (i.e., <250 μg/g for target range, 250-500 μg/g for undecisive range, and >500 μg/g for active disease). [7] Other outcome measures included the reading error rate (RER) per smartphone type, with reading error defined as an image of the test cassette not leading to quantitative result, and evaluation of the usability of the home test and the smartphone application on the first, seventh and last day of the home test experiments. For that purpose, the students completed the system usability scale (SUS). [8] The SUS is a simple, ten item scale giving a global view of subjective assessments of usability and learnability of a system. Scores have a range of 0 to 100, and should be considered as grade scales rather than percentages.

Sample size calculation
The primary outcome was taken as the overall percentage of agreement between the calprotectin home test and the corresponding ELISA test per manufacturer. We wished to detect a difference in agreement of at least 15% by a two-sided test. With the level of significance set at 5% and a study power of 80%, we aimed to include at least 110 paired measurements per manufacturer.

Statistical analysis
In Bland-Altman analysis, a scatter plot is constructed in which the difference between the paired measurements is plotted on the vertical axis and the average of the measures of two methods on the horizontal axis. The mean difference in values obtained with the two methods is called the bias and is represented by a central horizontal line on the plot. The standard deviation (SD) of differences between paired measurements is used to construct horizontal lines above and below the line to represent 95% limits of agreement (LOA). The plot enables the reader to visually assess the bias, data scatter and the relationship between magnitude of difference and size of measurement. ELISA measurements out of the measurable range of the home test were rounded to the upper limit of the home test range. In addition, we provide a distribution histogram of the differences between the paired measurements in the low range calprotectin concentrations. Concordance of home test and companion ELISA readings in each of the three calprotectin ranges are presented in a scatterplot. RERs per home test and smartphone type are reported as percentages with confidence intervals. Data were recorded electronically by using SPSS version 23.0 for Apple Mac (IBM Corporation, Armonk, NY). Graphs were constructed with GraphPad Prism version 7 for MacBook (GraphPad Software, San Diego, California USA). P-values <0.05 were considered statistically significant.

Ethical consideration
The

Bland-Altman analysis
In Table 2 we show the agreement between home test and corresponding ELISA per manufacturer using the Bland-Altman plot analysis. IBDoc underestimated the fCAL ELISA results over the whole range of potential calprotectin values with a mean bias of 105 μg/g (agreement interval 942 μg/g). QuantOn Cal overestimated IDK-Calprotectin ELISA results with a mean bias of 137 μg/g (agreement interval 1574 μg/g). CalproSmart overestimated Calprotectin-ALP ELISA results with a mean bias of 141 μg/g (agreement interval 914 μg/g). Fig 2 shows that in all comparisons the paired measurements were more clustered and closer to the zero-difference line in the low calprotectin range and more scattered in the high range. Low calprotectin range (� 500 μg/g). In the low calprotectin range the percentage of paired measurements within acceptable limits of difference for IBDoc, QuantOn Cal and Cal-proSmart were 87% (263/304), 82% (285/349) and 76% (235/310), respectively. Fig 3 shows that the difference scores for IBDoc, QuantOn Cal and CalproSmart were distributed in an approximately normal pattern around the bias of -19 μg/g (agreement interval 560 μg/g), -4 μg/g (agreement interval 455 μg/g) and 72 μg/g (agreement interval 458 μg/g), respectively. The agreement interval of all three comparisons exceeded the acceptable limits of difference in the low range (i.e. 200 μg/g).

Reading error rate
The CalproSmart and QuantOn Cal smartphone applications had significantly more reading errors than the IBDoc application, with rates of respectively 5.8% and 4.8% versus 1.9% (P = 0.002 and P = 0.012). Forty-three percent of the total amount of reading errors was with the Motorola device, in particular in combination with the QuantOn Cal application (Fig 5). Common reason for reading errors was an out of focus image.
System usability scale Fig 6 shows that mean SUS scores per home test were lowest on the first day of testing. On the last day of testing IBDoc was awarded the highest grade (B) of all home tests, mainly because the smartphone application was error-friendly and therefore less cumbersome to use.

Discussion
This is the first head-to-head method comparison study investigating three calprotectin home tests and their companion ELISA tests. We have shown that the majority of measurements performed with a lateral flow immunoassay and smartphone reader agreed sufficiently with the ELISA-based quantification of the same manufacturer, provided that calprotectin levels are below 500 μg/g. In the high calprotectin range a substantial proportion of measurement pairs exceeded the predefined limits of difference (±200 μg/g). QuantOn Cal performed significantly poorer in the high range than IBDoc and CalproSmart. IBDoc had significantly fewer reading errors than the other two applications. The Motorola device turned out to be less suitable for calprotectin readings than the other smartphone types. A complete summary of the test performance per manufacturer is shown in S1 Table. The possibility to measure calprotectin with a smartphone at home is a novelty in IBD-care. We identified three recently published studies on the IBDoc home test, [7,9,10] and one on the CalproSmart home test. [11] All studies showed acceptable agreement between home test and companion ELISA in the low range and lack of agreement in the high range. These studies differ from ours in several important ways. First, in three of the aforementioned studies, home tests were performed before shipping the unrefrigerated stool samples to the laboratory. [7,10,11] The possible decline in calprotectin during transport may have affected the agreement with the ELISA results, while in our experiment the transport of stool samples was taken out of the equation. Second, none of the studies performed a subgroup analysis per smartphone type, while we compared several iOS and Android devices. By doing so, we were able to show that not all smartphone cameras were suitable for calprotectin readings. Third, in this study we report an a priori decision of acceptable limits of difference for adequate interpretation of the Bland-Altman plot, a feature that was absent in three of the aforementioned studies. [9][10][11] Precision is of utmost importance in the low ranges of calprotectin values, where small deviations can lead to misclassification of disease activity and wrong treatment decisions. In our practice, however, imprecision in the high range is less of an issue, since we consider each calprotectin result >500 μg/g to reflect active disease, irrespective of the absolute concentration. A twofold increase in calprotectin, e.g. from 1000 to 2000 μg/g, does not necessarily mean that the inflamed surface area of the gastrointestinal tract doubled, neither that the intensity of the inflammation increased. Any shift of calprotectin values out of the target range, and into the action range (� 500 μg/g) is a trigger for us to change the treatment plan. In order to appreciate the true value of the home tests in the high range, we also evaluated the concordance with companion ELISAs. We observed that 119 of 125 IBDoc readings � 500 μg/g were concordant with fCAL results (95%), compared to 35% and 49% for the QuantOnCal-IDK-Calprotectin and CalproSmart-Calprotectin-ALP pairs.
There are some limitations in our study that need to be addressed. First, the home tests were not performed by real IBD patients. Instead, we had three 3 rd year medical students (AG, SA, JAAB) without any specific laboratory training to fill the lateral flow cassettes and execute the smartphone readings. They only had access to the instructions for use and video tutorials on the internet. We acknowledge that their level of education is higher than the "average IBD patient". Second, although we aimed to collect stool samples that covered the whole range of potential consistencies, we missed the liquid samples. For that reason we were unable to evaluate the effect of stool consistency on the precision of the lateral flow-based method.
The evidence base for calprotectin-guided treatment escalation and de-escalation is accumulating. [1,2,12] Simultaneously, the number of telemonitoring initiatives for IBD care is rising. [13][14][15][16] Telemonitoring with calprotectin home testing can make the service to IBD patients more efficient, as the technology allows to select and target patients that benefit from a face-to-face encounter with their IBD-team at short notice.
We evaluated by how much the home tests differed from the trusted ELISA method, and found that in the lower ranges the difference was small enough not to cause problems in interpretation. We conclude that the flow-based home-test and the companion ELISA method can be used interchangeably. To minimize wrongful interpretation of calprotectin changes over time it is essential to always use the home test and companion ELISA of one and the same manufacturer. This advice was already in force for the use of ELISA tests. [17][18][19][20] Ultimately, patients and health care providers would both benefit from better co-calibration of stool calprotectin assays.
Supporting information S1