Performance evaluation of the AiDx multi-diagnostic automated microscope for the detection of schistosomiasis in Abuja, Nigeria

In this research, we report on the performance of automated optical digital detection and quantification of Schistosoma haematobium provided by AiDx NTDx multi-diagnostic Assist microscope. Our study was community-based, and a convenient sampling method was used in 17 communities in Abuja Nigeria, based on the disease prevalence information extracted from the baseline database on schistosomiasis, NTD Division, of the Federal Ministry of Health. At baseline, samples from 869 participants were evaluated of which 358 (34.1%) tested S. haematobium positive by the reference diagnostic standard. Registered images from the fully automated (autofocusing, scanning, image registration and processing, AI image analysis and automatic parasite count) AiDx assist microscope were analyzed. The Semi automated (autofocusing, scanning, image registration & processing and manual parasite count) and the fully automated AiDx Assist showed comparable sensitivities and specificities of [90.3%, 98%] and [89%, 99%] respectively. Overall, estimated egg counts of the semi-automated & fully automated AiDx Assist correlated significantly with the egg counts of conventional microscopy (r = 0.93, p ≤ 0.001 and r = 0.89, p ≤ 0.001 respectively). The AiDx Assist device performance is consistent with requirement of the World Health Organization diagnostic target product profile for monitoring, evaluation, and surveillance of Schistosomiasis elimination Programs.


Description of the AiDx assist microscope. The multi-diagnostic AiDx Assist Microscope device
shown in Fig. 1, is a low-cost and compact automated single-slide scanner and microscope.The optical train consists of a 4× microscope objective (numerical aperture NA of 0.10 and a working distance of 18.0 mm).The current system model is designed with a Sony IMX 178 CMOS sensor 6.41 Mpix (3088 × 2076 pixel) with a pixel size of 2.40 μm.
The computer vision pipeline in this version of the device is powered by the U-Net segmentation model with a ResNet-18 network architecture.The approximate schistosome ova count was estimated using the predicted segmentation mask 23,24 .More recent approaches are being explored to improve the performance of the integrated algorithm.
The AiDx Assist multi-diagnostic device can detect microfilaria, S. haematobium and S. mansoni eggs in blood, urine, and stool samples respectively.Furthermore, the device has shown promising potential in the detection of malaria parasites and tuberculosis in prepared blood and sputum samples accordingly.
The AiDx Assist multi-diagnostic device can detect microfilaria, S. haematobium and S. mansoni eggs in blood, urine, and stool samples respectively.
The AiDx Assist Microscope can function in two modes at the choice of the operator: Semi-automated mode.In the semi-automated mode, the automatic parasite count algorithm is disabled.The device operates without the use of the integrated Artificial Intelligence software.Images registered and stored in the system are visually examined for the presence of the target parasite.Operating the device in this mode is very useful for experienced laboratory scientists who prefer to gain control of the device and navigate through the stored sample scan to detect the target of their interest without dependence on the AI.The integrated sample autofocusing mechanism and scanning system control algorithm reduces the time, error, fatigue, and other limitations associated with manual microscopy.www.nature.com/scientificreports/Fully automated mode.The fully automated AiDx Assist include an autofocusing algorithm which obtains an in-focus image obtained in a z-stack of image, a sample scanning algorithm to scan defined samples area.It also includes image registration, processing, AI image analysis and automatic parasite count.Detected targets are segmented and segmented images are stored in a folder for easy access and validation by the AiDx system operator.The AI Assist software outputs the number of estimated ova for the operator to validate and confirm the presence of the segmented Schistosoma ova.

Results
Three detection methods were analyzed: (i) The conventional microscopy which is the reference standard (ii) The expert analysis of the result of Semi-automated AiDx Assist and (iii) The fully automated AiDx Assist digital microscope with automated image interpretation.The three methods independently identified 358 (41.2%), 324 (37.3%) and 315 (36.2%) of the total slides as positive for S. haematobium respectively as shown in Table 1.
The egg count estimates per 10 mL of urine ranged from 1 to 2100 eggs/10 mL for conventional microscopy, 1 to 5451 eggs/10 mL for semi-automated, and 1 to 4064 eggs/10 mL for fully automated AiDx Assist.Compared with conventional microscopy, semi-automated and the fully automated AiDx assist showed an overall accuracy of 94% and 95%, respectively (Table 1).
The sensitivities of the semi-automated and fully automated AiDx Assist for the detection of S. haematobium eggs were comparable; 90.5% and 88%, respectively.Details of evaluated performance metrics is shown in (Table 2).Further review of the data showed that there is no erroneous choices overlap between the semiautomated and the fully-automated AiDx Assist.For instance, there was zero overlap between the false positive and the false negative samples detected by the semi and fully automated AiDx Assist respectively.
To characterize the performance of the AiDx Assist Microscope in the detection of low-intensity and high intensity infection, we compare the semi-automated and the fully automated functionalities of the AiDx Assist microscope with reference conventional microscopy.In this case, the conventional microscopy classified 244 (68.1%) as low-intensity infection and (31.8%) as high-intensity infection.The fully automated and Semi-automated AiDx Assist classified 246 (76.6%) and 237 (70.9%) as low-intensity infection respectively, 75 (33.9%)and 97 (29%) were classified as high-intensity infection.Sensitivities of the fully automated and semi-automated AiDx Assist for low-intensity infections were 86% and 82.38%, which increased to 100% for high-intensity infections in both cases accordingly.
In Fig. 2 the graphical representation of the correlation between fully/semi-automated AiDx Assist Microscope and the reference manual microscopy method is shown.The correlation between the fully automated AiDx Assist and the reference manual microscopy was lower (Pearson 0.89, p < 0.001) compared to the correlation between the Semi-automated AiDx Assist and reference manual microscopy (Pearson 0.93, p < 0.001).No significant change was observed in the correlation coefficient estimates when the outliers were removed.This implies that the model is robust to the removal of the outliers.
To further validate the agreement between the AiDx Assist and the reference manual microscopy, Bland-Altman analysis was performed on 93 samples collected from a highly endemic community.The sample size was chosen to represent the range of infection from low, medium, and high infection accordingly.Graphical representation of the correlation between manual microscopy (reference) and fully/semi-automated AiDx Assist www.nature.com/scientificreports/parasitemia count data is shown in Figs. 3 and 4. A logarithmic scale (log(x + 1)) transformation of the data is used to avoid the log(x) approaching negative infinity as x approaches zero.The Bland-Altman plot shows a strong agreement.Removing the outliers in the plot did not correspond to any significant change in the computed correlation coefficient.
False positives-misclassified samples.With respect to the reference standard conventional microscopy, a third trained reader examined digital images of all (869) samples registered by the semi-automated AiDx Assist microscope (the 3rd microscopist was blinded to the results obtained from the reference microscopy and the fully automated AiDx Assist digital microscope to avoid any form of bias).The result was compared to the average reading of the expert microscopist.10 out of the 10 false positives were confirmed to be true positive samples.The specific images were sent to the expert microscopist who validated the schistosome ova in the registered images.We observe however that the samples had very low egg count range (1-14 eggs/10 mL of urine).

Discussion
This study focused on the field performance evaluation of the AiDx Assist for the detection of S. haematobium eggs in infected urine samples.The system has been evaluated as (i) A semi-automated AiDx Assist microscope and (ii) fully automated AiDx Assist microscope.The integration of the AI algorithm as described in the previous section increased the specificity of the fully automated AiDx Assist.This is a unique hybrid (AI + Human operator) feature that demonstrates increase in specificity.Complete dependence on the AI analysis could result in low specificity which could significantly compromise the quality of the result.
The semi-automated AiDx Assist microscope demonstrated better performance than the manual microscope as it detected additional positive cases which were missed by the expert manual microscopy.Underestimation  of the intensity of infection was observed in some cases.This could be due to two primary reasons: Firstly, the autofocus technique implemented fails to converge to the optimal in-focus plane consistently.This increases the AI algorithm's poor generalization in defocused images.Secondly misrecognition of overlapping eggs as a single egg by the AI algorithm.Complete sample analysis was achieved in 15 min.During this field study performance evaluation this was a limitation of the AiDx Assist.In the current system development iteration of the device however, complete sample analysis is achieved in less than 6 min.While the AiDx Assist provides clear digital images of the stool parasite, a functional AI algorithm to detect S. mansoni and STHs is currently being developed.
The semi-automated AiDx Assist and fully automated AiDx Assist has proven to hold great potential for future and timely diagnosis of schistosomiasis as it has good sensitivity, specificity, positive and negative predicted values respectively.The AiDx Assist device performance evaluated in this study is consistent with requirement of the World Health Organization Diagnostic Target Product Profile for monitoring, evaluation, and surveillance of Schistosomiasis.The usefulness of this novel AI-based AiDx Assist device will be of great value in schistosomiasis diagnosis, evaluation, surveillance and may be incorporated into elimination programs agenda.

Methods
Study design and population.Participants aged 5 and above were eligible to participate in this study.
Study was community-based, and a convenient sampling method was used in 17 communities: 9 communities were selected from Gwagwalada Area council and 8 from Abuja Municipal Area Council (AMAC) of the Federal Capital Territory, Abuja Nigeria.The Area councils were selected based on the disease prevalence information extracted from the baseline database on schistosomiasis, NTD Division, of the Federal Ministry of Health.Participants collected their urine samples in the bathroom at collection sites.These were stored in temperaturecontrolled storage boxes (2-6 °C) and transported to the laboratory immediately after collection.The sample size of the study population was computed using the Cochran formula 24 .Given the baseline prevalence for Gwagwalada and AMAC at 51.9% and 21.12% respectively, a total of 869 (509 and 360) respondents were sampled from Gwagwalada and AMAC.They were randomly selected from 10 and 7 Communities in Gwagwalada and AMAC respectively.Smaller samples were sampled in the Area Council with lower prevalence to ensure that enough positive samples are collected.The power of the diagnostic test required to achieve adequate sensitivity or specificity is calculated based on the formula outlined in Jones et al. 25 .

Sample collection process.
Community consent forms which were filled and signed by community leaders on behalf of community members according to standard government protocol were administered by the data manager of the NTD unit, the Public Health Department of the Federal Capital Territory Abuja (FCTA).The hardcopy version of all the consent forms were then transferred to an electronic format and uploaded to the cloud server platform.To ensure full confidentiality, each participant was assigned a specific unique identification number.Each selected participant was provided with a capped sterile specimen bottle and instructed to fill it with about 20 mL of clean catch, midstream urine.Participants were subjected to a little exercise to agitate their bladder and carefully instructed with illustration aid.The bottles were labeled with unique identification (ID) numbers and samples were collected between 10.00 am and 14.00 h.The age and sex of each participant were noted and recorded.Samples were transported to the laboratory and processed 1-2 h after collection.In the laboratory, each urine sample was tested for hematuria using urinalysis reagent strip (Combi 9 Dipstick), then homogenized by gentle agitation and was filtered using urine filtration technique to concentrate eggs of schistosome on a membrane filter as described by WHO, 2010.The number of eggs were counted by a microscopist and recounted by a second independent microscopist both of whom were blinded to the urinalysis results to reduce margin of error and results expressed as number of eggs per 10 mL of urine.Same urine slide was then passed to the AiDx machine for image-based microscopy and count.The data collection form which contained data entry space for urine results from Dipstick and microscopists were submitted daily at the AiDx Medical Laboratory office after sample analysis by the technicians and the laboratory scientist.Quality control was instituted, and laboratory results verified by senior laboratory scientists and Federal Ministry of Health (FMoH) supervisors to ensure consistency in sample preparation and examination.A comparative flow chart of the sample collection and analysis of the urine samples is shown in Fig. 5. www.nature.com/scientificreports/ the fully automated AiDx Assist across the range of mean egg counts for both methods is shown in figure below (Fig. 6).The approximate ova count is estimated using the predicted segmentation mask.Images on the left are the captured images and images on the right are the annotations.Figure 7 shows S. mansoni eggs detected in stool samples by the automated AiDx Assist.The fecal samples are prepared based on the Kato Katz technique.AI algorithm for the detection of the S. mansoni in registered images has been developed for testing in a planned upcoming field study.

Statistical analysis.
Qualitative agreement between the automated AiDx Assist and conventional microscopy was assessed.Egg counts were categorized as low-intensity infection (< 50 eggs/10 mL of urine) or highintensity infection (> 50 eggs/10 mL urine).Data transformation was performed to increase the data cohesion.

Figure 1 .
Figure 1.AiDx Assist fully automated microscopes designed for laboratory and field use.

Figure 2 .
Figure 2. Graphic representation of the correlation between manual microscopy (reference) and AiDx assist methods.Data shown for (a) semi automated AiDx assist versus manual microscopy (reference) and (b) fullyautomated AiDx Assist versus manual microscopy (reference).

Figure 3 .
Figure 3. Bland-Altman plots showing the level of agreement between the manual microscope and semiautomated AiDx Assist.The plot shows strong agreement in measurement with (r = 0.93, p ≤ 0.001).

Figure 4 .
Figure 4. Bland-Altman plots showing the level of agreement between the manual microscope and fully automated AiDx Assist Microscope.The plot shows strong agreement in measurement with (r = 0.89, p ≤ 0.001).

Figure 5 .
Figure 5. Flow chart of urine sample collection and analysis comparing conventional microscopy with the AiDx assist automated and digital result analysis by expert.

Figure 6 .
Figure 6.Registered images of S. haematobium eggs.The approximate ova count is estimated using the predicted segmentation mask.Images on the left are the captured images and images on the right are the annotations.

Figure 7 .
Figure 7. Registered images of S. mansoni eggs in stool samples.The characteristic terminal spine is located at the side of the ova.Although the stool samples contain lots of debris and potential artifacts, the AiDx automated microscope sufficiently resolves the desired target.

Table 1 .
S. haematobium egg count by conventional manual microscopy and automated AiDx assist microscope.