Deep Multi-Modal Fusion Model for Identification of Eight Different Particles in Urinary Sediment

– Urine sediment examination (USE) is an essential aspect in detecting urinary system diseases, and it is a prerequisite for diagnostic procedures. Urine images are complex, containing numerous particles, which makes a detailed analysis and interpretation challenging. It is crucial for both patients and medical professionals to conduct urine analysis automatically, quickly and inexpensively, without compromising reliability. In this paper, we present a deep multi-modal fusion system, commonly employed in artificial intelligence, capable of automatically distinguishing particles in urine sediment. To achieve this objective, we first created a new dataset comprising erythrocytes, leukocytes, yeast, epithelium, bacteria, crystals, cylinders, and other particles (such as sperm). The data were gathered from urinalysis requests made between July 2022 and September 2022 at the biochemistry laboratory of Fethi Sekin Medical Center Hospital. A dataset containing 8509 images was compiled using the Optika B293PLi microscope with trinocular brightfield. We propose a 5-step process for detecting particles in the dataset using a multi-modal fusion deep learning model: i) The obtained images were augmented by applying affine transformation. ii) To distinguish images, we opted for ResNet18 and ResNet50 models, which yielded high performance in medical data. iii) Feature vectors from both models were fused to generate more consistent, accurate, and useful particle features. iv) We employed ReliefF, Neighborhood Component Analysis (NCA), and Minimum-Redundancy Maximum-Relevancy (mRMR) feature selection methods, widely used to determine features that maximise particle discrimination success. v) In the final step, Support Vector Machine (SVM) was utilised to distinguish the particles. The results demonstrate that the highest accuracy value achieved is 98.54 % when employing the ReliefF algorithm. Contributions of the study include eliminating standardisation differences in manual microscopy, achieving high accuracy in particle discrimination, offering an artificial intelligence-based system applicable in laboratory environments, and providing the dataset as educational and practical material for biochemistry professionals.


I. INTRODUCTION
Urine sediment analysis involves the microscopic examination of precipitated particles obtained by centrifugation of urine.It is a practical and preferred, inexpensive, and accessible test in the diagnosis of various diseases such as diabetes, liver problems, kidney disorders, and urinary tract infections [1], [2].In urine sediment examinations, substances that should or should not be present in the urine are observed.This examination, widely used and frequently requested by doctors, is a non-invasive method known as the Complete Urine Test (CUT) [3].In this test, sediment microscopy images, in which shaped elements are examined, as well as parameters observed with the strip such as Specific Gravity, cylinder, pH, bilirubin, erythrocyte, glucose, colour, clarity, leukocyte esterase, ketone, nitrite, protein, and urobilinogen are evaluated.Depending on the size of the laboratory and patient volume, these parameters are performed with manual or fully automatic urine analysers.One or more of these parameters can determine whether the individual is healthy or sick based on the value it shows.
The CUT, analysed by biochemistry experts, is conducted using visual inspection, Dipstick analysis, autoanalyzer, and microscope.Visual inspection involves examining properties such as colour, clarity, and odour, while chemical examination determines properties such as pH, specific gravity, and ketone levels.Microscopic examination involves scrutinizing epithelium, bacteria, leukocytes, erythrocytes, cylinders, yeast, crystals, and other particles (such as sperm).The possible values of these examined parameters are presented in Table I.Grey areas in Table I denote the normal values for urine strip analysis, while blue areas indicate the normal values for urine sediment microscopy.It is important to note that these normal values may vary slightly between laboratories.
Table I shows that detecting cells in the urine sediment and knowing their number in each sample is important for the accurate detection of the disease.In the case of many patients, the increase in workload and the complexity of urine images may cause errors in manual measurements, and standardisation cannot be achieved.In this respect, it is precious that the manual imaging process can be standardised and used as training material for assistant physicians and laboratory technicians, and the efforts to blend these materials with developing technologies are precious.

A. Why This Study?
Physical, chemical, and microscopic examinations are performed in urine analyses using fully automatic urine analysers.In case of incompatibility seen as a result of physical and chemical examination, the specialist has to make a microscopic examination of the manually prepared sediment.The following problems are frequently encountered in manual microscopic examination [4], [5]: • Microscopic examination takes a long time, especially in tests with object counting, and there is a high probability of manual errors; • The variability of the examination from person to person and the emergence of standardization differences; • The need for an experienced technician to perform the analysis; • Manual microscopy increases labour cost and is less suitable for high-volume laboratories; • depth field effect of the image, some particles sticking together and some background effect; • Report sharing (lack of clinical report sharing); • Difficult to get results in a reproducible manner; • Overlapping of cells, small cell size (like bacteria) are the main challenges.

B. Motivation
To minimise these disadvantageous situations in urine analysis and to increase standardisation and measurement accuracy, the need for systems that perform urine analysis automatically has emerged.In recent years, digital image processing and pattern recognition theory-based automatic analysis models have been developed for the identification of various particles in urinary sediment [6]- [10].As a result, the use of automatic classifiers such as deep learning is becoming widespread, especially in high-volume laboratories to save time and cost.
In this paper, a multi-modal fusion deep learning-based system that can automatically distinguish particles in urine sediment is presented in Fig. 1.A new dataset consisting of eight classes of Leukocytes, erythrocytes, crystals, epithelium, yeast, bacteria, cylinders, and others (such as sperm) was used.The features of the images were obtained and combined with the Resnet18 and Resnet50 models, which are used in the model and are preferred as feature extractors.The features that maximise the classification success from the feature vector are determined and classified by feature selection algorithms.As a result, the accuracy of urine analysis has been increased.

C. Contributions of the Paper
The main contributions of the paper are as follows: 1. Introducing a new dataset with eight classes, we believe it will significantly contribute to new research in this domain by expanding the scope of existing datasets and including more classes; 2. The resnet18 and resnet50 architectures were employed as feature extractors, effectively combining the features extracted from urine sediment images; 3. The most effective features for successful classification were determined using the relieff, mrmr, and nca feature selection algorithms; 4. Experimental results demonstrate that the proposed system exhibits superior discrimination ability compared to other studies and mitigates standardization differences.

D. Figures
The presented paper is organised as follows.Section II presents related works.Section III relates to the data-gathering process and data.The proposed method is given in Section IV.Section V contains results and discussions.Finally, the conclusion is presented.

II. RELATED WORKS
With the use of artificial intelligence in the field of medicine, there are important developments in health services.Successful results are obtained in many areas from the use of artificial intelligence-based robots in the field of health to decision support systems.This section summarises the literature on urinalysis.Due to the success of classical machine learning algorithms and CNN architectures in classification, many approaches focusing on urine analysis have been proposed in recent years.Liang  Experimental studies to estimate urine glucose levels have shown highly accurate results compared to standard methods [10].Suhail et al. focused on the classification of particles in images of urine sediments using different types of CNN.Classification of particles was performed without feature extraction and segmentation in the model.Thus, the problems encountered in the manual analysis of urinary sediment with the proposed model were eliminated [11].
Liu et al. classified particles such as crystal and red blood cells (RBCs) by SVM using digital microscopic images obtained from urine samples.A contour tracking algorithm is preferred for feature extraction.The proposed system is based on environment, area, regional density, etc.It measures and classifies various parameters.With this method, a classification success of 91 % was obtained.The advantage of this method is its high classification speed and lower cost.The disadvantage of the system is that it is difficult to select the SVM kernel [12].Shen et al. preferred a urine analysis method based on the SVM classifier and AdaBoost learning algorithm.The method uses Harr attributes obtained from the image.In the method, the AdaBoost algorithm was used to select Harr features.Six types of urine particles were classified with 95 % accuracy [13].Zhou et al. proposed a segmentation-based method to identify 12 different particles.28 features such as shape and texture were extracted from urine images by using a genetic algorithm.The classification was performed with the artificial neural network model (ANN), and 96.19 % accuracy was obtained [14].Tangsuksant et al. used the ANN classifier to count RBC and WBC cells.Centre coordinates of RBC and WBC cells were determined by Circular Hough.Particle detection error rate was 5.28 % and 8.35 %, respectively [15].Avcı et al. proposed a scheme combining Discrete Wavelet and Neural Network classifier to detect urine particles.Before classification, images were pre-processed, and entropy, energy, and discrete wavelet properties were extracted.3400 urine images containing 10 types of urine particles were classified with an accuracy of 97.58 % [16].Li et al. used the Gabor filter and scattering transform to classify urine particles RBC-A, RBC-B RBC-C, WBC, and crystal.with SVM and extract features from images.It was classified with an accuracy of 98.11 % and a recall of 100 % [17].Sun et al. proposed a model based on the AdaBoost algorithm to identify RBCs.The model has an accuracy of 98.02 % and a classification speed of 0.9182 s [18].
The methods used in the studies summarised above require data pre-processing, such as segmentation and extraction of handcrafted features.The efficiency of the models depends on the segmentation and the extracted features.However, the complex features of images of urine sediments complicate preprocessing and feature extraction of the conventional model.The deep learning-based CNN automatically learns the desired features and provides end-to-end detection without the preprocessing and segmentation, enabling the classification and detection of urine particles.In the literature, CNN [2], faster R-CNN [19], multi-scale Faster RCNN [20], online fixed sample mining faster RCNN [21], and Yolov5 [22], [23] models have been developed to detect particles in urine.In [21], pre-trained ZF networks, Resnet, VGG-16, and PVANet architectures were used for particle detection with FRCNN.As a result, mAP value of 84.1 % with PVANet and 77.1 % with single-shot detector (SSD) was obtained.Zhang et al. classified the RBC and WBC in the Faster R-CNN model by determining the positions of the urine particles [24].The dataset consists of over 6000 annotated images from urine samples of 100 patients.In general, Dysmorphic RBC is similar to WBC.The Faster RCNN model was used to overcome this problem, which reduces the classification accuracy.RBC and WBC were detected with an F1_score of 91.4 %.The shortcoming of the proposed model is that it can only detect RBC and WBC.Pan et al. developed a CNN model to train the network, with the input images having a fixed size, and classified urine particles into three categories, including RBCs, WBCs, and COAXs.The method has 98.07 % accuracy and 98.39 % recall rate [7].Ji et al. used the Area Feature Algorithm and CNN to classify urine particles.The proposed method consists of the main net, RBC-WBC, and HYAL-MUCS modules.Each module is responsible for the classification of different urine particles.The advantage of the model is that it can classify 10 urine particles.The accuracy of the model is 97 % [8].Li et al. classified urine images with the LeNet-5 architecture, focusing on shape analysis [25].RBC, WBC, crystals, and epithelial particles were successfully detected by varying the number of convolution and output layers of the Le-Net-5 architecture.In the study, 50 microscopic images were taken per sample and pre-processing techniques including four morphological operations, noise reduction and contrast enhancement, thresholding and dilatation, erosion, opening and closing, were applied to the images.The improved LeNet-5 network provided 92.89 % classification accuracy with 91.29 % sensitivity, and 97.45 % specificity, slightly better than the commonly used three-layer backpropagation neural network.The literature on the analysis of microscopic urine samples is given in Table II.

III. DATASET
For this study, the urine sediment image dataset was obtained from the urine samples of 409 people, 143 women and 266 men, who applied to Elazig Fethi Sekin Research and Training Hospital with the approval of the Ethics Committee dated 20 May 2022 and numbered 8455.Microscopic examination was carried out by the procedure outlined below.
The urine sample was taken for examination within the first 1 h.After the samples were accepted in the laboratory, they were centrifuged at 1500 rpm (400-450 g) for 5 min with BD brand 10 ml urine tubes.After centrifugation, the clear parts of the urine remaining on the precipitate were carefully emptied so that 250 μl of precipitate remained.After the remaining precipitate was thoroughly mixed, 20 μl was taken with a pipette, and a slide-to-lamella preparation was prepared.Microscopic examination was performed and the average number per field was calculated.Images were obtained with an Optika B293PLi camera-attached trinocular microscope using a 22×22 mm coverslip in the urine sediment.Patient sample clinical sediment examination values were also noted as HPF (high power field).The whole transaction: Care was taken to prevent evaporation, cell fragmentation, contamination, pH irregularities, heat, light, etc. interferences within a 30-min turnaround period until the patient's urination results [23].
This procedure was carried out by an expert biochemistry doctor.Figure 2 shows the data collection equipment of the experimental setup for obtaining the urine image.Images were taken with the Optika B293PLi microscope with trinocular brightfield and a camera device with high speed, user-friendly camera with high resolution (10 MP), CMOS sensor, and USB3.0 connection.Each analysed image is in 3664×2748 colour JPG format.The total number of particles examined in the images taken from a total of 409 patients is 8509.The size of the examined particle images was taken as 224×224.Damaged cells and overlapping cells were ignored.Labelling of each particle was carried out by a physician specialised in biochemistry.The dataset was increased five times by applying clockwise rotation at 90°, 180°, and 270° angles and symmetry in the horizontal and vertical axis to each image.70 % of the dataset is randomly allocated for training and 30 % for testing.Table III shows the number of images collected and the number of images obtained by augmentation by classes.Figure 3 illustrates a sample image belonging to each class.Stage 1: Feature Extraction Resnet50 and Resnet18 were used as feature extractors to extract the features of the images in each class [29], [30] Stage 2: Feature fusion was applied to produce more consistent, accurate, and useful features from feature vectors obtained with Resnet architectures.F1 and F2 feature vectors are 1×1000 in size and by combining the two feature vectors, a 1×2000 size F feature vector was created.F = F1UF2, Here U is the concatenation operator.Stage 3: Feature Selection The feature selection algorithm was applied to create a subfeature set from the F feature vector, and decide whether the related feature should be selected by evaluating the features in question, and maximise the prediction accuracy of the classification algorithms.ReliefF, mRMR and NCA algorithms were used for this task.mRMR: Minimum Redundancy Maximum Relevance algorithm is a filtering method that tries to select the most relevant attributes with the class labels, while simultaneously trying to minimise the redundancy between the selected attributes [31].The purpose of this algorithm is to find a set of F features that make the Vs value given in (1) maximum and the Ws value given in (2).
where, I is a mutual information value and the ratio of Vs to Ws is MIQ (mutual information quotient) value.It determines the feature that makes the MIQ value the largest at (3) Fc.
NCA: Neighbour Component Analysis is a transformation technique that reduces the size of the dataset, which includes a large number of interrelated variables, to a smaller size by preserving the existing changes in the data as much as possible [32].NCA pseudocode algorithm shows xi feature vector yi = {1,2,3,…,c}class label, c class number in multiclass S = {(xi,yi), I = 1,2,…,c} classifier.It is as in (2): η -small optimistic constant, σ -Kernel width, λ -regularization parameter, and α -initial step length.In the first step of the algorithm, the weight vector w(0) = (1,1,…1) error value ε(0) is finite.ReliefF: ReliefF is an advanced version of the Relief statistical model.The method first compares the sample from the dataset with other samples in its class.The sample taken creates a model according to the distance from the samples in different classes and makes a feature selection [33].
Here,  represents the weight (significance) of the jth feature, Hj is the relevant feature value in the closest sample with the same class, and Mj is the related feature value in the closest sample with a different class.The diff function in (4) calculates distances between samples and features. .
9. end Stage 4: Classifier In this study, SVM was used to distinguish particle images.

V. RESULTS AND DISCUSSION
In order to examine the performance of the model, we use the following metrics commonly used in the medical image classification field: accuracy (A), recall (R), precision (P), and F1_score, TP -number of true positive particles, FP -number of false positive particles, FN -number of false negative particles, and TN -number of true negative particles.The metrics are defined as follows: (5) The study was carried out on a 64-bit operating system computer with Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz, NVIDIA Corporation Geforce GTX 1050, and Matlab R2021a.In the training phase, we use the transfer learning mechanism for the Resnet18 and Resnet50 architectures.

Applied Computer Systems _________________________________________________________________________________________________2024/29
To demonstrate the superiority of the proposed model in recognising particles in urine, the presented paper offers three new contributions.These are i) collecting a new dataset with eight classes; ii) fusing the features of urine sediment images; iii) using feature selection algorithms to identify features that maximise classification accuracy.First, the particles in the dataset were distinguished using the Resnet50 and Resnet18 models.Accuracy based on standard performance measures was 96.92 % and 95.15 %, respectively.Figure 5 shows the confusion matrices for Resnet50 and Resnet18.Three scenarios were used in the proposed method to increase the particle discrimination accuracy.The most important factors affecting the success of the proposed model are the extraction of high-level features and the combination of features.ReliefF, mRMR, and NCA feature selection algorithms were used in each scenario to determine the features that maximised accuracy.In the first scenario where ReliefF was applied, the accuracy was 98.54 % with the first 944th feature of the feature vector.In Scenarios 2 and 3, accuracy with mRMR and NCA was 98.50 % and 98.45 %, respectively.The first 1604 and 1595 features of the feature vectors were used in these two scenarios.Figure 6(a) shows the confusion matrix obtained by using ReliefF feature selection and Figure 6(b) shows the variation in accuracy according to the number of features used.Table IV shows the performance parameters for Scenario 1 using the ReliefF algorithm.According to the results obtained, the proposed model increased the accuracy by 3-3.5 % more than the accuracy values obtained with Resnet models.Although the developed model exhibits a more complex structure compared to pre-trained models, its high accuracy value enhances its attractiveness.It would be logical to conduct an error analysis of the model for future research.It would be appropriate to mention two errors that affect the accuracy of our model.These are faulty decisions made by the specialist doctor when labelling the dataset due to the discrimination error and the shape similarity of some particles.However, precision, recall, and F1_score are reasonable for all classes.Therefore, the discrimination error rates are so low.According to the Confusion matrix in Figure 6(a), some erroneous results are striking.For example, the similarity of Erythrocyte-Leukocyte, Erythrocyte-yeast and Crystal-cylinder cell structures to each other is effective in reducing the overall accuracy.

VI. CONCLUSION
Examination of urine sediment microscopic images is a relatively complex procedure that requires experience due to the large diversity of urinary sediment particles, their overlapping or adherence, morphological differences of cells, unclear background of the image, lamella-lamella defects, and indistinguishable artefacts as a result of possible contamination.An experienced technician is needed, especially since microscopic examination will show individual variability and increase the amount of laboratory measurement and observation uncertainty.To overcome these difficulties encountered in the manual examination of urinary sediment, a deep learning-based model that can automatically distinguish particles is proposed.Initially, the Resnet50 and Resnet18 architectures were used as feature extractors in the model.The features of the urine sediment images were effectively extracted and combined.Then, the features that maximised accuracy were selected from the obtained feature vectors and classified.As a result, the accuracy value has been increased with the developed network structure.In addition, the urine dataset that we have made public in the article will expand the scope of existing datasets and contribute to new research in this direction as it includes more classes.Data Availability: The data that support the findings of this study are openly available at https://github.com/ttuncer/urinedatasetFunding: This study was supported by the Scientific and Technological Research Council of Turkey with project number 122E094.
Conflict of interest statement: The authors declare no competing interests.
Ethical statement: The dataset used in this study was collected with the decision of the ethics committee dated 20 May 2022 and numbered 8455.
. The dataset was randomly allocated as 70 % training data and 30 % test data.The 224×224 images(I) were fed into the Resnet50 and Resnet18 architectures inputs.Feature vectors were obtained from the fc1000 layer of Resnet50 and Resnet18.F1 = Resnet50(I); F2 = Resnet18(I).

Fig. 6 .
Fig. 6.Confusion matrix for ReliefF and accuracy variation according to the number of features used.

Figures 7 (
Figures 7(a) and 7(b) show the variation in accuracy obtained with mRMR and NCA feature selection.In the first scenario, where the highest accuracy value was obtained, fewer features were used compared to the other two scenarios.TableIVshows the performance parameters for Scenario 1 using the ReliefF algorithm.According to the results obtained, the proposed model increased the accuracy by 3-3.5 % more than the accuracy values obtained with Resnet models.

TABLE III DATASET
CLASS AND DATA COUNTS