Research on Preprocessing Method for Microscopic Image of Sputum Smear and Intelligent Counting for Tubercule Bacillus

In order to automatically detect bacilli in sputum smear with microscopy, an intelligent recognition method based on machine vision is presented. Firstly, a novel method with the fusion of multi-frame image information is presented to improve the quality of microscopic image of sputum smear by extending the dynamic range, and then the background filter was designed based on the single layer perceptron to recognise bacilli segmentation from background. After eliminating the short twig and small area noise, the suspicious goals and the image noise are separated. In the feature extraction, two important features are presented to solve the difficult problem of identification and counting for the overlapping and winding bacilli. Finally, based on the above research content an EBP neural network classifier is designed for the accurate identification and counting of the bacilli. Experimental results show that the method presented in this paper is a feasible and accurate solution for bacilli automatic identification.


Introduction
Tuberculosis (TB), known as the "white plague", is a chronic infectious disease caused by tubercle bacillus infection. The World Health Organization (WHO) has declared tuberculosis is a global emergency. It is estimated that around one-third of the World's population is infected with mycobacterium tuberculosis (M. tuberculosis) [1]. The majority of cases occur in sub-Saharan Africa, South and South-east Asia, Latin America, the Caribbean and China.
Tuberculosis is a communicable disease for which an early diagnosis is critical for disease control. Conventional microscopy is the most commonly technique. Used in the routine diagnosis of TB. In this method, the identification of tubercle bacilli is routinely done in sputum smears using a microscope with manual detection, which has been widely used in bacteriological examination of sputum smears obtained from suspected patients. In fact, the manual screening by using a microscope for the bacillus identification involves a labor-intensive task with a high false negative rate and requires sophisticated equipment and trained technicians [2]. Automatic detection by computer with machine vision might replace the current manual identification by the following advantages, e.g. a substantial reduction in the labor workload of clinicians, improving the sensitivity of the test and a better accuracy in diagnosis by increasing the number of images analyzed. The key of the method is to obtain the clear microscopic image of sputum smear and design an effective image processing algorithm to separate the target from the complex background so as to count the number of tubercle  2 bacilli accurately. Since there being the complex background with a lot of debris and the overlapping and conglutinate bacilli, the automated identification of such organisms is a challenging task.
There are many reports to realize automatic identification of tubercle bacilli [3][4][5][6][7][8][9][10][11]. However, due to the complex background and the fact that there is a large number of overlapping cells in microscopic images, a new method is necessary to obtain the information-rich, clear microscopic image and to count overlapping cells accurately. In this paper, a novel method based on the fusion of multi-frame image information is presented to extend the dynamic range of the microscopic image. In this method, the Laplacian pyramid algorithm is used to divide the microscopic image into some layers, and the respective weights of each layer are determined by the gradient and the entropy of the layer images. Finally, the inverse Laplacian pyramid algorithm is applied to synthesize the information-rich microscopic image. In order to accurately count the number of the overlapping bacilli in the microscopic image, this paper uses the head nodes and long branches for each connected region to describe the features of overlapping bacilli cells and a corresponding classifier is designed, which can achieve the identification and counting of bacilli accurately and effectively.

The Novel Pixel-Level Image Fusion Algorithm
It is an important method to improve the technological level of TB routine test by using the machine vision to achieve the intelligent identification of bacilli for regular smears. However, there are still many difficulties to be resolved. The sputum smear contains a great number of mucus, impurities, as well as necrotic tissues, which will lead to different sensitivity to light intensity in different regions of the same view in microscopic imaging process. To achieve the intelligent identification of bacilli, a wide dynamic range of the microscopic image is necessary to obtain the complete characteristic information of the detection zone of sputum smear. However, the dynamic range of CCD used for imaging sensor is generally restricted. Therefore, it is difficult to record the most of details of mycobacterium tuberculosis, such as over-saturation in bright areas, underexposed in dark areas, poor color reproduction and other issues, so the misidentification of mycobacterium tuberculosis is easily caused. In order to obtain the sufficient information of tuberculosis micro-images, a lot of research work has been carried out in recent years, such as using new materials and new technologies to improve the accuracy of detection [12][13], however, these methods are still difficult to meet the requirement of the intelligent detection of microscopic image.
Theoretically, the images obtained under dark conditions can record the details in the bright scene. Conversely, the images obtained under bright conditions can show the details of the shadow. All of the scene information can be recorded by changing the input light intensity to record the multi-frame images for the same scene. Consequently, the dynamic range of microscopic images can be extended. In this paper, we obtain multi-frame exposure TB microscopic images of the same view to create high dynamic range images, instead of the traditional method which enhances the amount of information by the single_frame image.  In this paper, the automatic detection device is designed and shown in Fig. 1. Among them, the automatic detection device mainly comprises an imaging system and a three-dimensional motion control executive mechanism. The imaging system comprises a CCD, lens and the detection platform constructed by the back light source. The motion control of the detection device is mainly composed of three sets of the motion controller and the motor which is for X, Y and Z direction, respectively. The whole system is controlled by a computer to achieve image acquisition, processing and view transformation.

The information fusion of sequence images of the same view
There are lots of methods to create high dynamic range images from multi-frame images with different exposure obtained in the same view field. In this paper, a new method is proposed based on the Laplacian pyramid algorithm [14] [15], in which the gradient and entropy of the hierarchical images of the different exposure images are combined to represent the information of the same view field, and the weights of hierarchical images are allocated according the amount of information of each layers of the pyramid of various images with different exposure. Finally, images are reconstructed by Laplace pyramid inverse algorithm. The high dynamic range micro-image created can not only highlights the details but also retains the most of information of the image as much as possible. It should be noted that the features can be enhanced for requirement of the intelligent detection of microscopic image by using this method.
The proposed method can be described as follows. Firstly, the Gaussian pyramid images are obtained. Assuming the original image is f with the size M*N, the image g1 with size (M/2)*(N/2) can be obtained by the low-pass filter and the interlaced sampling (Reduce operation(2.1) ). If the this process is repeated k times, a series of gradually smaller size of the low-pass images marked as g0, g1, g2,…gk can be obtained, which is called as Gaussian pyramid.

Reduce
x,y = m=1 Where w[m,n] is called as Gauss template. The Gaussian pyramid is given by the following equation.

Figure 2. Diagram of laplacian pyramid
After getting the Gaussian pyramid, the difference between two adjacent image layers is called as Laplace image. The series of l0,l1,l2,…,lk (here gk=lk) are called as the Laplace pyramid. The construction method is shown in Fig. 2. The mathematical fomula is as follows.
Laplacian pyramid inverse algorithm is the inverse process, and its role is to reconstruct a new image by merging the Laplace image layers. This process can be realized by interpolating and merging all layer images. The mathematical equation is as follows. Here lk,k is the result of interpolating and enlarging the kth layer of Laplacian pyramid image lk by k times, so its size is the same as the original image. A simple method is that lk is interpolated and enlarged to the size of lk-1 and embedded into lk-1 to get a new image, then the new image is interpolated and enlarged to the size of lk-2 and embedded into lk-2 to get another new image, we repeat this process again and again until level 0 layer image. At last g0 is the reconstructed image. Its mathematical description is as follows: As each layer (except the top layer) of Laplacian pyramid can retain the important characteristic information (such as edge information, etc.), and the characteristic information exists in different decomposition layers respectively in accordance with different scales. The important information plays a key role in synthesized image. All frames of different exposure images are stratified by Laplace pyramid algorithm and each layer of different exposure images is allocated different weights respectively according to the amount of information. And then these layers are synthesized into a high-quality micro-image by Laplace pyramid inverse algorithm. The Fig. 3 shows the method.
Generally speaking, the change of the pixel's brightness in image determines the outline of the image and the gradient of the image can exactly reflect the changes of pixel's brightness in image. The richer the image content, the greater the image gradient values. Therefore, the image gradient can stand for the detail information of image. Likewise the image entropy can reflect the amount of information contained in the images, the theory of entropy was first proposed by American scientist Shannon, which shows distribution of pixels information in an image. Basing on the above theory, we combine the image gradient with the image entropy to determine weights for hierarchical images of different exposure in order to synthesize final image which contain the more information and details. In this paper, the image gradient is written as the following equation (2.7) and (2.8): Normalized function is expressed as: I(x,y) is the pixel value at (x,y), δ is the value of ΔIx or ΔIy, Imax refers to the maximum pixel value of the image. If the image is 8-bit bitmap, then Imax is 255. The final normalized gradient can be expressed as: pi is the probability when image pixel value is i. The final weight of each level of the image is defined as: where i is ith exposure image of the same vision; k refers to the kth layer of pyramid. Ii,k is the kth layer of ith pyramid. n is the number of exposure images. α is regulator which is used to adjust the weights of image entropy and gradient information, and its value can be determined by experiments. According to the expression (2.12), we can calculate the weight value of each layers of different exposure pyramid, and then form a new Laplacian pyramid by fusion of corresponding layer of different exposure images based on different weight. The synthesized Laplace Pyramid is as follows:

13)
Ik is the kth layer of the composite pyramid images. At last, the high-quality micro-image by the Laplace pyramid inverse algorithm can be obtained. For the color images, we can process the each color space respectively. At same time, we also can attenuate or enhance some color components according to object features. We use four different exposure images of the same field of view to enhance image information. Fig.  4(a)~(d) are four microscopic images of the same field of view with different exposure: 20%, 40%, 60%, 80% respectively, while Fig.4(e) is composite image by the above algorithm. Obviously, the quality of composite image is improved significantly, not only for the amount of information of the micro-image, but also for the obvious features of details of the bacillus. It is very conducive for the further image segmentation and feature extraction operations.

Selection of Characteristic Parameters of Bacilli
In the process of establishing the model of pattern recognition, description of the shape feature for object is various and to find the most effective invariant parameters based on the standard of separability and classification is a key to extracte bacillic.

The characteristics of the single bacilli
After the processing of color segmentation, we select some basic morphological features in connected components according to characteristics of mycobacterium tuberculosis in micro-image, such as the length, width, perimeter, acreage, rectangle degrees, stretching length. Considering the rectangular features and the texture features were just global information extracted from microscopic images, and their effect is not significant for intelligent recognition, so we won't consider it in this paper. The main characteristics of TB we choose are as follow. (1) The length H and width W, they describe the height and width of identification box in each connected region of microscopic image. (2)The perimeter P, it refers to the number of edge pixels of object. (3)The acreage S, it refers to the number of pixels in the suspected target regions. (4)The rectangular degree R, it is used to describe the deviate degree between actual region and rectangular target region. This feature can be used to distinguish a number of small circular areas of impurities from bar TB. The rectangular degree R can be expressed as R=S/(W*H). (5) The elongation E. It describes the stretching degree of tuberculosis bacteria and the regional compact. Because TB is bar shape, so that elongation can better characterize the shape of TB. The E can be expressed as E=min {W, H}/max{W,H}.

The characteristics of the overlapping and intertwining bacilli
In the process of intelligent identification, there are many overlapping and intertwining bacilli in some detected visual field of the microscopic image. It is difficult to distinguish and count the number of TB just by the six basic characteristics described above. Considering there are many flagellums which are easy formed short-branches surrounding bacilli, we select the number of long branches and the nodes of branches of the detected object as two important features to distinguish the bacilli from debris. In addition, the nodes of branches are used to count number of overlapping and intertwining bacilli in the microscopic image. Usually there are two branch nodes in the single bacillus, two bacilli contain 3~4 branch nodes and three bacilli contain 4-6 branch nodes and so on.
The method to extract branch nodes of TB can be described as follows. Firstly, in order to eliminate a great majority of the debris generated in the previous processing, the entire microscopic image is smoothed, and the thinning algorithm is applied to each suspected bacilli region (a connected region). And then through scanning method, the entire image is processed. For ith connected region, if the value of a processed pixel is 0 (black spots) and its surrounding 8 pixels value is equal to 255*7, the point is regarded as a branch node of suspected bacilli region. We store the location of the pixel in a predefined structural container Ai, and the total number of the branch node of suspected bacilli region is added by 1. Finally, the total number Mi of all branch nodes in ith connected region of the microscopic image can be got. The above process is repeated for all connected region until the entire image has been processed, the M={ M1,M2..Mi..Mn } can be got. Where, the n is the number of suspected bacilli region.
The number N of long branches of bacillus is an important characteristic parameter to distinguish the real number of bacilli in each region. The specific extraction method is described as follow. Firstly, a threshold n_Limit of the branch length is set. Each sub-branches node is extracted one by one from the structural container Ai according to the location of the node, and the node is tracked from the location stored container Ai. If there are not intersections and the tracking number of pixels is greater than n_Limit, the branch was thought as a long branch and the number N is added by 1. If there is a cross-point or the tracking number of pixels is less than n_Limit, the branch was thought as a short branch and the number N is unchanged. After finished the tracking of the first node, we continue to track the next branch from the container Ai until the container Ai is empty. Finally we can get the number N in suspected target regions. Tab.1 shows the processing results (d).

EBP Neural Network Classifier
Neural network has a better classification and memory function by introduced the hidden layer neurons in the neural network, and the corresponding learning algorithm has become the research focus now. Rumelhart [16] raised the EBP algorithm which systematically solved the learning problems about connection of hidden layer in multi-layer neural network and gave a full derivation in mathematics. In this paper, we adopt EBP neural network for bacilli cell recognition and use the added momentum factor algorithm to prevent the network into the local minimum value of the error surface.

Variables of input and output
For artificial neural network, the choice of input variables needs to meet two basic principles. The first one is that the input variable has great impact on the output and easily to extract from the features we choose. The second one is that the correlation of all input variables is unrelated or very small. Therefore, we selected 8 representative feature parameters as input of neural network based on the discussion above. In addition, we found there are generally no more than 3 intertwined bacilli in the actual microscopic images, so we define four output variables ultimately for artificial neural network. The three-level structure of BP network has shown in Fig. 5. Two output nodes of the out layer can describe the four output variables of a suspected target region, Namely, (00) refers to 0 bacillus (non bacillus); (01) refers to one bacillus, (10) refers to 2 bacilli, and(11) refers to 3 bacilli. The corresponding connected region of suspected bacilli also can be divided into four categories. The connected region which has non-bacilli is defined as 0# class connected region; the connected region that contains 1 bacillus is defined as the 1# class connected region and so on.

The selection of nodes in hidden layer
There are many ways to determine the number of nodes of hidden layer. Generally, the node number of hidden layer is as one or twice as nodes of the input layer, and then the appropriate residuals is added [17]. In this paper, in order to determine the optimal number of nodes of hidden layer, 5-20 nodes of the hidden layer are trained, and 400 suspected view of microscopic images are adopted as training samples. In these 400 vision of images, the number of 0# images (it only contains 0# class connected region) is 190, the number of 1# images (the bacilli number of each connected region is no more than 2) is 150, the number of 2# image (the bacilli number of each connected region is no more than 3)is 42 and the number of 3# image (the bacilli number of each connected region is no more than 4) is 18.

Results and Discussion
After designing the neural network, we selected a set test samples to prove the validity of the system described above. Each full sample is composed between 10 and 100 RGB color images of 1600*1280 pixels acquired from each subject's sample of sputum. The test sample set includes 3000 negative images of different microscopic fields from 30 healthy subjects and 1114 positive images from 10 patients. For each detected view, the testing results by the method described in this paper are compared with results of manual screening. Diagnostic accuracy is traditionally expressed in terms of sensitivity and specificity. Sensitivity is the probability to assign a diagnostic test as positive when in fact is positive. It is also known as the fraction of true positives. The complement of sensitivity is the specificity which is the false negative rate. Tab.2 shows the testing result of some typical samples, which contained impurities, entanglement and fracture. Compared with the conventional microscopy, it proved this method has a high accuracy. Besides that, Tab.3 presents the specificity and sensitivity  From the results shown in Table 1, we can conclude that the developed technique appears as a feasible solution for bacilli identification in sputum samples.

Conclusion
The manual screening by using a microscope for the bacillus identification and counting involves a labor-intensive task with a high false negative rate and requires sophisticated equipment and trained technicians, therefore, to look for new methods to automate these procedures is necessary. Using machine vision can improve efficiency and increase the security of the people by minimizing contact with the samples and obtaining more accurate results. The major factors affecting the results of the bacillus identification are the microscopic image quality and there being many overlapping and intertwining bacilli in some detected visual field. In this paper, a new technique to realize automatic counting of bacilli cell based on the fusion of differently exposed microscopic image of sputum smears is presented; and the background filter was designed based on single layer perceptron to realize object segmentation from background; After eliminating short twig and small area noise, the suspicious goals and the image noise is separated. In the feature extraction, besides the base features of single bacilli two important features are presented to solve the difficult problem of identification and counting for the overlapping and winding bacilli cells. And at last, an EBP neural network classifier was designed for accurate classification and counting of bacilli. Through a large number of smear detection experiment, the effectiveness of the proposed method is proved.

Acknowledgments
This project is supported by the National Natural Science Foundation of China (Grant No.51575099)