Focus measure method based on the modulus of the gradient of the color planes for digital microscopy

Abstract. The modulus of the gradient of the color planes (MGC) is implemented to transform multichannel information to a grayscale image. This digital technique is used in two applications: (a) focus measurements during autofocusing (AF) process and (b) extending the depth of field (EDoF) by means of multifocus image fusion. In the first case, the MGC procedure is based on an edge detection technique and is implemented in over 15 focus metrics that are typically handled in digital microscopy. The MGC approach is tested on color images of histological sections for the selection of in-focus images. An appealing attribute of all the AF metrics working in the MGC space is their monotonic behavior even up to a magnification of 100×. An advantage of the MGC method is its computational simplicity and inherent parallelism. In the second application, a multifocus image fusion algorithm based on the MGC approach has been implemented on graphics processing units (GPUs). The resulting fused images are evaluated using a nonreference image quality metric. The proposed fusion method reveals a high-quality image independently of faulty illumination during the image acquisition. Finally, the three-dimensional visualization of the in-focus image is shown.


Introduction
Automatic autofocusing (AF) in digital microscopy is highly dependent on the sample topography variability and also its color distribution.As stated by Qu et al., 1 different focus criterion functions perform quite differently even for the same sample.7][8] Some research has determined that the best AF metric is based on the Brenner function; 2 other research gives priority to the variance, 9 Vollath-4, [10][11][12] or the sum-modified-Laplacian, 13 among other methods.
In the case of the RGB space, few works for AF have been reported. 14,15In addition, the effectiveness of the AF algorithms depends on the color space selection wherever the numerical computation is done. 16To avoid it, a waveletbased technique for converting multichannel (e.g., color) data to a single channel by principal components analysis has been reported for this task; 17 unfortunately, it is computationally intense.
9][20] Hence, it is possible to improve the performance of a large quantity of AF algorithms since all of them are capable of indicating a focused slice from the MGC image.Even more, because first derivative methods can be efficiently implemented in GPUs, the MGC algorithm can work in parallel.
In widefield microscopy, it may be possible to focus the transverse sections that are placed at the depth of field (DOF) of the objective lens.To record the three-dimensional (3-D) volume, it is necessary to axially scan the sample.Additionally, an extra difficulty arises: the DOF of the optical objectives decreases when the numerical aperture (NA) increases.It abruptly produces blurry images in the portion of the object that lies outside of the DOF.
A common approach to digitally extend depth of field (EDoF) is by the use of a digital image fusion scheme.Typically, the image fusion schemes select the in-focus pixels along the z-axis to reconstruct an all-in-focus composite image.2][23] In this work, a parallel implementation in GPU of a pixel-by-pixel image fusion of multifocus color images based on MGC is done.According to the image quality metrics, the proposed method is competitive to merge these kinds of images.The 3-D visualization of the in-focus images verifies the fusion results.
This work is organized as follows: in Sec. 2, the MGC transformation for multichannel to grayscale frames is briefly reviewed, and the AF functions and image fusion technique used in this paper are analyzed.In Sec. 3, the procedure for acquiring the different z-stacks of digital images is described.In this research, human and animal tissue samples have been employed as test objects to prove the proposed algorithms.The human tissue samples were prepared by Mikroskope.Net 24 , and the animal tissue came from the Human Connective Tissues Microscope Slide Set. 25 In Sec. 4, the AF and fusion results of the experiments, which we conducted to evaluate the algorithms are presented.Finally, the conclusions of the work are presented in Sec. 5.

Multichannel Conversion to a Grayscale Image
In the RGB space, the red, green, and blue components of a vector are commonly related to the pixels of an RGB image of size M × N.They can be represented by Cðx; yÞ, as in the following equation: ; t e m p : i n t r a l i n k -; e 0 0 1 ; 6 3 ; 5 8 7 Cðx; yÞ ¼ Rðx; yÞ î þ Gðx; yÞ ĵ þ Bðx; yÞ k; (1) where Rðx; yÞ, Gðx; yÞ, and Bðx; yÞ are the RGB space channels and î, ĵ, k are the unitary vectors, respectively.Typically, a compound gradient image g c ðx; yÞ is determined by 18,19 E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 2 ; 6 3 ; 5 0 8 g c ðx; yÞ where g R ðx; yÞ, g G ðx; yÞ, and g B ðx; yÞ are the gradient images for each channel.
In general, the modulus of the gradient of the color planes g c is computed using the Euclidean distance, 20 as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 3 ; 6 3 ; 4 2 5 g c ðx; yÞ ¼ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi X band i¼1 ∂Cðx; y; iÞ where i ¼ 1; : : : ; band is the dimensionality of the color space.An alternative representation of the MGC operator is the expression g c ðx; yÞ ¼ jMGC½Cðx; y; iÞj.Conventionally, the partial derivative along the x-axis of a two-dimensional function Cðx; y; iÞ can be numerically approximated as ∂Cðx;y;iÞ ∂x ≈Cðxþ1;y;iÞ−Cðx;y;iÞ.Likewise, the partial derivative along the Y-axis is given by ∂Cðx;y;iÞ ∂y ≈ Cðx; y þ 1; iÞ − Cðx; y; iÞ.In color image processing, the gradient is commonly used as a procedure of color edge detection.Therefore, the modulus of the gradient of the color planes is a sharp image, which can be computed using the equation E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 4 ; 6 3 ; 2 2 7 The color difference formula of Eq. ( 4) is valid in the RGB color space.
In addition to the RGB space, color images are also processed in the hue, saturation, and intensity (HSI) color space, because it is a suitable model for color description and analysis.The HSI space is modeled as a double cone where hue represents the dominant color, saturation represents the purity of the color, and intensity represents the brightness, respectively.As stated by Gonzalez and Woods, 26 this model decouples the intensity component from the colorcarrying information (hue-saturation) in a color image.The intensity channel is an essential descriptor of monochromatic images, and it is classically used for multichannel conversion to a grayscale image.The difference measurement when working in the HSI color space is modified as established by Koschan and Abidi. 18n this work, the multichannel conversion to a grayscale image has been done by means of the MGC operator as is shown in Fig. 1.As the MGC operator is a color edge detection technique for digital images, the MGC(RGB) and MGC (HSI) matrices show the high spatial frequency content of the input color images.Thereby, this makes it suitable for finding focused regions.

Autofocus Methods
In the literature, there exist some comparisons about the performance of AF algorithms. 4,6,8,9,12Each algorithm is capable of producing a figure of merit (FM) that is analyzed by taking into account the global or local variance in the image intensity values fðx; yÞ.Customarily, the AF algorithms can be classified into five groups according to their mathematical nature: derivative-based algorithms, 27,28 statistical algorithms, 10 histogram-based algorithms, 6,12 intuitive algorithms, 9 and image transformations-based algorithms. 3hroughout this paper 15 AF algorithms, which have been widely reported in the literature, are tested and compared using the MGC images.This task was carried out to improve the performance of AF algorithms.Table 1 summarizes the definitions of the most typical AF metrics defined in the new approach, namely the MGC transformation.The output of an ideal AF algorithm is commonly defined as having a maximum value in relation to the best focused image position.Moreover, this value clearly decreases as defocus increases.As noted by Tian, 3 the fundamental requirements for an FM are unimodality and monotonicity, which ensure that the FM has only one extreme value and is monotonic on each side of its peak or valley.Furthermore, Redondo et al. 4 defined the number η of local maxima, the width of the focus curve α∕β given by the width of the focus curve at 80% and 40%, and noise/illumination invariance as important features of the autofocus curve.Another complementary characteristic of the AF algorithms is their accuracy and fast response.To evaluate the AF performance (AP) of each AF algorithm, the following score is proposed: e m p : i n t r a l i n k -; e 0 0 5 ; 6 3 ; 1 3 7 APðFocus Measure; SpaceÞ ¼ 1− where Z ¼ z∕ΔZ represents the number of focal planes along the z-axis far away from the origin z ¼ 0, and ΔZ is the distance between axial planes.For instance, if Z ¼ 150 μm 50 μm ¼ 3 then the AP metric is equal to 0.7.This happens because the mentioned measurements do not locate the focused plane until precisely three ΔZ steps away from the plane z ¼ 0. At best, AP is equal to 1. Low AP results from two conditions: (a) the AF algorithms do not reveal the focal plane at z ¼ 0 or (b) the AF algorithms indicate a wrong focal plane, which is too far from z ¼ 0.

Multifocus Image Fusion
As mentioned previously, any microscopic imaging system can only focus the field of view (FOV) of the sample that is   inside the DOF of the objective lens.This means that only certain axial planes of the sample are in-focus.A current solution to this drawback is a multifocus image fusion to reconstruct an all-in-focus image of the complete FOV for a particular specimen.This can be done by capturing images of the sample on different focal axial planes.In this section, a color image fusion scheme based on the MGC method is proposed as shown in Fig. 2. Let C z ðx; y; iÞ be a set of input images, where z ¼ 1;2; : : : ; Z.The index i ¼ 1;2; 3 is related with the channel/band used.For each axial plane, Eq. ( 2) is computed to create a compound gradient image and then for each pixel ðx; yÞ the maximum value is selected using sapðx; yÞ ¼ max z fg c 1 ðx; yÞ; • • • g c Z ðx; yÞg.In other words, the sapðx; yÞ matrix denotes the slice axial position of in-focus pixels along the z-axis.A postprocessing stage involves a spatial consistency algorithm. 17This postprocessing is carried out by means of a low pass filtered f sapðx; yÞ matrix using a p × q median filter.This algorithm ensures that the majority of the intensity pixels in a p × q neighborhood of f sapðx; yÞ, come from the same z-slice or from the closest one.For example, the spatial consistency of the f sapðx; yÞ matrix is shown in Fig. 3. Figure 3(a) contains three p × q neighborhoods, where the value of the slice axial position is higher than its neighbors.Figure 3(b) shows these values adjusted to match the values of the p × q neighborhood of f sapðx; yÞ to conserve the continuity of the surface of the sample.
To avoid the introduction of artificial information, the fused image Φðx; y; iÞ is composed from the (multichannel) pixels that are present in the original input data C z ðx; y; iÞ, only if the slice position fulfills the condition z ∈ f sapðx; yÞ for each pixel ðx; yÞ.Therefore, a multifocus image fusion algorithm can be defined as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 6 ; 3 2 6 ; 3 6 3 Φðx; y; iÞ ¼ C e sapðx;yÞ ðx; y; iÞ: Schematically, the proposed image fusion procedure is shown in Fig. 2. As can be seen, the fused image Φðx; y; iÞ is composed of the sharp regions provided by the in-focus pixels of the input color images.To accelerate the numerical computation, the fusion process is migrated to GPU.The resulting fused images are evaluated with a nonreference image quality metric based on measuring the anisotropy of the images.The anisotropic quality index (AQI) of an image Φðx; y; iÞ is given by 29 E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 7 ; 3 2 6 ; 2 3 0   where μ Φ is the mean of the values of the Rényi entropy RðΦ; θ s Þ, measured in directions θ s ∈ ½θ 1 ; θ 2 ; : : : ; θ S .The Rényi entropy measures the frequency content of an image through its directional pseudo-Wigner distribution. 29

Simulated data
For testing purposes, a simulated stack of 20 frames is constructed from a color image of 2584 × 1936 pixels.
Figure 2(a) shows some digitally defocused slices using the software package extended EDoF plug-in. 17Each blurred image was obtained by convolving an image with a Gaussian point spread function (PSF) with increasing width.The 3-D visualization of the resulting in-focus image using the fused scheme of Eq. ( 6) is sketched in Fig. 2(d).Their spatial consistency of the f sapðx; yÞ matrix is shown in Fig. 3.In addition, the results of the AQI of the fused image and the normalized mean square error (NMSE) 30 between the original image and the merged image are shown in Table 2.

Image Acquisition of Histological Samples
A motorized Axio-Imager-M1 optical microscope system manufactured by Carl Zeiss is used to image the histological    3, it is evident that interplanar distance ΔZ between different optical sections is determined by the NA of the objective lens.4 Results and Discussions

Focusing Results
To obtain a performance evaluation of the 15 AF techniques on the MGC images, six z-stacks of 21 multichannel images are recorded using two histological samples.Each stack has a particular color that is highly dominant as shown in Fig. 4.This allows us to evaluate the MGC method for different color distributions and amplifications inside of the digital image.
Case I: Figures 5 and 6 show the focus measure graphs for a z-stack of images obtained from the human carotid tissue amplified at 2.5× and 40×, respectively.According to the focus measure curves, the MGC image turns out to be a suitable space for AF measurements because all the FM decrease monotonically as the defocus increases.Also, the focus curves of all the AF algorithms show a monotonic behavior, a single local maxima η, a narrow width α∕β, and they achieve the highest performance.Case II: Figure 7 shows an elastic cartilage sample amplified at 2.5×, 10×, 40×, and 100×.From this histological sample, four z-stacks of frames are acquired.Every RGB image is transformed into the MGC space to measure the focusing.As can be seen, when the objective lens is modified, it is necessary to modify the illumination intensity over the sample, which causes the color distribution to change.Data are processed and the results are graphed in Fig. 8.
Some research 4 has reported that beyond a magnification of 63×, the performance of the various AF metrics is drastically impaired.According to results shown in the graphs of Fig. 8, when using the MGC approach all the FM curves present monotonic behavior, even when the magnification is increased to 100× (oil immersion).This last experimental result supports the advantage of a color-to-MGC space transformation.Nevertheless, a problem arises when images of a sample of thickness t ≥ DOF are acquired at a magnification of 100×.There exist portions of the image partly in focus.In the graphs of Fig. 8(d), two regions in focus located at z ¼ 0 and z ¼ 6 μm can be seen.According to the results given in Tables 4 and 5, all the AF measures realized in the MGC space are accurate in spite of the different magnifications, unlike some typically used channels for focus measure.
Another advantage of the MGC method is its computational simplicity and inherent parallelism.Figure 9 shows Fig. 8 Focus measure from a z-stack of images, which are acquired using different objectives.All the FM present monotic behavior even up to an amplification of 100×, and the new color distribution inside of the images acquired from the amplified sample under inspection.
the computational cost of the MGC(RGB) method in a z-stack of digital images of 2584 × 1936 pixels, when they run on Intel © X © (R) 2.10 GHz, 16 GB RAM, NVIDIA Quadro K4000.The parallelized MGC method on the GPU is one order of magnitude faster than the same application implemented in CPU.

Multifocus Image Fusion Results
It is well known that the digital images of thick microscopic objects provided by an optical widefield microscope device are strongly blurred for the portion of the object that lies outside of the DOF of the objective lens.We can seek those regions of the FOV, which are conveniently located in-focus.The present subsection will describe the results of a method to merge multifocus frames based on the MGC approach.
Our experiment starts with the acquisition of a digital image z-stack from a histological sample.This set of z-images are obtained by moving the microscope stage along the optical axis.For this, the axial extension t of the sample is defined and then the axial stage with the sample is moving to cover this extension.The interplanar distance ΔZ between different optical sections is less than the axial resolution of the microscope, defined as the DOF in Table 3. From this table, it is evident that ΔZ is determined by the NA of the objective lens.
Digital images of a beetle shell are acquired with amplification of 10× and interplanar distance ΔZ ¼ 3 μm.The given z-stack is composed of 42 images with 1024 × 768 pixels.Figure 10(a) shows the in-focus image obtained with the software package EDoF plug-in 31 based on a complex wavelets algorithm for EDoF. 17The fusion process takes an average execution time of 52.2 s, whereas the fused image of Fig. 10(b) is based on the proposed MGC fusion method.In this technique, the resulted slice axial position matrix sapðx; yÞ is low-pass filtered using a p × q median filter with p ¼ q ¼ 3; 15; 35.The total execution time is 32.1 s.The 3-D visualization of the resulting infocus image is sketched in Fig. 10(c).Finally, the nonreference image quality metric of Eq. ( 7) is computed for the in-focus image quality assessment.The results are shown in Fig. 10(d).
Another example is the case of an umbilical cord that is imaged at the amplification of 10× and interplanar distance ΔZ ¼ 3 μm.The given z-stack is composed of 39 images with 2584 × 1936 pixels.Again, Fig. 11(a) shows the infocus image obtained with the EDoF plug-in. 17   again low-pass filtered using a p × q median filter with p ¼ q ¼ 3;15; 35.The total execution time is 238.9 s, and the fusion evaluation is shown in Fig. 11(d).As we can see, the proposed fusion method reveals a high-quality image independent of faulty illumination during the image acquisition.

Conclusions
In this research, the MGC operator has been applied to digital color images.This procedure transforms the multichannel information to a grayscale image, which is used for (a) focus measurements during the AF process and (b) for extending the DOF in the framework of digital microscopy applications.
The AF experimental results of this work demonstrate the effectiveness of the MGC method when it is applied to several z-stacks of images.From this point of view, we can conclude that the use of the proposed MGC image increases the performance of currently used passive AF algorithms and produces monotonic FM curves with an only one local maximum η and a similar width α∕β of the focus curve, as shown in Figs. 5, 6, and 8.The test frames have been acquired from two histological samples, which are amplified at the magnifications of 2.5×, 10×, 40×, and 100× (oil immersion).The AF graphs in Fig. 8 that are obtained by the MGC method present similar behaviors even up to a magnification of 100×.Therefore, all the AF algorithms reveal the image slice on z ¼ 0. Contrastingly, as shown in the AP results in Tables 4 and 5, the same AF algorithms in other color spaces only work properly in some cases.As can be seen in the same tables, the mean and the standard deviation of the AF performance for the MGC image are 1 and 0, respectively, for both amplifications.We can conclude that the effectiveness of the AF algorithms depends on several factors (1) the color space selection for doing the numerical computation, (2) the color distribution of the sample under inspection, and (3) the sample magnification.Only in the MGC space does the AF performance tend to be invariant according to these factors.Another remarkable characteristic of the MGC method is that it is computationally simple and inherently parallel.The computational cost of the MGC(RGB) algorithm implemented on a GPU can be reduced by an order of magnitude, for images with 2584 × 1936 pixels, as is shown in Fig. 9.
On the other hand, the fusion scheme Φðx; y; iÞ was implemented on an image z-stack for EDoF.The fused image is composed of the sharp regions provided by the in-focus pixels f sapðx; yÞ of the input data.Our fusion method has been quantitatively and qualitatively compared with the EDoF plug-in, which is widely used in digital microscopy for DOF extension.From a simulated image stack, the resulting image fusion was compared with the corresponding original images using the NMSE, as shown in Fig. 2. Also, a nonreference image quality metric AQI was implemented for image quality assessment.These quantitative evaluations given in Table 2 show that the quality of the resulting fused image Φðx; y; iÞ is better than the fused image given by the EDoF plug-in.
The 3-D visualization of the in-focus images verifies the fusion results.Based on the experimental results of Figs. 10 and 11, the MGC method is an algorithm sufficiently competitive to merge multifocus images.In general, the main advantages of the proposed fusion method based on MGC transformation are that it is computationally simpler, faster, and more efficient than other methods which have been typically used to fuse multifocus information.Additionally, the comparisons in Figs.11(a) and 11(b) show that our method reveals a high-quality image independent of faulty illumination during the image acquisition.

Fig. 1
Fig. 1 Multichannel conversion to grayscale images by means of the MGC operator from (a) defocused and (b) focused color images.The MGC(RGB) and MGC(HSI) images show sharp regions for the case of focused images, unlike the dark uniform intensity distributions for the case of defocused images.As can be seen, the MGC operator is sensitive to variations in intensity but not very sensitive to variations in hue and saturation.

Fig. 2
Fig. 2 DOF extension on GPU.(a) Source color images C z ðx; y; iÞ, (b) modulus of the gradient of color planes g c z ðx; yÞ, (c) fusion rule, and (d) 3-D visualization of an in-focus image.

Fig. 3
Fig. 3 (a) Slice axial position sapðx; yÞ of in-focus pixels ðx; yÞ along the z-axis, (b) postprocessing g sapðx; yÞ matrix by means of a low-pass filter to reach spatial consistency.

Fig. 4
Fig. 4 Digital images of histological tissue sections used to evaluate the performance of AF algorithms.(a) Human carotid and (b) elastic cartilage.The samples are amplified at 2.5×.The all-in-focus images fit within the DOF of the microscope system under use.

Fig. 5
Fig.5Focus measure for a z-stack of images amplified at 2.5×, which have been processed on the MGC (RGB) of color images.

Fig. 6
Fig.6Focus measure for a z-stack of images amplified at 40×, which have been processed on the MGC (HSI) of color images.
,31  It takes an average execution time of 477.43 s.The fused image of Fig.11(b) is based on the proposed MGC fusion method, where the resulted slice axial position matrix sapðx; yÞ is

Fig. 10
Fig. 10 Image fusion results using (a) the software package EDoF plug-in and (b) the MGC fusion method.(c) 3-D visualization of (b).(d) Fusion evaluation of in-focus images.The readjustment of in-focus pixels of g sapðx; yÞ along the z-axis avoid false edge detection.

Fig. 11
Fig. 11 Image fusion result using (a) the software package EDoF plug-in and (b) the MGC fusion method.(c) 3-D visualization of (b).(d) Image fusion evaluation.As it can be seen, the proposed fusion method reveals a high quality image independently of faulty illumination during image acquisition.

Table 2
Image quality assessment of in-focus images.

Table 4
Autofocusing Performance AP of all metrics in different grayscale channels and MGC images.The mean and standard deviation of AP is given in bold.
Fig. 9 Execution time of the MGC(RGB) method in a z-stack of digital images.