Flexible focus function consisting of convex function and image enhancement filter

Abstract: We propose a new focus function Λ that, like many of the existing focus functions, consists of a convex function and an image enhancement filter. Λ is rather flexible because for any convex function and image enhancement filter, it is a focus function. We proved that Λ is a focus function using a model and Jensen’s inequality. Furthermore, we generated random Λs and experimentally applied them to simulated and real blurred images, finding that 98% and 99% of the random Λs, respectively, have a maximum value at the best-focused image and most of them decrease as the defocus increases. We also applied random Λs to motion-blurred images, blurred images in different-sized windows, and blurred images with different types of noise. We found that Λ can be applied to motion blur and is robust to different-sized windows and different noise types.


Introduction
A focus function is always modeled as a function that has a maximum value at the bestfocused image and decreases as the defocus increases [1].Efforts are under way to develop focus functions relevant to widely used applications, such as automated microscopy, holography, deblurring, image refocusing, iris recognition, and depth of focus estimation [2].In the past 40 years, more than 30 focus functions have been proposed.Some of them have the same principles or similar patterns, and some have the same formulae but different parameters.Thus, researchers have presented several categories of focus functions.
A large category of focus functions is derivative-based functions, which assume that wellfocused images have more high-frequency content than defocused images [3,4], and apply image enhancement filters to extract the high-frequency content.(In this paper, an image enhancement filter is a convolution mask that uses smoothing or sharpening of the image contrast to make an image more useful for analysis.)Brenner et al. constructed a focus function by summing the squares of the second-order differences of the image [5].Groen et al. presented three categories of focus functions and applied methods to sum the normalized variation of the image [6].Krotov convolved the image by a Sobel filter and then summed the squares of the gradients of the image [7].Subbarao et al. convolved the image by a Laplace filter and then summed the squares of the gradients of the image [8].Nayar et al. convolved the image by a Laplacian filter and then used the l 1 -norm of the gradient to construct the focus function [9].Santos et al. presented several focus functions by summing the difference of the image higher than a threshold, summing the squares of the differences of the image higher than a threshold, and summing the squares of the image values higher than a threshold [10].Daugman proposed a sharpening filter and summed the total high-frequency power of the 2D Fourier spectrum of the image [11].Kang et al. proposed a sharpening filter smaller than Daugman's and argued that the filter contains more high-frequency power [12].Langehanenberg et al. used the determination of the logarithmically weighted cumulated Fourier spectra to determine the image sharpness of the reconstructed amplitude distributions [13].
A small category of focus functions is mid-frequency-based functions.Lee et al. detected objects distributed in the image using a mid-frequency discrete cosine transform (DCT) focus measure and then selected the target object through fuzzy reasoning [14].Feng et al. presented an image clarity evaluation function based on the center blocking DCT method [15].Wang et al. used combination filters to strengthen the image medium-frequency information in order to construct focus functions [16].Another small category of focus functions is those based on statistics, in which focus functions are constructed using variance and correlation.Mendelsohn et al. summed the image values higher than a threshold [17].Groen counted the number of pixels with values higher than a threshold [6].Another small category of focus functions is histogram-based functions.Firestone et al. argued that a focused image contains more information than a defocused image and used the image entropy [18].
There are also focus functions that cannot be classified into the above categories.For example, Brázdilová et al. selected only areas of interest within an image and proposed further enhancements of existing methods [19].Hamm et al. proposed a content-based focus searching method that uses a priori information on the observed objects by employing local object features and boosted learning [20].Tsai et al. proposed a focus function that transforms the focus profile to the reciprocal domain [21].Xu et al. proposed a focus detection criterion based on the phase contrast [22].Ferraro et al. used an approximation of the Tamura coefficient to construct a focus function [23].Gao et al. used the difference between the normalized amplitudes of red and green light [24].
Categorizing focus functions helps people understand how a focus function works.In addition, focus functions in one category always have one or more parameters, which enable researchers to construct new focus functions by tuning the parameters on the basis of the known focus functions.E.g., Elozory et al. proposed an indicator function that signifies whether the current pixel location is high contrast, a function that uses the binary median filter, and a function that returns the thresholded absolute gradient contrast strength in both the horizontal and vertical direction for a given pixel location [25].
However, no study has presented a focus function that both summarizes many known focus functions and has flexible and adjustable parameters.In this paper, we summarize 13 known focus functions [5][6][7][8][9][10][11][12][13] to produce the focus function Λ, which has two flexible and adjustable parameters: a convex function, and an image enhancement filter.For any given convex function and image enhancement filter, Λ is a focus function.
The proposed focus function Λ was first discovered using the observation that many of the known focus functions consist of a convex function and an image enhancement filter, as shown in Eq. ( 1), despite their differences.
( ( , )* ) , I x y g dxdy ϕ Λ =  (1) where I denotes the image, (x,y) denotes the coordinates of the image pixels, φ is a convex function, and g is an image enhancement filter.
A close observation of the focus functions that can be summarized by Eq. ( 1) yields two conclusions.First, the convex function φ can be a strictly convex function, such as the quadratic function x ϕ = when x ≥ 0).Further, the choice of the image enhancement filter g is rather arbitrary; it can be a one-dimensional image enhancement filter such as [ ] or a two-dimensional image enhancement filter such as . Most image enhancement filters are sharpening filters, such as the Laplacian filter; however, we found that smoothing filters, for example, the Gaussian filter, also work.
The observations above imply that the construction of the convex functions and image enhancement filters is rather free.Thus, we wondered whether Eq. ( 1) succeeds for any convex function and image enhancement filter.
If the answer is yes, we would have by far the most flexible focus function, because the construction of the convex function and image enhancement filter is rather arbitrary.Furthermore, the new function will deepen our understanding of how focus functions work because so many known focus functions can be summarized by Eq. ( 1).
This paper is arranged as follows.First, we propose the new focus function Λ.Second, we create a model of the defocus problem.Then, using the model, we give a brief proof that Λ is a focus function for any convex function and image enhancement filter.We must note that the proof was based on the model; however, the result might also works in situations that do not fit the model.Finally, we constructed random Λs and experimentally applied them to both out-of-focus blurred images and motion-blurred images.

The proposed Λ
We found that 13 known focus functions can be expressed by Eq. ( 1), as shown in Table 1.
Furthermore, we noticed that the construction of the convex functions and the image enhancement filters are rather free, as shown in Fig. 1.Therefore, we proposed that for any convex function and image enhancement filter, Λ is a focus function.The major challenge of proving that Λ is a focus function lies in how to model the defocus problems such that we can discuss it mathematically.


where S x and S y are Sobel operators.
where P(v) is the probability of a pixel with intensity v. )

Modeling the defocus problem
In optical systems, an out-of-focus image is always described as the real image convolved by a point spread function (PSF).In polar coordinates, a diffraction-limited PSF has the form of a Bessel function of the first kind.For a wavelength λ with an f/# of F focusing radiation, it is which can be approximated by the Gaussian function [26], as shown in Fig. 2. In addition, the Gaussian function is also always used to fit the modulation transfer function (MTF) [27][28][29][30].This supports the idea that the PSF can be approximated by the Gaussian function, because the MTF and PSF are connected by a Fourier transform, and the Fourier transform of a Gaussian function is also a Gaussian function.Thus, in image processing, we can obtain a blurred image by convolving a less-blurred image using a Gaussian filter.This is a blurring model, However, in practice, this model is too simple.The Gaussian filter and PSF are both Dispersion Functions (in this paper, a Dispersion Function is defined as a real function with two properties: first, it is non-negative everywhere; second, its integral is equal to 1; this is shown in Eqs. ( 14) and (15) in the appendix).Therefore, we extended the model by modeling the blurred image as the less-blurred image convolved by a Dispersion Function, as shown in Eq. (3).where I 0 is the best focused image, I 1 is the less blurred image, I 2 is the more blurred image, PSF 1 is the PSF of I 1 , PSF 2 is the PSF of I 2 , and G is a Dispersion Function.Further, when an image is convolved by an image enhancement filter, the input image values outside the bounds of the image are computed by implicitly assuming that the input image is periodic.

A brief proof
The proof of "Λ is a focus function" is equivalent to the proof of "Λ has a maximum value at the best-focused image and decreases as the defocus increases."

Statements
In the proof, we will need the Dispersion Inequality [an inequality we proposed that describes the inequality in the dispersing process, as shown in Eq. ( 16) in the appendix].

Continuous form of the problem
Continuous form of the problem: Consider I 1 and I 2 to be images of the same object plane, where I 2 is more defocused than I 1 .Let φ be a convex function in R 2 and g be an arbitrary function in R 2 ; then we have Here, R 2 denotes two-dimensional vector space, and * is the convolution operator.

Proof
We start from the left side of Eq. ( 4).According to Eq. ( 3), we have where G is a Dispersion Function.On the other hand, according to the two-dimensional form of Eq. ( 25) in the appendix, we have where G is a Dispersion Function.From Eqs. ( 5) and ( 6) we obtain Eq. ( 4). The proof is briefly illustrated in Fig. 3.

Discussion
The inequality might be the consequence of light energy dispersion during defocusing.
In real applications, images always have a finite height and width, in which condition Eq. ( 4) takes the form of where Ω is the image window.
The proof was based on a model, however, the result might also work in situations that do not fit the model, such as holography, which is a coherent imaging problem.Despite this limitation, the proof does suggest a focus function that always works under most situations.To study the performance of Λ in real situations, Λ was experimentally applied to blurred images.

Experimental results
The experiments were conducted in two parts.First, the experimental data, which contain the random Λs and the blurred images, were prepared.Then, we experimentally applied the constructed random Λs to the blurred images.

Preparation of experimental data
We began by constructing random Λs by generating random convex functions and random image enhancement filters.
First, random convex functions were constructed.In this paper, the convex function was modeled with a real polynomial in one variable.Because the second derivative of a polynomial is also a polynomial, and the second derivative of a convex function is nonnegative everywhere, the problem of constructing a convex polynomial was reduced to the easier problem of constructing a positive polynomial.Furthermore, Positive polynomials have the property that "Every real polynomial in one variable is non-negative on R if and only if it is a sum of two squares of real polynomials in one variable" [31], which enables us to model a positive polynomial as 2 2 , where p m represents an mth-order polynomial, and p n represents an nth-order polynomial.Integrating Eq. ( 8) twice, we obtain the convex function m n p p +  (9) In this paper, we set m = 3, n = 2; thus, the convex function was an eighth-order polynomial.
Random convex functions can be constructed by randomizing the parameters in Eq. ( 9).In this paper, the gray values were normalized in the range of [0,1].Consequently, to obtain convex functions significant in the range of [-1,1], the parameters were randomized in the range of [-1,1].Examples of the obtained random convex functions are shown in Fig. 4. Next, the random image enhancement filters g were constructed.They were constructed as 5 × 5 matrices, and the values of each matrix were randomized in the range of [-0.5, 0.5].There are two types of image filters in the known focus functions.One type is smoothing filters, such as Gaussian filters, in which the sum of all the elements is equal to 1; the other type is sharpening filters, such as a Laplacian filter, in which the sum of all the elements is equal to 0. Here, the random filters g act as sharpening filters.Thus, the random filters g were normalized such that the sum of all the elements is equal to 0. The normalization was carried out by moving g vertically along the y-axis such that the sum of all the elements is equal to 0. Examples of the random image enhancement filters are displayed in Fig. 5.The simulated out-of-focus blurred images were constructed by convolving the focused images (as shown in Fig. 6) by the simulated PSFs.Four types of images were selected: a portrait (the Lena image), an object, an outdoor landscape, and a microscopic image.The PSFs were simulated using PSF Lab, which simulates the illumination PSF of a confocal microscope under various imaging conditions [32].Most of the parameters in PSF Lab were set to the defaults except for the depth parameter, which denotes the distance from the microscope objective to the coverslip/sample interface.Adjusting the depth from  The real out-of-focus blurred images were captured by a microscope (XSP-06-1600).Here, a 5 × eyepiece and a 100 × objective were selected to observe a printed paper.The focal length was adjusted manually such that the object plane crossed the focal plane from the downside to the upside.Seventeen blurred images were captured and were indexed from 1 to 17 from the lowest to the highest object plane; examples are shown in Fig. 9.The blurred images were best focused at index 10, and the defocus increased as the index increased or decreased.

Experiments
The experiments were conducted as follows.
First, the constructed Λs were applied to the simulated out-of-focus blurred images.For convenience in comparing the performances of the Λs, their values were normalized in the range of [-1,1].For each Λ curve, normalization was performed in two steps.First, the curve was moved vertically along the y axis such that 1 '( ) 0 I Λ = ; then, Λ′ was normalized using Eq. ( 13).
'( ) '( ) , max( ( '( ))) where I denotes the blurred image, i denotes the index of the blurred image, and norm indicates normalization.The experimental results are displayed in Fig. 15, where the x axis denotes the image index, and the y axis denotes the Λ value.About 98% of the constructed Λs reach the maximum value at the best-focused image and most of them decrease as the defocus increases, which supports our proof in the previous section.Second, the constructed Λs were applied to the real out-of-focus blurred images.The focus value was calculated for the entire image.For convenient comparison, the same normalization method was used.The experimental results are displayed in Fig. 16.About 99% of the constructed Λs reach the maximum value at the best-focused image and almost all of them decrease as the defocus increases, which confirms that the focus function Λ can be used in real applications.From Fig. 15 and Fig. 16, we can see that about 2% of the constructed Λs do not work well as focus functions.This is because the proof was constructed using a model, whereas the real situation is far more complicated.First, when an image is convolved by an image enhancement filter, input image values outside the bounds of the image are used, which causes errors.Second, real objects are three-dimensional rather than two-dimensional, so the image plane is not an ideal plane, which also causes errors.Third, errors result from noise in the optical system.The experimental results indicated that the best focus function varies with the application.In fact, researchers have been comparing and choosing focus functions for specific applications for years, and to date, the best focus functions for different applications have always varied.
Third, the constructed focus functions Λ were applied to the motion-blurred images.As we know, focus functions are widely used in many applications; therefore, we proposed that Λ might also work on motion blur.For convenient comparison, the same normalization was used.The experimental results are displayed in Fig. 17.About 82% of the constructed Λs reach maximum values for the image without motion blurring and decrease as the motion blur increases, which confirms that the proposed focus function Λ works on not only out-of-focus blur, but also motion blur.Then, The constructed Λs were applied to the simulated out-of-focus blurred images in different-sized windows.For convenient comparison, the same normalization was used.The experimental results are displayed in Fig. 18.About 99.9% of the constructed Λs reach the maximum value at the best-focused image, and most of them decrease as the defocus increases, which confirms that the focus function Λ can be used when window is applied.
From Figs. 15 and 18, we can see that the Λs perform better for larger images or windows.That might be the result of the scale-space effect [34].When an image is convolved by an image enhancement filter, input image values outside the bounds of the image are used, which causes errors, and for a fixed-size PSF, the smaller the image is, the larger the errors are.
Finally, the constructed Λs were applied to the simulated out-of-focus blurred images with different noise.For convenient comparison, the same normalization was used.The experimental results are displayed in Fig. 19.About 98% of the constructed Λs reach the maximum value at the best-focused image, and most of them decrease as the defocus increases, which confirms that the focus function Λ is robust to Poisson and Gaussian noise, such as photon noise and thermal noise.

Conclusions
The known focus functions are summarized to create the flexible focus function Λ, which consists of a convex function and an image enhancement filter.Λ is rather flexible because for any convex function and any image enhancement filter, it is a focus function.A proof was also proposed using a model and Jensen's inequality.It should be noted that the proof is built on a Gaussian or Gaussian-alike PSF assumption, which may be incompatible with some imaging systems.Nevertheless, the proposed focus function is still acceptable in wide variety of applications.A method of constructing random Λs was developed, and experimental results for both simulated and real blurred images proved its feasibility.The developed technology can be especially helpful in practical applications such as microscopy, holography, depth of focus estimation, deblurring, image refocusing, and iris recognition because it provides significant convenience in constructing focus functions.Future studies will address questions such as how to construct Λ for a specific application.Another question of interest is how to construct the convex functions by adjusting the parameters.Furthermore, studies on how the criteria for good focus functions can be improved would also be of remarkable value.

   
Obviously, σ is a subset of p. Replacing p with σ in Eq. ( 20), we get From Eq. ( 21), we have From the right side of Eq. ( 22), we have  17), ( 22) and ( 23), we obtain Eq. ( 16). Discussion Special attention is called to the fact that if the convolution operator * in the Dispersion Inequality is replaced with the cross-correlation operator, the inequality still holds.The inequality also holds on higher dimensions.
Finally, we introduce two characteristics of the Dispersion Inequality that are applied directly to prove that Λ is a focus function in this article:

Transitivity
Let σ 1 and σ 2 be Dispersion Functions; then we have 2. Pre-proceeding invariance Let g be an integrable function in R 1 , then we have The proof is ignored.

Appendix 2
The finite form of Λ

Statements
In section 4, we proved the continuous form of Λ.However, the situation in real optical imaging systems is always discrete, where the images and image enhancement filters are matrices.In this section, a proof of the finite form of Λ is presented.First, we start from a lemma.

Lemma
The simplified finite form of Λ: Consider I 1 and I 2 to be images of the same object plane, where I 2 is more defocused than I 1 , I 1 and I 2 to be m × n matrices, and φ to be a convex function in R 2 ; then we have , , m n m n i j i j I i j I i j

Proof
According to Eq. ( 3), I 2 is equal to I 1 convolved by a Dispersion Function G that is a p × q matrix, which indicates that from the left side of Eq. ( 26), we have where "mod" denotes modulus.
On the other hand, G is a Dispersion Function; thus, we have the following two formulas: where "mod" denotes modulus.
Jensen's inequality states that for a real convex function φ, real numbers x 1 , …, x n , and positive weights a j , we have ( ) ( ) .
From Eqs. ( 28), (29), and ( 31) and the matrix of I 1 , we have , , p q p q r s r s G r s I i r j s G r s I i r j s From Eqs. ( 27), ( 30) and ( 33), we have Eq.(26). Now we prove the finite form of Λ.

The finite form of Λ
The finite form of Λ: Consider that I 1 and I 2 are images of the same object plane, where I 2 is more defocused than I 1 ; I 1 and I 2 are m × n matrices, φ is a convex function in R 2 , g is a p × q image enhancement filter, and The proof is ignored.

2 x
ϕ = , or a non-strictly convex function, such as the absolute value function x ϕ = .Second, it can be an axisymmetric function, such as the quadratic function 2 x ϕ = , or a non-axisymmetric function, such as the threshold absolute value function (e.g., 0 ϕ = when x < 0 and

Fig. 1 .
Fig. 1.Curves of convex functions φ in Table 1.Functions a, b, c, d, and e are convex; f and g are approximately convex.Function h is reversed convex because defocusing an image always shrinks the histogram, which reverses the smoothing process.

Fig. 9 .
Fig. 9. Examples of real out-of-focus blurred images captured by a microscope.The images were best focused at index 10, and the defocus increased as the indexes increased or decreased.Motion blur was simulated by a filter to approximate rapid linear motion of a camera along an angle of 45 degrees with blurring degrees of 0, 2, 4, …, 16 pixels.Nine PSFs were constructed; examples are shown in Fig. 10.Then, by convolving the original image Lena by the PSFs as shown in Fig. 6, the motion-blurred images were obtained; examples are shown in Fig. 11.The images were indexed from 1 to 9 with increasing blurring degree.The blurred images were the least motion-blurred at index 1, and the blurring degree increased as the index increased.

Fig. 15 .
Fig.15.Experimental results for 196 random Λs applied to simulated out-of-focus blurred images.The blurred images were best focused at index 8, and the defocus increased as the index increased or decreased.

Fig. 16 .
Fig.16.Experimental results for 196 constructed Λs applied to real out-of-focus blurred images.The images were best focused at index 10, and the defocus increased as the index increased or decreased.

Fig. 17 .
Fig.17.Experimental results for 196 random Λs applied to motion-blurred images.The images were the least motion blurred at index 1, and the blurring degree increased as the index increased.

Fig. 18 .
Fig.18.Experimental results for 196 constructed Λs applied to simulated out-of-focus blurred images in different-sized windows.The images were best focused at index 8, and the defocus increased as the index increased or decreased.

Fig. 19 .
Fig.19.Experimental results for 196 constructed Λs applied to simulated out-of-focus blurred images with different types of noise.The images were best focused at index 8, and the defocus increased as the index increased or decreased.