Matrix-masking to balance nonuniform illumination in microscopy

: With a perfectly uniform illumination, the amount and concentration of fluorophores in any (biological) sample can be read directly from fluorescence micrographs. However, non-uniform illumination in optical micrographs is a common, yet avoidable artefact, often caused by the setup of the microscope, or by inherent properties caused by the nature of the sample. In this paper, we demonstrate simple matrix-based methods using the common computing environments MATLAB and Python to correct nonuniform illumination, using either a background image or extracting illumination information directly from the sample image, together with subsequent image processing. We compare the processes, algorithms, and results obtained from both MATLAB (commercially available) and Python (freeware). Additionally, we validate our method by evaluating commonly used alternative approaches, demonstrating that the best nonuniform illumination correction can be achieved when a separate background image is available.


Introduction
In most common microscopy configurations, including those used in fluorescence microscopy, the light source is aligned for Koehler illumination [1], which ensures that the structure of the light source (as either rectangles generated by the LEDs or a light bulb's coil) does not introduce optical artefacts in the microscopy pictures. Such illumination provides a clean light with a radial symmetric Gaussian intensity gradient and thus a non-uniform illumination. This uneven illumination can cause significant issues in many fields (for example it can lead to problems in the assessment of dermoscopy images in the clinical sciences [2]).
In the last decades, the correction of uneven illumination (including vignetting) in microscopy has become common, with techniques relying upon a number of techniques, including: the acquisition of additional images to subtract the background; the use of inherent image features by blurring [3]; or by extracting illumination data from the image (e.g. the variational framework for Retinex (VFR) [2,[4][5][6][7]). These techniques often require the use of specific software and/or a deep mathematical understanding. Alternative approaches to background reduction include binarization [8] and spot detection [9] which can remove greyscale details of the original images (leading to micrographs that are unsuitable for detailed analysis or quantification).
Here, we present a simple method for matrix-based balancing of the nonuniform illumination and validate the technique in two common computing environments: the commercially available gold standard -MATLAB and its freeware alternative -Python. To demonstrate the practical application of this technique, we study images acquired using commercial fluorescent and brightfield microscopes. The image depict samples of high optical density [ Fig. 4] or a microfluidic device [10] (originally designed to study the diffusion of fluorophores and single cell motility in laminar flows [10]), [Figs. 2 and 3].

Materials
Raw images and fluorescence micrographs were obtained using an Olympus BX 61 upright microscope (Olympus, Germany) with a Sensicam camera (PCO, Germany) [ Fig. 3(a)]. A Xenon lamp was used as light source with a long pass filter set (λExc = 535 nm, AHF, Germany). Rhodamine B (0.5% aqueous solution) was used as the fluorophore, and images were acquired from within in a previously described microfluidic device [10]. A brief description of this device is shown in the Supporting Information [10]. Conventional bright field micrographs were collected using a Zeiss Observer A1, [Figs. 2(a) and 2(b)].
Image processing was performed with both MATLAB 2014 beta developer trial version (MathWorks, USA) and in Python 3.6 with the modules opencv2 to load the images into Python, Numpy to convert the images into arrays/matrices and process them, as well as the matplotlib.pyplot module to display the results using the Spyder and IPython 6.2.1 upgrade (all freeware). The code and all data used in this article is available open access for download [11] at https://nordlab.med.lu.se/?page_id=34.

Methodology
The image processing was performed using two central steps, namely, the acquisition or generation of background information and then, subsequently use this background data to process sample images. Depending on the available background data of the illumination, we will review four different approaches to balance out nonuniform illumination, as shown in the decision tree below [ Fig. 1].

Acquisition of background images
In any microscopic analysis, it is typically easy to obtain background data. For example, taking a dedicated background image (I) of a blank glass slide (i.e. without any sample). Ideally, the settings for the lamp brightness and camera exposure would be the same as for taking pictures of samples (such as a cell, a particle or bubble). A normalization step (see Table 1) may be performed during image processing to remove differences in overall brightness and the gamma value.   Table 1.
If it is not feasible to obtain a blank background image (e.g. one may be working with historical data sets), then it is possible to emulate (II) by recording multiple (hundreds) images of samples, collected with the same optical setup and then average them using e.g. Fiji (ImageJ) and its built-in feature Z-project (Image > Stack > Z-project) [12].
Similarly, but more cumbersome, images can be imported into bitmap-software like GIMP or Photoshop as layers and then be averaged. Both Matlab and Python could be used as well, by importing all relevant images, converting them to matrices, which can then be averaged. The random distribution of objects throughout the stack of images would ideally cancel out any bright and dark spots on the individual sample images. With the background image acquired, it is possible to balance nonuniform illumination, as described in Table 1, with the result shown in Fig. 2. If, however, it is not possible to acquire a background (i.e. both (I) and (II) are not viable), for example, because the objects on the available sample images are not distributed randomly but always centered, or because there is no background data available, there are options for image processing that remain. The route which is generally recommended for MATLAB is shown as (III) [13], an approach which "blurs" the image to obtain a background image. This method generally works well for small objects that are evenly distributed over an entire image and do not introduce large patches of brighter or darker background.

Image processing
For images, that do not meet these requirements, we present option (IV), as an alternative way to balance nonuniform illumination, by generating a background image de novo (see Table 2), a technique that is especially suitable for fluorescence microscopy. The method works by selecting the brightest line from the fluorescence image [ Fig. 3(a)] and smoothing it into a single vector. This vector represents the brightness ( = the grey value) of each pixel along the brightest lines of the fluorescence image [ Fig. 3(b)]. By multiplying the vector with itself (using the outer product) we generate a matrix which simulates the non-uniform illumination of the fluorescence image [ Fig. 3(c)]. This can be inverted to compute a second matrix, which will cancel out the non-uniform illumination [ Fig. 3(d)]. Simple multiplication of the normalized background with the sample image (see Table 3) provides a result which represents the original image taken with perfectly even and uniform illumination [ Fig. 3(e)].    Fig. 3.
For ease of use we color-coded: loops, comments and filenames. Full code available at [11].
A direct comparison between the MATLAB standard procedure (III) and our "matrixmask" method (IV) shows a significant difference in the quality of balancing out the nonuniform illumination. If the same fluorescence image used for the approach described above (IV) is treated using (III), the results [ Fig. 4] show an improved distribution of grey values all over the image but it introduces artefacts, especially around areas with a large variance of grey values within small areas [ Fig. 4(e)]. We extracted the grey values along the brightest lines in the center (N = 19) and plotted them over their x-position in the image, for both our newly proposed approach IV [ Fig. 3(c)] and the MATLAB standard procedure III [ Fig. 4(f)]. We also calculated the standard deviation of these plotted vectors [Figs. 4(c) and 4(f)] to be 0.31 for the original image (uncorrected), 0.11 for the MATLAB approach (III), and less than 0.01 for our "matrix-mask" approach (IV), which correspond best to the constant concentration of fluorophore within the microfluidic channel.  Fig. 3(e)]), analogous to Fig. 3(b). (d) The image, which results from subtracting the background image Fig. 4(b) from sample image Fig. 4(a). It is also possible to use this new "matrix-mask" approach to process images for which background data is available. The background image [ Fig. 4 (b)] can be used to create a matrix for the illumination, which is typically nonuniform, while the sample image is also converted into a second matrix; each pixel in the images generates one value in the corresponding matrix, whilst the pixel's grey value is represented in the value of the individual data points.

Alternative matrix-based approaches
The presented method proved to be the best and most reliable matrix-based approach using both MATLAB and Python. There are shorter operations that can be carried out in order to balance out uneven illumination, which are intuitively correct, but lead to sub-optimal results (for example, simple pixel-by-pixel subtraction or division of the background image from the sample image, or vice versa). Since these operations will result in mostly black images with poor contrast, a contrast maximization akin to the one presented on the bottom of Table 1 was carried out right afterwards. In Table 4, we have compiled all of these operations, the algorithms needed to perform them in both MATLAB and Python, together with the results obtained from these operations, demonstrating that our Python bundle with numpy and the opencv2 module often automatically performs a contrast maximization, especially after subtractions. Alternative approaches to balance out uneven illumination might feel intuitive yet lead to worse results. Due to different internal handling of the matrices, the same operations can lead to different results for MATLAB and Python, as shown above.

Discussion
The results shown in Fig. 3 demonstrate the effectiveness of this simple matrix-based approach to balance out uneven illumination, and also show, that it is possible to extract the illumination information from a single line of one image. The quality of the obtained illumination matrix can be further improved by averaging over several lines -preferably through the point of highest light intensity -and by additional smoothing to avoid the introduction of artefacts [ Fig. 3]. This approach, however, only works when the sample has the same brightness value along the selected line (e.g. due to the same amount of fluorophore present along the line). In the sample we chose, some artefacts to the left of the channel result in additional reflection and thus a non-normal distribution in the extracted vector [ Fig. 3(b)] and ultimately the illumination mask [ Fig. 3(d)]. The ideal solution for balancing out uneven illumination in micrographs is nonetheless, first taking a dedicated, sample free background image with the same optical parameters and subsequently performing the recipe shown in Table 1 [see also Fig. 2]. Nonetheless, our approach yielded better results than the goldstandard MATLAB procedure [ Fig. 4], and had so few artefacts that the grey value of the image could be taken to visualize the concentration of fluorophore within the device (see supplementary data of [10]). Additionally, other matrix-based approaches have been tested and shown to yield worse results in regards of balancing out uneven illumination, as shown in Table 4. For example, simply subtracting the background image from the sample image (see Table 4, "BAD1"), or vice versa (see Table 3, "BAD2") leads to images with poor contrast, loss of details and introducing artefacts. In the results obtained from Python, artefacts were found where the sample image was brighter than the background image, in unexpected switching of black and white.
Both, commercial MATLAB and freeware Python, offer solutions that ultimately lead to comparable results. Differences in the resulting images (see Table 4) can be explained by the different algorithms MATLAB and Python use in their cores for handling the matrices, which represent the images. As an example, there are salient spots in the first two right-hand side images in Table 4, areas of bright white surrounded by darkness and areas of deep black in a generally bright area. These black/white inversions could stem from misinterpreting negative numbers or a data compression step, where the matrices are no longer handled point-by-point but segmented to safe calculation space.
With the correct coding, however, both programs can be used to greatly improve image quality of micrographs, be it with dedicated background data, or by extracting illumination information from the sample images themselves and applying our matrix-mask method. This opens up many possibilities, including fast and easy balancing of sample micrographs illumination, or of using the grey value of a fluorescent image to measure the concentration of a fluorophore at any position in fluorescence micrographs.

Appendix A.
MATLAB processing results, for comparison with Figs. 2 and 3.