Underwater image enhancement via extended multi-scale Retinex

Underwater exploration has become an active research area over the past few decades. The image enhancement is one of the challenges for those computer vision based underwater researches because of the degradation of the images in the underwater environment. The scattering and absorption are the main causes in the underwater environment to make the images decrease their visibility, for example, blurry, low contrast, and reducing visual ranges. To tackle aforementioned problems, this paper presents a novel method for underwater image enhancement inspired by the Retinex framework, which simulates the human visual system. The term Retinex is created by the combinations of “Retina” and “Cortex”. The proposed method, namely LAB-MSR, is achieved by modifying the original Retinex algorithm. It utilizes the combination of the bilateral ﬁlter and trilateral ﬁlter on the three channels of the image in CIELAB color space according to the characteristics of each channel. With real world data, experiments are carried out to demonstrate both the degradation characteristics of the underwater images in different turbidities, and the competitive performance of the proposed method.


Introduction
Underwater exploration has become more active in recent years with the increasing application demands, such as the studies of the marine species [1] , wreckage exploration, inspection of the underwater cables and pipelines, underwater scene analysis, search and rescue, and mapping of the offshore seabed among others. There is also a strong interest in applying computer vision based algorithms to these research works. However, due to the scattering and absorption effect in the underwater environment, visual images often suffer from low visibilities, such as low contrast, blur, and color variations [2] . This is a real challenge for computer vision based underwater tasks, which require detailed information about the image for further operations, such as feature extraction.
The light rays suffer from the scattering and absorption effects in the underwater environment. The scattering effect is brought by the suspended particles in the water that reflect the light rays into other directions, which makes the image blurry. The absorption is caused by the medium of water that degrades the energies of light rays according to their wavelengths, which makes the image visually losing its contrast and reduces the visible ranges.
The absorption depends on the density and turbidity of the water. Studies have been conducted into this field, but most of them require a dedicated underwater imaging device. Although there are some computer vision based post-processing methods for underwater image enhancement, the performances are still limited. To tackle aforementioned problems, this paper presents a novel method that can process the underwater image to make it less affected by the underwater environment. The processed image has a clearer visibility and a higher dynamic range. The visual range is also improved. These factors are important for other computer vision tasks, which require observing underwater scenes. Two sets of the experiments included in this paper demonstrate the competitive performance of the proposed method.

Related work
Many methods have been proposed during recent years to de-noise and enhance the underwater images. These methods include the ones based on the deliberately designed hardware and the ones that are built on the computer vision algorithms. Schechner and Karpel [ 3 , 4 ] designed a dedicated hardware for underwater image enhancement. It utilized a polarized filter for the lens of the visual imaging device in the underwater environment. The scattering components in the medium of water are mostly associated with partial polarization of light. Since the main illumination in the ocean is the sun from above, a carefully placed polarized filter with a certain angle can remove most of the scattering components before they reach the lens. However, their method required to be prepared before the underwater images being taken, which leads to some limitations in certain applications. Iqbal et al. [5] proposed an unsupervised color correction method (UCM) to deal with the reduced contrast and non-uniform color cast in the underwater images brought by the scattering and absorption effects underwater. They utilized two sets of different histogram stretching strategies for Red color channel and Blue color channel respectively according to the characters of the underwater environment. And the HSI color model was also employed at the end of the process for color correction. However, their method was based upon the assumption that the color cast in the underwater environment was only dominated by a specific color spectrum, which was blue. They deliberately compressed the blue channel of the image, which might damage the perceptions of the original blue components in some scenes with different color casts.
The method presented in this paper is inspired by the framework of the Retinex [6] . The Retinex theory simulates the mechanism of the human vision system that perceives the world. The term of Retinex is created by the combination of the "retina" and "cortex". It attempts to achieve the color constancy when the scene is dominated by a certain illumination, which has a similar situation in the underwater environment. In the human visual neural system, the color constancy is fulfilled by both retinal and cortical mechanisms to discount changes in illuminations [7] . Some researchers believe that the local contrasts between adjacent cones contribute significantly to the color constancy [8] . Similarly, the traditional Retinex applies convolution process on local windows using Gaussian Filter to approximate the illumination component. The methods based on Retinex have been applied to the areas such as color image enhancement [9] , foggy image filtering [10] , and aero image process among others. Hurlbert [ 11 , 12 ] studied the properties of the Retinex framework and other lightness theories. He treated them as a learning problem for artificial neural networks, and found that the solution had a center/surround spatial form [13] . Wang et al. [14] employed the Retinex framework on the Y channel of the image in YCbCr color space for image enhancement. Jung et al. [15] applied Retinex for illumination effects removal for eye detection under varying lighting conditions. There are also some researchers employing the Retinex to enhance the underwater images. Joshi and Kamathe [16] conducted the experiments with Retinex to enhance the images degraded by weathers, which has the similar properties as the ones degraded by underwater environment. They applied the method of the traditional multi-scale Retinex with color restoration to the images in the weather conditions such as haze, poor light, rain, smoke, and as well as the underwater scene. In their experiments, the underwater image quality was improved. However, the enhancement was still limited compared to the results for the images that were captured in the air since that the medium of water did more damages to the image visibilities than the mediums in the air did. SM and Supriya [17] presented a single-scale Retinex based method for underwater image enhancement. They converted the image to YCbCr color space and used the Gaussian surround function for the process of convolution in the Retinex framework. The experiment results were positive. However, the visibilities of the processed underwater images were still polluted by the medium of water. Fu et al. [18] also presented a Retinex based enhancement approach for the single underwater image. They transferred the underwater image from RGB color space to LAB color space and applied Retinex only on L component. They utilized the Retinex theory to decompose the L channel of the underwater image for reflectance and illumination. However, their method involved 4-6 iterations of the process, and took several seconds to process one image. The method proposed in this paper extends the Retinex framework for underwater image enhancement. The bilateral filter and trilateral filter are utilized for different color channels of the underwater images to process the pixels according to different constraints. These two filters hold advantages over the classic Gaussian Filter employed in the traditional Retinex when applied on underwater images.
The rest of the paper is organized as following: Section 3 discusses the characters of the underwater image degradation; the proposed method is introduced in Section 4 ; the experiments that demonstrate both the visual degradation of the underwater image and the encouraging performance of the proposed method are shown in Section 5 ; the paper is then concluded in Section 6 .

The characteristics of the underwater image degradation and problem formulation
The visibility of the image is limited in the underwater environment. The colors in the image can be washed out in a distance about 12 m in clear water, and about 5 m or less in turbid water [19] . The higher the turbidity of the water is, the more the underwater images are attenuated. The main causes of this attenuation are the scattering and absorbing properties of the underwater environment. During the imaging processes, the scattering effect changes the directions of the light rays, while absorption effect reduces the energy of the light ray with respect to different wavelengths. The scattering effect is caused by the floating particles in the water that reflect the light rays into other directions. It follows the behaviors that are described by the point spread function (PSF) [ 20 , 21 ]. There are two parts for scattering effect, the forwardscattering and the back-scattering. The forward-scattering reflects the original light rays of the scene into multiple directions, which make the scene seem to be blurry. The back-scattering reflects the ambient light into multiple directions, and parts of them can go directly toward the imaging device. This interferes with the sensations of the original light signals from the scene, and casts the ambient illumination on the whole underwater scene. The underwater irradiance of the light ray can be expressed as following: where E (0) is the original irradiance of a target point in the water, E ( r ) is the observed irradiance of the target point with the distance of r from it. And c is the attenuation coefficient of the medium, which is affected by the scattering and absorbing effect of the medium together. Thus, Eq. (1) can be further expressed as below, where a and b are the absorbing coefficient and scattering coefficient respectively.
Different regions of the underwater image may have different depths, which refer to different r values in Eq. (2) . This leads to different levels of degradation in different regions of one underwater image. Moreover, the coefficient b is also spatial sensitive. This is because that the scattering effect is caused by the suspended particles in the water, and there is a high probability that the distribution of these particles is not spatially even in the field of view (FOV) of the image. Therefore, the degradation of the underwater image is not evenly distributed across the image plane spatially, as shown in Fig. 1 .
Additionally, the absorption effect in the underwater environment, which refers to the coefficient a in the Eq. (2) , varies with respect to the different wavelengths of the light rays. As pointed out by Sahu et al. [22] , Li et al. [23] and Li et al. [24] , the light rays with the wavelength below around 380 nm or above around 620 nm are more likely to be affected by underwater absorption effect. Fig. 2 shows the relationship between the absorption coefficient and the wavelength [25] . Thus, the attenuation of the  underwater image varies not only in spatial domain, but also in color range domain.
As a result, we find out that the underwater image degradation following uneven distributions in multiple domains. To tackle this problem, this paper presents a novel method for underwater image enhancement that is built on the framework of the biologically inspired Retinex. The proposed method utilize the multiple filters to preserve the original features in multiple domains during the enhancement process, which demonstrates the competitive results in the experiments with real world data.

Retinex applied on underwater images
Many approaches have been proposed for underwater image enhancement, as described in Section 2 . The methods are based on Retinex, which are biologically inspired ones that utilize the color constancy theory. In the human vision system, the color constancy is automatically applied to enable humans to perceive the world in different illumination conditions [16] . The first theory on color constancy is the Retinex presented by Land [6] . Recently, an algorithm named multi-scale Retinex with color restoration (MSRCR) has emerged [26] to make the algorithm more reliable. A correction step is added for each color channel to suppress the desaturation of the image processed by multi-scale Retinex (MSR) [13] . The proposed method follows the framework of the Retinex that estimates the illumination component from the observed image.
According to the lightness theory [27] , the observed image can be decomposed into two components, which are the luminance and the reflectance, as shown in Eq. (3) .
where S ( x, y ) is the observed pixel in the image at the location of ( x, y ). L ( x, y ) and R ( x, y ) denote the components of luminance and reflectance respectively. L ( x, y ) varies according to the differ- Eq. (4) demonstrates the process of MSR with one Gaussian Kernel by one weight for each iteration. Differing from the Single-Scale Retinex (SSR) [28] , the multiple scales can handle the different details at multiple levels in the image. Experimentally, three scales are appropriate enough to represent a small scale, a large scale and an intermediate scale respectively [13] . And generally three equal weights are used for these three scales.
For multi-channel images, RGB images for example, MSR is performed for each channel respectively. Then a color correction step is employed to make the color tone closer to the original one as much as possible, as shown in Eq. (5) : where C i is the coefficient for each channel. It stands for the proportions of three channels within an image. It is better to adjust the coefficient C i after applying the MSR to the image to make the color tone approximate to the original one. The function of Merge () stands for the process of merging three single-channel images into a multi-channel one. As the Retinex algorithm transfers the image Many researchers are trying to apply Retinex theory to the underwater image enhancement or restoration. For example, Jobson et al . [29] discussed the applications of MSR based Visual Servo on the underwater image with moderate turbidity, and the enhanced image was improved about 470% than the unenhanced image with VCM Aggregate Score [30] . As described in Section 3 , the method built on Retinex is suitable for underwater enhancement since the process of estimating the luminance in Retinex is also spatial various. The convolution using small scale Gaussian Filter on the image plane only considers the nearby intensities, which ignores the intensities far from the Kernel center that might have different attenuation situations. However, the method in the form of the original Retinex only takes into account the spatial difference with a Gaussian distribution, excluding the degradation various in color range domain, which is also an important part of the characters for underwater image attenuation as discussed in Section 3 . Thus, if a local region in the image has a wider spectrum of color, then the Retinex will only process this region with the Gaussian average of the colors within the filter region. Some colors may drift far from the original ones after the process, which might be unacceptable.
Another aspect of the improvement that is required for the Retinex is the Halo Artifacts around the edges in the image [31] . For example, as shown in the right column of the Fig. 3 , the halo artifacts can be observed clearly around the edges between the diver's body and the water in the processed image. The intensity contrast is relatively high near these edges, which leads to larger gradients. This is caused by the Gaussian Filter applied on the edges that makes brighter side of the edge (the water) overenhanced and darker side of the edge (the diver's body) insufficiently enhanced. The over-enhanced area is called positive halo and the insufficiently enhanced area is called negative halo.

Filter combination based LAB-MSR for underwater image
In this paper, we present a novel method for underwater image enhancement by proposing the LAB-MSR algorithm. It is inspired by the original MSR framework and processes the underwater image in the CIELAB color space. As discussed in the Section 3 , the degradations of the underwater image vary in multiple domains, while traditional Retinex with the Gaussian Filter only considers such variations in different spatial regions. Therefore, the proposed method utilizes a combination of bilateral filter and trilateral filter according to different color channels to process the underwater image. It takes into account the uneven distribution of the underwater image degradation in multiple domains.
The workflow of the proposed method is demonstrated in Fig. 4 , which can be divided into three main parts: (a) the pre-processing in the left, (b) the multi-channel enhancement with a combination of filters in the middle, and (c) the post-processing in the right. The whole process of the method consists of the following steps: (1) Calculate the original proportions of all three channels of the underwater image in RGB color space. (2) Convert the underwater image from RGB color space to CIELAB color space. (3) As described in Eq. (7) , apply convolutions with bilateral filter to the L channel of the underwater image to estimate and remove the luminance component in this channel of the image. (4) As described in Eq. (8) , apply convolutions with Trilateral Filter to the A and the B channels of the image respectively to estimate and remove the luminance components in these two channels of the image. (5) Convert the underwater image back from CIELAB to RGB color space. Then adjust the proportions of three channels to make them close to the original ones, as described in Eq. (5) .
The CIELAB color space uses a three-dimensional lookup table for color definitions. The three dimensions are L, A and B channel respectively. L channel is related to the lightness. A and B channels are the coordinates of two axes for a predefined chromatic table. The color encoding mechanism for CIELAB color space is similar to the human visual system, which also consists of one luminance channel and two chromatic channels [ 32 , 33 ]. In the human visual neural system, the color information are conveyed from eye to the brain by three channels  color space can better describe how human perceive a color. Moreover, the shadow gradients seem to be stronger in the L channel [34] . Research has shown that the colors appear to have a bluer shade in shadows [33] . This is coincidently similar with the underwater environment, where the surrounding seems to be in a bluish tone due to the absorption and scattering effects. Therefore, in the proposed method, the underwater images are converted from RGB color space to LAB color space for better analysis.
The bilateral filter and the trilateral filter are introduced as the edge-preserving filters, which are both treated as an alternative of the classic Gaussian Filter in some areas. The bilateral filter, which was first presented by Tomasi and Manduchi [35] , considers the uneven distribution not only in the spatial domain, but also in the intensity values. Therefore, the bilateral filter is included in the proposed method for illumination estimation. This process is expressed as Eq. (7) .
where F( σ i ) shares the same function as in Eq. (4) , which stands for the Gaussian Distribution in a spatial domain. H( σ ' i ) denotes the Gaussian Distribution in the domain of intensity values within the filter range. The bilateral filter has the advantages over the classic Gaussian Filter when applied in Retinex. It can alleviate the over-smooth of the edges in the enhanced images to a certain degree according to the similarity of the intensity values.
Following some drawbacks of anisotropic diffusion for contrast reduction in bilateral filter, a new nonlinear filter named trilateral filter was proposed by Choudhury and Tumblin [36] . Compared to the bilateral filter, trilateral filter includes an additional analysis that involves the distribution of the intensity gradients within the filter range. Thus, the filter considers more on the pixels that hold larger similarities to the filter's central pixel with respect to their spatial locations, intensity values and local gradients altogether.
Eq. (8) shows the process of the proposed method that utilizes the Trilateral Filter.
where F( σ i ) and H( σ ' i ) hold the same functions as the ones in Eq. (7) , and G( σ " i ) stands for the Gaussian Filter on local gradients of the pixels. The Trilateral Filter can suppress the halo artifacts to a certain extent since it processes the pixels only with the neighboring pixels that have higher similarities in gradients.
The proposed method utilizes bilateral filter and trilateral filter on different channels of the underwater image for MSR processing according to the different characteristics of these channels. As described in Section 3 , the underwater image degradation holds variations both in spatial domain and intensity value range domain. Therefore, the proposed method employs the bilateral filter and trilateral filter instead of classic Gaussian Filter due to that the Bilateral and Trilateral Filter both take into the considerations of the similarities in spatial domain and intensity value range simultaneously. The A and B channels together represent the chromatic characters of the underwater image, which hold the color information of the scene. The L channel represents the luminance component of the image. Therefore, the proposed method tends to process more pixels in the L channel by applying the bilateral filter with less constraints, and applies Trilateral Filter to the A and B channels to further preserve the edge signals.

Experiments and evaluations
In this paper, we evaluate the proposed method by two types of the experiments including both the subjective tests and the objective tests. The first type is about the psychophysical experiments related to the underwater image perceptions. This type of the experiments demonstrates both the psychophysical characters of the degraded underwater images, and the performance of the proposed method which improves the visibility of the underwater images. In the second type of the experiments, the proposed method is further tested with the undersea images in the real world.

Subjective tests
In this part, the experiments are conducted with 30 individual subjects observing images of two pads with 16 different testing color patches on each. These images are captured when the camera and the pads are both placed in the water with different turbidity conditions. Moreover, a pad with reference color patches is also captured in the air. The subjects are instructed to find the most similar reference color patch from each testing color patch. All the pads are printed by the same printer. All the images are captured  by the same camera, and are displayed to the subjects by the same monitor. To demonstrate the performance of the proposed method, the experiments are divided into two sections. In the first section, the original images of the underwater testing color patches are provided to the subjects; in the second section, the processed underwater images containing the testing color patches are provided to the subjects. The processed images are all enhanced by the proposed method.
In the experiments, the underwater turbidity conditions are simulated by using a mixture of whole milk and grape juice in the water. Existing research has demonstrated that the milk can simulate the scattering effect, while the grape juice provides the absorbing effect [37] . With different proportions of milk and juice to water, 4 different turbidity conditions are applied during the experiments. The pads that contain color patches are shown in Fig. 5 . (

a) Experiment with original underwater image
In the first section of the experiment, the 30 subjects have been recruited to recognize the testing patches in the images captured underwater with 4 different turbidity conditions and 2 different distances. Examples of the images are shown in Fig. 6 . For the convenience of the experimental result analysis, the reference color patches are grouped into 7 groups according to 3 channels in RGB color space, as shown in Fig. 5 (c), where each row is a group: ( A ) In the color group 1, colors only vary in the green and blue channels with the values in the red channel fixed at the maximum.  The experiment results are shown in Table 1 , which demonstrates the percentage when perceiving one color group as a certain color group. As it can be observed that: ( A ) the color groups 1 and 4 are most likely to be confused with each other. Beside this confusion, the colors in group 1 are most likely to be confused with the ones in group 5. ( B ) The color groups 2 and 6 are most likely to be confused with each other. Beside this confusion, the colors in group 2 are most likely to be confused with the ones in group 7. ( C ) A few subjects recognize the colors in group 3 as the colors in group 7, but a majority of subjects recognizes the colors in group 7 as the colors in group 3. ( D ) The colors in group 5 are recognized correctly at most time.
As it can be observed in the results of the first section of the experiments, the visual images can be highly affected in the underwater environment. The degradation of the images increases when turbidity condition of the water increases, as shown in Fig. 7 . With the turbidity in the water, which mostly causes the scattering and absorption, the perception of the blue and green components of the color becomes insensitive. This is likely due to the loss of the information in the red channel of the color, which makes the ambient lights seem to be dominated by green-blue. This demonstrates that the underwater image degradation follows the uneven distribution in color range domain. At the same time, the image seems to be blurrier with the increase of the turbidity and the object's distance. It can also be noted in the Fig. 6 that the attenuation of the underwater image is spatially different. (

b) Experiment with underwater image enhanced by proposed method
In the second section of the experiment, the underwater images are processed using the proposed method, and then provided to subjects for testing. The proposed method is implemented using C ++ with OpenCV libs. Three scales are applied with equal weights for them. Bilateral filter and trilateral filter share the same σ values for the spatial domain, which are 15, 50 and 120 for three scales. The σ ' for the distribution of color range domain equals σ /2, and the σ " is the same value as the σ ' . Those parameters affect the sensitive ranges of the filters in three domains. The smaller σ values lead to tight constraints when evaluating the similarities of the pixels. This will make the processed image de-saturated.
The larger σ values let the filters accept more pixels with larger weights in the convolution process, which results in over-saturated with more halo artifact.
The processed underwater images are shown in Fig. 8 , and the experiment results are demonstrated in Table 2 . As it can be observed in the results, the visibilities of the underwater images enhanced by the proposed method are much better than the original ones. According to the Table 2 , the accuracies of the color Table 1 The statistics of the underwater color perceptions . The percentages with underline belong to the color group that is perceived by most participants. The percentages with dashed underline belong to the second most perceived color group. to σ for each scale respectively.

Table 2
The accuracy of the perceptions for enhanced underwater images using proposed method. perceptions by the subjects are also well improved compared to the ones seated on the diagonal elements in Table 1 .

Performance comparison of the proposed method and the existing one
In this part, experiments are conducted to demonstrate the performance of the proposed method using real-world undersea images. The parameter settings of the proposed method are the same as the ones in the Section 6 . A.2. For comparison, we apply the method in the form of the multi-scale Retinex [16] to the underwater images. All the images are scaled to VGA size for the convenience of the comparison. The results are demonstrated in Fig. 9 , where the first, the second and the third columns contain the original undersea images, the undersea images processed using the MSR in [16] , and the enhancement results using the proposed methods respectively. The comparisons of the image quality measurement are also demonstrated in Table 3 , which demonstrates the better performance of the proposed method for underwater image enhancement.
As it can be observed from the experiment results, the images enhanced by the proposed method are significantly improved from the original ones. With the proposed method, the contents in the dark areas of the original underwater images are brightened. The decayed colors in the underwater environment are corrected. Compared to the method of MSRCR, the proposed method alleviates the halo artifacts in the enhanced images to a great extent. The halo artifacts are obviously noticeable in the processed image by the MSRCR, especially where edges are sharp. The proposed method suppresses these effects due to the utilization of the filters with the consideration of the distributions in all three aspects. Furthermore, the excessively high contrast is avoided for the enhanced images with the proposed method.

Conclusion
In this paper we propose a novel method for underwater image enhancement. The proposed method, LAB-MSR, is inspired by the original MSR framework. It processes the underwater images in CIELAB color space. The combination of the bilateral filter and trilateral filter is applied to the three channels according to the characteristics of each channel. Compared to the Gaussian Filter employed in the original Retinex, these filters consider not only the degradation distribution in the spatial domain, but also the ones in terms of neighboring intensity values and neighboring gradients. This makes the proposed method more suitable for the Fig. 9. The comparison of the enhanced underwater images using MSRCR and the proposed method respectively. The left column shows the original images; the middle column shows the enhanced images by MSRCR; the right column shows the enhanced images by the proposed method.

Table 3
The comparison of the quality measurements of the experiment in Fig. 9 . situations in the underwater environment. Moreover, these filters enable the proposed method to suppress the halo artifacts that are found severely in the traditional Retinex when applied to images with sharp edges. This paper also discusses the degradation characteristics of the underwater images, which supports the construction of the proposed method. The experiments carried out in this paper include two aspects: (a) demonstrate the degradation of the underwater images with real-world data and statistics; (b) demonstrate the competitive performance of the proposed method. In future, we will further explore the temporal information carried in the LAB channels of the underwater image with the wavelet filters [38] . The potential applications of the proposed method include the underwater robot control and manipulation [39] , the underwater scene analysis [40] , etc.