Underwater Image Restoration Based on Adaptive Color Compensation and Dual Transmission Estimation

Visibility restoration of an underwater image degraded by turbid water (due to scattering, absorption, and reflection) is a challenging task which requires careful design of computational imaging methods. The fundamental problems which limit image restoration are random diffusion and absorption of light in turbid water. Recently, the red-channel underwater image restoration method has emerged as an effective approach for visibility restoration under turbid water. However, this method fails to restore the visibility in a degraded image captured under non-uniform turbid water. The method fails due to its reliance on single transmission which cannot correctly express light propagation in an underwater environment. To overcome these problems, we propose a novel image restoration method based on adaptive color compensation and dual transmission estimation (ACDTE) to restore the visibility of underwater images degraded by non-uniform turbid water. The proposed method uses color-tone adaption to determine the hue of underwater images, and estimates global water light with quadtree decomposition. The method estimates dual transmission in the media through coefficient modification for R/G/B channels. Simulation and experimental results show that the proposed method can correct color deviations and has the advantage of visibility restoration of underwater images. The proposed method can be used for marine exploration, underwater rescue, and environmental monitoring.


I. INTRODUCTION
Recently, a lot of research in computational imaging is directed towards underwater image restoration. The visibility restoration of an underwater image improves the visual distance in a scene, which is conducive to underwater rescue and surveys. Studies on underwater image visibility restoration conducted in the past can be categorized into three types, namely (a) improved dark channel prior methods [1]- [9], (b) fusion-based methods [10]- [14], and (c) deep learning based methods [15]- [18]. The underwater image visibility restoration method based on deep learning requires synthetic datasets, which do not contain realistic (practical) The associate editor coordinating the review of this manuscript and approving it for publication was Jiajia Jiang . information, and usually lead to model over-fitting. The fusion-based methods compute feature weights, which in case of inaccurate estimation lead to image restoration failures. The improved dark channel methods based on physical model are more efficient (compared to two abovementioned categories) which consider light scattering and absorption in turbid water. Therefore, the proposed method is a type of underwater image restoration based on improved dark channel prior.
He et al. [19] proposed the dark channel prior method for image dehazing under bad weather. Although this method can be used for image dehazing, it only considers atmospheric scattering and does not provide a mechanism to counter absorption of light under turbid water. Therefore, the method fails to restore visibility of underwater images.
Galdran et al. [6] proposed a new red-channel method, which can be interpreted as a variant of the dark channel method. This method can be used to tackle both the visibility loss and color distortion of underwater images. However, this method only estimates a single transmission, and it is difficult for it to express real underwater light propagation. In addition, this method needs to estimate many unknown parameters (such as attenuation coefficients, weights for each component, saturation etc.) during image recovery, and inaccuracy of these parameters affects the quality of restored image. Li et al. [8] proposed a method based on blue-green channels dehazing and red channel correction for underwater image restoration. However, the method employs contrast stretching for correcting red-channel, which makes it difficult to achieve red component restoration. Li and Li [20] built an underwater image restoration system by measuring the currents and sediment movement at the sea floor. However, this method needs to deploy a free-ascending tripod under water to measure the scene information of the seafloor. Li and Zhang [9] proposed a new algorithm for underwater image restoration based on improved background light estimation and automatic white balance. This method can reduce the influence of light and white objects under water, and improves underwater light estimation accuracy. However, the color distortion of underwater image is over-compensated, and the red channel color saturation of the restored image is too high. Lu et al. [21] proposed a novel method for enhancing optical images using weighted guided trigonometric filter and spectral properties in turbid water. However, the turbid sediments cause inaccurate estimation of water light in dense turbid water. Bryson et al. [22] proposed a method to reconstruct true colors in an underwater image by using image formation model that accounts for attenuation, scattering, and artificial lighting influence. However, this method requires 3D structural information of the terrain to be imaged, which is not available or cannot be ascertained from images.
To summarize, in order to recover a clean image (with clear scene visibility) from a degraded underwater image (i.e., image captured underwater by a camera), different methods can be used [1]- [18], amongst which the improved dark channel method based on physical model is often used. There are many types of underwater image restoration methods based on improved dark channel prior, and the most commonly used method is the red-channel underwater restoration method. However, this method fails to restore the degraded image visibility in a non-uniform or dense turbid water. In addition, the method does not have adaptive color compensation (e.g., red-component compensation).
In this paper, we report an underwater image restoration method based on adaptive color compensation and dual transmission estimation (ACDTE) to solve the aforementioned problems. The proposed method employs adaptive underwater color-tone matching to estimate global water light, and subsequently use it to estimate dual transmission (and refine them rapidly). Considering the limitations of existing schemes, a novel underwater restoration method with the following salient features is proposed: (1) the proposed method automatically adapts to multi-tone underwater images with efficient color-ton matching, and estimates global water light by using the quadtree decomposition, (2) improved image quality under dense turbid water due to its ability to estimate dual transmission (direct transmission and backscattered transmission) from the color-tone of underwater images, (3) effective rectification of color distortions and blurs.

II. ADAPTIVE COLOR COMPENSATION AND DUAL TRANSMISSION ESTIMATION
ACDTE involves three processes. First, the underwater image color-tone is automatically matched (to blue or green tone), and the image region is searched by the quadtree decomposition for global water light estimation. Second, the dual transmission is computed by the improved dark channel prior based on the color-tone of underwater image. Finally, the scene radiance is restored by the underwater formation model.

A. COLOR-TONE ADAPTION AND WATER LIGHT ESTIMATION
Jaffe-McGlamery underwater image formation model [23], [24] points out that the intensity of an underwater image captured by a camera includes three components i.e., the direct incident component D λ , the fore-scattered component F λ , and the backscattered component B λ , shown in Figure 1. Thus, the intensity of degraded underwater image can be written as: where I λ (x) is the observed intensity, J λ (x) is the scene radiance, g λ (x) denotes the point spread function (expressing light diffusion due to forward scattering), A λ (x) denotes the VOLUME 8, 2020 global water light, t λ = e −cλd(x) is the transmission describing the portion of light that is not scattered and reaches the camera, c λ is the attenuation coefficient of light wavelength λ, λ ∈ {r, g, b}, d(x) is the distance between the scene and camera, c λ = a λ + b λ , a λ is the absorption coefficient, and b λ is the scattering coefficient. We assume that the distance between the scene and camera is small such that the influence of fore-scattered component can be ignored. Furthermore, Ref [25] points out that the absorption and scattering effects of water on light cause inconsistent light attenuation at different wavelengths; suggesting the use of same transmission for light of different wavelengths in Eq. (1) to be inappropriate. Therefore, the underwater image formation model can be simplified as: where the transmission can be divided into direct component transmission t D λ (x) and backscattered component transmis- . The global water light should be taken from the infinite distance of the scene. The greater the scene depth, the greater the influence of background scattering and the lower the transmission. Assuming the background scattering at infinite distance to be uniform, the quadtree decomposition [26] is applied on the underwater image to obtain the region for estimating global water light. In quadtree decomposition, the region with the largest average intensity is used to carry out (continue) decomposition. If the region size is less than L × L (we use an empirically optimized window size of 30 × 30 pixels throughout this article), the decomposition is stopped, and the convergence region I is obtained. The global water light is estimated by adaptive underwater image color-tone written as: where max(·) denotes the maximum extraction operation, (·) Index denotes the index value of the row and column for the current value in I .

B. ESTIMATION OF DUAL TRANSMISSION 1) ESTIMATION OF DIRECT COMPONENT TRANSMISSION
Assuming the turbid water to be homogeneous, according to the Lambert-Beer theorem [27], the direct component transmission of dual transmission can be expressed as: where c D λ is the attenuation coefficient of the direct incident component, which is related to the distance d(x) and wavelength λ. The distance d(x) is small when the images are captured by an underwater camera, and the change of d(x) has little effect on c D λ . Therefore, the model ignores the influence of d(x) on c D λ , and only considers the change of c D λ with wavelength in R/G/B channels. He et al. [19] proposed a dark channel prior (DCP) method for image dehazing. The DCP is based on the observation about haze-free images that: in most of the non-sky regions, at least one color channel has very low intensity in some pixels, defined by: (x) is a local patch centered at x. In order to ensure that the transmission is effective for the high intensity region under water, the dark channel extremum serves as the correction coefficient for the transmission. Referring to our transmission optimization research [28], the optimized transmission can be written as (6), shown at the bottom of the page, where U represents the identity matrix, and '•' denotes the Hadamard product. The absorption of light by water results in a bluegreen tone in underwater images. The red channel component can be compensated by adaptively judging the color-tone of underwater images. Therefore, the primary color transmission of the t D λ (x) can be estimated by adaptive image colortone judgment as: The global water light in an underwater image is proportional to the scattering coefficient and inversely proportional to the attenuation coefficient [29]: The relationship between the scattering coefficient of water and the wavelength of light can be expressed as [30]: where b(λ r ) is the scattering coefficient of reference wavelength. The blue/green channel transmissions of t D λ (x) can then be written as: where

2) ESTIMATION OF BACKSCATTERED COMPONENT TRANSMISSION
The backscattered component transmission of dual transmission can be expressed as: where c B λ is the attenuation coefficient of the backscattered component, which is weakly correlated with the wavelength of light [31]. Ignoring the effect of light wavelength λ on c B λ , the t B λ can be written as: The backscattered component transmission is mainly affected by the distance. When the underwater scene depth is large, the image will show a blue-green tone, and the backscattered component transmission is also the largest under the image color-tone. Therefore, according to the adaptive color-tone of underwater image, the backscattered component transmission is estimated as: In order to eliminate the block effects and halo artifacts in the restored images, guided filter [32] is employed to refine the transmissions t D λ (x) and t B λ (x). The guided filter can be used as an edge-preserving smoothing operator like the popular bilateral filter [33], with better smoothing performance near edges. The refined transmissions t D λ (x) and t B λ (x) are further used to restore degraded underwater images.

C. UNDERWATER SCENE RADIANCE RESTORATION
Once the t D λ (x), t B λ (x) and A λ (x) are estimated, the scene radiance J λ (x) can be restored. The flowchart of the proposed method for underwater image restoration based on adaptive color compensation and dual transmission estimation is shown in Figure 2. According to Eq. (2), the scene radiance of underwater image is restored by: where t 0 is a critical value set to prevent t D λ (x) from being too small. This value can effectively prevent over-bright pixels in the restored image. The value of t 0 is set as 0.1 [1], [14], [19].
The process of visibility restoration (of a degraded underwater image) by the proposed method is shown in Figure 3. It can be seen from Figure 3   shown in Figure 3(f). The scene depth estimation is not accurate unless the c B λ is accurately measured in an underwater environment. The attenuation coefficient for backscattering strongly depends on many factors (such as sensor response, imaging range, and veiling light etc.). However, we do not accurately estimate the attenuation coefficient of backscattered component during the restoration of degraded underwater image, and this part is left as future work. The result VOLUME 8, 2020 of underwater visibility restoration and compensation of red component by the proposed method is shown in Figure 3(g).
To evaluate the color compensation capability of the proposed method, the RGB space before and after restoration is shown in Figure 4(a). It can be seen from the results that the intensity range of each color channel is stretched after restoration. For the red and blue channels: high intensity pixels are compensated (shown by Figure 4(b1) and Figure  4(b3)), whereas for the green channel: intensity values (show as Figure 4(b2)) are reduced to correct the distorted color of the degraded underwater image. To evaluate detail enhancement capability of our method, we apply canny algorithm [34] on both degraded and restored images. The result in Figure 4(c) shows that the visible edges are increased after image restoration.
The visibility restoration performance of our proposed method can be further evaluated with degraded underwater images shown in Figure 5. For the images with blue tone, before and after restoration results are shown in Figure 5(a). The results of restored images clearly show the increase in visibility achieved by our method along with rich color enhancement. The restoration results of greenish degraded images are shown in Figure 5 (b). It can be seen from these results that the red component has been significantly enhanced, and the color perception is greatly improved.

III. EXPERIMENTAL COMPARISON AND ANALYSES A. UNDERWATER IMAGE VISIBILITY RESTORATION ANALYSIS 1) QUALITATIVE ANALYSIS OF UNDERWATER IMAGE RESTORATION METHODS
To verify the superiority of ACDTE method, we compare the restoration results of our method with the results of three existing algorithms namely He's method [19], Fusion method [10], and Red-channel method [6]. The qualitative comparison is presented in Figure 6. In Figure 6(a), image 1 and image 2 are the green tone underwater images, whereas  [35], (a3), (b1) and (b2) are from [13], (b3) and (b4) are from [11]. image 3 and image 4 are the blue tone images. The results of He's method (in Figure 6(b)) indicate that the method cannot correct color distortions in underwater imaging. In case of Fusion method, the color is over-corrected, especially the red component is over-compensated as shown in Figure 6(c). From Figure 6(d), it can be seen that the Red-channel method restores the visibility of image 2 and image 4, but fails to correct the colors of image 1 and image 3. In contrast, the proposed method (Figure 6(e)) not only restores the visibility of degraded underwater images (by eliminating scattering blur), but also compensates for color distortion effectively. In order to clearly show the effectiveness of the proposed method in terms of color compensation, the color histograms of the underwater images before and after restoration are presented in Figure 7 for comparison. It can be seen from Figure 7(b) that the red component in each histogram for the 'after restoration' images has been compensated by our method.

2) QUANTITATIVE ANALYSIS OF UNDERWATER IMAGE RESTORATION METHODS
To quantitatively evaluate the visibility restoration performance of the tested methods, three evaluation metrics are used i.e., underwater image quality measure (UIQM) [36], information entropy (E) [37], mean gradient (Ag) [38]. The UIQM is used to understand the overall underwater image restoration performance without reference to the ground truth VOLUME 8, 2020 image. The UIQM metric is a linear combination of three independent image quality measures, given by: (15) where UICM is the measure of colorfulness, UISM is the measure of sharpness, and UIConM is the measure of sharpness, and UIConM is the measure of contrast. We have set the weights as follows: c 1 = 0.3282, c 2 = 0.2953, c 3 = 3.5753. A greater value of the UIQM indicates superior image quality. The information entropy is introduced to quantify the amount of information in the image, and it is defined as: where n represents the total gray-level number of the image, p i represents the probability that the image gray-level i appears. A big score of E indicates rich amount of information in the image. The mean gradient is used to evaluate image sharpness, and is written as: where f (i, j) denotes the gray value of coordinate point (i, j) in the image, ∇ i f (i, j) and ∇ j f (i, j) denotes the gradient of the image point in the row and column directions, respectively. A greater value of Ag suggests higher image sharpness. The quantitative comparison for different underwater image visibility restoration methods are shown in Figure 8. From Figure 8(a), it can be seen that our method has the highest UIQM score compared to other methods, confirming superior visibility restoration capability of our method. The results in Figure 8(b) highlight the fact that the proposed method extracts more amount of information than other during recovery. Although the Ag value of our method is smaller than that of fusion method, our method still obtains better image sharpness indicated by Figure 8(c). In addition, to evaluate image detail information, the matching number of local feature point is counted by SIFT operator. Figure 9 is a comparison of local feature point matching before and after the visibility restoration of underwater degraded images. The matching number of local feature points is counted as shown in Figure 9(c), which shows that the underwater image details are improved by our method.

B. EXPERIMETNAL RESULTS AND ANALYSIS
Experiments are carried out to verify the visibility restoration capability of ACDTE on the images degraded by non-uniform turbid water. Experiments are performed under turbid water of different densities. The turbid water is generated by adding milk in water, and the concentration of turbid medium cannot be quantified due to non-uniform of milk. The scene objects are: 3D print model, wire stripper, and cup. The degraded images are captured by MV-SUA630C/M, Mind Vision camera. The distance between the camera and water is 50 cm, and this distance is fixed during image capturing. The volume of the water tank used is 32cm×32cm×60cm.
The results of visibility restoration by the ACDTE method applied on degraded images captured under non-uniform turbid water are shown in Figure 10(b). Figure 10(b) shows that the scene visibility in the degraded images under turbid water (of varying density) is significantly improved by the proposed method. Targets with different scene depths can be restored by the ACDTE method even under highly dense turbid water. The visibility restoration results for the wire stripper and cup targets are shown in Figure 11. Figure 11(a) shows that the color compensation of the proposed method does not over-compensate the red component. From Figure 11, image insets of the zoomed areas marked by red rectangles indicate that the proposed method can recover fine details under strong scattering condition.
To quantitatively evaluate the visibility restoration performance of ACDTE method under different conditions (of varying density), targets marked with 'green dotted boxes'  [4], and the third group is from [21]. in Figure 10(b) and Figure 11 are used for comparison. The quantitative results are presented in Table 1. The results show that the three metrics of UIQM, Ag and E of ACDTE improve significantly after restoration (compared to before restoration images), highlighting the superior underwater image restoration performance of ACDTE.

C. DISCUSSION
Although the ACDTE method has good underwater image restoration capability, there are some problems that affect restoration performance. Here, we discuss some potential limitations and provide solutions as future direction to improve the performance of our method.
First, the real-time performance of ACDTE method needs to be improved for processing underwater videos. The ACDTE method requires 2s to restore an underwater image (600 × 400 pixels), mainly due to the use of iterative template matching algorithm (used to find water light region). To cater real-time needs, as future work, a matrix operation can be employed instead of loop nesting in template matching algorithm, and a GPU can be used for acceleration in our method.
Second, although our algorithm can process images degraded by water scattering and absorption, it cannot process blurred defocused images. In our future work, we will add super-resolution de-blurring technology to our imaging model to enhance its underwater restoration capability.

IV. CONCLUSION
This paper presents a novel method (called ACDTE) for the visibility restoration of images degraded by non-uniform turbid water. The proposed method is based on adaptive color compensation and dual transmission estimation for efficient restoration capability. The method is tested on different underwater images such as blue-tone green-tone images, and images captured under different scattering conditions (i.e., under different turbidity). The simulation and experimental results show that the proposed method has superior visibility restoration capability. The proposed method not only restores the visibility in degraded underwater images, but also compensates for color losses underwater. To achieve real-time performance for underwater video processing, the algorithm should be optimized by matrix operation, and requires GPU acceleration. The proposed method can effectively be used for different application such as marine exploration, underwater rescue, and various military applications. YAMI FANG, photographs and biographies not available at the time of publication. VOLUME 8, 2020