Reflection Removal Algorithms : A Review

This study describes some algorithms in literature review that tried to computationally remove reflections from images over the past years and explains the advantages, limitations or drawbacks of these algorithms. With the popularity of all kinds of digital instrument and digital products, image has become one of the most commonly information carriers in the human activity. The image quality is often influenced by a variety of external factors. When you're going to travel, in some cases, you may have to take pictures through the dividing glass because you are unable to get off the vehicle. You may want to take photographs through windows when you are in a coffee shop or high rise building, this resulted to undesired reflections in the photograph and loss of information in the scene we actually want to capture and destroy the picture, so reflection in images is a big problem in photography. In computer vision, removing reflections is a very useful pre-processing step to remove undesired information from the image to be analyzed, to reach this goal many algorithms have been developed.


INTRODUCTION
Besides computer vision, reflection removal is also widely needed in our daily lives, especially with the spread of the camera equipped with smart phones. With their mobile phones, when people take pictures of clothes behind windows, paintings behind glass covers or figures on computer screens, the reflections always severely reduce the quality and usage of resulting images (Geng and Jiang, 2013).
One of the most difficult tasks in computer vision is the process of identifying objects and people also tracking their movements in real complex scenes using a series of images, shadows or reflections on the ground associated with each object are greatly reduced in the localization accuracy of moving objects that must be present in indoor and outdoor scenes. Shadows and reflections have, in fact, a bad effect on localization and tracking of moving objects (Teschioni and Regazzoni, 1998).
Both photographers and entertainment professionals have a need to remove reflections from color images acquired through a semi transparent medium. This removal is necessary to improve human cognition and applies many of the computer vision algorithms later. This problem can also happen in the field of cultural heritage, where historical works such as paintings, murals and precious manuscripts are usually framed behind glass for conservation objectives (Bedini et al., 2015).
The superposition of two layers (the scene in front and behind of the camera) getting reflected due to the semi reflective surface resulted in an difficult problem which is reflection removal .
Polarizing lenses are used by professional photographers to decrease the impact of reflection, through rotating the polarizing lens that can be minimized the reflection but not eliminate it .
As a general rule, the picture we get through windows or glass panes is a super-position of two sources: a scene behind it and a reflection of a scene on the same side of it. We need to keep the transmission scene and remove extraneous reflection information from the original image at the same time (Liao et al., 2016).
The reflection removal can be classified as one of the layer separation tasks with reflection stains being addressed as a different layer from the remainder of the image. These two layers form a linear combination that can be simply addressed in the following equation: where, I B and I R are background and reflection images respectively (Sirinukulwattana et al., 2015). The goal is to eliminate the reflection I R and to reconstruct the background I B (Wan et al., 2016).

REFLECTION REMOVAL CATEGORIES
Reflection removal is intended to separate the mixture from the desired scene and unwanted reflections. From Eq. (1) clearly extra information or constraints are needed to solve this problem and in previous research works various priors have been introduced. Existing methods for reflection removal can be grouped into three classes depended on the number of images utilized as input.

Class (1):
In this class, a single image is used, so image priors, including statistics of derivative filters and corner detectors in natural images were used to separate the image into two different layers. Removing the reflection from one image remains a difficult problem in spite of the presence of these priors.

Class (2):
Two images are used in this class to remove reflections, for instance, some of them used two images taken with different angles by rotating the polarizer to separate the reflections. This class needs special requirements when capturing input data. These methods are characterized by the ability to remove reflections efficiently.

Class (3)
: an image sequence is used in this class as input, different approaches are adopted to separate the reflections; one of them assumed that the motion of each layer is a homography and a rank minimization method is implemented to solve this problem strongly. The possibility of these methods is limited to the camera motion condition for a dynamic scene (Wan et al., 2016).

ALGORITHMS DESCRIPTION
This section illustrates several separate algorithms to remove the reflection that described in Table 1 in terms of working methods and the number of images used as inputs in these algorithms:

CONCLUSION
The algorithms with single image input require some image priors to remove reflections from the input image, which is modeled as linear combination of the scene through the window and the reflected image: where, I = The resulting image T = The transmission layer R = The reflection layer Therefore, it is ill posed problem to obtain two variables from single image. This means algorithms that work on a set of images achieve higher performance than algorithms that work on a single image.
The existence or absence of the prior knowledge led to propose a variety number of separation methods. User can choose an appropriate reflection removal algorithm utilizing Table 2 that highlight the advantages, limitations and drawbacks of the reflection removal algorithms that described in Table 1. In 2012 (Gai et al., 2012) A series of images -This study proposed a sparse blind separation with motions algorithm (SPBS-M) to guess all mixing parameters from the moving superimposed images. -(SPBS-M) algorithm is depended on image statistics to estimate mixing parameters in order to separate layers by generating assumptions on: • The sparsity of image gradients.
• The noncorrelation of the gradients of different locations in the same image.
• The joint behaviors of various gradients in the same image. 2 In 2014 (Guo et al., 2014) Multiple images -A robust method to separate the transmitted layer and the reflected layer from multiple images is proposed in this study, taking advantage of the correlation of the transmitted layer across multiple images, the sparsity and independence of the gradient fields of the two layers. -To solve the separation problem effectively, an algorithm depended on an augmented lagrangian multiplier is designed. 3 In 2015 .
Single input image -The proposed solution in this study is derived from the method suggested by (Levin and Weiss, 2004). -The suggested modification in this study is to make the gradient marking process automatically and eliminate user support by taking advantage of the information about the reflected scene. -This solution has been developed to be applied in smart phones or smart devices with less user support. 4 In 2015 (Shih et al., 2015) Single image -In this study, a new algorithm has been introduced to exploit ghosting cues for layer separation in images taken through glass windows. -A critical hint is provided through Ghosting to separate the reflection and transmission layers, because it breaks the symmetry between the two layers. -A double impulse convolution kernel is used as a paradigm to represent the ghosted reflection and estimated the spatial separation automatically. In 2015 (Bedini et al., 2015) Single image -The reflection in this work is supposed as a monochromatic or achromatic image that incorporates additively with the real transmitted image. -A blind source separation technique that takes advantage of the reliance of the three color channels of the original image and the independence of the reflected image is used in this work. -The estimation step of the model parameters through ICA followed by image's components estimation step via regularization are formed the algorithm working steps. 6 In 2015  Two images are taken simultaneously.
-The suggested method in this study is an expansion of the Independent Component Analysis (ICA) method, utilizing a back camera (capturing actual scene) and front facing cameras. -The smart device is used to captures two images at two different directions of the polarizer, using a polarizer gives a linearly mixed set of images, which is the basic supposition for the captured scenes. -The joint pdf (probability distribution function) of two mixed images looks like a parallelogram and this is the fundamental principle of this algorithm. 7 In 2015 (Simon and Kyu Park, 2015) Video is captured by a black box in a moving car.
-The proposed method is taken advantage of the spatio temporal consistency of reflection, which is based on the movement of the vehicle is forward while the internal objects' reflection remains static, in order to remove the reflection on the windshield from in vehicle black box videos. -To remove the reflection on the windscreen an effective technique is provided by using the suggested average image prior and the region based optimization technique. -The average image prior is defined as the average of the frame sequence has been taken to produce a very low pass image in which the reflection still remains sharp. 8 In 2015 (Xue et al., 2015) A series of images are captured by the camera phone while the user is lightly scanning the scene -This algorithm produces the wanted background scene as if the visual obstacles were not there and it is working fully automatically. -The background and the obstacles from the camera are separated according to the differences that are existed in the relative position of them depending on their motions. -The obstructing layer can be removed using five images taken along a small, almost horizontal baseline of a few centimeters. 9 In 2016 (Sun et al., 2016) Two photos.
-The proposed method requires a minimum number of images namely, two images in this study, to provide the motion cues. -The main observation is that the background edges tend to have uniform motions across the hierarchy while the reflected edges possess larger motion variations. -To distinguish background edges and reflection edges this method first calculates two priors for each edge pixel, namely, the intensity score (gradient magnitude) and the aggregated motion score (from SIFT-flow hierarchy built upon a number of smoothness levels). 10 In 2016 (Wan et al., 2016) Single image -Photographers always concentrate on the background at a certain depth when taking pictures, which is the basic hypothesis in this work. -The idea: background and reflection edges are labeled by using depth of field (DoF). -Authors developed a multi scale inference strategy to elect background and reflection edges to remove the reflections. 11 In 2016 (Liao et al., 2016) Single image -An approach has been presented in this study to increase the speed of the reflection removal algorithm by using the GPGPU technology. -CUDA is a parallel computing architecture. GPU is a high-performance manycore processor, which can be used to accelerate many applications. CUDA's programmatic interfaces can be exploited to call resources of GPU directly. -The image quality had a higher priority than the algorithm performance where the researchers were focusing on it. It is important to maximize the use of hardware resources to enhance the performance of the algorithm, since the amount of the calculation of the reflection removal algorithm is large. In 2012 (Gai et al., 2012) -The experiments on both real and simulated superimposed images were shown the success of this technology -Translations, scalings, rotation and other transitions can be treated in this approach.
-The SPBS-M algorithm is performed well under a relatively steady scene.

2
In 2014 (Guo et al., 2014) -This algorithm is flexible in image translation and transformation. -The outperformance of the proposed method in accuracy and simplicity on both simulated and real data is demonstrated by the empirical results in comparison with the SIUA, SPBS-M and RASL methods.
-User interaction is required to select the area of interest in the very first picture frame. In 2015  -The removal of user support is resulted in better results than the original algorithm. -The proposed algorithm works like to the original algorithm when user's markings are excellent.
-The supposition that is made during the selection of points may result in treating the weak textures in the image as a reflection which may be led to loss of these weak textures if it is assigned incorrectly. 4 In 2015 (Shih et al., 2015) -A large portion of reflections on both synthetic and real world inputs was removed by this method and it was worked automatically. -Image classification is one of the potential applications of this algorithm.
-This method was limited to spatially invariant ghosting kernels.

5
In 2015 (Bedini et al., 2015) -The advantages of this algorithm: high speed and producing promising results. -This method is developed to apply to color images of any type of scene that is affected by an achromatic reflection.
-The estimation of the component images step in this method is one of the aspects that need to be improved by finding an effective method to skip the permutation indeterminacy of ICA and exploring for more effective constraints about the component images to get a more true color of the recovered images. 6 In 2015  -The PSNR of the image is enhanced by an average of 10% in this method compared with the original ICA implementation. -This method is achieved a good separation due to the exploitation of the front camera image in supplying more information about individual sources.
-Images have been captured without considering the motion 7 In 2015 (Simon and Kyu Park, 2015) -The reflection layer in many real black box videos has been successfully separated using the proposed approach. -The suggested method first restores the reflection layer instead of the scene behind the glass surface; this is contrary to the previous methods.
-The proposed method has limitations because the using of average image prior and the supposition of static reflection restrict the general cases' application. 8 In 2015 (Xue et al., 2015) -In this study the algorithm for estimating motion in the existence of obstacles is characterized by strength and reliability. It is success lies beyond the ability to dealing with the depth variation in addition to small motions within each layer and produces a strong initial estimate of the motion of each layer in the existence of visual obstacles. -A unified approach is presented in this study to deal with reflections and physical occlusions such as fences.
-Some restrictions were imposed on the scene, such as being almost static during the taking of photographs, but the authors found that many common shooting conditions fit it well -The approach in this study will not operate if the motion of the camera is completely rotational.

9
In 2016 (Sun et al., 2016) -The effectiveness of this method was validated by testing various challenging examples. -Compared to the methods that require a sequence of images or a video clip for the separation, this method only needs two images, which largely improves its usefulness.
Input used: Two images that are taken in front of a transparent glass under a little different view.

10
In 2016 (Wan et al., 2016) -The effectiveness of this method is verified through experimental results using real-world images. -This method showed better separation results. -The majority of reflections can be removed by this algorithm.
This method cannot handle an image containing very small details, including the small structures in some objects.

11
In 2016 (Liao et al., 2016) -Compared with CPU, The GPU-based implementation is faster (up to 18 times) than the CPU-based implementation and GPU has the characteristic of low cost and high performance. The high performance feature of GPU is obvious in the arithmetic speed of floating point and memory bandwidth.
-This work can produce good results from a single image if it contains that double reflection, but depending on the geometry, not all reflections are doubled, or the doubling may be difficult to see.