Real-time robust direct and indirect photon separation with polarization imaging

Separation of reflections, such as superimposed scenes behind and in front of a glass window and semi-diffuse surfaces, permits the imaging of objects that are not in direct line of sight and field of view. Existing separation techniques are often computational intensive, time consuming, and not easily applicable to real-time, outdoor situations. In this work, we apply Stokes algebra and Mueller calculus formulae with a novel edge-based correlation technique to the problem of separating reflections in the visible, near infrared, and long wave infrared wavelengths. Our method exploits spectral information and patch-wise operation for improved robustness and can be applied to optically smooth reflecting and partially transmitting surfaces such as glass and optically semi-diffuse surfaces such as floors, glossy paper, and white painted walls. We demonstrate robust image separation in a variety of indoor and outdoor scenes and real-time acquisition using a division-of-focal plane polarization camera. © 2017 Optical Society of America under the terms of the OSA Open Access Publishing Agreement OCIS codes: (110.5405) Polarimetric imaging; (100.2960) Image analysis; (290.5855) Scattering, polarization. References and links 1. A. Hyvärinen and E. Oja, “Independent component analysis: algorithms and applications,” Neural Netw. 13(45), 411–430 (2000). 2. E. C. Cherry, “Some experiments on the recognition of speech, with one and with two ears,” J. Acoust. Soc. Am. 25, 975–979 (1953). 3. S. Haykin and Z. Chen, “The cocktail party problem,” Neural Comput. 17(9), 1875–1902 (2005). 4. A. J. Simpson, G. Roma, and M. D. Plumbley, “Deep karaoke: Extracting vocals from musical mixtures using a convolutional deep neural network,” in International Conference on Latent Variable Analysis and Signal Separation, (Springer, 2015), pp. 429–436. 5. D. Maino, A. Farusi, C. Baccigalupi, F. Perrotta, A. J. Banday, L. Bedini, C. Burigana, G. De Zotti, K. M. Górski, and E. Salerno, “All-sky astrophysical component separation with Fast Independent Component Analysis (FASTICA),” Mon. Not. R. Astron. Soc. 334, 53–68 (2002). 6. S. Makeig, A. J. Bell, T.-P. Jung, and T. J. Sejnowski, “Independent component analysis of electroencephalographic data,” Adv. Neural Inf. Process. Syst. 8, 145–151 (1996). 7. Y. Y. Schechner, “Inversion by P4: polarization-picture post-processing,” Philos. Trans. R. Soc. Lond. B Biol. Sci. 366(1565), 638–648 (2011). 8. H. Farid and E. H. Adelson, “Separating reflections from images by use of independent component analysis,” J. Opt. Soc. Am. A 16(9), 2136–2145 (1999). 9. A. Levin and Y. Weiss, “User assisted separation of reflections from a single image using a sparsity prior,” IEEE Trans. Pattern Anal. Mach. Intell. 29(9), 1647–1654 (2007). 10. Y. Shih, D. Krishnan, F. Durand, and W. T. Freeman, “Reflection removal using ghosting cues,” in IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2015), pp. 3193–3201. 11. M. Zibulevsky and B. A. Pearlmutter, “Blind source separation by sparse decomposition in a signal dictionary,” Neural Comput. 13(4), 863–882 (2001). 12. Y. Y. Schechner, J. Shamir, and N. Kiryati, “Polarization and statistical analysis of scenes containing a semireflector,” J. Opt. Soc. Am. A 17(2), 276–284 (2000). 13. N. Kong, Y. W. Tai, and J. S. Shin, “A physically-based approach to reflection separation: from physical modeling to constrained optimization,” IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 209–221 (2014). 14. K. Xu, S. Liu, W. Sun, Z. Ma, Z. Li, Q. Yu, and G. Li, “Design and fabrication of a monolithic optoelectronic integrated Si CMOS LED based on hot-carrier effect,” IEEE J. Sel. Top. Quantum Electron. 22, 70–77 (2016). 15. J. S. Tyo, D. L. Goldstein, D. B. Chenault, and J. A. Shaw, “Review of passive imaging polarimetry for remote sensing applications,” Appl. Opt. 45(22), 5453–5469 (2006). 16. V. Gruev, R. Perkins, and T. York, “CCD polarization imaging sensor with aluminum nanowire optical filters,” Opt. Express 18(18), 19087–19094 (2010). Vol. 25, No. 23 | 13 Nov 2017 | OPTICS EXPRESS 29432 #305545 https://doi.org/10.1364/OE.25.029432 Journal © 2017 Received 24 Aug 2017; revised 1 Nov 2017; accepted 3 Nov 2017; published 10 Nov 2017 17. G. Myhre, W.-L. Hsu, A. Peinado, C. LaCasse, N. Brock, R. A. Chipman, and S. Pau, “Liquid crystal polymer full-stokes division of focal plane polarimeter,” Opt. Express 20(25), 27393–27409 (2012). 18. W. L. Hsu, G. Myhre, K. Balakrishnan, N. Brock, M. Ibn-Elhaj, and S. Pau, “Full-Stokes imaging polarimeter using an array of elliptical polarizer,” Opt. Express 22(3), 3063–3074 (2014). 19. X. D. He, K. E. Torrance, F. X. Sillion, and D. P. Greenberg, “A comprehensive physical model for light reflection,” in SIGGRAPH, (ACM Press, 1991), pp. 175–186. 20. K. Xu, “Current-voltage characteristics and increase in the quantum efficiency of three-terminal gate and avalanche-based silicon LEDs,” Appl. Opt. 52(27), 6669–6675 (2013). 21. W. K. Pratt, “Correlation techniques of image registration,” IEEE Trans. Aerosp. Electron. Syst. 10, 353–358 (1974). 22. A. J. Fitch, A. Kadyrov, W. J. Christmas, and J. Kittler, “Fast robust correlation,” IEEE Trans. Image Process. 14(8), 1063–1073 (2005). 23. S. R. Curtis and A. V. Oppenheim, “Reconstruction of multidimensional signals from zero crossings,” J. Opt. Soc. Am. A 4, 221–231 (1987). 24. J. Canny, “A computational Approach to Edge Detection,” IEEE Trans. Pattern Anal. Mach. Intell. 8(6), 679– 698 (1986). 25. M. Jacob and M. Unser, “Design of steerable filters for feature detection using canny-like criteria,” IEEE Trans. Pattern Anal. Mach. Intell. 26(8), 1007–1019 (2004). 26. P. Bao, L. Zhang, and X. Wu, “Canny edge detection enhancement by scale multiplication,” IEEE Trans. Pattern Anal. Mach. Intell. 27(9), 1485–1490 (2005). 27. W. Rong, Z. Li, W. Zhang, and L. Sun, “An improved CANNY edge detection algorithm,” in IEEE International Conference on Mechatronics and Automation (IEEE, 2014), pp. 577–582. 28. G. Bradski and A. Kaehler, Learning OpenCV: Computer Vision with the OpenCV Library (O'Reilly Media, 2008). 29. W. L. Hsu, K. Balakrishnan, M. Ibn-Elhaj, and S. Pau, “Infrared liquid crystal polymer micropolarizer,” Appl. Opt. 53(23), 5252–5258 (2014). 30. W. L. Hsu, J. Davis, K. Balakrishnan, M. Ibn-Elhaj, S. Kroto, N. Brock, and S. Pau, “Polarization microscope using a near infrared full-Stokes imaging polarimeter,” Opt. Express 23(4), 4357–4368 (2015). 31. E. J. Elton and J. S. Preston, “Effective refractive index and the porosity of coated paper,” (Surfoptic, 2007). 32. G. Ghosh, “Dispersion-equation coefficients for the refractive index and birefringence of calcite and quartz crystals,” Opt. Commun. 163, 95–102 (1999). 33. L. L. Long, M. R. Querry, R. J. Bell, and R. W. Alexander, “Optical properties of calcite and gypsum in crystalline and powdered form in the infrared and far-infrared,” Infrared Phys. 34, 191–201 (1993). 34. R. Gonzalez, R. Zallen, and H. Berger, “Infrared reflectivity and lattice fundamentals in anatase TiO2s,” Phys. Rev. B 55, 7014 (1997). 35. SCHOTT, “Interactive Abbe-Diagram” (2016), retrieved http://www.us.schott.com/advanced_optics/english/knowledge-center/technical-articles-and-tools/abbediagramm.html?us-origin=en-US#. 36. R. A. Chipman, Mueller Matrices, in Handbook of Optics, M. Bass, ed. (McGraw-Hill Education, 2010). 37. I. H. Malitson, “Interspecimen comparison of the refractive index of fused silica,” J. Opt. Soc. Am. 55, 1205– 1209 (1965).


Introduction
Conventional imaging only exploits the direct (line of sight) photons emitted or reflected from an object. In a typical scene, light from an illumination source is reflected from an object of interest to the camera, which measures the direct, single-bounce photons. The camera provides valuable information about the shape, color, and location of the object of interest as long as that the object is located within the field-of-view of the camera lens. Here, we consider the problem of extracting information from indirect photons, which are the photons that undergo multiple bounces and follow an indirect pathway to the camera. Examples of indirect photons include photons that bounce off an object of interest and then reflect on a wall or other object before reaching the camera. Such reflections can be specular, diffuse, or both. Accurate separation of indirect photons that come from multiple pathways, if possible, can provide valuable information of hidden objects that are outside the field of view and thus considered invisible in conventional imaging techniques.
Depending on the available prior information, separation of direct and indirect photons can be performed with little, partial, or full information of the object of interest and reflected surfaces with decreasing difficulty. When there is little to no prior information available, blind source separation methods such as the independent component analysis (ICA) [1] are typically employed for separation. ICA has been successfully applied to audio signal separation in the so-called 'cocktail party' problem [2][3][4], image separation problems such as astrophysical component separation [5], electroencephalographic data collection [6] as well as for feature extraction and noise removal. Robustness of blind source separation, such as ICA, is highly dependent on the correlation of the underlying components, and therefore, its performance is limited by the inherent noise in the different components [7,8]. Other relevant separation approaches include user-assisted separation [9], reflection removal using ghosting cues [10], and machine learning [11] methods, which require a training image library. On the other hand, when partial prior information is available, the fidelity of separation can often be improved by exploiting a physical model of the scene [12,13], thereby reducing the sensitivity to correlation of the underlying components [7]. Existing approaches make use of the polarization effect on the reflecting light and assume a prior knowledge of the medium of the reflection surface to calculate the reflectance and transmittance. This knowledge includes the surface being flat and uniform and statistical independence of transmitting and reflecting object [12][13][14]. The surface flatness and uniformity requirements are typically satisfied in scenarios like a reflection from a window or floor, while the requirement on statistical independence cannot be generally applied to all scenes, for example, the separation of objects that are similar in shape and intensity [8].
In this work, we demonstrate a robust physics-based model approach that exploits spectral and polarization information to achieve real-time, pixel-wise image separation of direct and indirect photons for indoor and outdoor scenes. Polarization information tends to be largely uncorrelated from intensity, spectral, and coherence information, and it has been commonly employed in remote sensing applications where surface features like material, roughness, and shading play a more important role [15]. Direct measurement of polarization images, which can include all or partial components of the Stokes vector, can be measured in real-time by using an imaging polarimeter [16][17][18]. Advantages of our technique include (1) good separation for a variety of scenes, (2) generalization to various angles and surfaces by patchwise analysis, (3) separation of both black and white and color images, (4) applicability to reflection from transparent or opaque semi-glossy objects, and (5) ability to estimate surface reflection angle, i.e. 3D information about the reflector. The scene with object T and object R. (c) Patch-wise separation of an outdoor scene. In (b), φ is incident angle, and θ is the polarization orientation. In c, sc d is the distance from the camera to the plane of incidence (the window), and cam h is the height of the camera.

Transmission and reflection on a double-surfaced window
The bidirectional reflectance distribution function (BRDF) describes the reflectance of a surface for any given incoming and outgoing angle. This function can be decomposed into three components: specular reflection (SR), directional diffuse reflection (DDR), and uniform diffuse reflection (UDR) [19,20]. Among the three components, SR can be explicitly modeled with different polarization components described by the Fresnel equations; UDR is Lambertian and therefore, unpolarized. At the outgoing angle of SR, the DDR component is equivalent to the SR component, which is attenuated solely in its intensity (Appendix) and consequently indistinguishable using polarization information. Without loss of generality, we consider the BRDF is comprised of only the SR and UDR components [ Fig. 1(a)]. We consider a scene with a thin transparent medium and two objects: object T behind the medium and object R in front of the medium [ Fig. 1(b)]. We assume that the light reflected from both objects arises from UDR and is therefore not polarized. To quantitatively characterize the polarization property of light, we employ the Stokes vector representation of an optical field. The Stokes vector of light from object T and object R is defined as:

Polarimetric imaging and separation of light
We calculate the Stokes vectors of the direct and the indirect optical beams that reach the camera using Mueller calculus. In this calculation, we assume the refractive index of the surface, n, is known a priori so that , s p T and , s p R are functions of the incident angle alone. Inaccuracy in the knowledge of the refractive index value adds a perturbation to the separation results (Appendix). As the Stokes vectors of the beams leaving the medium are defined in the local coordinate on the medium, which is not necessarily aligned with the local coordinate on the camera, a general two-dimensional (2D) rotation matrix is inserted for coordinate transformation from the medium to the camera: where θ denotes the counterclockwise rotation angle. cam S is then a function of the , θ φ , window transmission and reflection, and the Stokes parameters of object T and R [ Fig. 2 To back-calculate the Stokes parameters of object T and R [ Fig. 2(b)], we define two diattenuation terms for window transmission and window reflection [12]: Tr Re S φ . This is because the computation of the intensity of object T (object R) includes the latter fraction term as the intensity to be removed from 0 cam S , and this term is essentially related only to the intensity of object R (object T) per Eq. (4). If this magnitude of the intensity to be removed is inaccurate, the 'other' object becomes visually observable in the recovered image of the object of interest and is referred to as a residual.

Two metric functions of incident angle estimation
To estimate the incidence angle φ required for image separation, we define a metric to quantify the cross correlation between the separated images 0 ( ) Tr S φ and 0 ( ) Fig. 2(b)]. In most real-world scenarios, object T and object R are unrelated to each other, so φ can be estimated by minimizing the cross correlation between the two images. One commonly employed metric for incidence angle estimation and image separation is the mutual information (MI). While the cross correlation metric is based on a pixel-wise comparison of images that capture different sub-regions of the same object [21, 22] and has been applied to estimate image shifts and distortion, the MI metric is based on image statistics and is applied to evaluate two overlapping grayscale images [12]. The MI metric evaluates the statistical similarity (or distance) between two statistical distributions (histograms) defined on the intensities of the two objects. A second metric function, which we formulate in this work, is a variation of the cross-correlation metric and is defined based on the edge map of the two separated images instead of the intensity images. We are motivated to employ image edge information, for it is well-known in both image analysis and image understanding that edge information encodes significant symbolic information about the scene. For example, for a band limited image which is irreducible as polynomial, it is possible to fully reconstruct the image only given the edge information [23]. We define the edge overlap (EO) cross correlation metric, evaluated on each of the RGB color channels separately, to equal the number of pixels at the same location where both images have an edge present. For the edge detection itself, we apply the Canny edge detector [24-27] with an optimized threshold that minimizes the number of false edges. The mathematical definition of the two aforementioned metrics functions is as follows: where P in ( ) , which is then used to recover the separate image component for each color. Next, we examine the consistency of the incident angle estimate across the three RGB color channels. For simplicity, we chose an artificial limit of 10° as maximum incident angle difference across RGB channels. If the incident angle estimates, , EO MI φ , across all three channels are within 10° of each other, the three values are averaged to yield a single final incident angle estimate; otherwise, the incident angle estimation is rendered inconsistent and unsuccessful for the given metric.

Patch-wise separation and physical calculation of the incident angle
In a scene where the size of object R is comparable to its distance to the camera, variation of incident angle across the surface of the medium needs to be considered [13]. Assuming the surface is smooth, φ is a continuous and smooth function of the positions on the surface , phys phys x y and therefore, can be modeled as approximately constant over a small region. We thus divide the surface of the medium into multiple non-overlapping patches, each with a slightly different incident angle as shown in Fig. 1(c). For each patch, we estimate the local incident angle and perform the image separation. To evaluate the fidelity of the incident angle estimate, we calculated the incident angle for each patch based on the physical measurement of various distances and size of objects (e.g. glass window pane) comprising the scene. This patch-wise separation can be applied to analysis of both grayscale images and color images, often with increasing accuracy in the latter case. In the scenario where different parts of the object have significant spectral variations, errors can result from over/under exposure and low signal-to-noise ratio in one or more spectral channels. Such errors can be reduced or eliminated by selecting the same area in a complementary color channel or by averaging estimates from different color channels.
To calculate the incident angle on each patch using the measured distances in the scene ( ,  In mutual information, the objects are the same, and the comparison of their histograms is shown.

Analysis of signal-to-noise ratio of reflected light having both diffuse and specular components
Reflection from a flat surface, such as a mirror, obeys the law of reflection, where the angle of incidence is equal to the angle of reflection.
As SNR is a function of the refractive index of the reflector and the ratio / Alternatively, reflectors with a higher scattering have a lower SNR.

Experimental setup
For indoor scenes in the visible spectrum, as shown in Fig. 3, the transmission and reflection objects were illuminated with an incandescent lamp (5500K, 900 lumens) and a picoprojector, respectively. To avoid saturation in the measurement, illumination sources were not part of the scene. The pico-projector, Model AAXA P2 Jr., was operated under low brightness mode to project a 5500 K uniform white image. For outdoor scenes, the illumination sources were the sun, as shown in Fig. 4, and the fluorescent lamps inside the building, as shown in Fig. 5. Static images in the visible spectrum were acquired using a Sony DSLR-A350 camera with a Sigma 17-35 mm aspherical lens. A linear polarizer (HOYA 72 mm linear polarizing filter) and a circular polarizer (for right-handed circular polarization: HOYA 72 mm circular polarizing filter; for left-handed circular polarization: B + W 72 mm HTC KSM circular polarizer) were placed in front of the camera and rotated manually to measure different polarization states. Static images in the long wave infrared spectrum were taken using a Seek Thermal CompactPRO thermal camera. A ZnSe based linear polarizer (THORLABS WP50H-Z holographic wire grid polarizer) was used in front of the camera and was rotated manually for different measurements. The glass cover in Fig. 6 was heated with an infrared light bulb (RubyLux NIR-A infrared bulb). Object R was illuminated with a StudioPRO standard 45W photo fluorescent spiral daylight light bulb.
Videos in near infrared were taken using a customized IMPERX ICL-B1620W-KC000 CCD camera with combination of a zoom lens (Computar H6Z0812 C-Mount) and a close-up lens (Vivitar 49 mm close-up lens). A 760 nm bandpass filter and a micro polarizer array of 2 2 × elliptical micro polarizers [29] were mounted in front of the CCD sensor to form a full-Stokes division of focal plane imaging polarimeter [30]. Resolution of the polarimeter was 1608 1208 14 × × bit, and the frame rate was 12 frames per second. Video in the visible spectrum was taken by a commercial PolarCam camera from 4D Technology using the same lens system as the near infrared. This division of focal plane imaging polarimeter utilized linear wire grid micro-polarizer and had a resolution of 2400 1800 16 × × bit and a frame rate of 18 frames per second. The calibration of the cameras and computation of the Stokes images are described in reference [30].

Separation results in the visible spectrum
We tested our proposed photon separation method both indoors and outside. For the indoor scene, we considered a glass from a common photo frame as the reflection (indirect)/transmission (direct) surface. Object T and object R were illuminated individually. We assume that the glass is BK7 material ( 1.52 n = ). While the dispersion effect between RGB channels can be included, our calculation indicates that the reflectance change is smaller than 3.5% in the visible range (400 -700 nm). The measured incident angle is 61.9°. Minimizing ( ) EO φ yields incident angles of 62°, 67° and 66° for RGB channels, while minimizing ( ) MI φ leads to incident angle estimates of 42°, 38° and 35° for RGB channels respectively. Residual exists in separations using both metrics, especially in separated object R image. We attribute the residual in part to the underestimation of the refractive index (Appendix). Next, we quantified the fidelity of the separated images using reference individual images of the objects T and R that were taken separately. The individual image of object T (object R) was taken with the two illumination sources on and a black scattering cardboard inserted between object R (object T) and the glass window. We employed the median absolute deviation (MAD) metric for normalized images to quantify the fidelity of the separated images. Compared with the mean squared error (MSE), MAD is more resilient to outliers in an image and thus is more robust.  Table 1 and show that separation using EO metric outperforms the MI metric. We then replaced the glass surface by a glossy printer paper for an indoor diffuse surface separation experiment. For this experiment, the scene was illuminated with a single source. This results in a slightly polarized incident light onto the coating-paper surface; nevertheless, our technique provides good separation of object R. Coated paper has an effective refractive index around 1. Both metrics yield incident angle estimates that are close to the measured value, and yet the saturation of colors in the separated object R image from the MI method is higher than that of the real object. To evaluate the separated image fidelity, we inserted a cardboard between the color blocks and the glossy paper and took the individual image of the surface as the reference image. Under this configuration, the paper was weakly yet uniformly illuminated by the ambient light. The MAD image fidelity metric for object T shows that minimizing the ( ) EO φ metric gives a result that is closer to ground truth (Table 1). Our outdoor scene is centered on large glass window panes of a library situated next to a cluster of trees. Because the window panes are the only region of interest, the raw image is masked to eliminate other parts of the scene and thus, it is not included in the metric evaluation [ Fig. 4(a)]. As the size of the trees is comparable to their-distance to the camera, we employ a patch-wise separation approach. Note that the accuracy of incident angle estimate reduces for patches that contain the masked area due to fewer sampling points. For each patch, we compute the incident angle estimate using the same approach as the indoor scene separation. We fit a second order polynomial to the incident angle estimates of different patches and obtain a smooth map of incident angles across the entire scene. As shown in Fig.  4(f), a shift in color can be observed in the separation results obtained by minimizing ( )  Fig. 4(d)]. Furthermore, compared with non-patch-wise separation approach, the incident angles estimated using the EO metric are closer to the actual incident angles from physical measurements; the average incident angle deviation using patch-wise separation is 5.1°, while that of non-patch-wise separation is 8.6°. Thus, the patch-wise separation provides a more accurate representation of the outdoor scene.   Table 1 summarizes the MAD in various scenes in the experiments. All the reference images were taken under the same conditions (ISO, shutter speed, f-number, etc.) as the overlapping images. A smaller MAD corresponds to a higher fidelity in the recovered image. In all four experiments, the EO method demonstrates a smaller MAD and is considered a more truthful image separation method.

Separation results in the long-wave infrared spectrum
We also performed an experiment in the long-wave infrared (LWIR) spectral band, where the object acts as the source of radiation. The scene consists of a glass plate in front of a papermade 'OSC' pattern as the object T and an incandescent light bulb as the object R. The paper is heated by another incandescent bulb placed behind the glass plate, providing a thermal patterned background. The goal of this experiment is to separate the thermal pattern of the object R from the background thermal pattern of the object T. We adjusted the glass plate orientation to setup three different incident angles at 48°, 60° and 66°. The actual angles meas φ are measured from the scene and compared with the angles estimated using the EO method, EO φ , and MI methods, MI φ [ Fig. 6(a)]. In all three angles, the EO method provides a closer estimation of φ than the MI method. Incidentally, the MI method gives a close estimation in the 60° scene. A plot of MI versus φ shows that the curve is flat for a large range [ Fig. 6(c)], and thus the method is sensitive to noise. As shown in Fig. 6(d), an incorrect estimation of the angle of incident can lead to deterioration in the quality of the separated image. When the incident angle is 48° and the MI method yields a rather poor underestimation of 24 MI φ =°, the residual is readily seen in the separated object R image. When the incident angle is 66° and the MI method leads to an overestimation of 80 MI φ =°, the overlapping area is dark and the object R can be seen in the separated object T image. This is consistent with the analysis section for an overestimated incident angle (Appendix). For glossy surfaces in the LWIR spectrum, the separation experiments were performed using two types of reflector surface, a glossy paper surface and a white paint surface on dry wall (BEHR Premium Plus Ultra Pure White Eggshell Zero VOC Interior Paint) in the LWIR spectrum. The surfaces were chosen for their different diffuse and specular reflection components. The object R consists of a halogen lamp behind a paper mask in the shape of the letter A, the logo for the University of Arizona, and the incident angle is fixed at 76°.  A set of indoor and outdoor videos in visible and near infrared spectral bands were also analyzed (Visualization 1, Visualization 2, Visualization 3, and Visualization 4). These videos, including the original and the separated scenes, are available in the supplementary materials. Although the analysis is not performed in real time, the results demonstrate that real time acquisition and separation of overlapping scenes is possible and that our proposed separation technique can be applied to visible and near infrared spectrum.

Conclusion
The accuracy of image separation is affected primarily by the polarization of the light reflected from the original objects, the accuracy of the refractive index (medium) and the incident angle estimates, and the BRDF of the transparent or semi-glossy reflector, i.e. strength of the UDR component due to diffuse surface scattering. A detailed analysis of the image separation fidelity with respect to refractive index and incident angle is described in Appendix. The assumption that the light from the two objects is unpolarized does not hold under all circumstances, for example, when the object surface is optically smooth, which gives rise to strong SR, i.e. glass and polished ceramics, or when the light source itself is polarized, i.e. computer screens. Nevertheless, our technique can be generalized to arbitrary polarization, if we can independently estimate the polarization of the light coming from the objects.
An inaccurate refractive index estimate can lead to inaccuracy in , ( ) s p R φ and subsequently to artifacts (residuals and shifts in color) in the separated images. However, we show that for materials commonly-used in daily life, indices range from 1.3 n = [31] (precipitated calcium carbonate, PCC) to 1.79 n = [35] (SF11 glass). This inaccuracy does not affect the separated object T image significantly, and it has no effect when B φ φ = , the Brewster angle. In addition, minimization of the same metric by treating the value of the refractive index as an optimization variable may further improve the quality of the object R image. Meanwhile, inaccuracies in incident angle estimate can also lead to artifacts in the separated images. The diattenuation term ( ) Tr D φ is a monotonically decreasing function of the incident angle. Therefore, when recovering the object R image, either more intensity is removed if the incident angle is underestimated or less intensity is removed if the incident angle is overestimated. This conclusion applies to object T, image separation, when both the estimated incident angle and the actual incident angle are below B φ , while more (less) intensity that is removed corresponds to an overestimated (underestimated) incident angle when both the estimated and the actual incident angles are above B φ . Therefore, compared with the MI metric based separation method, the EO metric typically reduces the artifacts in the separated images with a more accurate estimation of φ .
In conclusion, our separation technique is robust and can be applied in real-time in both indoor and outdoor environments. Objects that are not in the field of view can be imaged and measured in visible, near infrared and long wave infrared spectrum. The ideal choice of the spectrum is determined by the properties of the reflecting surface, and the ideal wavelength range corresponds to where the SNR of light received from the reflector is the highest.

A1 Justification of directional diffuse reflection as attenuated specular reflection
In this section, we show that for a given reflection angle r φ , the polarization effect of the directional diffuse reflection (DDR) component is equal to a specular reflection (SR) component that is attenuated in the intensity, that is, the Stokes vector of DDR ( DDR S ) is equal to the Stokes vector of SR ( SR S ) scaled by a real scalar. We describe how the reflectance of DDR is related to the Fresnel coefficients, and how DDR S is related to SR S . Our theoretical framework is identical to that in the reference [19]. Relevant equations are reproduced here for completeness. Figure 8 shows a schematic to facilitate understanding of the notations used in the derivation.  In addition, as 0 S is generally greater than 0, the incident Stokes vector can be written as where the polarization effect is embedded in the normalized Stokes vector ' s , defined as: where T refers to transpose operation. Thus, the normalized Stokes vector of the reflected light ′ s is a function of the ratio of reflectance ratio r and the normalized Stokes vector of incoming light. Two identical normalized Stokes vectors represent the same polarization state on the Poincare sphere.
In general, the reflectance has contributions from both DDR and SR components of the BRDF. In the BRDF theory, the bidirectional reflectivity (BR), denoted by , s p ρ , characterizes the reflecting powers for the s and p modes respectively. The reflectance ratio r is related to the bidirectional reflectivity as / s p r ρ ρ = .
We consider first the BR of SR [19],