Separation and contrast enhancement of overlapping cast shadow components using polarization

Shadow is an inseparable aspect of all natural scenes. When there are multiple light sources or multiple reflections several different shadows may overlap at the same location and create complicated patterns. Shadows are a potentially good source of information about a scene if the shadow regions can be properly identified and segmented. However, shadow region identification and segmentation is a difficult task and improperly identified shadows often interfere with machine vision tasks like object recognition and tracking. We propose here a new shadow separation and contrast enhancement method based on the polarization of light. Polarization information of the scene captured by our polarization-sensitive camera is shown to separate shadows from different light sources effectively. Such shadow separation is almost impossible to realize with conventional, polarization-insensitive imaging. Comments Copyright 2006 Optical Society of America, Inc. Postprint version. Published in Optics Express, Volume 14, Issue 16, August 2006, pages 7099-7108. (The manuscript can be found at http://www.opticsexpress.org/abstract.cfm?id=96192) This journal article is available at ScholarlyCommons: http://repository.upenn.edu/ese_papers/262 Separation and contrast enhancement of overlapping cast shadow components using polarization Shih-Schön Lin, Konstantin M. Yemelyanov Electrical and Systems Engineering Department, University of Pennsylvania, 220 South 33 Street Moore 203 Philadelphia, PA 19104-6390, USA shschon@seas.upenn.edu, kostya@ee.upenn.edu.


Introduction
Shadows are formed whenever an occlusion partially blocks the illumination of a surface or object by a light source. With the exception of the ambient light, which is assumed to be omnidirectional, light sources illuminate surfaces from only one specific direction. In addition to classification by the source direction, shadows are further classified into "self" and "cast". A "self" shadow refers to the regions of an object not directly illuminated by a light source due to its surface orientation, whereas "cast" shadow refers to a region not illuminated by a source due to occlusion by other objects. Shadowed regions usually appear darker than the lit regions and their color properties (e.g., hue and saturation) can also appear different than the directly illuminated regions. Such differences in intensity and color create patterns and boundaries/edges that often confuse human observers or machine vision algorithms which attempt to segment scenes and identify objects using these cues. For this reason many techniques have been developed to identify, segment, and remove shadows from an image or a video sequence [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19]. However, all previously published methods use only two aspects of light, its intensity and/or spectral ("color") distribution as information in shadow segmentation, though; in some cases these are combined with temporal and geometric information available. It appears that a third fundamental property of light, its polarization, has not heretofore been used for the purpose of shadow segmentation. Furthermore, most existing shadow segmentation algorithms assume a relatively simple shadow model: an area of a scene is classified either as shadow or non-shadow. In fact it is possible for a specific region of a scene to be both shadow for one source and illuminated simultaneously by another source or sources, as explained below, and polarization information can assist in "parsing" such complications in scene segmentation.
Polarization is an intrinsic property of light. Light from the dominant natural source, the sun, is not polarized but light scattered from small particles in the sky and most light reflected or scattered from object surfaces is partially polarized. The unaided human eye and most machine vision cameras are "blind" to polarization, but some animal species can detect and utilize polarization information and use it for a variety of purposes, including navigation and object recognition [20][21][22][23]. Inspired by biological polarization vision, our group has developed polarization sensitive cameras and processing methods for the detection of targets in scattering media, detecting latent fingerprints and enhancing surveillance [24][25][26][27]. We have also developed methods for displaying polarization information effectively to human observers [28,29]. It has been reported that polarization increase in dark surface area [30] and Duggin [31] used polarization to enhance details in shadow. Goldstein, Chenault, and Pezzaniti [32] reported that polarization increases with increasing incident light angle. In this investigation we show that complex overlapping cast shadows that are almost impossible to distinguish in images generated with only intensity and color information can be readily segmented from each other in images generated from the polarization parameters of a scene. According to the generally accepted macroscopic description of the interaction of light with a surface, reflected light can be subdivided into specular and diffuse components ( Fig. 1(a)). The ratio of energy carried by the diffuse and specular components depends on the angle of incidence and the material properties of the surface. The diffusely reflected components often undergo multiple random reflections microscopically, so statistically they tend to be unpolarized. In contrast, the specularly reflected component is usually at least partially polarized, with the polarization direction (dominant axis of E-field oscillation) parallel to the local tangent plane of the surface ( Fig. 1(b)). These physical phenomena can be formalized through appropriate application of Fresnel's analysis and equations [33].

Specular reflection Diffuse reflection
In addition to the scattering by surfaces, another important natural source of polarization is the scattering of light by the atmosphere of the earth. The polarization of sun light by the air particles can be explained by the theory of Rayleigh scattering [33], which describes the particles as electric dipoles: because oscillating dipoles do not radiate in the direction of oscillation, a polarization-sensitive observer will see the dome of the sky to exhibits polarization pattern that depends on the location of the sun. Since pioneering investigations of von Frisch it has been well established that many insects can avail themselves of this polarization for navigation [20][21][22][23]. Such polarization has consequences for the segmentation of shadows: thus, as we will show below, an area that is inside a shadow cast by direct sunlight, but which is lit by the polarized ambient sky light will show a distinctive polarization, whereas an area that is inside both the shadow cast by sunlight and the shadow cast by skylight will show no polarization at all.
Because most imaging devices integrate light energy over a time epoch that is long relative to the oscillation period (fs), phase information is not recorded. With the phase information lost, when a linear polarization analyzer is placed in front of the camera, the measured intensity I at a specific image location or pixel, as a function of the angle of orientation , of the polarization analyzer is given by where . is the orientation angle of the major axis of the polarization ellipse, I U is 50% of the total intensity at each pixel, and / A U p I I defines the degree of linear polarization at the pixel. The reference axis for the two angles , and . can be arbitrarily chosen, and complete information about the polarization state of the light can be obtained by capturing images with the polarizer oriented at three different angles, for example , =0, 45 and 90 degrees [26,34] From these three images, one can recover I U , I A , and . for each pixel of the image using the following expressions: Here indices 0, 45, and 90 indicate the orientation of the polarizer in degrees when each specific image was taken. Because . and .+0 are indistinguishable for phase-blind sensors, the meaningful range of . is restricted to 0, and . ranges from 0 to 0. In the work presented here we sample three angles (0, 45 and 90) by manually or mechanically rotating a single linear polarizer mounted in front of an intensity integrating camera. The camera used in our experiments is a calibrated Olympus E-10 digital camera with 4 Mega pixels and 10 bit pixel depth (we use the RAW mode).

Experiments
The first example is an outdoor scene of a walkway in front of a building with all-glass walls ( Fig. 2 to Fig. 4, the glass-walled building is visible in Fig. 4). Note that in order to help reader grasp the relationship between pictures in Fig. 2 to Fig. 4, we overlay a yellow circle over a sewer drainage cover that is visible in all pictures but Fig. 4 Left to call attention to readers that this is the exact same object in all the pictures. Then a green square is overlayed on the same glass door that is visible in both Fig. 4 Left and Fig. 4 Right. The sun illuminated the scene from the right hand side of the picture: shadows cast by trees are seen along the walkway, most clearly in the upper portion of the image. Most existing shadow handling algorithms would simply segment the dark areas as shadows, reducing or eliminating the contrast in brightness caused by the shadow. However there is a more complicated overlapping shadow pattern hidden inside the scene that is not detectable from analysis of the intensity distribution. In this scene the glass-wall building to the left rear side of the scene reflected sunlight from its glass panels, but not from the thinner frames around each piece of glass. The reflected light was partially polarized, and the reflection pattern cast on the scene overlapped with the shadow pattern cast by the direct sunlight. The light reflected by the glass was weaker than the direct sunlight, and the pattern it creates is essentially invisible in the "intensity-only" image at left. However, when our polarization-sensitive camera was used to extract the "degree of polarization" image, the hidden pattern of overlapping shadow was revealed (Fig. 2, right  panel). The area that was lit neither by direct sunlight, nor by the reflected light from the glass, is both dark and unpolarized, and thus appears dark in both images. These are the cast pattern of the glass frames of the glass-wall building to the left of the picture. The areas that were not lit by direct sunlight -and thus appear as shadows in the intensity-only image -but which were illuminated by the partially polarized reflected light from the glass-wall building, exhibit strong polarization. The degree-of-polarization image normalizes the polarization signal with respect to the total intensity (Eq. (2)), so these areas show up as a bright pattern in the degree-of polarization-image (Fig. 2, right). To establish that this pattern is unique to the polarization analysis, and not hidden in the intensity-only image due to low contrast in the shadow area, we performed linear contrast enhancement, followed by gamma correction of 0.5 to both images of Fig. 2: from the results (Fig. 3) it is clear that the shadow patterns are only revealed in the degree-of-polarization image. To further document the nature of the sunlight and glass wall sources to the shadows revealed by polarization analysis we provide images of the glass-wall and frames of the building (Fig. 4, left), and of the walkway when the bright direct sunlight is blocked (Fig. 4, right). Note that pictures in Fig. 4 are taken with the camera at about the same position and general view direction as when taking pictures in Fig. 2 and Fig. 3. The only difference is that in Fig. 4 the camera zooms out and points more upward in order to put the tall glass-walled building into view. In sum, the patterns revealed in the degree-of-polarization image are indeed caused by shadows created by polarized light reflected from the glass source.   Fig. 2 and Fig. 3 taken another day when the direct sun light is blocked due to nearby construction scaffolding. The shadow pattern cast on the walk way by the glass-wall and frames is visible. The yellow circle in the Right picture points out the same sewer drain cover as seen in Fig. 2 and Fig. 3. The left and right pictures in this figure can be related by the same glass door bracketed by the overlayed green rectangle. Note that pictures in Fig. 4 are taken with the camera at about the same position and general view direction as when taking pictures in Fig. 2 and Fig. 3. The only difference is that in Fig. 4 the camera zooms out and points more upward in order to put the tall glass-walled building into view.
We performed a controlled laboratory experiment to further confirm the results obtained outdoors. The setup comprises a 150W incandescent light source illuminating a scene from the direction opposite the camera, and a 15W fluorescent light illuminating the same scene from the direction corresponding to right hand side of the picture (Fig. 5). In the intensity-only image only the shadow of the knob cast by the dominant light source is visible. However, in the degree-of-polarization image, additional information is visible and separated clearly in high contrast. Specifically, a "shadow" cast by the much weaker light from the right hand side is revealed as a bright area to the left of the metal pillar. The reason that this region appears bright in the degree-of-polarization image is due to the viewing geometry: the strong light reflected from the table is highly polarized, whereas the light reflected to the camera from the side source is only weakly polarized. As a result, the area that is not illuminated by either source is very dark in the intensity-only image and is least polarized and seen as the darkest area in the degree-of-polarization image. The polarization of the image regions corresponding to areas lit by both strong and weak light sources is lessened by the unpolarized light reflected to the camera from the weak source at the right hand side of the picture. Segmentation algorithms operating on the degree-of-polarization image can readily extract the distinctive "shadow" cast by the weak source. A sample analysis (Fig. 6 left) shows a segmentation obtained by a growing algorithm that starts with 2 × 2 regions. The side shadow area is cleanly separated from the image when 21 or more regions are segmented (Fig. 6, right).

Discussion
The processing of shadows in images presents many difficulties for scene segmentation, and all existing methods for analyzing shadows based on intensity-only information have limitations. Many methods are designed for specific applications like aerial photography or traffic monitoring [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19], so that the lighting condition is simplified or known a priori. Many applications using extant methods require a specific geometry of the illumination and camera, and/or very precise calibrations of the pixel sensitivity of the camera. The use of polarization in shadow analysis and segmentation appears to be robust and certainly provides new and useful information that may facilitate segmentation and reveal new features of the scene and the sources that illuminate it. The polarization based shadow segmentation method suggested here does have its own limitations. While this method is not strictly tied to specific scene geometry, the method does not work when the scene signals happens to be unpolarized everywhere, a rare but possible scenario. Nonetheless, because signals extracted with Eq. (2) are strongest when there is specular reflection, the use of the degree-of-polarization image for segmentation can be expected to give the best results when the source is opposite and directed toward the imaging system. A valuable feature of the method presented here is that it can readily reveal the presence of multiple sources of widely varying "strength". As methods have already been developed for estimating the illumination directions of multiple light sources from information in the image [35][36][37][38], it can be anticipated that combining polarization analysis with these methods will produce valuable new tools for determining the direction of illumination sources. Investigations along these lines are underway. This use of polarization information in shadow detection, separation and contrast enhancement will also be further enhanced when it is combined with other well known cues like intensity, color, and geometry to achieve more accurate shadow segmentation and give more detailed information on the origin of distinct shadow components. While the present investigation has based its shadow-segmentation solely on degree-of-polarization information, the additional cue provided by the orientation of the local polarization ellipse ( in Eq. (1)) can also be used for image segmentation (in much the manner in which color is used), and it can also be anticipated that this will further contribute to the method. Moreover, as expected from the independence of polarization from the other physical attributes of light and demonstrated by our experiments, information extracted by polarization about shadows is unique and in general cannot be extracted from other cues alone.

Conclusion
We have presented a novel method of shadow segmentation based on the local degree of polarization in images captured by a polarization-sensitive imaging system. This analysis reveals that the polarization of light conveys distinct and valuable information about a scene that can be extracted at modest cost -three channels. Polarization have been used in many other vision tasks such as removing glare and target detection, but to the best of our knowledge has not previously been used to aid the segmentation of complex shadows in a scene. Polarization information enables our system to extract information about the complexities of multiple light sources and the overlapping shadows they create. Such shadows are very difficult even to detect in intensity-only images, and their extraction with polarization analysis provides a new means of identifying the direction and nature of light sources illuminating a scene.