Residual interpolation for division of focal plane polarization image sensors

Division of focal plane (DoFP) polarization image sensors capture polarization properties of light at every imaging frame. However, these imaging sensors capture only partial polarization information, resulting in reduced spatial resolution output and a varying instantaneous field of overview (IFoV). Interpolation methods are used to reduce the drawbacks and recover the missing polarization information. In this paper, we propose residual interpolation as an alternative to normal interpolation for division of focal plane polarization image sensors, where the residual is the difference between an observed and a tentatively estimated pixel value. Our results validate that our proposed algorithm using residual interpolation can give state-of-the-art performance over several previously published interpolation methods, namely bilinear, bicubic, spline and gradient-based interpolation. Visual image evaluation as well as mean square error analysis is applied to test images. For an outdoor polarized image of a car, residual interpolation has less mean square error and better visual evaluation results. © 2017 Optical Society of America OCIS codes: (260.5430) Polarization; (230.5440) Polarization-selective devices; (110.5405) Polarimetric imaging. References and links 1. N. M. Garcia, I. de Erausquin, C. Edmiston, and V. Gruev, “Surface normal reconstruction using circularly polarized light,” Opt. Express 23(11), 14391–14406 (2015). 2. D. Miyazaki, T. Shigetomi, M. Baba, R. Furukawa, S. Hiura, and N. Asada, “Surface normal estimation of black specular objects from multiview polarization images,” Opt. Eng. 56(4), 041303 (2016). 3. H. Zhan and D. G. Voelz, “Modified polarimetric bidirectional reflectance distribution function with diffuse scattering: surface parameter estimation,” Opt. Eng. 55(12), 123103 (2016). 4. V. Thilak, D. G. Voelz, and C. D. Creusere, “Polarization-based index of refraction and reflection angle estimation for remote sensing applications,” Appl. Opt. 46(30), 7527–7536 (2007). 5. B. Shen, P. Wang, R. Polson, and R. Menon, “Ultra-high-efficiency metamaterial polarizer,” Optica 1(5), 356– 360 (2014). 6. P. Terrier, V. Devlaminck, and J. M. Charbois, “Segmentation of rough surfaces using a polarization imaging system,” J. Opt. Soc. Am. A 25(2), 423–430 (2008). 7. O. Morel, R. Seulin, and D. Fofi, “Handy method to calibrate division-of-amplitude polarimeters for the first three Stokes parameters,” Opt. Express 24(12), 13634–13646 (2016). 8. M. W. Hyde 4th, J. D. Schmidt, M. J. Havrilla, and S. C. Cain, “Enhanced material classification using turbulence-degraded polarimetric imagery,” Opt. Lett. 35(21), 3601–3603 (2010). 9. S. Alali and A. Vitkin, “Polarized light imaging in biomedicine: emerging Mueller matrix methodologies for bulk tissue assessment,” J. Biomed. Opt. 20(6), 061104 (2015). 10. T. York, S. B. Powell, S. Gao, L. Kahan, T. Charanya, D. Saha, N. W. Roberts, T. W. Cronin, J. Marshall, S. Achilefu, S. P. Lake, B. Raman, and V. Gruev, “Bioinspired polarization imaging sensors: from circuits and optics to signal processing algorithms and biomedical applications: analysis at the focal plane emulates nature’s method in sensors to image and diagnose with polarized light,” Proc IEEE Inst Electr Electron Eng 102(10), 1450–1469 (2014). 11. N. W. Roberts, M. J. How, M. L. Porter, S. E. Temple, R. L. Caldwell, S. B. Powell, V. Gruev, N. J. Marshall, and T. W. Cronin, “Animal polarization imaging and implications for optical processing,” Proc. IEEE 102(10), 1427–1434 (2014). #286440 https://doi.org/10.1364/OE.25.010651 Journal © 2017 Received 14 Feb 2017; revised 11 Apr 2017; accepted 18 Apr 2017; published 28 Apr 2017 Vol. 25, No. 9 | 1 May 2017 | OPTICS EXPRESS 10651


Background
The vital physical parameters of light are intensity (I), wavelength (λ), and polarization (Vector E).In the past, polarization has been ignored by imaging technology, as the human eye is insensitive to the polarization factor of light.Polarization information provides orthogonal information compared to intensity and color, and it captures information about the target 3-D surface normals [1][2][3][4], material composition, roughness and ultra-high efficiency metamaterial polarizers [5][6][7][8].In bioengineering research, polarization imaging is used to discriminate healthy from diseased tissue without the use of molecular markers [9][10][11].
Various techniques and instruments have been developed to record the polarization parameters of light [12].With developments in nanofabrication technology, compact, inexpensive and high-resolution polarization sensors called division of focal plane (DoFP) polarization image sensors have been realized [13][14][15][16][17][18].These developments in nanofabrication and nanomaterials allow for fabrication of pixelated nanowire filters on the top surface of the imaging sensor and help realize robust DoFP polarization imaging sensors.The imaging elements, i.e., photodetectors and micro polarization filter arrays, are included on the same substrate as the DoFP image sensor.The main advantage of DoFP image sensors over division of time (DoT) sensors is their capability of seizing polarization information at each frame and avoiding incorrect polarization information in moving targets [19].DoFP sensors integrate pixelated polarization filters with an array of imaging elements and are organized in a super-pixel [20,21] configuration containing four distinct pixelated polarization filters with transmission axes oriented at 0°, 90°, 45° and 135°, respectively (see Fig. 1).The super pixel holds all the required information to obtain a useful polarized image, recording the first three (S 0 , S 1 , S 2 ) or four Stokes (S 3 ) parameters at every frame [22].
The image obtained from a DoFP sensor has a lower accuracy of polarization information as each individual pixel within the super pixel has a slightly different field of view.To reconstruct the polarization information, missing pixel information is estimated across the imaging array [23,24].Normally DoFP polarization sensors lose spatial resolution and capture erroneous polarization information [22,23,25,26].Due to the four spatially distributed pixelated polarization filters, the instantaneous fields of view for the neighboring pixels in a super-pixel configuration can be different from each other [24,[27][28][29][30].Therefore, the first three Stokes parameters (S 0 , S 1 , S 2 ), angle of linear polarization (AoP) and degree of linear polarization (DoLP) will contain error and are different from the true polarization component.Such edge artifacts can be easily observed in AoP and DoLP images.These drawbacks need to be resolved to obtain the real-time advantage of DoFP image sensors.
The polarization-imaging sensor shares many similarities with color imaging using a Bayer color filter array [31].The 2 × 2 super pixel of a color filter array consists of three wavelength channels of red, green and blue.The blue and red colors are down-sampled by a factor of 25% each, while green is down-sampled by a factor of 50%.As the color filters are placed on the imaging element array, spatial resolution is reduced in the different color channels during the above-mentioned down-sampling.Since the sensor for each channel only perceives partial information, interpolation algorithms are used to recover the lost spatial resolution with minimal artifacts.
In color image demosaicking, the G image is first interpolated, and then the tentatively estimated R image ( R ) is generated.The residuals are created between observed and tentatively estimated R pixels (R-R ) at the R pixels.Then the interpolated residuals are added to the tentatively estimated R to get the interpolated image [32][33][34].The interpolation techniques used for a color filter array cannot be directly employed on the polarization domain due to the essential difference between these two modalities.We have borrowed the tentative estimation of pixels from the residual interpolation technique used for a color filter array.In DoFP, we apply the residual interpolation method to the '4' polarized images separately before calculating the DoLP and AoP.
Interpolation methods are applied to recover some of the lost spatial resolution and improve the accuracy of the captured polarization information.The following methods have been traditionally used to interpolate polarization information: bilinear, bicubic, spline and gradient-based methods [22,23,[25][26][27][28].For each method, four polarization-filtered images are required to obtain the necessary polarization information, such as Stokes parameters and angle and degree of linear polarization.The bilinear, bicubic, and spline methods are essentially low-pass filters, which smooth out the intensity information obtained by the four polarization-filtered images and create saw tooth artifacts at the edges.In the case of multiple-object images against a background, their continuity for low-resolution images fails and false polarization signatures are generated.In the gradient-based interpolation technique, interleaved gradients are used, and these will introduce nonconformities due to the instantaneous field of overview (IFoV).However, the errors can be reduced clearly if a proper interpolation technique is used to reduce incorrect polarization information.Therefore, a novel residual interpolation method with edge preservation and interpolation of residuals obtained by tentatively estimating and observing pixel difference is developed to provide more accuracy.In this paper, we propose residual interpolation for division of focal plane imaging sensors, where the interpolation is executed in a "residual" domain.We interpolated lowresolution polarized images, generated tentative estimates of 0°, 45°, 90° and 135° ( 0 45 90 135 , , , q q q q ) and calculated their residuals, which are the differences between the observed and the tentatively estimated pixel values (i.e., 0° -0 i q ,45° -45 , i q 90° -90 , i q and 135° -135 i q ).We used a guided filter for edge preservation and to accurately generate the tentative estimates of the pixel values [35].The advantage of the guided filter is that its computing time is independent of filter size.The performance of the residual interpolation method is compared with several previously published interpolation methods, the bilinear, bicubic, spline and gradient-based methods.Based on the results, it is clear that residual interpolation outperforms the others in terms of both mean square error and visual evaluation.

Linear polarization imaging calculations
A DoFP imaging sensor captures both the intensity and polarization information of a scene.The sensor samples the scene through 0°, 45°, 90° and 135° polarization filters, and registers the four sub-sampled images.The intensity and polarization are then worked out from images with 0°, 45°, 90° and 135° linear polarization filters.To observe polarization, two polarization properties are of most interest, DoLP, and AoP.The intensity, polarization difference, DoLP and AoP are computed via the following equations:

Residual interpolation
In this section, the bilinear interpolation method is first briefly reviewed, followed by an overview of the proposed residual interpolation method.We used bilinear interpolation due to its low computational complexity.The basic principle of bilinear interpolation is to estimate the pixel values in two dimensions.The distance weighted average of the four nearest pixel values is used to estimate a new pixel value.Based on the four neighboring pixel points (see Fig. 2), f (i, j), f (i + 1, j), f (i, j + 1) and f (i + 1, j + 1), of the interpolating point f (x, y), the mathematical formula for bilinear interpolation can be written as follows: 1, 1 . . 1, . . 1 .
We can calculate the pixel (2, 2) at 90° polarization orientation by bilinear interpolation for a 4 X 4 block as follows [22] (see Fig. 3): We used a guided filter for edge-preserving smoothing of the images taken at 0 , 45 , 90 and1 35 ).The filter output ' i q ' is a linear transform of the guided image in window k w at pixel k, and this model is applied on all four images: 0 , .
Similarly, this model can be applied on 45 , 90 and1  for and .
The guided filter provides the tentatively estimated pixel values for each 4-polarizer filter array ( 0 , 45 , 90 , 1 35   o o o o ).The residuals (∆) can be calculated from the original and guided filter output pixel as follows: I i j I i j qi i j I i j I i j qi i j I i j I i j qi i j I i j I i j qi i j The residuals (∆) can be further interpolated and then added to the tentatively obtained pixel values.The ∆ int I for 0 , 45 , 90 , and1 35 The difference between the residual and bilinear interpolations is that bilinear estimates the new pixel from the four nearest neighbors and residual interpolates, using bilinear interpolation, the residuals obtained from the tentatively estimated and observed pixel values.The interpolated residuals are added to the tentatively estimated pixel values to get the net residual interpolation.
In Fig. 5, the flow chart of residual interpolation is presented.First, the low-resolution polarization images are up-sampled using bilinear interpolation to generate high-resolution images.With the guided-filter, the proposed algorithm can up sample the sparse data by using the above-mentioned interpolated images and the high-resolution guided images ( 0 I 45 90 135 , , I I I ).Therefore, the image structures of the interpolated images are preserved.We generated the tentative estimation of the 0°, 45°, 90° and 135° images ( 0 45 90 135 , , , q q q q ) and calculated the residuals, as presented in Eq. ( 15).The residuals were again interpolated using bilinear interpolation, and those tentatively estimated were augmented to get the residual interpolation, as shown in Eq. ( 19).

Modulation transfer function
The modulation transfer function (MTF) of an imaging system is a measure of the contrast being transferred by the lens.The MTF measures the magnitude response of an imaging system to a varying sinusoidal pattern at different spatial frequencies.Simply, it measures how a camera can see small things [36].At a range of spatial frequency, the MTF can be calculated as the ratio of the contrast at the output to the input sinusoidal patterns.
The polarization image sensor captures the polarization information at each imaging frame.An input target image can be defined as varying sinusoidal patterns at different frequencies.We generated an artificial sinusoidal image in MATLAB at each frame, i.e, 0°, 45°, 90° and 135°, as follows [22]: where x f and y f are the spatial frequency components in the horizontal and vertical directions.All patterns were down-sampled in order to check the accuracy of the interpolation algorithms.We interpolated the down-sampled polarized images to get the high resolution image according to the bilinear, bicubic, spline, gradient and residual interpolation algorithms.The 0 and S 2 S parameters are variables based on a sine function, while 1 S is constant .We then matched our imager as to how it would sample such a signal and applied interpolation techniques.The ratio of the original sinusoidal signal to the interpolated signal gives us the MTF at one frequency.We changed the frequency, and for each frequency, we got another MTF point.In this way, at a range of frequencies swept from 0 to 0.5 cycles per pixel, the MTF curve was plotted, as shown in Fig. 6.In Fig. 6(a) to (e) the 3-D MTF chart of bilinear, bicubic, spline and gradient-based and residual interpolation is represented, showing the MTF along x f and y f .The horizontal frequency x f and vertical frequency y f were swept from 0 to 0.5 cycles per pixel.Figure 6(f) shows the MTF response of spline interpolation in cyan color, bilinear shown in green, bicubic in blue, gradient in yellow and residual interpolation in red along x f = y f .The ideal MTF, shown by the dotted purple line, has unity gain for 0 to 0.5 and zero gain at higher frequencies.All interpolation algorithms other than residual interpolation give low gain below 0.25 cycles per second and zero gain at higher frequencies.The residual interpolation has a higher gain at low frequencies beyond 0.25 cycles per pixel than the other methods.At higher frequencies, greater than 0.375, residual interpolation again provides increased gain less than 0.5 cycles per pixel as compared to other interpolation methods.

Experimental setup
To assess the accuracy of different interpolation methods, the "true" high-resolution polarization image must be known beforehand, and with the DoFP polarization imaging sensors, we can only generate low-resolution images.Four images at 0°, 45°, 90° and 135° orientations were taken of a car in an outdoor environment with a CMLN-13S2M-CS CCD camera mounted with a linear polarization filter.These true high-resolution images were down-sampled by following the sampling pattern of the DoFP polarization imaging sensor, and four low-resolution images at 0°, 45°, 90° and 135° feature angles were obtained, like those acquired from a DoFP sensor.After applying the interpolation algorithms, the final high-resolution interpolated images were compared against the true high-resolution images that were originally obtained.The images obtained at 0°, 45°, 90° and 135° orientations are grayscale images.A further four true high-resolution images of a car were captured in an outdoor environment.The intensity, DoLP and AoP images for the car are shown in Fig. 7.
Potential error in the original high-resolution images due to optical misalignment is not an experimental concern.Our concern is only to test the interpolation algorithm in terms of mean square error and visual evaluation because the algorithms are tested on low-resolution images, and any original error will be the same in both the low and high resolution images.Our setup provides a fair comparison of the reconstruction error among the bilinear, bicubic, spline and gradient based and residual interpolation methods.

Performance estimation
In this section, we adopt mean square error (MSE) and visual evaluation to compare the performance of the different interpolation algorithms.The interpolation methods are used to get high-resolution images from the low-resolution images.The interpolated images are matched with the true high-resolution image, and the characteristics of polarization are explored on the images.In Section 5.1 and 5.2 the visual image evaluation and the MSE of both test images are given, respectively.

Visual image evaluation
In Fig. 7, the intensity, DoLP and AoP images computed from the high-resolution car image are shown.The row contains the true high-resolution polarization image, and this is used to visually compare the reconstruction accuracy of the different interpolation methods presented in Fig. 8 using small patches.In the first column of Fig. 8, the original intensity, DoLP and AoP are given, while the second to the sixth columns show bilinear, bicubic, spline, gradient, and residual interpolation images, respectively.In Fig. 7, the DoLP values are lower in the red areas and higher on the car's glass windows, with the light blue spot on the glass marked with a white oval showing medium DoLP.The AoP value is low, medium, and high in the red, light blue and purple areas, respectively.
In Fig. 8, small patches of the car image are shown.In the figure, the purple ovals on the original and residual intensity images show how many artifacts have been effectively recovered.The image artifacts and glitches are effectively reduced by residual interpolation, with the interpolated images near to the original images.DoLP and AoP patches of the car show the accuracy of residual interpolation as compared to the bilinear, bicubic, gradient and spline algorithms.We used parallel programming to speed up the processing to real-time.On our system Intel (R) core (TM) i5-3470 CPU @ 3.20GHz, 8-Gb RAM, bilinear interpolation computes the AoP image (960 x1280) in 40 milliseconds, bicubic in 47 milliseconds, gradient in 45 milliseconds, spline in 57 milliseconds and residual interpolation in 61 milliseconds.The most important is that in terms of the polarization information recovery, mean square error, visual evaluation and MTF, the residual interpolation performance is significantly better than other interpolation methods.

MSE comparison
The MSE for the different interpolation algorithms is found using the following equation: where img O (i, j) is the actual target pixel, img i (i, j) is the interpolated actual intensity image and M and N are the number of rows and columns in the image array, respectively.The mean square error results for the different interpolation methods are shown in Table 1 for the car image.The minimum MSE for the I(0°), I(45°), I(90°), I(135°), intensity, DoLP and AoP images is obtained via the residual interpolation method.The spline interpolation method introduces the largest error, while the bicubic and gradient interpolation methods show similar error performance, with the latter being computationally efficient.

Conclusion
In this paper, we proposed the residual interpolation algorithm for a division of focal plane image sensor.We have compared the structure of the gradient, bilinear, bicubic and spline interpolation algorithms with residual interpolation.The performance was compared visually, by modulation transfer function (MTF) and with a MSE matrix through a CCD camera and a linear polarization filter turned around the sensor.The interpolation algorithms were applied on low-resolution images and compared statistically against the true high-resolution polarization images.We applied the algorithms on intensity (S 0 ), angle of linear polarization (AoP) and degree of linear polarization (DoLP) to observe the accuracy of edge recovery and polarization information.The improvements in the reconstruction accuracy using the proposed residual interpolation method were shown both with the MSE and visually in comparison with the bilinear, bicubic, spline and gradient-based algorithms.This demonstrates that residual interpolation could bring a large improvement to the output quality in terms of edge artifacts for a real DoFP polarization image sensor.Most importantly, the residual method outperforms and shows its advantage over other leading methods.
filter has been used to find the Stokes parameters; however, the Stokes parameter ( 3 S ) is not captured for the DoFP sensor shown in Fig.1.The above equations show that a polarization imaging sensor has to sample the images with four linear polarization filters offset by 45°[29].
guided filter is a linear model between high-resolution guided images ( 0 I , 45 90 135 , , I I I )and filtered images (

35
k b are the linear coefficients constant in window k w .These coefficients can be determined by minimizing the cost function in window 0 0

and 2 kσ
13) Similarly, cost functions E45, E90 and E135 can be calculated for are the mean and variance of 0 I in k w , w is the number of pixels and 0 k Int is the mean of0 int I Similarly, k ka and b values can be calculated for E45, E90 and E135, respectively.The filter output for all the tentatively estimated images of the four polarizers can be found as follows:

Fig. 5 .
Fig. 5. Flow chart for residual interpolation.The net residual interpolation is the addition of each pixel value of the residuals (∆), and the tentative estimates for each polarized image are shown in Fig.4(b).This can be represented as follows:

Fig. 8 .
Fig. 8.The true high-resolution image and comparison of interpolation methods on the (a) intensity, (b) DoLP, and (c)AoP, showing the effect of interpolation on the artifacts in the patches.