Simulated validation and quantitative analysis of the blur of an integral image related to the pickup sampling effects

In an integral imaging (II) system, the pickup sampling effects play an important role in affecting the blur of an integral image. In this paper, the blur property of an integral image due to the pickup sampling artifacts is ﬁrst analyzed. Then, a ﬁgure of merit-the edge blur width (EBW) of a white and black bar object is proposed to characterize the blur of the reconstructed image, and its theoretical model is derived in detail based on a continuous/discrete (C/D) sampling mechanism by considering both the pickup sampling and the reconstruction process. Further, the quantitative relationships of the blur with the pickup sampling parameters (the pixel number of each elemental image, the number of elemental images) are calculated by the EBW model and measured by C/D sampling II simulation experiments, respectively. We ﬁnd out that the theoretical results have a good agreement with the estimated ones, and the minimum values of the EBW occurred periodically when the pixel number of an elemental image is an integral multiple of the magniﬁcation ratio. [DOI: http://dx.doi.org/10.2971/jeos.2014.14037]


INTRODUCTION
Integral imaging (II) is a promising three-dimensional (3-D) display technique that has been considered for 3-D television and visualization for its inherent advantages, such as fullparallax, continuous viewing points, operating without coherent light, etc.A typical II system consists of two parts: a pickup unit and a display unit.In the pickup process, each elemental lens that constitutes the lens array forms an elemental image (EI) according to its relative position to the object.These EIs from slightly different perspectives are recorded by an image detector such as a charge-coupled device (CCD).In the reconstruction process, the EIs displayed on a display panel are integrated by a second microlens array and reconstructed to form a 3-D image of the object.
Over the past decades, several theoretical models have been established for the analysis of the resolution of an integral image [1]- [6].H. Hoshino et al. [1] analyzed the resolution limitation of viewpoint reconstructed image.But they only discussed the effect of the reconstruction microlens pitch on the resolution.To analyze the resolution of an integral image, Refs.[2,3] also derived the total modulation transfer function (MTF) of an II system based on wave optical theory.Ju-Seog Jang et al. [4] presented the relationship between the resolution and the depth of a 3D integral image and proposed the use of the product of depth and resolution square as a figure merit in II systems, which is limited by the inverse of illumination wave length λ.However, the sampling artifacts of the pickup unit were not mentioned in their reports.Actually, the cut-off frequency determined by the detector is generally smaller than that by the optical system.Thus, the sampling effects of a detector array mainly limit the resolution of the reconstructed image.Fushou Jin et al. [5] presented the effects of a finite number of pixels in EIs on the resolution and the depth of focus in 3D II.Chunhong Wu et al. [6] discussed the effect of planar recording/displaying resolution, microlens pitch size, microlens focal length on the 3D resolvability.The authors [7] also proposed the perceptual contrast threshold (PCT) surface for characterizing the systematic performance of II systems by considering the sampling artifact of the detector as the equivalent MTF, which can correctly describe the lateral resolution, contrast, depth distance and their trade-off relationships in II.However, this equivalent method is relatively simple, which cannot provide accurate description for continuous/discrete (C/D) sampling mechanism of EIs.Furthermore, we introduce a squeezed MTF concept to quantitatively characterize the spurious response due to the sampling aliasing of the reconstruction microlens array and characterize its effects on the viewing resolution of an integral image [8].Jun Arai et al. [9] analyzed the sampling effects on depth control in II.But they only analyzed the relationship between the spatial frequency and the reconstructed distance.Besides, current computational reconstruction methods of integral image are mainly performed based on the planeto-plane reconstruction, viewpoint reconstruction, and pixel- to-pixel mapping methods, and do not consider the pickup sampling effects of continuous scene object [10]- [15].
Although there are above research works on the sampling effect in II, few efforts have been made to quantitatively evaluate the relationship of the resolution of an integral image with the pickup sampling effect.In this paper, the EBW model is proposed and derived based on C/D sampling mechanism to characterize the effects of the pickup sampling aliasing on the resolution of integral image.Furthermore, the feasibility of this model is verified by utilizing actual simulation results and demonstrates some valuable conclusions.
This paper is organized as follows.Section 2 is devoted to blur property analysis of an integral image due to the C/D sampling in the pickup process.In Section 3, the EBW model is derived in detail for characterizing integral image quality due to the pickup sampling artifacts.Section 4 provides the simulated experiment verifications of the EBW model and some important discussions.Finally, conclusions are summarized in Section 5.

BLUR PROPERTY ANALYSIS OF AN INTEGRAL IMAGE DUE TO THE PICKUP SAMPLING
Figure 1 shows the pickup process of an II system.The light which comes from the continuous object passes through the pickup microlens array to form an EI array on the aerial image plane.Then the aerial EI passes through the relay lens, and is recorded as spatially discrete EIs by an image detector such as a CCD, which is used to reconstruct an integral image.
Assume that in the pick up process, an edge of a bar object is recorded as three different pixels in three adjacent EIs in image space, respectively, such as Pixel 1, Pixel 2 and Pixel 3 in the grey values of the bar objects, no longer 0 or 255, become a spread distribution.
Since EBW is an important variable to characterize the blur of an integral image.The proposed EBW is similar to generally used line spread function (LSF) in imaging field, it is reasonable to introduce the edge blur width (EBW) of a bar object as a figure of merit of II quality in this paper.Because of the pickup sampling effects, some pixels in EIs not only record the continuous light field of the bar edge, but also the black and white region around the bar edge.In the reconstruction process, these pixels cannot be reconstructed to a line edge but a blur region.The width of the blur region is denoted as the edge blur width (EBW).The blur of an integral image, also called as cut-off frequency f , can be further expressed as the reciprocal of the EBW.That is, To analyze the pickup sampling effects on the blur of an integral image, it is necessary to first determine a quantitative relationship between the EBW of the reconstructed image and the pickup parameters, such as the number and size of the detector unit, the number and size of microlens array.

THEORETICAL MODEL FOR THE EBW OF A BAR INTEGRAL IMAGE
According to the pickup and conversion principle of an II system, the recorded EI array from a continuous 3D object can be described by combining ray-tracing model and C/D sampling model.Hence, the EBW of the bar can be derived based on these two models.To quantitatively analyze the blur width of the bar edge in an integral image, we limit the following discussions to the one-dimensional (1-D) case considering the independence of horizontal and vertical sampling of an II system.
Let D e denotes the size of a detector unit and D o denotes the corresponding continuous object field in object space.To determine the quantitative relationship of the EBW with the parameters of the pickup microlens array and EIs, assume that the pitch of EIs is φ, the EI array is composed of N EIs and the pixel number of each EI is n.Then the relationship between them can be expressed as: Assuming that the position of the bar edge in the first EI is x 1 , its corresponding position in the ith EI can be calculated by: Because of the pickup sampling effects, some pixels in EIs not only record the continuous light field of the bar edge, but also the black and white region around the bar object edge.That is, the gray values of pixels of the bar edge in image space are between 0 and 255.And other pixels in image space record only the continuous light field of black region or white region in object space.
As shown in Figure 3, in the pickup process, the edge of the bar object is recorded is recorded in Pixel 1 in the first EI (the red area), Pixel 2 in the second EI (the blue area) and Pixel 3 in the third EI (the green area), respectively.In the reconstruction process, Pixel 1, Pixel 2 and Pixel 3 are mapped and overlapped in the reconstructed image space, as shown in the yellow area of Figure 3, the width of which is the EBW that will be calculated.Assuming that the edge of the bar object is recorded as the m i th pixel in the ith EI, ∆ i and ∆ i are the distance of the left edge and the right edge of the m i th detector pixel projection region from the object bar edge in object space, respectively (Figure 3).Thus, where 0 m i n − 1 and f loor(x) denotes the maximum integer no larger than x; i max and i min denote the maximum and the minimum index of the EIs, respectively.Thus, and where ceil(x) denotes the minimum integer which is not smaller than x, Z expresses the distance between the cental depth plane and the microlens array and g is the distance between the EI array and the microlens array.Combining the above equations gives EBW =max{∆ i min , ∆ i min +1 , . . . ,∆ i max } + max{∆ i min , ∆ i min +1 , . . . ,∆ i max } Hence, the cut-off frequency can be further calculated based on Eqs. ( 9) and (1).

SIMULATED EXPERIMENTAL VALIDATION AND DISCUSSIONS
To verify the feasibility of the theoretical model of the EBW and the cut-off frequency, II simulation experiments are carried out based on C/D sampling process and ray tracing theory, as shown in Figure 4(a) and Figure 4(b).Specific steps are summarized as follows: (1) Build a white and black bar object in high fine lattice 3D space by using 3DS Max software.
(2) Record the EI array based on C/D sampling mechanism (the discrete object has much finer lattice than that of discrete detector array, and the object seems to be approximately continuous for the detector array) by virtual camera array in Direct3D software.
(3) Reconstruct an integral image of the bar object by the conventional computational II reconstruction (CIIR) method [16].
(4) Count the pixel number of the edge blur of the reconstructed bar image n blur .Then calculate the estimated cut-off frequency by f blur = 1/EBW = 1/(n blur • D r ) and compare it with the theoretical one which is obtained based on Eqs. ( 9) and ( 1), where D r is the pixel size of the reconstructed image.
In this experiment, the bar object is set to be 20 mm × 20 mm, and the periodic width of the bar object is 4.44 mm.The reconstructed image has 640 × 480 pixels and the pixel size of the reconstructed image D r is 0.0485 mm.Then, the quantitative relationships of the resolution of an integral image with the size of the detector element and the number of EIs are analyzed based on the calculated and experimental results, respectively.

Effect of the size of the detector element on blur
When the overall size of the detector array is a constant, the changed size of each detector element will result in a variation of the pixel number of each EI.Thus, the sampling effect of the detector is also changed.We set that g = 4 mm, L = 96 mm and φ = 1 mm are fixed.The reconstruction plane is at the central depth plane, where Z = 96 mm.The virtual camera array is composed of 10 × 10 cameras.The size of each EI is 1 mm × 1 mm and the pixel number of each EI n is a variable.The pinhole array is composed of 10 × 10 pinholes.To analyze the quantitative relationship of the cut-off frequency with the pixel number, the actual EBW can be obtained by calculating the actual pixel number of the bar edge blur in reconstructed image and the pixel size of the reconstructed image.Then, the actual cut-off frequencies can be calculated for different pixel numbers of EIs.At the same time, the theoretical cut-off frequencies are also obtained for different pixel numbers of EIs based on Eq. ( 9).The actual and theoretical results are plotted as a function of the pixel number of EI in Figure 6, respectively.
Figure 6 shows that both theoretical cut-off frequencies and actual ones increase gradually with an increment of n, and the theoretical curve has a good agreement with the actual one.Their peak values of cut-off frequency exit at n = 48 and n = 72 at the same time.We find out that the peak value points of cut-off frequency always exist for the pixel number of EIs where k is a positive integer and M = Z/g is the magnification ratio.That is, when the pixel number of each EI is an integral multiple of the magnification ratio, the peak value of cut-off frequency of the reconstructed image will be obtained.To verify the above conclusion, more peak values of the cut-off frequency are calculated and measured, respec- tively.Figure 7 illustrates that the peak value of the cut-off frequency increases with the increment of pixel number of each EI.From Figure 7, we can see that there are many peak value points in the actual reconstruction process, and the theoretical peak values are almost consistent with the actual ones, and always occurred when the pixel number of elemental image is integer times of the magnification ratio.

Effect of the size of the detector element on blur
But the cutoff frequency is restricted by the pixel size of the reconstructed image.When n achieves some value, the resolution of the reconstructed image does not increase.The EBW is larger than the pixel size of the reconstructed image and smaller than two times of it, that is, D r < EBW < 2D r .Because of n = k • r and at the peak value point, EBW = φ/k.Thus, the maximum cutoff frequency is f max = 1 D r .Hence, in this experiment, the maximum cut off frequency is a constant.

Effect of the number of EIs on blur
In this experiment, the size of each detector unit is assumed to be constant and the number of EIs is a variable.Thus, the pixel number of each EI and the sampling effect of the detector array are variable, too.Additionally, we set that g = 4 mm and L = 96 mm.The size of the detector array S is 10 mm × 10 mm and the pixel size of each EI D e is 0.00833 mm.The reconstruction plane is at the central depth plane, where Z = 96 mm.The camera array is composed of N × N cameras and the pinhole array is also composed of N × N pinholes , where N is a variable.Thus, both the pitch of the cameras and the pinholes are S/N.More experimental results are measured for different number of EIs N and the theoretical results are calculated for the same pickup sampling parameters.The theoretical and actual cut off frequencies are plotted as a function of the number of EIs in Figure 9, respectively.Figure 9 shows that the actual curve is in a good accordance with the theoretical one, and the cut-off frequency decreases slightly with an increment of the number of EIs.Furthermore, φ can be calculated based on the number of EIs N corresponding to the peak value of the cut-off fre- quency.We also find out when the cut off frequency reaches the peak value, φ = kD o , and N = k × r = k • L/g.In this case, the EBW is equal to D o , and the maximum cut-off frequency f max = 1/EBW = 1/D o can be further obtained.Because the pixel size in image space D e is a constant, the sampling width in object space corresponding to D o in image space is a constant, and the peak value f max is also a constant.

CONCLUSIONS
In this paper, the EBW is proposed and derived to characterize the effect of the pickup sampling on the blur of the reconstructed image in an II system.We further conduct some computational experiments to verify the correctness of the EBW theoretical model when the pixel number and the number of EIs are changed, respectively.Some valuable achievements are summarized as: (1) When the number of EIs, the size of each EI and the total size of the detector are constants, the reconstructed image becomes less blurred with the increment of the pixel number of each EI.When the pixel number is an integral multiple of the magnification ratio, the cut-off frequency of an integral image will have a peak value and the peak value increases with an increment of the ratio.And the peak value has the maximum limitation, which is determined by the pixel size of the reconstructed image.
(2) When the overall dimension of the detector array and the size of the detector unit are constants, the reconstructed image becomes less blurred slightly as the increment of the number of the EIs, i.e., the decrement of the size of the EI and the pixel number of each EI.When the pixel number of each EI is an integer time of the magnification ratio, the cut-off frequency has a peak value and this value is a constant.
The above research result will be helpful for an optimum design of an II system to obtain high quality reconstructed images.Therefore, our work is of great importance for both the theoretical research and the actual II system design.

Figure 2 .FIG. 2
FIG.2Illustration of blur mechanism of the bar edge due to the pickup sampling in II.

3 FIG. 3
FIG. 3 Schematic illustration of the calculation of EBW.

Figure 5
Figure 5 shows the reconstructed images of the bar object for two EIs of different pixel numbers n = 105 and n = 325, respectively.The lower right images are the corresponding enlarged area of the red rectangle parts and the length of the purple line in the lower right image is the blur width of the bar object EBW. Figure 5(b) is more clear than Figure 5(a) and the EBW of Figure 5(b) is smaller than that of Figure 5(a).

FIG. 5 FIG. 6
FIG. 5 Reconstructed images of the bar object for different pixel numbers of each EI.

FIG. 7
FIG. 7 Peak value of the cut off frequency as a function of the pixel number of each EIs.

Figure 8
Figure 8 shows the reconstructed images of the bar object when N = 5, N = 10 and N = 15, respectively.The lower right images are the enlarged images of the corresponding red rectangles and the length of the purple line in the lower right image is the blur width of the bar object is the EBW.The EBW of Figure 8(a) is equal to that of Figure 8(b) and smaller than that in Figure 8(c).When N = 5 and N = 10, the cut-off frequency has peak values.That is, the resolution of the image shown in Figure 8(a) is close to that in Figure 8(b), which is higher than that in Figure 8(c).

FIG. 8
FIG. 8 Reconstructed images of the bar object for different numbers of EIs.

FIG. 9
FIG.9Curves of the cut off frequency variation with the number of EIs.