Point light source integral imaging with improved resolution and viewing angle by the use of electrically movable pinhole array

A resolution and viewing-angle enhanced integral imaging system using electrically movable pinhole array is proposed. A pinhole array on liquid crystal is adopted as dynamic pinhole array in integral imaging. The location of the pinhole array is controlled electrically. The pinhole array is expected to be moved fast enough to make an after-image effect, and the corresponding elemental images are displayed synchronously without reducing the 3D viewing aspect of the reconstructed image. With the proposed technique, the resolution and the viewing angle can be improved remarkably, and the upper resolution limit imposed by the Nyquist sampling theorem is overcome. The explanation of the proposed system is provided and the experimental results are also presented. ©2007 Optical Society of America OCIS codes: (100.6890) Three-dimensional image processing; (110.2990) Image formation theory; (220.2740) Geometric optics; optical design. References and links 1. G. Lippmann, “La photograhie integrale,” Comptes Rendus Acad. Sci. 146, 446-451 (1908). 2. F. Okano, H. Hoshino, J. Arai, and I. Yuyama, “Gradient-index lens-array method based on real-time integral photography for three-dimensional images,” Appl. Opt. 36, 1598-1603 (1997). 3. N. Davies, M. McCormick, and L. Yang, “Three-dimensional imaging systems: a new development,” Appl. Opt. 27, 4520-4528 (1988). 4. M. C. Forman, N. Davies, and M. McCormick, “Continuous parallax in discrete pixilated integral threedimensional displays,” J. Opt. Soc. Am. A 20, 411-420 (2003). 5. M. Martínez-Corral, B. Javidi, R. Martínez-Cuenca, and G. Saavedra, “Integral imaging with improved depth of field by use of amplitude-modulated microlens arrays,” Appl. Opt. 43, 5806-5813 (2004). 6. H. Liao, M. Iwahara, N. Hata, and T. Dohi, “High-quality integral videography using a multiprojector,” Opt. Express 12, 1067-1076 (2004), http://www.opticsinfobase.org/abstract.cfm?URI=oe-12-6-1067. 7. S.-W. Min, M. Hahn, J. Kim, and B. Lee, “Three-dimensional electro-floating display system using an integral imaging method,” Opt. Express 13, 4358-4369 (2005), http://www.opticsinfobase.org/abstract.cfm?URI=oe-13-12-4358. 8. Y. Kim, J.-H. Park, H. Choi, S. Jung, S.-W, Min, and B. Lee, “Viewing-angle-enhanced integral imaging system using a curved lens array,” Opt. Express 12, 421-429 (2004), http://www.opticsinfobase.org/abstract.cfm?URI=oe-12-3-421. 9. S. Jung, J. Hong, J.-H. Park, Y. Kim, and B. Lee, “Depth-enhanced integral-imaging 3D display using different optical path lengths by polarization devices or mirror barrier array,” J. Soc. Inf. Display 12, 461467 (2004). 10. M. Okui, J. Arai, Y. Nojiri, and F. Okano, “Optical screen for direct projection of integral imaging,” Appl. Opt. 45, 9132-9139 (2006). 11. D.-H. Shin, S.-H. Lee, and E.-S. Kim, “Optical display of true 3D objects in depth-priority integral imaging using an active sensor,” Opt. Comm. 275, 330-334 (2007). 12. J.-H. Park, H.-R. Kim, Y. Kim, J. Kim, J. Hong, S.-D. Lee, and B. Lee, “Depth-enhanced threedimensional-two-dimensional convertible display based on modified integral imaging,” Opt. Lett. 29, 27342736 (2004). #87153 $15.00 USD Received 4 Sep 2007; revised 13 Nov 2007; accepted 18 Dec 2007; published 20 Dec 2007 (C) 2007 OSA 24 December 2007 / Vol. 15, No. 26 / OPTICS EXPRESS 18253 13. H. Choi, J.-H. Park, J. Kim, S.-W. Cho, and B. Lee, “Wide-viewing-angle 3D/2D convertible display system uing two display devices and a lens array,” Opt. Express 13, 8424-8432 (2005), http://www.opticsinfobase.org/abstract.cfm?URI=oe-13-21-8424. 14. H. Choi, Y. Kim, S.-W. Cho, and B. Lee, “A 3D/2D convertible display with pinhole array on a LC panel,” Proc. of the 13th International Display Workshops, Otsu, Japan, 2, Dec. 2006, pp. 1361-1364. 15. S.-W. Cho, J.-H. Park, Y. Kim, H. Choi, J. Kim, and B. Lee, “Convertible two-dimensional-threedimensional display using an LED array based on modified integral imaging,” Opt. Lett. 31, 2852-2854 (2006). 16. C. B. Burckhardt, “Optimum parameters and resolution limitation of integral photography,” J. Opt. Soc. Am. 58, 71-76 (1967). 17. T. Okoshi, “Optimum design and depth resolution of lens-sheet and projection-type three-dimensional displays,” Appl. Opt. 10, 2284-2291 (1971). 18. H. Hoshino, F. Okano, H. Isono, and I. Yuyama, “Analysis of resolution limitation of integral photography,” J. Opt. Soc. Am. A 15, 2059-2065 (1998). 19. K. Perlin, S. Paxia, and J. S. Kollin, “An autostereoscopic display,” Proc. of the 27th Ann. Conf. on Computer Graphics and Interactive Techniques (ACM Press/Addison-Wesley, 2000), pp.319-326. 20. J.-S. Jang and B. Javidi, “Improved viewing resolution of three-dimensional integral imaging by use of nonstationary micro-optics,” Opt. Lett. 27, 324-326 (2002). 21. H. Liao, T. Dohi, and M. Iwahara, “Improved viewing resolution of integral videography by use of rotated prism sheets,” Opt. Express 15, 4814-4822 (2007), http://www.opticsinfobase.org/abstract.cfm?URI=oe-15-8-4814. 22. J. S. Jang and B. Javidi, “Three-dimensional integral imaging with electronically synthesized lenslet arrays,” Opt. Lett. 27, 1767-1769 (2002). 23. J. S. Jang and B. Javidi, “Improvement of viewing angle in integral imaging by use of moving lenslet arrays with low fill factor,” Appl. Opt. 42, 1996-2002 (2003). 24. J.-H. Park, J. Kim, Y. Kim, and B. Lee, “Resolution-enhanced three-dimension / two-dimension convertible display based on integral imaging,” Opt. Express 13, 1875-1884 (2005), http://www.opticsinfobase.org/abstract.cfm?URI=oe-13-6-1875. 25. M. Levoy and P. Hanrahan, “Light field rendering,” Proc. SIGGRAPH, New Orleans, Louisiana, Aug. 1996, pp. 31-42. 26. B. Lee, S. Jung, and J.-H. Park, “Viewing-angle-enhanced integral imaging by lens switching,” Opt. Lett. 27, pp. 818-820 (2002).


Introduction
Integral imaging (integral photography) is a three-dimensional (3D) display technique first proposed by Lippmann in 1908 [1].Although it had used a lens array and a film originally, the film has been replaced by a pickup device and a display device nowadays [2,3].Integral imaging is composed of two basic steps -pickup and display.In the pickup step lights from an object go through each lens in a lens array, and a two-dimensional (2D) image array, which is called elemental images, is generated.The elemental images are recorded on a pickup device such as charge coupled device (CCD) camera.In the display step, the recorded elemental images are displayed using a display device.The elemental images are integrated through a lens array in a reverse manner, and they form a 3D image.
Integral imaging has attracted much attention as an autostereoscopic 3D display owing to its many advantages [2][3][4].It does not need any special glasses.It provides almost continuous full-color viewpoints within a viewing angle, while other autostereoscopic methods using lenticular or parallax barriers have only discrete viewpoints.Unlike the other autostereoscopic methods, integral imaging provides not only horizontal parallax, but vertical parallax also.Lately, it has been possible to display real-time 3D movies by the virtue of the advancement of electronic devices such as high-definition television camera, spatial light modulator, liquid crystal display (LCD) and so on.In addition, some problems in the early stage of the integral imaging have been overcome by incessant researches [2][3][4][5][6][7][8][9][10][11].
Some integral imaging schemes with 2D/3D convertibility have been reported [12][13][14][15].Recently, an improved 2D/3D convertible display using pinhole array on a liquid crystal (LC) panel has been proposed [14].This scheme uses two LC panels, one for displaying elemental images and the other for generation or elimination of the pinhole array.Here, the pinhole array plays the same role as the lens array.The pinhole array on LC is located between the LC for display and the backlight unit (BLU) as shown in Fig. 1.If the pinhole array is generated, the system operates in the 3D mode.The principle of the 3D display with point light sources has already been used [14].In the 2D mode, the pinhole array is eliminated, and the LC for pinhole array becomes transparent.The 2D image displayed on display LC is observed by the transmitted backlight as shown in Fig. 1(b).Using this method, the 2D display mode is easily converted into the 3D display mode.However, the resolution of the 3D display is remarkably lower than that of the current 2D display.

Limitation of resolution
Even though the integral imaging has many advantages, the low resolution of the integrated 3D image is still a problem.The resolution of the reconstructed 3D image is determined by various system parameters such as the size and the pitch of lens array, the distance between the apertures, the resolution of the CCD camera and the display device [16][17][18].The resolutions of pickup and display devices are important factors for the sufficient resolution of the implemented 3D images.However, the most fundamental factor that limits the resolution of 3D image is the pitch of elemental lens or the pitch of exit pupil.The pitch of the lens or the pitch of exit pupil determines the sampling rate of the elemental images in the spatial dimension as shown in Fig. 2.

Fig. 2. Limitation of the resolution in integral imaging
Figure 2 shows the sampling of the real image according to the pitch of the exit pupil.From the Nyquist sampling theorem, the upper limit of the viewing resolution, nyq β (cycles per radian), in a lateral direction is determined as follows: where p is the pitch of the exit pupil and L is the viewing distance as shown in Fig. 2. Integral imaging reconstructs the original 3D object only with the limited viewing resolution.The viewing resolution of 3D image is limited due to the sampling rate fundamentally, and the image quality is degraded in 3D display.The detailed theoretical analysis on the viewing resolution of integral imaging is given in Refs [16][17][18].
To overcome the low resolution problem, many studies have been done [19][20][21][22][23][24].In the stereoscopic method, using dynamically varying parallax barrier, which changes the width and positions of its stripes as the observer moves, has been proposed [19].However, the location of the observer is confined to a specific position, or it requires eye tracking since it is stereoscopic.In integral imaging, a method that moves the lens arrays for image pickup and display synchronously has been proposed [20].Another method that rotates a pair of prism sheets in front of a display panel instead of the lens array has been reported recently [21].These methods improved the viewing resolution.However, both of them require the rapid mechanical movement of the lens array or the prism sheet.It causes problems such as vibration, air resistance and noise, which deteriorate the stability of the systems.To avoid the mechanical movement, an electrical synthesis of lens array using an array of Fresnel zone plates in a LC has been studied [22].A resolution-enhanced 2D/3D display based on integral imaging using an additional lens array has been proposed [23].However, the system is bulky, and the viewing angle is narrow.Although the concept of the light field rendering was introduced in 1996 [25], it was mainly for the pickup and intermediate (or new) view generation, while our work is for practical dynamic integral imaging display using practical LCD systems.
In this paper, we propose a thin and compact enhanced integral imaging by moving pinhole array electrically.For electrically movable pinhole array, the pinhole array on LC is adopted in the proposed integral imaging, and the location of the pinhole array is electrically controlled.For the enhancement of resolution, the pinhole array on LC moves fast enough to make an after-image effect and the corresponding elemental images are displayed synchronously without any mechanical movement.In our experiment, we implement the fast electrical movement of pinhole array and the synchronization with the elemental images actually.As a result, the upper resolution limit imposed by the Nyquist sampling theorem is overcome.In addition, the viewing angle is enhanced by making the neighboring pinholes on or off in turn and by displaying the elemental images synchronously.The explanation of the proposed system is provided, and the experimental results clearly show the enhancement of resolution.

Resolution enhancement
In conventional integral imaging, the pinhole array is fixed, i.e., remains stationary, and the resolution of 3D display is determined by the number of pinholes.Inherently the number of pinholes should be smaller than that of pixels in 2D display to provide different perspectives for 3D images.It is natural that the resolution of 3D display be lower than that of 2D display.However, in our proposed method, we adopt the pinhole array on LC, which has been proposed for 2D/3D convertible display, as a dynamic pinhole array system.It enables the pinhole array to be shifted to the other location electrically.
Figure 3 shows the configuration of our proposed method.The moving pinhole array is displayed on LCD 1, and elemental images are on LCD 2 in Fig. 3.However, the moving pinhole array can be implemented on LCD 2, and the elemental images can be on LCD 1 inversely.Both of them are justifiable in the proposed system, if the pinhole array can be implemented on LC by turning on or off the pixels.In this paper, we choose the configuration that the pinhole array is displayed on LCD 1 and elemental images are on LCD 2. Figure 4 shows the principle of our proposed method.In Fig. 4, one-dimensional (1D) case is represented as simple schematic diagram.Figure 4(a) corresponds to the conventional method.The arrow image is observed with the limited resolution determined by the number of pinholes.However, if the pinhole array in the case 1 is shifted to the pinhole array in the case 2 with time-multiplexing and the corresponding elemental images are displayed synchronously, the resolution-enhanced integrated image is observed by the after-image effect as shown in Fig. 4

(b).
As mentioned above, in our proposed method, the pinhole array can be implemented on LC by regularly turning on or off the pixels.Figure 5 shows an example of this situation in detail.Although there are many apertures in pinhole array used in integral imaging, we consider a portion of pinhole array for simplification as shown in Fig. 5(a).An aperture is placed every eight pixels in horizontal and vertical directions respectively.The locations of apertures are periodic.Hence, we can name the part indicated by the green rectangular line in Fig. 5(a) as one unit of the pinhole array.Four apertures are shown in Fig. 5(a).
In our proposed method, the pinhole array is shifted sequentially as shown in Fig. 5(b).The unit in the conventional pinhole array in Fig. 5(a) is magnified as shown in the mode 1 of Fig. 5(b).The location of aperture can be represented as (1, 1) in the mode 1. Next, the aperture is shifted four pixels down or up (Either of the case is the same.) to (5, 1) as shown in the mode 2. The pinhole array is shifted four pixels right or left to (5,5) as the mode 3 shows.The aperture is shifted four pixels up to (1,5) as the mode 4 shows, and it returns to (1, 1) of the mode 1 and repeats the movement.
According to the location of the aperture, the corresponding elemental images that should appear on LCD 2 in Fig. 3 are calculated by computer.It adopts ray optics in a reverse manner.In other words, to integrate points in the object, we consider an imaginary ray that originates from the object point and that goes through the center of corresponding aperture.The crossing point on the front display panel for this line is the elemental image point corresponding to the object point.We perform this process for all apertures and all points in the object.In this calculation, computer-generated integral imaging is used.
The synchronization between the pinhole array and the displayed elemental images is important to integrate a correct 3D image.We control the shifting of pinhole array and make the elemental images be synchronized with the pinhole array movement by the computer as shown in Fig. 3.When the pinhole array is in the mode 1, the elemental images for the mode 1 are displayed on the LCD 2 at the same time.If the pinhole array is in the mode 2, the elemental images for the mode 2 are displayed.In this way, the elemental images for the mode 3 and the mode 4 are displayed, and they are electrically synchronized with the movement of pinhole array.The elemental images and the pinhole arrays according to each mode are displayed successively, which provide the pixels of the integrated 3D image.If this movement is fast enough to raise the after-image effect of human eyes, the four apertures are observed in one unit as shown in Fig. 5(c).Four kinds of elemental images are integrated by the four apertures in one unit without reducing the 3D viewing aspect of the reconstructed image.Each mode makes each pixel of the 3D image in a unit at the location of the aperture.The resolution depends not only on the number of pinholes, but also on the number of modes, i.e., the number of times that the pinhole array is time-multiplexed in our proposed method.We denote the number of modes as T. If T is four as shown in Fig. 5 and if the modes are time-multiplexed fast enough, the resolution is enhanced four times higher than that of the conventional case [14].As the T increases, the resolution of 3D display increases in proportion to T. Consequently, the resolution of 3D display is determined as T times the number of pinholes in our proposed method.This is the main characteristic that enhances the resolution of the 3D image.As T increases, the number of pixels in a unit also increases.When the movement of aperture covers all the pixels in one unit, the viewing resolution of the 3D integrated image can be up to the resolution of the 2D display theoretically.For example, when there are 64 pixels in one unit of pinhole array as shown in Fig. 5, the number of modes can increase up to 64.If T can be 64, the 3D image is observed with full pixels of the used 2D display panel.We can expect that the high resolution 3D display, the resolution of which is up to that of the 2D display, can be realized.

Viewing angle enhancement
We adopted the 2D/3D convertible display using pinhole array on an LC panel in the proposed integral imaging, the principle of which is based on point light source array [14].The light rays emerging from each aperture are modulated by the display LC and integrated into 3D images.Each aperture has its own elemental image region on display LC.For the integration of the point O in Fig. 6(a), only the elemental images that are within their corresponding elemental image regions are displayed on the display LC.The viewing angle c θ of O is approximately the same as the angle that covers the corresponding elemental image region [12] from the aperture as shown in Fig. 6(a), and the corresponding equation is given by where d is the gap between the pinhole array and the elemental images as shown in Fig. 6(a).
Figure 6(b) shows an example of conventional stationary pinhole array.However, in our proposed method the pinhole array can be shifted electrically, and this feature can enhance the viewing angle of 3D display.The concept of the proposed method is motivated by the work using lens switching [26].Figure 7 shows the scheme of our proposed method.In the mode 1, the apertures in (1, 9) and (9, 9) are off.In the mode 2, the apertures in (1, 1) and (9, 1) are off.Then in each mode, the horizontal distance between the apertures becomes twice larger than that of the conventional method.The elemental image region in our proposed method becomes twice of p , and the viewing angle p θ in the proposed method can be derived from This is the horizontal viewing angle of the proposed system.The vertical viewing angle is the same as the conventional viewing angle because the height of the elemental image region is identical.The elemental images are calculated for each mode of pinhole array, which has a wider elemental image region.The corresponding elemental images are displayed synchronously with the shifted pinhole array.If the movement is sufficiently fast, the four apertures are observed like the conventional stationary apertures shown in Fig. 7(b).In our proposed method, the resolution of 3D display is the same as that of the conventional method.However, the viewing angle is approximately two times wider than that of the conventional method.
As mentioned in the previous subsection, if the number of modes, T, increases, the elemental image region for an aperture is wider in proportion to the number of modes.As a result, the viewing angle of our 3D display increases.

Resolution and viewing angle of the proposed method
In our proposed system, the resolution and the viewing angle can be enhanced independently.When the viewing angle of the system is equal and when the number of modes, T, attributes only to the resolution enhancement, the resolution p R of 3D display in our proposed system can be derived as follows according to the relation of the conventional resolution c R of the previous system: (4) The 1D resolution and viewing angle are considered in this derivation.Since the size of the display is constant, the resolution of 3D display is proportional to the inverse of the pixel pitch, i.e., the distance between the apertures.Regarding the resolution, the dimension of pixels per millimeter is used in this paper.The resolution p R can be derived as The resolution of our proposed system is enhanced in proportion to T.
When the resolution of the system remains unchanged and when T attributes only to the viewing angle enhancement, the viewing angle p θ of the proposed system is derived as follows from equations ( 2) and (3): The viewing angle becomes wider as T increases.
The resolution and the viewing angle can be enhanced simultaneously in our proposed system.Also, T attributes to both the resolution and the viewing angle.The resolution is enhanced k times, and T/k contributes to the viewing angle as follows: In this case, the relation between the viewing angle and the resolution can be induced from the above equations.
From the above equation, we can see that there is a trade-off relationship between the resolution and the viewing angle.Figure 8(a) shows the relation between the viewing angle and the resolution with the dimension of pixels per millimeter.The resolution of the used 2D display is about 3.7 pixels per millimeter in 1D.This is the possible maximum resolution of the 3D display in our proposed method.When T is one, it corresponds to the conventional case.There are no movements of pinhole array.As T increases, both the resolution and the viewing angle increase.This corresponds to the proposed case.Considering the 2D resolution and the 2D viewing angle, the above equations are required to be modified.The resolution, the viewing angle and the relation between the resolution and the viewing angle are respectively given as follows: The viewing angle versus the resolution with the dimension of pixels per millimeter square according to T is shown in Fig. 8(b).The resolution of the 2D display is 14.3 pixels per millimeter square.When T is one, it corresponds to the conventional case.When T is 4, it corresponds to the 2 by 2 T in both horizontal and vertical directions.When T is 9, it corresponds to the 3 by 3 enhancement.As T increases, both the resolution and the viewing angle increase.This corresponds to our proposed case.The quality of our 3D display is enhanced as T increases by the electrical movement of the pinhole array.

Experimental results and discussions
In our experiment, one LCD panel and one LCD monitor are used for LCD 2 and LCD 1, respectively.Figure 9 shows our experimental setup.The two LCDs have the same specification.The panel size is 17 inch in diagonal direction, and the resolution is 1280 (Horizontal: H) by 1024 (Vertical: V).The size of each pixel is 0.264 mm (H) by 0.264 mm (V).LCD 2 is located in front of the LCD 1, and it is used for displaying elemental images.A pinhole array is implemented on LCD 1, and it is used instead of the lens system.Polarization sheets are placed between LCDs and in front of LCD 2 to display the color of image correctly.They are adjusted to minimize the optical loss.The pinhole array consists of 100 by 100 square apertures.An aperture is implemented by turning on a pixel, which is the smallest electrically controllable unit in the LCD.The size of each aperture is 0.264 mm (H) by 0.264 mm (V), and the distance between the apertures is 2.112 mm.The gap between the LCD 1 and the LCD 2 is 6 mm.The maximum frequency of LC panel is 75 Hz.We activated the pinhole array to have four modes in this experiment.Since the aperture is eight pixels distant away from each other, we set the four modes for regular pixel distribution as follows.As mentioned above, in the mode 1, all the apertures are located at (1,1) in the corresponding unit.The apertures move four pixels down, located at (5, 1) in the mode 2. In the mode 3, they move four pixels right, located at (5,5).In the mode 4, they are located at (1,5).Next, all the apertures go back to the mode 1 again and repeat their movements electrically.At the same time, the corresponding elemental images should be displayed in LCD 2 according to the movement of the pinhole array.To display exact integrated images, it is important that the displayed elemental images are synchronized with the movement of pinhole array correctly.We connected the two display devices to a computer and used a program that synchronizes the two panels.Elemental images are generated by the computer launching with our ray tracing program.The elemental images for the same 3D object are calculated according to the locations of the apertures, and they are displayed on LCD 2 in synchronization with the moving pinhole array.In this experiment, we displayed one real image and one virtual image.Two character images, '3' and 'D' are used.In this experimental setup, the expressible depth range, within which an observer recognizes 3D images well, is from about 40 mm behind to 40 mm in front of the pinhole array.The image '3' is located 20 mm in front of the pinhole array and the 'D' is located at 20 mm behind the pinhole array within the depth region.
Figure 10 shows the experimental results.Figure 10(a) shows integrated 3D images according to each mode of pinhole array when the pinhole array is stationary.If the viewpoint is changed, the 3D images, '3' and 'D', show different perspectives.However, all the figures in Fig. 10 are taken at the same location to compare the integrated 3D images with different modes.Since elemental images for each mode are generated from the same 3D object, the four images in Fig. 10(a) represent the same 3D images.However, the locations of the pixels of the integrated 3D image are slightly different, 4 pixels shifted.The four images represent the same 3D image with slightly different pixels as Fig. 4 describes.
When the pinhole array is stationary, the result of the mode 1 corresponds to the conventional integral imaging that uses a fixed pinhole array.The resolution is determined by the number of apertures.One aperture provides one pixel in the integrated image.However, in our proposed method, the integrated images by the four modes in Fig. 10(a) are electrically time-multiplexed fast.The pinhole array is shifted electrically and the corresponding elemental images are displayed synchronously.Figure 10(b) shows the resolution-enhanced result using the proposed method.It is obtained by digital composition of the integrated 3D images in four modes.The resolution of the 3D image in our proposed method shown in Fig. 10(b) is four times higher than that of the conventional method shown in each mode of Fig. 10(a).
To investigate the resolution enhancement, we compare the integrated image of the conventional scheme with that of the proposed scheme.Figure 11 shows this in detail.We take pictures of the different perspectives of the two images as the observation direction changes.The resolution of 3D image using our proposed method is higher than that of the conventional method.For precise comparison, we chose the same part of the 3D integrated image indicated by the square line and magnified it.Figure 11 shows the magnified images and the resolution in detail.The results show that the quality of 3D image integrated by our proposed method is much higher than that by the conventional method.The proposed method that moves pinhole array electrically enables to enhance the resolution of integrated 3D images remarkably.Another experiment is carried out with respect to the increase of the viewing angle.The size of each aperture is the same as 0.264 mm.The distance between the apertures is 4.224 mm in horizontal direction and 2.112 mm in vertical direction.
We activated the pinhole array to have two modes in this experiment for the viewing angle enhancement in the horizontal direction.Since the aperture is 16 pixels distant away from the horizontal direction, we set the two modes for regular pixel distribution as follows.In the mode 1, two apertures located at (1, 1) and (9,1) are open in the corresponding units.The apertures are moved by eight pixels to the right, located at (1,9) and (9,9) in the mode 2. Next, both of the two apertures go back to the mode 1 again and repeat their movements electrically.At the same time, the corresponding elemental images, which have a wider elemental image region, should be displayed in LCD 2 according to the movement of the pinhole array.It is necessary that the displayed elemental images be synchronized with the correct movement of pinhole array.
Figure 12 shows the experimental results.The upper two images in Fig. 12 show integrated 3D images according to the two modes of pinhole array when the pinhole array is stationary.The result of the composition of two modes is shown in the left lower corner of Fig. 12, and the result when the actual time-multiplexing is implemented in the right lower corner of Fig. 12.In the upper two modes, the location of the pixels of integrated 3D image are slightly different, 8 pixels shifted in horizontal direction.This feature enables a wider viewing angle.
To demonstrate the viewing angle enhancement, we compare the integrated image of the conventional scheme with that of the proposed scheme.Figure 13 shows the related results.We can see the different perspectives of the two images as the viewing angle changes.In the conventional case, the viewing angle is approximately 18°, and the characters, '3' and 'D', show perspectives within the viewing angle.However, in the proposed case, the viewing angle is 36° and we can see a lager parallax between the '3' and 'D', and between the images observed from left and right, than that of the conventional case as shown in Fig. 13.For the precise comparison, the same part of the 3D integrated image indicated by the square is magnified.In the magnified images, we can see the difference in parallax indicated by the lines.The results show that the viewing angle of 3D image integrated by the proposed method is wider than that by the conventional method.The moving of the pinhole array electrically enables the enhancement of the viewing angle of the integrated 3D image, too.
However, the optical efficiency of 3D display is low because the proposed system uses a pinhole array, and a moiré pattern is observed owing to the overlapping of two LCD panels.These problems can be overcome by using the brightness-enhanced backlight or by eliminating the color filter in pinhole array LC.Also, the polarization sheet between the LCDs results in the low efficiency.If we use another type display like cathode-ray tube display instead of the pinhole array LC, the brightness will increase.
Even though the efficiency of pinhole array is lower than that of lens array, using pinhole array in our proposed method has an important merit in the quality of the integrated 3D image.In the case of using lens array, each lens samples a pixel in the corresponding elemental image and represents the pixel with the same size of the lens.The 3D pixel size is the same as the size of the lens aperture.When the lens array moves fast for resolution enhancement, the blurs of the image boundary is inevitable because the sampled pixel is magnified and added in the process of time-multiplexing.However, a pinhole represents a sampled pixel with the size of the pinhole aperture, the size of which is the same as the pixel size.If the moving pinhole is used in the proposed method, the result image has sharp image boundaries because the sampled pixel is not magnified by the lens.We can obtain a resolution-enhanced image with clear image boundaries.This is the advantage of the proposed system using pinhole array.Another important advantage is that our pinhole array is implemented on LCD and can be time-multiplexed fast without any mechanical movement.
As mentioned in the principle of our proposed method, if we make more modes of pinhole array in the proposed system to have better resolution and control the movement fast, the resolution of 3D display can increase up to the resolution of the used 2D display.There is no limitation in viewing resolution theoretically.In addition, the viewing angle can be enhanced remarkably.However, the driving frequency of LC for moving pinhole array is not so high, yet.The maximum frequency of LC used in our experiment is 75 Hz now.Three or maximum four modes are able to increase the after-image effect of human eye.As the number of modes increases, the trembling of 3D image is observed.However, if the fast response LC is developed (Very recently, 120 Hz LCDs were introduced by some display companies.),we expect that high resolution and wide viewing angle 3D display can be realized by electrically moving pinhole array.

Conclusion
We proposed a resolution-enhanced integral imaging with pinhole arrays on LC panel.The pinhole array on LC is adopted as pinhole system in integral imaging and enables the fast movement of pinhole array without any mechanical movement.Since light through a pinhole corresponds to a pixel in 3D image, we electrically move the pinhole array fast enough to make the after-image effect and display corresponding elemental images synchronously without reducing the 3D viewing aspect of the reconstructed image.Also, the viewing angle is improved electrically by shifting the pinhole array.The proposed system is simple and easy to enhance the quality of 3D display.The experimental results show that the resolution and the viewing angle of the 3D integrated image are enhanced remarkably when compared with those in the conventional method.The proposed method has a good possibility for high resolution and wide viewing angle 3D display.

Fig. 3 .
Fig. 3. Configuration of the proposed integral imaging system

Fig. 4 .
Fig. 4. Principle of the proposed method: (a) resolution of integrated image in case 1 and case 2 (b) resolution enhancement by time-multiplexing of cases 1 and 2

Fig. 5 .
Fig. 5. Schematic diagram of the moving pinhole array: (a) conventional fixed pinhole array (b) magnified units of the pinhole array according to the shift for each mode (c) proposed pinhole array by time-multiplexing of the modes in (b)

Fig. 8 .
Fig. 8. Viewing angle versus the resolution according to T considering (a) 1D case (b) 2D case

Fig. 10 .Fig. 11 .
Fig. 10.Experimental results for resolution test: integrated 3D images (a) according to the four modes (b) using the proposed method

Fig. 12 .Fig. 13 .
Fig.12.Experimental results for viewing angle test: integrated 3D images according to the modes and the integrated image using the proposed method