A Reflective Augmented Reality Integral Imaging 3D Display by Using a Mirror-Based Pinhole Array

Abstract

In this paper, we propose a reflective augmented reality (AR) display system based on integral imaging (II) using a mirror-based pinhole array (MBPA). The MBPA, obtained by punching pinholes on a mirror, functions as a three-dimensional (3D) imaging device, as well as an image combiner. The pinhole array of MBPA can realize a pinhole array-based II display, while the mirror of MBPA can image the real objects, so as to combine the images of the real objects with the reconstructed 3D images. The structure of the proposed reflective AR display is very simple, and only a projection system or a two-dimensional display screen is needed to combine with the MBPA. In our experiment, a 25cm × 14cm sized AR display was built up, a combination of a 3D virtual image and a real 3D object was presented by the proposed AR 3D display. The proposed device could realize an AR display of large size due to its compact form factor and low weight.

1. Introduction

With the development of display technologies, electronic display devices have to meet people’s higher requirements. Augmented reality (AR) displays, which allow the overlaying of digital information onto a direct view of the real world so that the two-dimensional (2D) or three-dimensional (3D) images are seamlessly combined with the real-world scene, have been widely of interest to researchers and the industry [1,2,3,4]. There has been great desire for AR in many applications, such as medical visualization, surgical training, flight simulation, and entertainment [5,6,7,8]. Basically, there are two types of AR displays, which are optical see-through AR displays and video see-through AR displays [9,10,11,12,13]. Optical see-through AR displays, compared to video see-through AR displays [14,15], can provide a non-obstructed visualization of the real environment that guarantees that the visual and proprioception information will be synchronized and maintains see-through vision to the real world, and therefore has attracted much research interest. Additionally, integral imaging (II), a 3D imaging technique which provides auto-stereoscopic images without the help of special glasses [16,17,18], is very suitable for displaying 3D images in the optical see-through AR display system. Moreover, the structure of II is very compact, which makes it suitable for the AR system to display 3D images.

Some works have been proposed to achieve AR display with 3D images, a method of using free-form optics to magnify the 3D images reconstructed by a micro-II display unit and to overlap the 3D images with the real world, was reported by Hua [19]; another useful way to transfer 3D images into the viewer’s eyes through total internal reflection in an optical waveguide was also proposed, and achieved a good transmittance of the real-world lights [20]. Researchers have also proposed to realize an AR display by folding virtual 3D images into space with a semi-transparent mirror [21]. However, AR systems using these methods are more suitable for near-eye displays because of the size limitation, known as one of the common shortcomings, of the display screen or view box. To achieve a more compact structure for an AR system, the use of a lens array holographic optical element (HOE) has been proposed [22,23], but the size of the lens array HOE only reached several centimeters [24,25].

In this paper we propose, for the first time, a reflective AR system using a mirror-based pinhole array (MBPA). The real-world scene is reflected by the mirror area of the MBPA, and the 3D images are reconstructed by a pinhole array-based II unit. Due to its compact form factor, the proposed AR display system has the advantages of lower cost, no distortions, and more importantly, ease of manufacturing large-sized display screens.

2. Principle

2.1. Pinhole Array Based II Display

Figure 1 shows the schematic diagram of a pinhole array-based II technology including two parts: the pickup process and the display process [26]. In the pickup process, the light emitted from the 3D object is transmitted through pinholes in the pinhole array, and the lights go through each pinhole form an elemental image (EI) on the imaging sensor. The set of EIs recorded on the imaging sensor form an elemental image array (EIA), which contains different perspectives on the 3D object. In the display process, the captured EIA is displayed on a 2D display screen. The pinhole array is located in front of the 2D display screen to modulate the lights from the EIA. As shown by the blue dots in Figure 1b, lights emitted from the homologous pixels of a 3D object point intersect in the space to form a 3D voxel in front of the pinhole array. All of the 3D voxels constitute the 3D image.

2.2. Proposed Reflective AR Display

The schematic diagram of the proposed reflective AR display system using an MBPA is shown in Figure 2. Figure 2a shows the structure of the MBPA, which contains the pinholes and mirror area. The pinholes on the MBPA are white circular, while the other parts of the MBPA represent the mirror area. The MBPA can be obtained by punching pinholes with uniform horizontal and vertical spacing on a mirror. The pinholes on MBPA can transmit lights and form a 3D image like the traditional pinhole array-based II display does, as shown in Figure 2b. The mirror area of the MBPA can reflect the lights from the real scene, and a mirror image of the real scene is imaged on the opposite side of the MBPA, shown in Figure 2c. As a result, the viewers in front of the display device can receive the transmitted lights (the 3D image) and the reflected lights (the mirror image), simultaneously. In this way the combined AR contents are obtained. The MBPA functions as a 3D imaging device as well as an image combiner.

3. Experiments and Results

To verify the feasibility of the proposed reflective AR display system, an experimental system was built, as shown in Figure 3. The experimental system consists of a projector, a rear projection screen and a MBPA. The projector with high brightness was used to displace the 2D display screen in order to improve the brightness of the 3D images. According to the projection ratio of the projector we used, the size of the AR display was determined to be 25 cm × 14 cm. A 25 cm × 14 cm sized reflective film on which the pinholes were engraved by a laser engraving machine was attached on a transparent acrylic plate to implement the MBPA. The pinholes on the MBPA can transmit the light from the EIA, while the other parts of the MBPA can reflect the light from the real object. The rear projection screen was attached to the other side of the acrylic plate substrate to receive EIA projected by the high luminance projector. From our previous research work [27], the viewing angle was calculated, and the spatial angular resolution referred to the viewpoints number per degree was 0.575. The parameters of the reflective AR display system developed are shown in

Table 1. Specifications of the AR display system developed.

Components	Specifications	Values
Projector	Resolution Product model luminance	3840 pixels × 2160 pixels BenQ HD2934 2000-2999 lm
EIA	Resolution Size Pixel pitch Resolution of each EI	3840 pixels × 2160 pixels 250 mm × 140 mm 65.1 µm 23 pixels × 23 pixels
MBPA	Pinhole size Pinhole pitch Numbers of pinholes Thickness of MBPA viewing angle Spatial angle resolution	0.25 mm 1.5 mm 167 × 94 0.2 mm 41.1° 0.575

.

As shown in Figure 4a, a 3D scene consisting of eight apples was built up at different depths in 3d Max software, and their depths were −4 mm, −3 mm, −2 mm, −1 mm, 2 mm, 3 mm, 4 mm, and 5 mm, respectively. An EIA was generated by using the sparse camera array method [28]. As shown in Figure 4b, the EIA contained 167 × 94 elemental images, and each elemental image had 23 × 23 pixels. A portion of the EIA in Figure 4b has been enlarged, from which we can see several perspectives on the apple. A 4K projector was used to project the EIA onto the rear projection screen, and the re-projection transformation algorithm was used to align the projected EIA and the pinhole array on the MBPA. A coconut palm tree, which represents the real object, was located in front of the MBPA.

Figure 5 shows the different AR views of the reconstructed 3D images and the mirror image. The reference rectangles were drawn in Figure 5 to illustrate that the relative positions of the eight apples and the coconut palm tree changed when the viewpoints moved from left to right. The left four apples and the viewpoints move in the opposite direction, while the right four apples move in the same direction with the viewpoint. This proves that the reconstructed 3D images of the left four apples were located in front of the MBPA, and 3D images of the right four apples were located behind of the MBPA. The apples with large depths moved faster than those with small depths. The experimental results showed that the proposed system can realize a good AR display effect by combining the reconstructed 3D image and the mirror image with the MBPA.

The proposed AR display can be used in AR shopping. As shown in Figure 6, we designed an application scenario for AR fitting. A doll stood in front of the reflective AR display system, and a 3D image of a dress was displayed on her mirror image, as if she was wearing the dress. The experimental results confirm that the proposed AR display can be used for AR fitting. Just by changing the EIA, the virtual 3D dress can be changed very fast. A great deal of time will be saved when selecting clothes at mall or at home.

4. Discussions

The resolution of the system is determined by the number of pinholes, because the viewer sees one pixel from each EI, and the brightness of the system is directly affected by the pinhole size. The 3D resolution of the pinhole array-based II 3D display can be improved by increasing the number of pinholes. With the increase in the number of pinholes, however, the number of EIs increases accordingly (since each a pinhole corresponds to an EI). As a result, a 2D display screen with ultra-high resolution is needed. So the 4K high-resolution projector was used in our experiments. In addition, some research groups have also improved the resolution of pinhole array-based IIs by using time-division or space-division multiplexing technologies [29,30].The brightness of the system is affected by the pinhole size. When the pinhole size increases, the brightness and contrast of the system both increase. However, a large pinhole size will cause more pixels to be observed at a given viewing point through each pinhole (which results in crosstalk image). To compensate for the loss of brightness, a high-brightness projector was used in our experiment. The brightness of the projector was several times that of a 2D display screen.

5. Conclusions

In conclusion, in this paper we have proposed a reflective AR display using an MBPA. The MBPA functions using 3D imaging and mirror imaging. The 3D image was reconstructed through the pinhole array of the MBPA, and the real object was reflected by the mirror area of the MBPA, so as to achieve the effect of AR display by allowing viewers to receive the 3D image and the virtual image of a real object simultaneously. The proposed AR display system is simple in its structure and is easy to realize as a large-size display. It could be widely applied in business, such as AR shopping and more. However, one of the remaining problems is the low brightness of the reconstructed 3D image; it is possible, as we expected, to improve on this by properly enlarging the size of pinholes, or attaching another reflective film on the reverse side of the MBPA to recycle the light emitted from the display screen.