Holographic Three-Dimensional Virtual Reality and Augmented Reality Display Based on 4K-Spatial Light Modulators

Abstract

In this paper, we propose a holographic three-dimensional (3D) head-mounted display based on 4K-spatial light modulators (SLMs). This work is to overcome the limitation of stereoscopic 3D virtual reality and augmented reality head-mounted display. We build and compare two systems using 2K and 4K SLMs with pixel pitches 8.1 μm and 3.74 μm, respectively. One is a monocular system for each eye, and the other is a binocular system using two tiled SLMs for two eyes. The viewing angle of the holographic head-mounted 3D display is enlarged from ${3.8}^{\circ }$ to ${16.4}^{\circ }$ by SLM tiling, which demonstrates potential applications of true 3D displays in virtual reality and augmented reality.

1. Introduction

Virtual reality (VR) and augmented reality (AR) have been hot topics recently, and 3D technology is an important part of VR technology [1]. VR is widely used in various fields, such as architecture, gaming, and education, where it has a promising future in the development of VR. Compared to holographic displays, conventional optics present several disadvantages for near-to-eye 3D products for VR and AR [2]. Although 3D techniques are popular in current VR and AR applications, most 3D head-mounted displays (HMD) are based on stereoscopic 3D display technology, i.e., left and right eyes get two images with binocular parallax. The human brain can combine these two images into a 3D image. However, these images in the viewing angle of the stereoscopic 3D display are separate two-dimensional (2D) images, which are different from the continuous wavefront of 3D objects or 3D digital models. When images were displayed outside the range of accommodation, visual fatigue was induced [3,4,5,6]. The stereoscopic 3D display is not suitable for everyone to watch for a long time, because not everyone’s pupil distance is within the applicable range. Therefore, 3D techniques should be improved to overcome the drawback in VR and AR. It is a conventional 3D technology in VR head-mounted displays, as shown in Figure 1.

To overcome the limitation of the stereoscopic 3D technique, some researchers have studied some other 3D techniques in HMD. Eunkyong Moon et al. used an LED light source instead of lasers as the reading light source in the holographic HMD and produced a nice effect [7]. Han-Ju Yeom et al. did some work on astigmatism aberration compensation of the holographic HMD, and their system can present holographic 3D images in a see-through fashion clearly in a wide depth range [8]. J. Han et al. presented a compact waveguide display system integrating free-form elements and volume holograms; this design makes the system more compact because of the waveguide [9]. Mei-Lan Piao et al. used a transparent volume hologram for a holographic projection HMD and filmed the holograms to show that the see-through holographic projection HMD system can present three-dimensional images clearly [10]. Galeotti et al. designed real-time tomographic holography for AR, which used a large-aperture holographic optical element (HOE) and obtained good results, using a large-aperture HOE to project an off-axis, viewpoint-independent virtual image 1 m away [11]. H.-E. Kim et al. used an active shutter for HMD application to get a more precise system [12]. J. Piao et al. used photopolymer for a full-color waveguide-type HMD and obtained a compact system, and it provided wide angular selectivity and can fabricate high-quality full-color HOEs [13]. Gang Li et al. used a mirror-lens to propose a see-through AR display and achieve an optimized optical recording condition of the mirror-lens HOE [14]. Jong-Young Hong et al. used an index-matched anisotropic crystal lens to propose a holographic see-through near-eye display, and the system showed the possibility of a holographic display with a large field of view [15]. Peng Sun et al. proposed a holographic near-eye display system based on a double-convergence light algorithm and provided a promising solution in future 3D AR realization [16]. Jisoo Hong et al. proposed a near-eye foveated holographic display [17]. Among these 3D techniques, holography is thought to be a good candidate for true 3D VR, because digital 3D holography based on spatial light modulators (SLMs) can provide the holographic 3D property of wavefront reconstruction and realize real-time video rate dynamic display [18,19,20,21]. In this paper, we present a holographic 3D HMD and study its 3D properties, such as viewing angle and depth of field dependent on the pixel size of SLMs. We believe that with the development of SLMs and digital information technology, holographic 3D VR and AR can be applied in the future.

2. Monocular System

Holographic VR is a true 3D display, which is different from that based on stereoscopic 3D technology. Stereoscopic displays can cause visual fatigue, which is not suitable for most people to watch for a long time. In holographic 3D VR, the image has a certain depth of field, which is the distance between the nearest and the furthest objects that are in acceptably-sharp focus in an image [22], and the maximum diffraction angle depends on the pixel pitch of SLM, while the viewing angle of the reconstructed images, $\theta$, is twice the maximum diffraction angle, which is the maximum angle at which an image can be viewed with acceptable visual performance. Equation (1) shows the relationship between the pixel pitch pand the viewing angle $\theta$.

$$\theta =2arcsin\frac{\lambda }{2p}$$

(1)

where $\lambda$ is the wavelength of reading light. For instance, here, the wavelength of reading light is 532.8 nm. When the pixel pitch of SLMs is 8.1 μm and 3.74 μm, respectively, obtained by calculation, the viewing angle should be ${3.81}^{\circ }$ ($\theta$${}_1$) and ${8.2}^{\circ }$ ($\theta$${}_2$), respectively, as shown in Figure 2a. It is obvious that the smaller pixel pitch of SLM leads to a larger viewing angle, as shown in Figure 2b.

In Figure 3a, a is the distance between image and eyes and b is the width in the pupil plane. The image plane is the hologram plane, and it is a Fresnel hologram. Equation (2) shows the relationship between a, b, and $\theta$.

$$a=b/(2tan(\theta /2\left)\right)$$

(2)

The pupillary distance is the distance between a human’s two pupils, which is 58–64 mm for adults, and 52–64 mm for children. It is different for different humans. The difference in the range of pupillary distance between children and adults is 12 mm. We compare distances between displayed images and eyes based on SLM with a pixel pitch of 3.74 μm and 8.1 μm, respectively. Setting the pupillary distance of d and the difference in the range of pupillary distance of Δd, Δb = (Δd)/2 = 6 mm, because the two eyes are symmetrical, obtained by calculation; a₁ is 42.1 mm, and a₂ is 90.2 mm. Obviously, the distance between the displayed image and eyes in the system with a 3.74-μm pixel pitch SLM can be much smaller than that with the 8.- μm pixel pitch SLM. Figure 3b shows the dependence of the distance between the image and eyes on the pixel pitch of SLM. Figure 4 shows the schematic layout of the monocular system, which consists of two SLMs (SLM1 and SLM2) for two eyes, two beam splitters (BS1 and BS2), a half wave plate, etc. Some mirrors were used to make the whole system more compact.

In Figure 5a, the image plane and convergence plane of the stereoscopic virtual reality HMD are shown. If they are in different planes, once all the elements of the HMD are fixed, there will be difficulty in making them be in the same plane. The image plane and convergence plane of the holographic virtual reality HMD can easily be in the same plane, as shown in Figure 5b, because the depth of field of the holographic 3D display can be set in the process of digital hologram computation (we changed some parameters in our MATLAB program of the computer-generated hologram to change the depth of the holographic reconstruction image). The computer-generated holograms are displayed in SLMs. This can improve the interaction and can eliminate the limitation of stereoscopic VR HMD in Figure 5a.

3. Binocular System

Although holography is a true 3D display, the limitation of the holographic device, such as a big pixel pitch of SLM, leads to a small viewing angle. To increase the viewing angle, it is necessary to tile multiple SLMs. Therefore, we propose a binocular holographic HMD system using two tiled SLMs for holographic HMD. As shown in Figure 6, there are two SLMs tiled in a curved configuration with tiled angle $\alpha$ in order to ensure accurate tiling. When $\alpha$ is smaller than $\theta$, the viewing angle is increased to $\theta$+$\alpha$ in this case [23], otherwise, the reconstructed image is not seen continuously. Under ideal conditions, i.e., $\alpha$ = $\theta$, the dependence of the viewing angle on the pixel pitch of SLMs is as shown in Figure 7a. The dependence of the distance between the image and eyes on the pixel pitch of SLMs is shown in Figure 7b.

Figure 8 shows a schematic diagram of the proposed binocular holographic HMD system. The tiled SLM unit consists of two SLMs (SLM1 and SLM2 with the same pixel pitch) and a beam splitter (BS). Some of the laser beam is reflected by the BS to SLM1 and reads out one computer-generated hologram display in SLM1, and then, the reconstructed image transmits through the BS and combines with the reconstructed image from SLM2, which is reflected by the BS after it is read out from the other computer-generated hologram display in SLM2 by the laser beam, which transmits through the BS. These two holograms displayed in SLM1 and SLM2 have different viewing angles of an object or a scene.

4. Experiments and Results

In this experiment, we took pictures of the reconstruction from the hologram at different viewpoints, which were left view, middle view, and right view, using a single SLM with pixel pitch of 8.1 μm and 3.74 μm, respectively. The SLMs were liquid-crystal-on-silicon (LCoS) SLMs, Holoeye Pluto (resolution: 1920 × 1080, pixel pitch: 8.1 μm), and Jasper (resolution: 4094 × 2400, pixel pitch: 3.74 μm). In Figure 9, the image is captured from the left, middle, and right viewpoints in the optical system using an SLM with a pixel pitch of 8.1 μm, and the reconstructed image with the viewing angle, ${3.8}^{\circ }$, is a cube with the letters V and R; the measured distance from the image to the camera was about 300 mm, and the lateral motion of the camera was about 20 mm. Figure 10 shows the captured image from three viewpoints from the left to the right in the optical system, which used an SLM with a 3.74-μm pixel pitch. The viewing angle of this system was up to ${8.2}^{\circ }$, and the measured distance from the image to the camera was about 300 mm, while the lateral motion of the camera was about 43 mm. By comparing Figure 9 and Figure 10, the reconstruction quality of Figure 10 is better than that in Figure 9, because the SLM in Figure 10 has a higher resolution and smaller pixel pitch.

We tiled two SLMs with a pixel pitch of 3.74 μm and used a full-color light source, red, green, and blue lasers at wavelengths of 632.8 nm, 532.8 nm, and 473 nm, respectively. The reconstructed color image is shown in Figure 11, and the measured distance from the image to the camera was about 300 mm, while the lateral motion of the camera was about 86 mm. Compared to the system based on single SLM, its viewing angle increased significantly. It was demonstrated that large viewing angles can be obtained in this binocular holographic HMD.

According to Equation (1), the viewing angle can reach ${16.4}^{\circ }$, in the two tiled SLMs with a pixel pitch of 3.74 μm, and it can be further increased, if the 4f system can be used in the display system to reduce the pixel pitch. The 4f system consists of two convex lenses. When the focal length of the lens close to the SLM is larger than the focal length of the lens close to the human eye, the 3D image viewing angle can be enlarged.

Through the above work, we proved the advantages of holographic display applications in HMD, such as more realistic images, meeting human perception habits. Compared to current 3D VR and AR, holography has more feasibility for future near-eye 3D display.

5. Feasibility Analysis

Currently, VR and AR glasses on the market usually use liquid crystal displays (LCD) or organic light-emitting diodes (OLED), which are very good for 2D display, but cannot be used as SLM. LCoS is a kind of display module in SLM, and it has a compact structure, which is suitable for 3D HMDs; because its size is about on inch, its resolution is currently up to 4K, and its minimum pixel pitch reaches 3.74 μm. The viewing angle of holographic HMD can be ${16.4}^{\circ }$, which is wide enough for near-eye display. Furthermore, with the development of lasers and other light sources, it is easy to get enough high intensity of readout light with a small size and low energy consumption. Therefore, true 3D HMD with a high brightness, high resolution, large viewing angle, and depth cue will be easy to obtain by holography with the development of SLMs. We believe that holographic 3D display for VR and AR can meet the requirements of near-eye display.

6. Conclusions

In this paper, holographic 3D HMD suitable for observers with different pupillary distance is proposed. We built a monocular system and a binocular system using SLMs with pixel pitches of 3.74 μm and 8.1 μm for 3D HMD. Based on the experimental results, we demonstrated that the viewing angle of the reconstructed 3D image of the holographic HMD can be improved by tiled SLMs and the 4f system. In our binocular system, it reached ${16.4}^{\circ }$, which was two-times wider than the original viewing angle formed by the monocular system. This is meaningful for holographic application in VR and AR. Holographic 3D VR and AR displays can be good candidates for true 3D near-eye display, and compared with stereoscopic display, they can give viewers a more realistic visual experience with continuous wavefront reconstruction. Thus, we demonstrate that holographic technology can be combined with VR and AR to achieve true 3D HMD, providing technical reference for 3D VR and AR applications in various fields.