Electrically adjustable location of a projected image in augmented reality via a liquid-crystal lens

An augmented reality (AR) system involving the electrically tunable location of a projected image is implemented using a liquid-crystal (LC) lens. The projected image is either real or virtual. By effectively doubling the LC lens power following light reflection, the position of a projected virtual image can be made to vary from 42 to 360 cm, while the tunable range for a projected real image is from 27 to 52 cm on the opposite side. The optical principle of the AR system is introduced and could be further developed for other tunable focusing lenses, even those with a lower lens power. The benefits of this study could be extended to head-mounted display systems for vision correction or vision compensation. We believe that tunable focusing LC optical elements are promising developments in the thriving field of AR applications. ©2015 Optical Society of America OCIS codes: (250.0250) Optoelectronics; (230.3720) Liquid-crystal devices; (230.2090) Electro-optical devices. References and links 1. B. Kress, E. Saeedi, and V. Brac-de-la-Perriere, “The segmentation of the HMD market: Optics for smart glasses, smart eyewear, AR and VR headsets,” Proc. SPIE 9202, 92020D (2014). 2. B. Kress and T. Starner, “A review of head-mounted displays (HMD) technologies and applications for consumer electronics,” Proc. SPIE 8720, 87200A (2013). 3. O. Cakmakci and J. Rolland, “Head-worn displays: a review,” J. Disp. Technol. 2(3), 199–216 (2006). 4. J. P. Rolland and H. Hua, “Head-Mounted Display Systems,” in Encyclopedia of Optical Engineering, R. B. Johnson and R. G. Driggers, eds. (Taylor and Francis, 2005), pp. 1–13. 5. S. Liu, H. Hua, and D. Cheng, “A Novel Prototype for an Optical See-Through Head-Mounted Display with Addressable Focus Cues,” IEEE Trans. Vis. Comput. Graph. 16(3), 381–393 (2010). 6. S. Sato, “Liquid-Crystal Lens-Cells with Variable Focal Length,” Jpn. J. Appl. Phys. 18(9), 1679–1684 (1979). 7. L. Li, D. Bryant, T. Van Heugten, and P. J. Bos, “Speed, optical power, and off-axis imaging improvement of refractive liquid crystal lenses,” Appl. Opt. 53(6), 1124–1131 (2014). 8. H. S. Chen, Y. J. Wang, C. M. Chang, and Y. H. Lin, “A polarizer-free liquid crystal lens exploiting an embedded-multilayered structure,” IEEE Photonics Technol. Lett. 27(8), 899–902 (2015). 9. M. Ye, B. Wang, M. Uchida, S. Yanase, S. Takahashi, M. Yamaguchi, and S. Sato, “Low-voltage-driving liquid crystal lens,” Jpn, J. Appl. Phys. 49(10), 100204 (2010). 10. H. Ren, D. W. Fox, B. Wu, and S. T. Wu, “Liquid crystal lens with large focal length tunability and low operating voltage,” Opt. Express 15(18), 11328–11335 (2007). 11. H. S. Chen, Y. H. Lin, A. K. Srivastava, V. G. Chigrinov, C. M. Chang, and Y. J. Wang, “A large bistable negative lens by integrating a polarization switch with a passively anisotropic focusing element,” Opt. Express 22(11), 13138–13145 (2014). 12. G. Q. Li, P. Valley, P. Ayras, D. L. Mathine, S. Honkanen, and N. Peyghambarian, “High-efficiency switchable flat diffractive ophthalmic lens with three-layer electrode pattern and two-layer via structures,” Appl. Phys. Lett. 90(11), 111105 (2007). 13. Y. H. Fan, H. Ren, and S. T. Wu, “Switchable Fresnel lens using polymer-stabilized liquid crystals,” Opt. Express 11(23), 3080–3086 (2003). 14. H. S. Chen, Y. H. Lin, C. M. Chang, Y. J. Wang, A. K. Srivastava, J. T. Sun, and V. G. Chigrinov, “A polarized bifocal switch based on liquid crystals operated electrically and optically,” J. Appl. Phys. 117(4), 044502 (2015). 15. Y. H. Lin and H. S. Chen, “Electrically tunable-focusing and polarizer-free liquid crystal lenses for ophthalmic applications,” Opt. Express 21(8), 9428–9436 (2013). 16. G. M. Dai, Wavefront Optics for Vision Correction (SPIE Press, 2008). #246005 Received 15 Jul 2015; revised 8 Oct 2015; accepted 15 Oct 2015; published 19 Oct 2015 © 2015 OSA 2 Nov 2015 | Vol. 23, No. 22 | DOI:10.1364/OE.23.028154 | OPTICS EXPRESS 28154 17. F. W. Campbell and G. Westheimer, “Dynamics of accommodation responses of the human eye,” J. Physiol. 151(2), 285–295 (1960). 18. Z. Y. Luo, F. L. Peng, H. W. Chen, M. G. Hu, J. Li, Z. W. An, and S. T. Wu, “Fast-response liquid crystals for high image quality wearable displays,” Opt. Mater. Express 5(3), 603–610 (2015). 19. H. C. Lin and Y. H. Lin, “A fast response and large electrically tunable-focusing imaging system based on switching of two modes of a liquid crystal lens,” Appl. Phys. Lett. 97(6), 063505 (2010). 20. Y. H. Lin, J. M. Yang, Y. R. Lin, S. C. Jeng, and C. C. Liao, “A polarizer-free flexible and reflective electrooptical switch using dye-doped liquid crystal gels,” Opt. Express 16(3), 1777–1785 (2008).


Introduction
Augmented reality (AR) is a technology that visually augments the real-world environment using computer-generated information [1][2][3][4].Such information is usually made visible in the form of a virtual image viewed with a wearable AR device consisting mainly of a liquidcrystal display (LCD) panel, projection lens modules, and light guides.However, the plane of focus of this projected virtual image may not coincide with that of the physical object (socalled image registration) owing to the fixed lens power of conventional lenses.This discrepancy can generate visual fatigue in the user, who must continuously adjust the crystalline lens in the eye.This problem therefore motivates the development of electrically adaptive optical elements that can register a projected virtual image with objects in the real world.Examples include tunable focusing liquid crystal (LC) lenses, liquid lenses, and spatial light modulators [3][4][5][6][7].S. Liu et al. reported a head-mounted display (HMD) that used a liquid lens to adjust the location of a virtual image [5].However, the aperture size of this lens was small (~2.5 mm), which limited the size of the exit pupil and reduced the field of view.The virtual image could be adjusted only between 40 and 100 cm.High power consumption and bulky structures are obstacles to the application of liquid lenses to portable devices [2].On the other hand, LC lenses are compact, light, electrically tunable, economical, and therefore more promising for use in AR systems.LC lens powers are electrically and continuously tunable via the spatial orientations of LC molecules in external electrical fields [8].Nevertheless, conventional LC lenses are not suitable for AR applications because of their polarization dependency and small aperture size (~2 mm), which result in a low optical efficiency, a low spatial cutoff frequency that affects the image resolution, a low transmittance, and vignetting [6,7,9,10].The aperture size may be enlarged in Fresnel-type LC lenses, and polarization-switching-type LC lenses could also be adopted.The drawback of both of these solutions is that the focal length is not continuously tunable, and that they involve only two discrete lens powers in general [11][12][13][14].Recently, we developed polarizerfree LC lenses with a large aperture size (6-10 mm) and where the lens power is made continuously tunable by adopting a multilayered structure [8,15].Nevertheless, the small lens power of an LC lens with a large aperture (< 2 D) is impractical for AR applications.In this paper, we demonstrate an AR system where the location of the projected image is electrically tunable using a LC lens.This projected image can be virtual or real.The location of the virtual image can be made to vary from 42 to 360 cm away from the AR system, while the tunable range for the real image is from 27 to 52 cm on the opposite side.We first introduce the optical principle of the AR system.The tunable range of the projected image is related to the tunable range of the power of the LC lens in the system.The concept could be extended to other tunable focusing lenses, even those with a low lens power.Our results could benefit the development of head-mounted display systems used for vision correction or vision compensation, and the thriving research field of AR applications in general.

Operating principle and sample preparation
The proposed AR system, depicted in Fig. 1, employs a light source, a liquid-crystal-onsilicon (LCoS) display, a polarizing beam splitter (PBS), a beam splitter (BS), an LC lens, and a concave mirror.When the polarized light reflected from the LCoS panel traverses the PBS, BS, LC lens, and concave mirror, the image (the text "NCTU" in Fig. 1(a)) in the LCoS panel is reflected by the BS into an observer's line of sight.The observer can see this light and also the light or images originating from the surroundings (e.g., the building in Fig. 1(a)).
However, one of the images is clear ("NCTU") while the other is blurred (the building) when using the zero-lens power LC lens, because the real object and virtual image are not in the same plane (Fig. 1(a)).The eye sees either the real object (the building) or the virtual image ("NCTU") clearly by dynamically adjusting (i.e., accommodating) the lens power of the crystalline lens in the eye, which can cause considerable fatigue.When the LC lens functions as a negative lens, the virtual image is projected onto the plane that also contains the building.The observer can then see both objects ("NCTU" and the building) in focus simultaneously, as shown in Fig. 1(b).Regardless of the building location, the virtual mage ("NCTU") is kept aligned with it by electrically changing the LC lens power.This also means that the real and virtual worlds can "coexist".The view of the real world is augmented by the virtual world.When the LC lens functions as a positive lens, the projected image is a real image, as shown in Fig. 1(c).To explain the concept further theoretically, we define p eff as the distance (or effective object distance) between the LCoS display and the LC lens, q eff (V) as the voltagedependent distance between the projected image and the LC lens.Thus, p eff and q eff (V) satisfy the thin-lens equation: where V is the voltage applied to the LC lens, P LC (V) is the LC lens power, and P mirror is the power of the concave mirror.After re-arranging Eq. ( 1), the image distance ( ) eff q V is then expressed as: In Eq. ( 2), , where p x and n 1x are the propagation distance in the object space and the refractive index of each optical medium during light propagation, respectively.Similarly, , where q y and n 2y are the propagation distance in the image space and the refractive index of each optical medium, respectively.According to Eq. ( 2), the location of the projected image is electrically adjustable by changing the LC lens power.When P LC (V = 0) = 0, q eff (V = 0) = 1/(P mirror -1/p eff ) is a finite negative value, implying that the projected image is an upright virtual image and the furthest observable distance is also predictable, as depicted in Fig. 1(a).We can estimate q eff (V = 0) ~6.51 m after substituting the parameters p eff ~30.79 mm and P mirror ~32.32 D. When P LC (V) < 0, the projected image distance is still negative and larger than q eff (V = 0) (i.e., q eff (V = 0) < q eff (V) < 0).This indicates that the location of the virtual image could be shifted to the real-object plane, as depicted in Fig. 1

(b).
The real-world object and the projected virtual world image can align mutually by electrically adjusting the LC lens power.When P LC (V) is positive and large enough to compensate the negative lens power caused by the concave mirror and the effective objective distance in Eq.
(2), q eff (V) is positive.The projected image is real and projected between the BS and the observer, as shown in Fig. 1(c).Its location is also electrically tunable.We can therefore achieve AR by matching the real and virtual worlds by introducing an LC lens with tunable power into a head-mounted projected image system.Although the LC lens power is relatively weak, especially for large apertures (< 2 D), light is made to pass through it twice by a process of reflection, thereby effectively doubling the lens power.In addition, an LC lens is polarization-dependent because the light from the LCoS panel is polarized.The concept we propose here applies not only to LC lenses, but also to other lenses with a small tunable lens power, provided that the lens power can be altered actively by an external electric field.
Figures 1(d)-1(f) outline the structure and operating principles of the LC lens.To demonstrate the concept, we adopted a double-layered LC lens structure with a large aperture.The molecular alignments of two LC layers, separated by a polymeric layer, are parallel.Without an externally applied voltage (i.e., V 1 = V 2 = 0), the LC lens power is zero because there is no spatial phase difference within the LC lens for the linearly x-polarized incoming light in Fig. 1(d).When V 1 < V 2 , the LC molecules near the aperture center are more perpendicular, and those near the aperture edge more parallel, to the glass substrates, as depicted in Fig. 1(e).When the incident light is a linearly x-polarized planar wavefront, the light near the aperture center travels faster than that near the edge.As a result, the LC lens functions as a negative lens because the incoming plane wavefront is converted to a diverging parabolic wavefront.Similarly, when V 1 > V 2 in Fig. 1(f), the LC lens functions as a positive lens because the plane wave is converted to a converging parabolic wave.The lens power is expressed as [15]: where δn is the difference in refractive index between the center and rim of the LC lens, d the LC layer thickness, and r the aperture radius.
To fabricate the LC lens illustrated in Fig. 1(d), we adopted a double-layered structure with the two LC layers identically aligned.The LC lens structure consists of three glass substrates, indium tin oxide (ITO) layers serving as electrodes, an insulating layer of the polymer NOA81 (Norland), a highly resistive ZnO layer (sheet resistance: ~10 6 Ohm/square), alignment layers of polyvinyl alcohol (PVA), two LC layers, and a polymeric layer separator.The middle polymeric layers are used not only to align the LC directors but also to separate the two LC layers.The polymeric layer fabrication process is as follows.We filled nematic LC (Merck, MLC 2144, Δn = 0.2493 for λ = 589.3nm at 20°C), reactive mesogen (Merck, RM-257), and a photoinitiator (Merck, IRG-184) with a 20:79:1 wt% ratio between two ITO glass substrates coated with mechanically buffered polyimide (PI) with anti-parallel rubbing directions.The separation of the two ITO glass substrates was 50 μm.A voltage of 300 V rms at a frequency f of 1 kHz and ~3m W/cm 2 UV light was applied to the ITO glass substrates for 1 hour to enable photo-polymerization.We then peeled off the two ITO glass substrates by a thermal releasing process to obtain the 50-μm-thick polymeric layer.The material of two LC layers was a nematic LC mixture of LCM-1790 (LCMatter Corp., LCM-1790, Δn = 0.4172 for λ = 589 nm at 21°C).The thickness of each LC layer in Fig. 1(d) was 35 μm.One of the ITO layers in Fig. 1(d) was etched with a 10-mm-diameter hole, which defined the LC lens aperture size.

Experiment results and discussion
We used a Shack-Hartmann wavefront sensor (Thorlabs, WFS-150-7AR) to measure the optical phase difference (OPD) of the wavefront and the LC lens power after light passed through the LC lens [8,15].Unpolarized light from the laser diode (λ = 532 nm, GM-CW04-005) was coupled to a fiber and the light exiting the fiber was incident onto a lens so as to generate a collimated light beam.This beam then impinged on a polarizer, the LC lens, relay lenses used for adjusting the beam size, and the wavefront sensor.The spatial OPDs or wavefront was recorded by the wavefront sensor.The wavefront was expanded using Zernike polynomials and the LC lens power was calculated according to the Minus Cylinder Notation of the Zernike polynomial fittings [16].The measured lens power is plotted in Fig. 2   To demonstrate the AR system, we constructed the optical system depicted in Fig. 1(a).A LED (TouchBright X3 TB-X3-RCP, LCI) was used as a light source and an image on the LCoS panel (HX7033, resolution of 320 × 240 pixels, Himax Display Inc.) served as a projected target with a linear polarization.The LED light was shone onto the LCoS panel and this image propagated from the polarizing beam splitter (PBS) to the beam splitter (BS), the LC lens, and the concave mirror (focal length 30.94 mm).Thereafter, the reflected light continued onto the LC lens and BS, and hence to the observer (a Canon EOS 500D camera).
The distance between the BS and camera was approximately 10 cm.The camera captured both the real object and the virtual image projected from AR system when the LC lens power was negative.To measure the distance between the BS and the projected image (resolution chart), we found the location where a target was well focused on the camera We then recorded the distance between the BS and the target before taking the photos.A piece of black paper was placed behind the projected image (resolution chart) before photographing the images.Voltages V 1 = 0 and V 2 = 50 V rms were applied to the LC lens, giving a lens power of −1.32 D. The recorded image was clear when the camera focused 42 cm away from the AR system (Fig. 3(a)), but blurred when the camera focused 360 cm away (Fig. 3(b)).This implies that the virtual image is located 42 cm away.The projected image resolution can be calculated by converting the CCD camera pixel size (2.75 μm) to approximately 10 lp/mm.By analyzing the recorded images, we plotted the contrast ratio of the virtual image (resolution chart) as a function of the distance between the BS and the resolution chart, as shown in Fig. 3(c).The contrast ratio (CR) is defined as (I max -I min )/(I max + I min ), where I max and I min are the maximum and minimum irradiances, respectively.In Fig. 3(c), the highest CR occurs at distances 42, 48, 58, 81, 145, and 360 cm between the BS and image, for LC lens powers of −1.32, −1.32, −1.26, −0.99, −0.51, and + 0.1 D, respectively.This variation confirms the electrical adjustability of the virtual image location.We also plotted the image distance as a function of the LC lens power in Fig. 4. The image distance is the sum of two components: the distance between the BS and the projected image, and the distance between the LC lens and the BS (~1 cm).A positive image distance signifies that a real image is located between the BS and the observer (In Fig. 1(a)), whereas a negative image distance signifies that a virtual image is located in the opposite direction to the real image.To observe the real image, we placed a diffuser between the BS and the camera before photographing it.In Fig. 4, the projected image is virtual when the LC lens power is negative, and real when it is positive.The virtual image distance changes from 360 cm (P LC = + 0.1 D) to 42.0 cm (P LC = −1.32D) and the real image distance from 54 cm (P LC = 0.83 D) to 27 cm (P LC = 1.71D).Equation ( 2) also yields the image distance as a function of the LC lens power (solid line in Fig. 4) after substituting the experimental parameters (P mirror = 32.32D and p eff = 30.79mm).The deviation between the measured and calculated results might result from the depth of field of the camera lens and the thin-lens approximation.According to the experimental results shown in Fig. 4, the tunable LC lens power range is (0.To further test this scenario of augmented reality, we used two printed photographs as the real scenes, placed at 360 cm (tall building) and 42 cm (short building) in front of the BS, upstream of the camera.With the LCoS panel turned off, the camera could capture either one of the buildings clearly, while the other remained blurred, by adjusting the camera lens power, as shown in Figs.5(a) and 5(b).When the LCoS panel was turned on, displaying the text "Taiwan 101", with the LC lens power set to −1.32 D, the camera clearly captured the object at 360 cm, whereas the projected image of "Taiwan 101" was blurred, as shown in Fig. 5(c).This is because the projected image of "Taiwan 101" is not located at 360 cm.However, when we applied voltages V 1 = 0 and V 2 = 50 V rms , the lens power became 0.1 D (Fig. 5(c)) and both the real object (tall building) and the virtual image ("Taiwan 101") were clearly captured because the virtual image was shifted to the same plane as the tall building, as shown in Fig. 5(d).With the LCoS panel turned on (and displaying the text "NCTU") and the LC lens power at 0.1 D, the camera captured the short building at 42 cm, but the virtual image of "NCTU" was blurred.Both the object (short building) and the virtual image of "NCTU" could then be clearly observed with the LC lens power set to −1.32 D, as shown in Fig. 5(f).The real image could thus be augmented with the virtual image, as the virtual image location was electrically adjustable through the control of the LC lens power.We also observed a slight difference in scale between Figs. 5(d) and 5(f).The character "T" in Fig. 5(d) is magnified by a factor of 1.11 relative to Fig. 5(f), similar to the calculate value 1.12.The projected image could be real when the LC lens functions as a positive lens in the AR system.To demonstrate this, we removed the camera 85 cm away from the BS.A diffuser was placed between the BS and the camera, 27 cm away from BS.A real image was then projected onto the diffuser and photographed.For an LC lens power of 1.72 D, the projected image was real and inverted, as shown in Fig. 6(a), but when it was 0.03 D, the projected image was out of focus, as shown in Fig. 6(b).Fig. 5. Image performance of the AR system.The camera captures two objects, located (a) 360 cm (tall building) and (b) 42 cm (short building) away, by adjusting the camera lens power with the LCoS panel turned off.(c) When the LCoS panel is turned on and displays the text "Taiwan101" and the LC lens power is −1.32 D with V 1 = 0 and V 2 = 50 V rms , the camera captures the more distant object (360 cm away) clearly, while the projected image of "Taiwan101" is blurred.(d) Both the object (tall building) and the virtual image ("Taiwan101") become clear as the LC lens power changes to 0.1 D by applying V 1 = V 2 = 50 V rms .(e) When the LCoS panel, displaying "NCTU", is turned on and the LC lens power is 0.1 D (V 1 = V 2 = 50 V rms , the camera is readjusted so as to capture the object at 42 cm, but "NCTU" is blurred.(f) Both the object (short building) and the virtual image ("NCTU") then become clear as the LC lens power becomes −1.32 D (V 1 = 0, V 2 = 50 V rms ).

Conclusion
We demonstrated an AR system, involving an LC lens with electrically tunable lens power, that registers a projected virtual image with real world objects.The advantage of our system is that the effective tunable lens power is doubled (~6.08 D), resulting in a longer tunable range of projected image locations despite the small tunable LC lens power (~3.04 D).The position of the virtual image can thus be tuned from 42 to 360 cm, while the tunable range for the position of the real image is 27 to 52 cm on the opposite side.Our study solves the problem of achieving a fixed virtual image location in AR.The LC lens rise and fall response times are, resepectively, 1.7 and 3.1 s (data not shown here), and these can be compensated through the dynamic manipulation of the LCoS panel or light source [17][18][19].A polarizer-free amplitude modulator can also be used to correct for contrast degradation in the virtual image [20].The system field of view remains limited and changes with position.The concept could be extended to other tunable focusing lenses, even with a small lens power, or to vision correction in AR and virtual reality systems.

Fig. 1 .
Fig. 1.Proposed optical system for augmented reality.PBS: polarizing beam splitter.BS: beam splitter.LCoS: liquid-crystal-on-silicon. (a) The virtual image written on the LCoS panel is projected at infinity as the LC lens power is zero.(b) The virtual image, written on the LCoS panel, is projected beside the real object because the LC lens is a negative lens.(c) The projected image is a real image near the observer because the LC lens is a positive lens.(d) LC lens structure (d) without an applied voltage, or when the LC lens is (e) a negative lens (V 1 < V 2 ) or (f) a positive lens(V 1 > V 2 ).
as a function of the applied voltage.Because the LCoS panel size (~4.5 × 6 mm 2 ) is smaller than the LC lens aperture size, the measurements are shown for the original aperture diameter of 10 mm (solid dots and solid triangles) and an effective aperture diameter of 4 mm (hollow circles and hollow triangles).The LC lens operates as a positive lens at a different voltage V 2 (f = 17.5 kHz), while voltage V 1 = 70 V rms (f = 17.5 kHz).Similarly, the LC lens operates as a negative lens at a different voltage V 1 (f = 5.2 kHz), while voltage V 2 = 50 V rms (f = 5.2 kHz).The lens power for unpolarized light can vary from −2.21 to + 2.20 D with the 10-mm aperture, and from −1.34 to + 1.82 D with the 4-mm aperture.The total tunable ranges for the aperture sizes 4 and 10 mm are thus 3.16 and −4.41 D6, respectively.

Fig. 2 .
Fig.2.LC lens power as a function of applied voltage.The LC lens with a 10 mm aperture is a positive lens (red solid dots) at a fixed V 1 of 70 V rms at a frequency f = 17.5 kHz, and a negative lens (blue solid triangles) at a fixed V 2 of 50 V rms at f = 5.2 kHz.The LC lens with an effective 4 mm aperture is a positive lens (red hollow circles) at a fixed V 1 of 70V rms at f = 17.5 kHz, but is a negative lens (blue hollow triangles) at a fixed V 2 of 50 V rms at f = 5.2 kHz.

Fig. 3 .
Fig. 3. Projected virtual images recorded by the camera for an LC lens power of −1.32 D (V 1 = 0, V 2 = 50 V rms .The camera was focused (a) 42 cm or (b) 360 cm away from the BS.(c) Contrast ratio versus distance to the projected image for different LC lens powers, adjusted via the voltage pair (V 1 , V 2 ).

Fig. 4 .
Fig. 4. Image distance as a function of the LC lens power.The solid lines and the hollow circles represent the calculated and experimental results, respectively.Calculations assumed an effective object distance p eff of 30.79 mm.

Fig. 6 .
Fig. 6.The projected image is real when the LC lens is a positive lens.(a) The LC lens power is 1.72 D when V 1 = 70 V rms , V 2 = 10 V rms , and (b) 0.03 D when V 1 = V 2 = 70 V rms .