Lissajous scanned variable structured illumination for dynamic stereo depth map.

Structured illumination plays an important role in advanced photographic and microscopic imaging applications. Here we report variable structured illumination (VSI) using Lissajous scanning techniques. The variable structured illumination module comprises Lissajous scanning micromirror and fiber-based diode pumped solid state (DPSS) laser with intensity modulation, combined with a stereo camera for dynamic stereo depth map. The micromirror projects static and discrete patterns by modulating the intensity of a laser beam at the least common multiple (LCM) of two scanning frequencies. The pattern density is increased by either decreasing the greatest common divisor (GCD) of scanning frequencies or decreasing the duty rate of the laser modulation. The scanning amplitude also controls the field-of-view (FOV) for the exact illumination of a target object for dynamic stereo depth map. The variable structured illumination module provides a new route for advanced imaging applications such as high-quality depth map, super-resolution, or motion recognition.


Introduction
Structured illumination opens a new chapter in 3D imaging [1][2][3] or super-resolution imaging [4][5]. Discrete pattern of illumination substantially improves the depth resolution of stereoscopic images by measuring the distortion of illumination patterns [6][7][8]. Unlike other non-contact 3D imaging techniques such as conventional stereo vision [9][10] or time-of-flight (TOF) [11][12], structured illumination gains more attention in 3D depth imaging due to its high quality and high uniformity [6][7][8]. Structured illumination patterns are simply realized by a diffractive optical element (DOE), combined with a single laser diode [13][14][15] or vertical cavity surface emitting laser (VCSEL) array [15][16]. The VCSEL array serves as a key element for compact 3D depth cameras in mobile-phones or laptops. However, such illumination patterns constrain high tunability of the field-of-view (FOV) and the illumination pattern density [17]. Recently, spatial light modulators (SLMs) such as digital micromirror device (DMD) [3,4] and liquid crystal display (LCD) [5,18] allow structured illumination, however, they still have technical limitations in system miniaturization or low power laser operation. For instance, the active elements of DMD [3,4] or LCD [5,18] split up incident light and thus they require highly intense laser beam to obtain distinct structured illumination patterns. They also entail a constant distance between the projector and an object to realize high contrast patterns [7][8]. In contrast, scanning micromirror allows highly intense structured illumination owing to low diffraction loss and also controls either field-of-view or the depth of focus by forming a solid scan trajectory along various object distances with a low power laser [19][20][21].
Recently, structured illumination has been demonstrated by using scanning micromirrors such as 1D micromirror with a diffractive microstructure [14][15], with a cylindrical lens [20] and 2D raster scanning micromirror [19,21]. Raster scanning micromirrors exhibit technical benefits for uniform scanning and simple reconstruction process, however, they have some bottlenecks such as low external shock-resistance and high operation voltage [24]. In contrast, Lissajous scanning micromirrors provide uniform illumination, high mechanical stability, and comparatively low operation voltage [22][23][24]. Recent studies have demonstrated the scanning frequency selection rule for high definition high frame-rate (HDHF) Lissajous scanning, which overcomes the existing trade-off between fill-factor and frame-rate [25][26].
Here we report variable structured illumination (VSI) for dynamic stereo depth map using Lissajous scanning micromirror. The stereo depth map is obtained by incorporating a stereo camera with variable structured illumination, employing Lissajous scanning micromirror and fiber-based diode pumped solid state (DPSS) laser with intensity modulation ( Fig. 1(a)). The micromirror scans at the pseudo-resonance of each axis within the resonant bandwidth ( Fig. 1(b)). The scanning frequency set is determined by the frequency selection rule to achieve high fill-factor scanning. Two-dimensional static and discrete pattern is then obtained by modulating a laser beam at the least common multiple (LCM) of two scanning frequencies. The pattern density and the FOV are flexibly controlled by changing the greatest common divisor (GCD) of two scanning frequencies, the duty rate of the laser modulation, and the operation voltage of Lissajous scanning micromirror. Note that the frequency set of high LCM at pseudo-resonance results in low GCD. The pattern density increases as the GCD and the duty rate decrease. (a) A schematic illustration of high-resolution 3D imaging using variable structured illumination module. The module consists of a Lissajous scanning micromirror and fiberoptic modulator. High resolution 3D imaging is obtained by using a variable structured illumination module and a stereo camera. (b) Working principle of variable structured illumination. The micromirror is two-dimensionally scanned at pseudo-resonant frequencies within the resonant bandwidth. Two scanning frequencies are determined by the frequency selection rule for high definition high frame-rate (HDHF) Lissajous scanning. The static patterns of structured illumination are obtained by modulating a laser beam at the least common multiple (LCM) of two selected scanning frequencies.

Lissajous scanned variable structured illumination module and opto-mechanical property
Variable structured illumination is achieved by Lissajous scanning and intensity modulation of a laser beam. Lissajous scanning micromirror was fabricated by silicon on insulator (SOI) micro-electro-mechanical-system (MEMS) process including deep reactive ion etching (DRIE). The micromirror has a physical dimension of 1.2 mm x 1.2 mm x 0.43 mm ( Fig. 2(a)). The variable structured illumination module contains micromirror, fiber-optic collimator inside a metal housing, and 532 nm fiber-based DPSS laser (MGL-III-532-AOM-60 mW, CNI Optoelectronics Tech). The micromirror is wired to the PCB board with epoxy encapsulant, mounted on 45-degree wedge in the housing, precisely aligned with the fiber-optic collimator (50-630-APC, Thorlabs), and finally sealed with a glass window (Fig. 2(b)). The physical dimension of the fully packaged structured illumination module is 16 mm x 10 mm x 8.5 mm. The micromirror is two-dimensionally operated with a low Q-factor inner mirror and outer frame. The scanning angles of the inner mirror and the outer frame were measured by shifting the scanning frequency (Fig. 2(c)). The inner mirror and the outer frame have resonance peaks at 5,164 Hz and 6,760 Hz, respectively. The micromirror performs 22-and 25-degrees of scanning angle in the inner mirror and the outer frame within 40 peak-to-peak volts, respectively. The inner mirror has a low Q-factor (bandwidth : 36 Hz) for a wide selection range of scanning frequency while as the outer frame has a relatively high Q-factor (bandwidth : 11 Hz) for less mechanical coupling between the inner mirror and the outer frame scanning. The color region in Fig. 2(c) indicates the frequency sets for high fill-factor Lissajous scanning. Lissajous scanning trajectory was captured at 1/320 sec and 1/30 sec in scanning time, respectively (Figs. 2(d)-2(e)). Lissajous scanning exhibits over 95% fill factor at 1/30 sec, faster than the frame rate of stereo camera. Finally, a prototype of high-resolution 3D camera system was packaged with the variable structured illumination module and the stereo camera (oCamS-1CGN-U, Withrobot, 120 mm base line distance) (Fig. 2(f)).

Variable pattern density
The scanning frequency sets for variable structured illumination are selected by using a color map of the GCD of two scanning frequencies (Fig. 3(a)). The frequency sets for Figs. 3(g)-3(j) were selected in the green region of the color map. A small GCD allows high laser modulation frequency, which results in high density pattern of structured illumination. However, the modulation frequency is limited by 1 MHz, i.e., the maximum modulation frequency of acousto optic modulator. The minimum GCD also corresponds to 35 for the scanning frequencies near 6 kHz. The pattern density, i.e., the number of spots, of variable structured illumination is numerically calculated depending on the GCD and the duty rate of laser modulation. The calculated results show that the pattern density is increased by either decreasing the GCD or decreasing the duty rate under a constant GCD condition (Fig. 3(b)). The experimental results also clearly show that the pattern density increases as the GCD decreases, as shown in Figs. 3(c)-3(e). In this experiment, three different sets of scanning frequencies were selected from GCD of 520, 94 to 61. The duty rate of laser modulation was constantly set to 0.15. The disparity map of a sphere plaster model was obtained by using stereo camera without structured illumination (Fig. 3(f)). Distance between the camera and the sphere model is 0.7 m. A non-textured smooth white plaster on a white background often provides low quality 3D imaging due to the substantial loss of depth information [27]. The experimental results display that the pattern density of sphere plaster model increases as the GCD decreases from 226, 123, 94 to 61 (Figs. 3(g)-3(j)). The resultant disparity maps are well matched with high uniformity and high resolution. The R-square value of a sphere model increases as the GCD decreases, however, the lowest GCD value corresponding to the highest laser modulation frequency causes imprecise pattern recognition and therefore some pixels are unclear under the stereo camera algorithm in MATLAB. This phenomenon can be further solved by improving the stereoscopic image reconstruction algorithm.

Variable field-of-view
Various FOV 3D imaging are achieved by controlling the scanning amplitude of Lissajous scanning micromirror. Unlike conventional stereo images, the 3D point-cloud images are substantially improved by using the stereo images of two plaster models captured with structured illumination (Figs. 4(a)-4(c) & See Visualization 1). In this experiment, the operation voltages of the inner mirror and the outer frame for the GCD of 61 were 36 V pp and 38 V pp for Fig. 4(b) and 28 V pp and 30 V pp for Fig. 4(c), respectively. In this experiment, the decrease of scanning amplitudes linearly reduces the FOV but substantially increases both the pattern density and the local intensity of structured illumination patterns. As a result, the reduced FOV of structured illumination effectively improves the imaging resolution of dynamic stereo depth map (Figs. 4(b)  Highly dense pattern of structured illumination provides high resolution 3D imaging.
provides flexible FOV depending on the dimension of target object and thus high quality dynamic stereo depth map is acquired. Fig. 4. Various field-of-view (FOV) 3D imaging (a) The optical images and 3D point-cloud images of (a) two plaster models without structured illumination, (b) two plaster models with structured illumination (GCD = 61), (c) only statue plaster model with smaller scanning angle, (d)-(e) Plush doll and a stomach phantom with FOV-controlled structured illumination (Scale bar : 4 cm). The depth map driven by the structured illumination results in high resolution 3D imaging, whereas many parts are broken and missing in conventional stereoscopy. Besides, the dynamic stereo depth map is achieved by varying the pattern density and the FOV with Lissajous scanning micromirror.

Conclusion
To conclude, this work has successfully demonstrated the variable structured illumination for dynamic stereo depth map using Lissajous laser scanning. The fully packaged system contains Lissajous scanning micromirror, fiber-based DPSS laser with intensity modulation, and stereo camera. The Lissajous scanned structured illumination was achieved by modulating a laser beam at the LCM of two scanning frequencies, based on the frequency selection rule for HDHF Lissajous scanning. The variable structured illumination was further driven by changing the set of scanning frequencies or the scanning amplitude of micromirror. The dynamic stereo depth map was finally obtained by controlling the pattern density or the FOV. The high resolution 3D camera system can open a new perspective for dynamic 3D imaging applications such as 3D depth map, super-resolution, motion recognition, and light detection and ranging (LIDAR).