Wide-angle Structured Light with a Scanning MEMS Mirror in Liquid

Microelectromechanical (MEMS) mirrors have extended imaging and vision capabilities onto mobile platforms such as hand-held projectors. However, the field-of-view (FOV) of these MEMS mirrors is usually less than 90◦ and any increase in the MEMS mirror scanning angle has design and fabrication trade-offs in terms of power, size, speed and stability. Therefore, we need techniques to increase the scanning range while still maintaining a small form factor. In this paper we exploit our recent breakthrough that has enabled the immersion of MEMS mirrors in liquid. While allowing the MEMS to move, the liquid additionally provides a “Snell’s window” effect and enables an enlarged FOV (≈ 150◦). We present an optimized MEMS mirror design and use it to demonstrate applications in extreme wide-angle structured light. © 2015 Optical Society of America OCIS codes: (130.3990) Micro-optical devices; (150.0150) Machine vision References and links 1. S. K. Nayar, V. Branzoi, and T. E. Boult, “Programmable Imaging: Towards a Flexible Camera,” IJCV, 2006. 2. S. J. Koppal, S. Yamazaki, and S. G. Narasimhan, “Exploiting dlp illumination dithering for reconstruction and photography of high-speed scenes,” International journal of computer vision, vol. 96, no. 1, pp. 125–144, 2012. 3. M. Duarte, M. Davenport, D. Takhar, J. Laska, T. Sun, K. Kelly, and R. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Processing Magazine, 2008. 4. Microvision, “Picop technology,” www.microvision.com, 2015. 5. S. T. Holmstrom, U. Baran, and H. Urey, “Mems laser scanners: a review,” Microelectromechanical Systems, Journal of, vol. 23, no. 2, pp. 259–275, 2014. 6. Velodyne, “Velodyne lidar,” velodynelidar.com, 2015. 7. X. Zhang, R. Zhang, S. Koppal, L. Butler, X. Cheng, and H. Xie, “Mems mirrors submerged in liquid for wideangle scanning,” Transducer, 2015. 8. R. W. Wood, “Physical optics,” Macmillan, 1911. 9. S. J. Koppal, I. Gkioulekas, T. Young, H. Park, K. B. Crozier, G. L. Barrows, and T. Zickler, “Towards wide angle micro-sensors,” Pattern Analysis and Machine Intelligence, IEEE transactions on, 2013. 10. A. Zomet and S. Nayar, “Lensless imaging with a controllable aperture,” CVPR, 2006. 11. O. Cossairt, D. Miau, and S. Nayar, “A scaling law for computational imaging with spherical optics,” JOSA, 2011. 12. D. Brady, M. Gehm, R. Stack, D. Marks, D. Kittle, D. Golish, E. Vera, and S. Feller, “Multiscale gigapixel photography,” Nature, 2012. 13. H. Ko, G. Shin, S. Wang, M. Stoykovich, J. Lee, D. Kim, J. Ha, Y. Huang, K. Hwang, and J. Rogers, “Curvilinear electronics formed using silicon membrane circuits and elastomeric transfer elements,” Small, 2009. 14. K. Jeong, J. Kim, and L. Lee, “Biologically inspired artificial compound eyes,” Science, 2006. 15. S. Hiura, A. Mohan, and R. Raskar, “Krill-eye : Superposition compound eye for wide-angle imaging via grin lenses,” OMNIVIS, 2009. 16. K. E. Petersen, “Silicon torsional scanning mirror,” IBM Journal on Research and Development, 1980.


Introduction
The advent of microelectromechanical (MEMS) spatial light modulators such as DMD (digital micro-mirrors) have allowed fast and accurate applications such as coded aperture [1], analysis of fast moving scenes [2] and compressive sensing [3].These devices provide an excellent platform to extend active computer vision techniques to the low-power and miniature domains.For example, Microvision [4] offers a single raster scanning MEMS micro-mirror, which is implemented on a 5W mobile system.
In this paper we focus on the field-of-view (FOV) of these MEMS-enabled low-power vision platforms.Current designs and fabrication techniques offer MEMS optical scanning ranges of up to 90 • [5].While this is sufficient for many tasks, vision techniques that require panoramic imaging or surround-view reconstruction would require a greater viewing angle.In macro-scale systems that require a wide FOV, such as the LIDAR structured light system from [6], mechanical rotation increases the sensing range of the device.For mobile and miniature platforms, such continuous mechanical movement would place an unacceptably high power and weight cost.Fig. 1: MEMS mirror submerged in liquid In (I) we show a ray diagram of our setup, where a MEMS mirror is immersed in liquid.We used this setup [7] to induce the well-known Snell's window effect that allows for modulating light over a large FOV, as shown in (II).In this paper, we demonstrate, for the first time, wide-angle structured light applications using an optimized MEMS mirror that is designed for use in liquid.
To enable futuristic applications such as micro/nano LIDAR or mobile 360 • coded aperture photography, wide-angle optics is required for MEMS modulators.Recently, we demonstrated a working MEMS device immersed inside a liquid whose refractive index is greater than that of air [7].This allows the mirror to scan freely and induces the well-known "Snell's window" optical effect [8].In air, such a mirror rotates over a 2θ air FOV.Due to Snell's law, a submerged mirror would have a FOV of 2θ liq = 2 arcsin(n sin θ air ).When the critical angle is reached and θ air = arcsin( 1 n ), the modulator's FOV can theoretically be 180 • .Figure 1 shows our prototype, where our MEMS mirror modulates an external laser.
In this paper, we harness and extend this MEMS breakthrough to build a wide-angle and miniature light modulator prototype for vision and imaging applications.Our contributions are: (1) We design a new MEMS mirror construction that is fabricated to achieve a larger mechanical angle.This design allows nearly 150 • FOV with a modulating form factor of around 1inch 3 , compared with 120 • FOV in [7].
(2) We demonstrate wide-angle structured light reconstruction using a laser-based line striping vision sensor, using our proposed optical design.For the first time, we can enable reconstruction results at nearly 130 • FOV and with a modulator size of around 1inch 3 .
Note: All the prototypes discussed in this paper use 1D scanning mirrors and require additional optics that are not miniaturized.However, this is an artificial constraint since there is no practical or theoretical limit to using 2D MEMS mirrors in our designs.

Related Work
Our work shares connections with vision, imaging, robotics, MEMS and optical 3D printing.
Vision and Computational Photography: The Snell's window effect has been used previously [9] for a passive vision sensor using a refractive slab.In contrast, we use liquid media that allows mechanical motion inside the optics and enables both passive and active sensing.It is worth noting that MEMS and other spatial light modulators have been widely used in vision and imaging as programmable templates [1,10].While we use a single MEMS sensor in this work, much of our optical designs will carry over to such devices.Curved optical designs can enhance the FOV of our devices, and our ideas complement work in computational photography to increase imaging bandwidth using curved sensors [11,12,13] and artificial insect eyes [14,15].
MEMS Mirrors: Since the first MEMS scanning mirror by K. E. Peterson [16], MEMS mirrors have enabled applications such as optical coherence tomography [17], projectors and displays [18] and printers [19].Wide-angle MEMS mirrors in air include different driving mechanisms such as electrostatic [20,21], piezoelectric [22], electromagnetic [23], and electrothermal [24].Some of these technologies have problems when operating in liquid.For example, due to viscosity damping, most resonance-based electrostatic or piezoelectric mirrors may have significant decrease in performance.We focus on electrothermal devices since electromagnetic mirrors require magnets or coils which increase fabrication complexity.In addition, electrothermal devices have a large spacing underneath the mirror, which allows liquid to fully surround the mirror and prevents air bubbles from forming.
Although Huang et al. [25] investigated the impact of acoustic MEMS devices in water, we were the first [7] to show that MEMS mirrors immersed in liquid could scan over a large range at moderate speed and could also have interesting control properties such as voltage, temperature gradient, etc.In particular, we showed an increased optical viewing angle is possible.In this paper we propose a design with a wider FOV than [7] and we are the first to use MEMS mirrors immersed in liquid for structured light reconstruction.
Other Miniature Wide-FOV Optics: Wide FOV optics [26] and micro-optics are active research fields [27,28,29].Our liquid MEMS design offers a new axis of optical modulation when compared to other available options.For example, commercially available fish lenses (such as [30]) and those custom made for research purposes (e.g.[31]) consist of multiple optical elements at cm scales.However, using these for MEMS modulation, as done by [31], has resulted in only 120 • FOV being demonstrated.Miniature polygonal mirror wheels allow fast laser modulation for a variety of applications such as range scanning [32].However, when compared with MEMS devices, polygonal mirrors face a disadvantage with respect to raster scanning; 2D MEMS mirrors can switch very quickly to reflect light in different directions, which is not possible with a polygonal mirror wheel.Curved mirrors allow panoramic imaging for computer vision applications [33] and have no dispersion related problems.Some designs, such as hyperbolic and ellipsoidal mirrors, have single-viewpoints, which allow many computational advantages.Unfortunately, to the best of our knowledge, the state-of-the-art for miniature curved mirrors does not appear to have a greater FOV than 45 • [34].

MEMS Mirror Immersed in Liquid
Electrothermal lateral shift free (LSF) mirrors with three bimorph segments can cancel the lateral shift and achieve large actuation range [24].We created a device using this principle, shown in Fig. 2(I-II).The large initial elevation and separation of the bimorph beams make the mirror less likely to suffer from the damping or stiction problems that occur in a liquid medium.
Only [7] has demonstrated immersing a MEMS mirror in liquid.Compared with [7] the embedded heater Pt layer in our design is increased from 0.1µm to 0.2µm while the active Al and SiO 2 layers of the bimorphs are still kept at 1µm (see Fig. 2(I)).The thickness increase strengthens the bimorphs which increases the maximum achievable mechanical angle.As the amplification of angle increases drastically when the mechanical angle is approaching the critical angle ( 21.5 • ), the actual mechanical angle reached in this new device is about 20 • compared Fig. 2: Wide-angle results Our MEMS device design (I) is imaged in (II).We show a photograph of the packaged MEMS device in (III).Using a laser setup similar to Fig. 1, we measured the optical angle scanned (using an optical table and graph paper) for multiple MEMS mirror positions (IV), given by the voltage applied to the MEMS device.We performed this experiment in air and then in mineral oil (n = 1.5).We were able to obtain close to 150 • FOV with the use of mineral oil, due to the Snell's window effect.In (V) we show a long exposure image of the device as it modulates the laser in three distinct directions, illustrating the wide FOV. with 18 • of [7], resulting in the 150 • optical scanning range compared with 120 • .Moreover, with the increase of Pt thickness, the resistance of embedded heater changes from about 80Ω to 40Ω, thus the actuation voltage needed to deliver the same amount of electrical power to the bimorphs is reduced, we can reach about 150 • with less than 7V , compared 120 • with 10V reported in [7].
The footprint of the LSF mirror is 3.2mm×3.2mmwith the mirror plate size of 1.7mm×1.8mm.Two sets of LSF actuators are symmetrically arranged on both sides of the mirror plate along one axis, as shown in Fig. 2. With differential control of each actuator, the mirror plate can scan as a seesaw, and thus a symmetric scan range can be achieved.
We utilize mineral oil for the surrounding liquid, as in Fig. 1, since it does not conduct electricity and since it has a high refractive index.Once immersed in the oil, the same mechanical rotation is able to achieve a ±75 • optical scanning range at only 6.7V.We measured the fields of view for different MEMS locations using the setup in Fig. 1 for both air and liquid immersion, and these are reported in Fig. 2(IV).Our device demonstrates a field-of-view of 150 • depicted in Fig. 2(V) using a long exposure.

Wide-angle Structured Light
We use MEMS mirror design to modulate a laser for structured light line striping [35].Since our prototype uses a 1D scanning mirror that modulates a single laser, we use a custom cylindrical   3 is kept at a "close position" within a 92 • FOV.In (II) the same scene is moved to a "far position", and is still illuminated by our sensor at around 128 • FOV.Note that our sensor has a 150 • FOV and our darkroom was not big enough (at that scene depth) to demonstrate the full FOV of the device.In (II) and (IV) we use a stereo camera viewing the line striped scene to recover depths.We obtained an average error of ±2 inches over the entire scene depth of 30 inches.Since each of the five boxes were fronto-parallel, we also calculated the RMS distance by fitting a plane to each box and the average such error was 0.56 inches.lens of 6mm focal length to convert the laser spot into a light plane.Our ray diagram and optical setup is given in Fig. 3 (I-III), where the MEMS sensor is packaged in mineral oil.We use a 1mm thick glass slide (whose refractive index of 1.46 is similar to that of mineral oil) and a 3D printed part made with an Objet Connex 20 micron plastic printer to form the housing of the MEMS mirror and the liquid.This design uses gravity to ensure that there are no air pockets on the liquid-glass boundary and that the liquid-air boundary is not in the optical path.In Fig. 3 (IV) we show an image captured from our setup, revealing a bright laser line stripe.
As our device sweeps a laser light plane over the scene, we use spacetime stereo [36,37] to exploit the illumination variation to recover horizontal disparity.We use a Point Grey Bumble-Fig.5: Reconstruction of a toy doll: We use the setup in Fig. 4 to reconstruct a toy doll (I).The object was placed around 50 • from the viewing direction.We show the depth map (II) and a 3D reconstruction (III) that is rendered from a different viewpoint.Note that the color image was not taken from either stereo pair but from an SLR camera placed close to the stereo pair.bee2 stereo camera that provides calibrated, synchronized and rectified video streams to obtain depth from disparity.As a review, spacetime-stereo matches the intensity profile of a pixel in one camera to the closest match along the epipolar line in the other camera.Therefore, if we are considering pixel (x, y) in the left camera image I l , then the search optimization becomes, arg min where d is the stereo disparity, t is the variable corresponding to the video time sampling and the epipolar line is horizontal since the images are rectified.We place a scene consisting of four boxes of sizes, ranging from a few inches to approximately a foot in width, in front a white lambertian plane such that each box is at a random distance from the plane.The scene is then moved to different positions in the field of view of the sensor; first at 46 • from the normal viewing direction (Fig. 4(I)) and then at 64 • from the normal (Fig. 4(III)).The latter represents a 128 • sensor FOV, which is lower than the achievable 150 • FOV due to limited size of our optical dark room.Our optical setup is in exactly the same location in the two experiments.What changes is the scene location and the camera positions that measure image data.We demonstrate reasonably good depth estimates both qualitatively from the depth maps in the figure and quantitatively, by measuring the inter-box distance using a tape measure and comparing with results from the stereo reconstruction.The latter revealed a ±2 inches error over the entire scene depth of 30 inches.The depth maps provide a correct reconstruction of the scene, as demonstrated by the rendered point cloud, and are consistent across the FOV.Since the boxes were fronto-parallel we also calculated the RMS distance by fitting a plane to each box and this average error was 0.56 inches, which is confirmed by the mostly error free depth map.
Finally, we applied the same method to reconstruct a toy doll, as shown in Fig. 5.The doll was placed approximately at 50 • from the normal direction.In (I) we show a color image of the doll, demonstrating that it consists of non-lambertian materials, such as fur and plastic.In (II) we show a depth map from our method, without any post-processing, which is almost noisefree.A rendering from a different angle in (III) was created by estimating surface normals and rendering under different lighting.Note that the thumb of the doll's right hand and the stub of its toes are clearly visible in the reconstructed result.

Summary and Future Work
In summary, we have presented a new MEMS mirror design that allows for a large scanning angle and have demonstrated its ability to move in liquid and modulate light over a large FOV.This allowed for the construction of a line striping projector with almost a 150 • FOV in a small, compact form.The main challenge we have overcome is to enable, for the first time, vision applications for a MEMS mirror modulating light while remaining fully submerged in liquid.This opens up a new range of questions of optical design, MEMS engineering and vision sensing.An important issue that we face is the decrease in bandwidth of the MEMS scanning due to being inside the liquid.In this paper we have avoided this problem by considering static scenes.For dynamic scenes, MEMS motion will result in the issues of heat transfer inside the liquid, turbulence and the relationship of these physical effects to motion blur.We hope to explore these types of research questions in the future.

Fig. 3 :
Fig. 3: Optical setup: (I) shows a diagram of the setup in (II), which allows us to scan a laser line across a 150 • FOV, due to the refractive effect of the mineral oil shown in (III).In (IV) a scene consisting of cardboard boxes is shown with the projected laser line stripe.

Fig. 4 :
Fig.4: Wide-angle laser line striping: In (I) the cardboard scene shown in Fig.3is kept at a "close position" within a 92 • FOV.In (II) the same scene is moved to a "far position", and is still illuminated by our sensor at around 128 • FOV.Note that our sensor has a 150 • FOV and our darkroom was not big enough (at that scene depth) to demonstrate the full FOV of the device.In (II) and (IV) we use a stereo camera viewing the line striped scene to recover depths.We obtained an average error of ±2 inches over the entire scene depth of 30 inches.Since each of the five boxes were fronto-parallel, we also calculated the RMS distance by fitting a plane to each box and the average such error was 0.56 inches.