Microlens array camera with variable apertures for single-shot high dynamic range (HDR) imaging

: We report a microlens array camera with variable apertures (MACVA) for high dynamic range (HDR) imaging by using microlens arrays with various sizes of apertures. The MACVA comprises variable apertures, microlens arrays, gap spacers


Introduction
High dynamic range (HDR) imaging surpassed the contrast ratio of conventional digital cameras [1,2].A single CMOS image sensor captured low dynamic range (LDR) images that falls short in capturing the full range of lighting information found in the real world.HDR imaging provided a broader range of brightness levels, resulting in more vivid and realistic images [3,4].Conventional HDR techniques rely on the combination of multiple LDR images captured at different exposure times.However, these methods had some technical limitations in capturing dynamic scenes, resulting in motion artifacts such as ghosting or blurring [5][6][7].In addition, they still required high hardware cost such as multiple optical elements [8][9][10], non-conventional image sensor fabrication [11], spatial light modulators (DMD or LCD) [12,13] or even high computational power for learning algorithms [14][15][16].
Microlens array (MLA) cameras have actively provided a promising avenue for low-cost compact imaging solutions [17][18][19][20].They reduced the total track length (TTL) of cameras with large depth of field and small optical aberration [21,22].Consequently, the MLA cameras were actively utilized for diverse applications such as wide field-of-view imaging [23], contact imaging [24], multifocal imaging [25], 3D imaging [26,27], and portable point of care testing device [28].In addition, the MLA camera incorporated either anti-reflection structures [29,30] to improve the image brightness or light absorber structures [31][32][33] to enhance the image resolution and contrast by reducing optical crosstalk between microlenses.However, the captured images still showed some technical limitations in increasing the dynamic range of the MLA camera.
Here we report microlens array camera with variable apertures (MACVA) for single-shot HDR imaging.The MACVA comprises aperture arrays with various hole sizes, formed by a patterned chromium (Cr) thin film, microlens arrays, and packaged with CMOS image sensor arrays (Sony IMX 477, pixel size: 1.55 µm × 1.55 µm).The unique configuration allowed each microlens to collect different amount of illuminating light, producing slightly different LDR image array with various exposure levels and small visual disparities when objects were located in a far-field plane (Fig. 1).The camera finally captures a single HDR image reconstructed from different exposure bracketing LDR array images under single-shot exposure.

Microfabrication of MACVA
The MACVA involves wafer-level microfabrication techniques microlens arrays (MLAs), various aperture arrays (AAs), and the compact integration of a CMOS image sensor (Fig. 2(a)).Lift-off photoresist (DNR L-300, Dong-jin Semichem, Co., Ltd, Korea) was spin-coated onto a 4-inch borosilicate wafer substrate and then photolithographically patterned to create micropatterns with various aperture diameters using an MA-6 mask aligner.An electron beam was then used to evaporate a 100 nm thick Cr layer onto the patterns and the remaining resist was removed by using a resist stripper (DPS-7300, Dong-jin Semichem.Co., Ltd, Korea) for Cr lift-off process.A 22 µm thick negative-tone photoresist (DNR L-4615, Dong-jin Semichem, Co., Ltd, Korea) was photolithographically defined on the patterned metal layer.Microlens arrays were uniformly formed in a convection oven at 140°C for 30 minutes using a resist reflow process.The microfabricated microlens has f /1.4, a focal length of 420 µm, and a radius of curvature of 284 µm.The MLAs were precisely attached using a flip-chip bonder, with 420 µm tall alumina spacers on a single CMOS image sensor.The microscope images show the microfabricated microlens plate with variable apertures and microlens, where the Cr apertures completely block visible light (Fig. 2(b) and 2(c)).The captured images display a perspective and the side view of fully packaged MACVA, confirming that image sensor is accurately placed at the focal length of MLAs (Figs. 2(d) and 2(e)).The total track length (TTL) of the camera including the window glass is 920 µm.

MACVA characterization and HDR imaging
The MACVA demonstrates single-shot array imaging with different exposure values (Fig. 3(a)).A confocal laser scanning microscope (CLSM) displays light adjusting properties of the microlens with aperture via a collimated 532 nm laser beam.The MLAs focus a laser beam on the same focal plane, and the amount of light entering the lens clearly decreases with the aperture size.The modulation transfer function (MTF) of the captured images was also analyzed by using the slanted-edge method with the ISO 12233 target.Depending on the aperture size, each image was captured under a constant exposure value by adjusting the shutter speed.The MTF50 value was measured at different aperture sizes, with a decrease in aperture size.(Figure 3(b)).The MTF50 value shows the maximum at 156.4 cycle/mm for 2-stop (f /2.8), which is 1.97 times higher than the MTF50 value at 0-stop (f /1.4).The MTF50 value initially increases due to the decrease in lens aberration as the aperture size decreases.However, the MTF50 value decreases after f /4 due to the increase of aperture diffraction.The MACVA captures array images of a printed 'Lena' image 10 cm far from the camera under single-shot exposure, showing the multiple LDR array images at different exposure values (Fig. 3(c)).The image histogram exhibits the distribution of pixel intensities depending on the f-stop (Fig. 3(d)). 2 and 3 stops show the pixel intensity distributions of bell-shape, while 1 or fewer stops and 4 or more stops lead to skewed distributions, indicating overexposure or underexposure respectively.As a result, the MACVA allows single-shot array imaging with different exposure levels, selecting a set of the proper f-stops for HDR imaging.
HDR images were finally reconstructed by developing an HDR imaging algorithm.LDR images depending on different f -stops were used as inputs and combined for a single HDR image through weighted average of the pixel intensity values.The radiance value (E i ) at each pixel of the HDR image was determined using the following equation.
, where Z ij is the pixel intensity at the i-th pixel of the j-th image, and F j is the f-stop of j-th image.
The weighting factor w(z) as determined by using a normalized pyramid function, where w(z) takes the value of (z − Z min ) for z ≤ 1 2 (Z min + Z max ) and (Z max − z) for z ≥ 1 2 (Z min + Z max ).Tone mapping was finally employed to spatially render HDR images on LDR formats [14].The MTF curves were evaluated for HDR images formed by reconstructing LDR images captured at three different sets of f-stops, as well as a single f -stop (f /4) image, all at optimal exposure of 3-stop.(Fig. 4(a)).The MTF50 values for HDR1 (from three LDR images for f /2, f /4, and f /8, i.e., 2-stop difference) and HDR2 (from three f /2.8, f /4, and f/5.6, i.e., 1-stop difference) are comparable with a single f-stop (f /4) image at 153.2 cycles/mm.However, the MTF50 value for HDR3 (from three LDR images for f /1.4,f /4, and f /11.2, i.e., 3-stop difference) is reduced to 127.8 cycles/mm.This decrease is attributed to the low MTF50 values of the individual LDR images captured at f /1.4 and f /11.2, as depicted in Fig. 3(b).Consequently, a set of LDR images with 2-stop difference provides both a broad dynamic range and high resolution for HDR imaging.Single-shot HDR imaging was experimentally demonstrated and compared with multi-shot HDR imaging (Fig. 4(b)).The single-shot HDR images were reconstructed from three LDR images for f /2, f /4, and f /8.The MACVA captures array images of a chip-on-board at a distance of 5 cm.The reconstructed HDR image clearly exhibits a broad dynamic range without data loss unlike the LDR images.In addition, the MACVA captures a rotating color disc (40 rpm) and a lighting lamp at a distance of 20 cm, allowing for a comparison between single-shot and multi-shot exposures.For the single-shot exposure, LDR images at different exposure values were simultaneously captured with a shutter speed of 16 ms.On the other hand, LDR images for f /2 were obtained through multi-shot exposures with shutter speeds of 16 ms, 4 ms, and 1 ms.The reconstructed single-shot HDR image clearly exhibits minimal motion artifacts, unlike the multi-shot HDR image [34], yet both HDR images provide a broad dynamic range.

Conclusions
In summary, this work has successfully demonstrated an ultrathin arrayed camera for high dynamic range imaging.The MACVA features microlens arrays with variable apertures, which capture LDR images with different brightness levels under single-shot exposure.The reconstructed HDR images clearly display not only expanded dynamic ranges surpassing LDR images, but also high resolution comparable to the maximum MTF50 value among the LDR images.Unlike conventional HDR techniques, the MACVA minimize motion artifacts effectively.Furthermore, the potential for future developments, such as adjusting the MACVA's fill factor, suggests the capability not only for high dynamic range but also high-resolution imaging.This camera provides a new platform for diverse compact and affordable imaging solution in machine vision or mobile devices.

Fig. 1 .
Fig. 1.Microlens array camera with variable apertures (MACVA) for high-dynamic-range (HDR) imaging.The MACVA features a variable aperture arrays (AAs), microlens arrays, gap spacers, and a CMOS image sensor.Objects situated in the far-field plane are imaged uniformly across each channel, because of the minimal visual disparity of individual channels.The variable apertures collect different amount of light to form array images under single-shot exposure so that array images of multiple exposure bracketing are simultaneously acquired via microlens arrays.The HDR image reconstructed by LDR images with HDR imaging and tone mapping algorithms.

Fig. 2 .
Fig. 2. Microfabrication and configuration of the MACVA.(a) The microfabrication steps of MACVA.The patterned Cr layer was formed by using metal lift-off process.After photolithographic patterning, the microlens arrays were created by using thermal reflow.The microlens plate was fully packaged with gap-spacers on a single CMOS image sensor.(b) A transmission microscope image of microlens plate.(c) A captured image of microlens plate.(d) Optically captured side view of MACVA after packaging.(e) A photograph of a fully packaged MACVA.

Fig. 3 .
Fig. 3. MTF and image histogram measurement depending on the f-stops.(a) Optically sectioned beam profiles through microlenses (scale bar: 100 µm) and captured ISO 12233 target images with different f -numbers.(b) Measured MTF50 at constant exposure value and luminance mean under single-shot exposure for the f -stops.(c) "Lena" array image captured under single-shot exposure.(d) The intensity histogram exhibits the distribution of pixel intensity depending on the f -stop.

Fig. 4 .
Fig. 4. HDR imaging through the MACVA.(a) The measured MTF curves of tone-mapped HDR images, and f /4 LDR image.HDR1 image was reconstructed by using f /1.4,f /4, and f /11.2 LDR images, HDR2 image was reconstructed by using f /2, f /4, and f /8 LDR images, and HDR3 image was reconstructed by using f /2.8, f /4, and f /5.6 LDR images.(b) Comparison of single-shot and multi-shot HDR imaging.The single-shot HDR images reconstructed by f /2, f /4, and f /8 LDR images with a shutter speed of 16 ms and the multi-shot HDR image is created by LDR images for f /2 with shutter speeds of 16 ms, 4 ms, and 1 ms.