High-speed phase-shifting incoherent digital holography (invited)

We propose high-speed incoherent digital holography with a recording speed of 1,000 fps, which is designed for the three-dimensional (3D) motion-picture measurement of moving objects with high temporal resolution. We adopt single-shot phase-shifting incoherent digital holography, a high-speed polarization-imaging camera system, and a palm-sized self-interference interferometer designed for spatially and temporally incoherent light, to implement a high-speed phase-shifting incoherent digital holography system. High-speed four-dimensional (4D) (3D and time) measurement capability is experimentally demonstrated, using the constructed incoherent digital holography system set on a wagon and a daily-use white-color light-emitting diode.


Introduction
Three-dimensional (3D) image information is essential for a person, an animal, and a machine to perceive and investigate the 3D structures of materials and samples. Both scientific research and industry always require the advancement of 3D measurement technology for accurate microscopic/ nanoscopic 3D observation, accurate and rapid control of a machine, product inspection, and self-driving of an automobile and other moving bodies. Digital holography (DH) [1][2][3][4][5] has been actively studied as a technique to obtain quantitative information on the 3D position [6], shape [7,8], and image of an object [9] with high accuracy using a single image sensor. In DH, a laser light source irradiates spatially and temporally coherent light to illuminate a 3D object. An image sensor records an interference fringe image, which is called a digital hologram and is generated by the light diffracted from the object, called an object wave, and the light irradiated from the same laser and not diffracted from the object, called a reference wave. 3D information is contained in the phase distribution of the light wave, and the phase information is converted to the intensity distribution of a digital hologram by light interference. A computer numerically reconstructs the 3D image of the object through calculations, fringe analysis such as the Fourier transform method [10] and phase-shifting interferometry (PSI) [11,12] and numerical wave propagation based on diffraction integrals [3][4][5]. The 3D motion-picture (so-called 4D (3D + time)) measurement of multiple objects has been performed with DH and a single lensless camera [13][14][15][16]. However, DH requires a laser light source to generate a digital hologram. Therefore, it is difficult to record a digital hologram of selfluminous light such as luminescent and fluorescent light, light from a flame, and sunlight, and to obtain a 3D image of the self-luminous light.
On the other hand, a method to obtain a digital hologram of spatially incoherent light, termed incoherent digital holography (IDH) [17][18][19][20][21][22][23][24][25][26], has been successively studied. IDH exploits the phenomenon of either self-interference or self-reference [27][28][29] to digitally record an interference fringe image of an object illuminated by spatially incoherent light-such an image is called an incoherent digital hologram-and the calculations of DH to reconstruct the 3D image of the object from the recorded incoherent digital hologram. IDH has the following attractive features: lensless 3D imaging is achieved [30], application to 3D fluorescence microscopy is developed [31][32][33][34], a single-path IDH system is highly robust against external vibration, a deep depth range is measured, a thumb-sized IDH optical setup is constructed [35], fully passive holographic 3D imaging is achieved with sunlight [36], and no a priori acquisition of a point spread function (PSF) is required for 3D imaging thanks to the use of a Gabor-zone plate pattern. IDH techniques for single-shot 3D imaging have been proposed [37][38][39][40][41][42][43][44][45][46], and 4D measurement has been performed by exploiting single-shot IDH techniques [22,47,48]. Singleshot phase-shifting incoherent digital holography (SSPS-IDH) [40][41][42][43][44] is a single-shot IDH technique exploiting SSPS interferometry [49][50][51]. Most of the SSPS-IDH techniques are implemented with polarimetric optical elements and an array of either polarizers or wave plates. Such the SSPS-IDH system is adopted by many IDH experts worldwide as the high-specification polarization-imaging camera can be obtained at a low cost. However, the frame rate of SSPS-IDH has been at most 100 fps until now [22].
We propose high-speed IDH with a recording speed of 1,000-fps order for the high-temporal-resolution 4D measurement of moving objects. We adopt SSPS-IDH, a highspeed polarization-imaging camera, and a palm-sized IDH optical system designed for spatially and temporally incoherent light to implement a high-speed IDH system. A compact and portable high-speed IDH optical setup with the high visibility of interference fringes is constructed to achieve high-speed 4D recordings with a daily-use light source. High-speed 4D measurement capability is experimentally demonstrated.
2 High-speed phase-shifting incoherent digital holography (HSPS-IDH) Figure 1a shows the schematic of high-speed phase-shifting incoherent digital holography (HSPS-IDH). HSPS-IDH adopts SSPS-IDH to conduct PSI from a single recorded incoherent image. The constructed HSPS-IDH system is designed to obtain high temporal resolution. Different from the previous SSPS-IDH systems, HSPS-IDH does not always Fig. 1 Schematic of high-speed phase-shifting incoherent digital holography (HSPS-IDH). a Illustration of the whole system and b photograph of the constructed palm-sized self-interference holography optical setup adopt Fresnel incoherent correlation holography (FINCH), a liquid-crystal spatial light modulator (SLM), a polarizationsensitive Fresnel phase lens displayed on the SLM, a polarimetric diffractive lens, or a grating. Instead of those devices, polarizers, birefringent materials, and a polarimetric and variable phase modulator to introduce a uniform phase shift, which is implemented by a liquid-crystal phase modulator (LC-PM), are adopted to generate a spatially and temporally incoherent hologram with high visibility and high transmittance. In contrast to a diffractive optical element, an LC-PM and other birefringent materials do not generate multiple-order diffraction waves for temporally incoherent light. In Fig. 1a, an incoherent object light wave is generated by a self-luminous object and an object illuminated by an incoherent light source. A polarizer is set in HSPS-IDH to obtain linear polarization because an incoherent light wave has generally random polarization. A birefringent lens generates two incoherent light waves whose wavefront curvature radii are different, and polarization directions are orthogonal to each other from an incoherent light wave, which is to obtain a self-interference fringe image, that is, an incoherent hologram. A birefringent variable phase modulator such as an LC-PM and a birefringent plate are set to carefully adjust the optical-path-length difference between the two incoherent light waves. Note that careful adjustment is important in IDH with a daily-use light source because the temporal coherence of a daily-use light source is severely limited, and interference fringes disappear in the case where the optical-path-length difference is more than 100 µm. To obtain an incoherent hologram with high visibility, highprecision adjustment for the optical-path-length difference is conducted by using an LC-PM. A quarter-wave plate converts the linear polarizations of two incoherent light waves into circularly polarized incoherent light waves with opposite handedness. A band-pass filter improves the temporal coherence of incoherent light. A micropolarizer array in a high-speed polarization image sensor consists of polarizers with four transmission axes. Each micropolarizer aligns the polarization directions of the two light waves. A phase shift between the two light waves differs along the transmission axis of the micropolarizer. One micropolarizer is attached to one photodetector of the image sensor. Therefore, the highspeed polarization image sensor digitally records a single image containing four phase-shifted incoherent holograms, which is based on the space-division multiplexing of phaseshifted holograms. Image reconstruction procedures are conducted on a computer. A demosaicking calculation is conducted for the recorded image, and four phase-shifted incoherent digital holograms are numerically generated. Four-step PSI is applied using the four phase-shifted incoherent digital holograms, and the complex amplitude distribution in the digital holograms is obtained. Focused images at arbitrary depth planes are reconstructed by applying the calculations of the diffraction integrals. As a result, the 3D image information of the incoherent object light wave is reconstructed from the recorded single image. As seen in Fig. 1b, a palm-sized optical setup of HSPS-IDH can be constructed. Figure 2 illustrates the flow of the polarizations of the two object light waves and the relationship between the polarization and the phase shift. As an example, we set the transmission axis of the polarizer placed between the object and the birefringent lens as a 45-degree direction against the horizontal direction, the fast and slow axes of the birefringent lens as horizontal and vertical directions, the fast and slow axes of the birefringent variable phase modulator and the birefringent plate as vertical and horizontal directions, the fast axis of the quarter-wave plate as a 45-degree direction, and the four transmission axes θ of the micropolarizer array as 0-, 45-, 90-, and 135-degree directions. The important relationships are that the fast axes of the birefringent lens and plate should be orthogonal to adjust the optical-pathlength difference, and the angle between the fast axes of the birefringent and quarter-wave plates should be 45 degrees to generate circularly polarized light waves with opposite handedness. The transmission axis of the polarizer placed in front of the lens is related to the intensity ratio of the two light waves. The fast axis of the birefringent variable phase modulator is along that of either the birefringent lens or birefringent plate to carefully adjust the optical-path-length difference. The transmission axes θ of the micropolarizers are related to the phase shifts. The phase shifts of the holograms passing through the micropolarizer array with transmission axes in the θ = 0-, 45-, 90-, and 135-degree directions are π/2, 0, 3π/2, andπ, respectively, in the case where the fast axis of the quarter-wave plate is set as a 45-degree direction. A self-interference hologram, I(x,y;r o ,θ), of an object point located at r o = (x o ,y o ,z o ) is mathematically expressed as. and because an incoherent hologram H(x,y;θ) is the incoherent sum of holograms of multiple object points, H(x,y;θ) is expressed as where H 0th (x,y) and I 0th (x,y) are 0th-order diffraction waves of H(x,y;θ) and I(x,y;r o ,θ), respectively; i is the imaginary unit; U(x,y) is an incoherent object wave, which contains 3D information of the object and is generated from multiple object points; C.C. and c.c. are the complex conjugates of U(x,y) and the second term of Eq. (1), respectively; C(r o ), C'(r o ), C''(r o ), and C'''(r o ) are coefficients; z 1 is the depth difference between an object point and the birefringent lens; [30]; * indicates a convolution; z 2 is the depth difference between the birefringent lens and the image sensor; ; d p is the thickness of the birefringent plate; n f and n s are refractive indices of the fast and slow axes of the birefringent plate, respectively; ϕ is the relative phase shift from the object point to the pixel placed on the optical axis, which is induced by the birefringent lens, variable phase modulator, and birefringent plate; f 1 and f 2 are the focal lengths of the birefringent lens in the fast and slow axes, respectively; is the numerical focusing distance. z 2 ′and z 2 ′′ indicate that the focal length shifts slightly owing to the birefringent plate and are derived with paraxial approximation [52]. The adjustment of ϕ is a key to improving the visibility of the interference fringes of the incoherent light. Where d l is the thickness of the birefringent lens on the optical axis, the refractive indices of the fast and slow axes of the birefringent lens are the same as those of the birefringent plate, ϕ pm is the phase shift generated by the birefringent variable phase modulator, and k c is the central wavenumber, we set the following relationship as a guideline for the adjustment: Static k c can be set using the bandpass filter. Equation (4) indicates that ϕ pm is set to be ϕ = 0, and the incoherent light wave incident from the optical axis is recorded as an incoherent hologram with zero opticalpath-length difference. As a result, interference fringes with high visibility are obtained for the object placed on the optical axis. When the material of the lens is the same as that of the plate, ϕ = 0 is obtained by setting d l = d p accurately. As another aspect, the visibility of interference fringes generated by an arbitrary wave-vector component is flexibly adjusted by changing ϕ pm . The attenuation of interference fringes is related to the shape of the spectrum, the optical-path-length difference, and the spectral bandwidth [52]. Equation (2) indicates that the phase shift of an incoherent hologram depends on θ, which changes pixel by pixel. Where ϕ = 0 is set, p x and p y are the pixel pitches of the image sensor in the x-and y-axis directions, x/p x = − N x /2, …, 0, …, N x /2 − 1 and y/p y = − N y /2, …, 0, …, N y /2 − 1 have discrete integer values, which means the address of the photodetector of the image sensor, and The meaning of Eq. (5) is the same as that of Eq. (1) of ref. [40]. Equation (5) indicates that four phase-shifted holograms can be obtained by calculating the demosaicking. Four-step PSI such as that described in refs. [40] and [53] and diffraction integrals are applied to obtain the 3D image of an object illuminated by incoherent light.
In high-speed IDH imaging, we should consider light intensity. After introducing a band-pass filter, the illumination-light intensity of a daily-use light source is much poorer than that of a laser. Random noise becomes a serious problem as the dynamic range of the recorded incoherent hologram is limited owing to the low light intensity. Thus, denoising processing for the reconstructed image is important for improving image quality. We set a deep learning model based on convolutional neural networks, MIRNet [54], for 1,000-fps HSPS-IDH as a denoising method. Figure 3 shows the block diagram of MIRNet. It consists of blocks called recursive residual groups (RRGs). RRGs generate multi-resolution feature maps and exchange the feature maps inside them, that is, high-resolution features, while maintaining accurate spatial information. By stacking RRGs, MIRNet can extract higher-order feature maps and produce denoised images.

Experiments
We constructed the HSPS-IDH system shown in Fig. 1 and conducted experiments to demonstrate its high-speed 4D measurement capability. We adopted a wire-grid polarizer, birefringent crystal lenses and plates, and an achromatic quarter-wave plate to generate a self-interference hologram.
We set a multi-order LC-PM (LCC2415-VIS, Thorlabs) as a birefringent variable phase modulator and ϕ pm = 4π using the LC-PM. We have introduced a high-speed polarization imaging camera system CRYSTA PI-5 (Photoron. Ltd.) with 2560 × 2048 pixels at 250 fps and 2560 × 480 pixels at 1,000 fps to conduct the high-speed recording of 4D information of moving objects. A bandpass filter whose central wavelength and full width at half-maximum (FWHM) were 532 nm and 10 nm, respectively, was selected considering the working wavelength bandwidth of the camera. This HSPS-IDH system was constructed on a wagon. We prepared a USAF1951 test target attached to a cell phone (SONY XPERIA SO-52B) as an illuminated object. A white-color light-emitting diode (LED) with a Fresnel zone aperture, which is set in the cell phone, was used as a self-luminous object. The test target was illuminated by the LED light, and both the test target and the LED were randomly moved by hand. Exposure time was 1 ms per frame to record a series of self-interference incoherent holograms of the motions of the test target and the LED of the cell phone at 1000 fps. Initially, we examined the effectiveness of high-speed recording with a short exposure time. We obtained holograms recorded with exposure times of 1 ms and 33 ms. We set the exposure time of 33 ms for comparison because such the time was set for video-rate recording. A digital hologram with an exposure time of 33 ms was generated by synthesizing 33 digital holograms obtained at1,000 fps. Figure 4 shows the experimental results. Images blurred to the vertical direction were reconstructed on respective objects when the exposure time was 33 ms. These images indicate that, in the case where objects dynamically move during the exposure, object images can be reconstructed with numerical refocusing even blurring fringes but these images are blurred. In contrast, images of the LED and group 2 of the test target were successfully reconstructed without blurring when the exposure time was 1 ms. From the results, it was clarified that the blurring problem can be solved by shortening the exposure time, which is the same as the solution for blurring in general imaging systems. Then, we carried out experiments of 1,000 fps recording of incoherent holograms for moving objects. Figure 5 and supplemental file 1 show the experimental results. The results indicate that the images of the objects moving in the in-plane direction, the LED and group 2 of the test target, were successfully reconstructed from respective incoherent holograms taken at different times. Although random noise generated from an image sensor becomes serious with increasing frame rate since the light intensity per exposure is limited in passive imaging, the image quality can be improved by signal processing for denoising based on a machine learning technique. Figure 6 shows examples of the images obtained after applying the signal processing with MIRNet to the images in Fig. 5.  Figure 6 indicates that the random noise in the reconstructed images was effectively suppressed by MIRNet. We measured the entropy and speckle contrast of the images in Fig. 6 for the quantitative and comparative evaluations of the noise level. The entropy values in the cases of Fig. 6a-d  Thus, 4D information was successfully recorded and reconstructed by HSPS-IDH, and the denoising with a deep learning method to IDH was experimentally demonstrated. Next, we set the condition that the moving speed of the objects was increased and objects were moved in 3D directions. Figure 7 shows the experimental results. A series of incoherent holograms were recorded at 1,000 fps. The displacement speed in the 3D direction was 9.13 pixels/ms on the reconstructed image plane. Figure 7 indicates that the focused images of both objects were successfully reconstructed while obtaining quantitative depth information, although the depths of the objects rapidly changed. Figure 7 also indicates that the resolution in the vertical direction was low in comparison with that in the horizontal direction. This is because the speed of the displacement was higher along the vertical direction than along the horizontal direction, and blurring in the vertical direction was severe. MIRNet suppressed the random noise in the reconstructed images, as shown in Fig. 7. The motion pictures shown in Fig. 8 and supplemental file 2 were obtained from the reconstructed images of the test target after applying MIRNet. Figure 8 and supplemental file 2 indicate the high-speed 4D recording and 3D tracking abilities of HSPS-IDH for an object moving in 3D directions. Thus, high-speed 4D measurement capability was successfully and experimentally demonstrated. We have conducted an additional experiment for demonstrating the 4D measurement capability for a moving and reflective 3D object. Figure 9 shows the experimental setup. We set a 30-mm-diameter rotator as a moving and reflective 3D object. A white-color LED (RC220D, Shenzen Leqi Network Technology Co. Ltd.) was prepared as a spatially incoherent illumination light source. In this experiment, bandpass filters, by which only light with wavelength bandwidths between 510 and 550 nm was detected, were set in the constructed HSPS-IDH system. A camera lens (Canon, EF 11-24 mm) was set between the object and the HSPS-IDH system. This HSPS-IDH system with the lens was used as a high-speed incoherent digital holographic camera. The frame rate of the high-speed polarization-imaging camera was set as 250 fps. We have obtained a series of holograms for an object rotated by a hand-operated air blower. Object images focused on different depths were successfully reconstructed. A clear motion picture of the moving and reflective 3D object was obtained using the frame rate of 250 fps, although blurring is seen along the rotation direction of the object with increasing rotation speed. Note that a video-rate image sensor fails to detect the rotation frequency and direction of the object owing to the undersampling of the temporal frequency of the rotation. It is also difficult to conduct the 4D measurement of a moving and reflective 3D object using a general laser DH system unless a pulsed laser is adopted as a light source, owing to extremely high sensitivity against the movement along the depth direction. In comparison, the constructed high-speed IDH system can be a powerful apparatus for the high-speed 4D measurement of a reflective 3D object dynamically moving in 3D space.

Discussion and conclusion
We discuss the imaging properties such as resolution, lightuse efficiency, and blur against a moving object, and the constructed HSPS-IDH system. The resolution in the physically Fig. 6 Noise suppression using MIRNet. Images a before and b after applying MIRnet at a frame. Images c before and d after applying MIRnet at another frame. Rectangles in a-d are the areas used for quantitative evaluations focused plane is the same as that of the regular imaging with a polarization-imaging camera. The resolution in the physically defocused and numerically focused plane is determined by the maximum spatial frequency of the Gabor zone plate pattern on the image sensor plane and the specifications of the image sensor, which is similar to that of 3D imaging with a Fresnel zone aperture. In general self-interference IDH, the spatial frequency quadratically decreases as the object becomes linearly distant from the IDH system. As a result, the depth resolution of IDH is less than that of 3D imaging with a daily-use light source exploiting a random pattern [25] and a plane reference wave [47]. An additional or a different PSF engineering with a known PSF, such as that described in ref. [55] is valid to improve spatial resolution. The fabrication of an optical device to implement a coded phase aperture will be an important research challenge for the improvement of HSPS-IDH.
The light-use efficiency can be optimized by improving the design of a polarizer with high transmittance and extinction ratio and a band-pass filter. The efficiency of the constructed IDH system after removing a band-pass filter is about 10% although the efficiency is up to 25% in principle. This is because the transmittance of the polarizer in the transmission axis was about 60%. Two polarizers were used in the constructed IDH system and the efficiency was decreased. The use of a polarization prism improves the efficiency but the size of the optical setup increases. A 50 nm FWHM band-pass filter can be used to optimize the efficiency, considering the working wavelength bandwidth of the polarization-imaging camera of the constructed IDH system. The use of a wide-FWHM band-pass filter limits the resolution of the IDH system and affects the PSF in the 3D direction because the number of lines of the interference fringes becomes limited as the FWHM increases [52]. A light source with high light intensity at a narrow bandwidth is required when a band-pass filter with a narrow FWHM is adopted to maintain the resolution in 3D directions.
In Fig. 4, blurred images are reconstructed when the exposure time is too long to capture a moving object as an incoherent digital hologram. This phenomenon is common in regular imaging but is not always common in DH. In laser DH using single-shot phase-shifting, a two-arm interferometer, and a plane reference wave, it is difficult to conduct the measurement when the distances among a light source, an object, and an image sensor change during an exposure as the object moves in the depth direction with λ/4. This is because the phase of each phase-shifted interference fringe image changes with π. In contrast, a blurred object image was reconstructed even if it seems that the distance changes by more than λ in the depth direction. Figures 4a, c indicate that IDH has the same nature as regular imaging.
The recording speed was far from the video rate in the first experimental demonstration of SSPS-IDH [40]. In contrast, the constructed HSPS-IDH system succeeded in performing motion-picture recording at 1000 fps, using a commercially available white-color LED. The constructed HSPS-IDH system was designed to obtain a high temporal resolution, as described in Sect. 2. SSPS interferometry is useful for conducting high-speed PSI. A polarization-imaging camera is an important device to implement SSPS interferometry. A commercially available high-speed polarization-imaging camera has a limited working wavelength band [56]. On the other hand, full-color imaging is conducted with a commercially available video-rate color polarizationimaging camera [43]. Therefore, the constructed HSPS-IDH system can be used for not full-color imaging but single wavelength-band high-speed imaging, or multispectral highspeed imaging with the limited wavelength band, using a dispersive optical system.
We have proposed high-speed IDH exploiting SSPS with a recording speed of 1,000 fps for the high-temporalresolution 4D measurement of moving objects. We have constructed an HSPS-IDH system consisting of SSPS-IDH, a high-speed polarization-imaging camera system, and a palm-sized IDH optical system designed for spatially and temporally incoherent light. A compact HSPS-IDH setup was constructed to achieve the recording of high-speed 4D information with a daily-use light source and a portable IDH optical system. High-speed 4D measurement capability was experimentally demonstrated on a wagon that is an example  Experimental setup for a moving and reflective 3D object of a moving body. Experimental results clarify that our HSPS-IDH system can be set on a moving body and operated without calibration. A daily-use white-color LED can be used to achieve the 4D measurement of moving objects at 1,000 fps. The speed for recording incoherent holograms can be increased at the cost of the number of pixels and can be set as 3,500 fps with 1280 × 256 pixels, 7,000 fps with 1280 × 128 pixels, and 30,000 fps with 2560 × 4 pixels. Our IDH system adopts a high-speed polarization-imaging camera in which motion-picture holograms are immediately transferred to a memory unit of a personal computer of the camera system through the PCI Express interface. Therefore, a graphical processing unit (GPU) can be easily combined with our HSPS-IDH system, and a high measurement speed is expected, because the calculations for the image reconstructions are significantly accelerated using a GPU [57,58]. HSPS-IDH enables us to detect 4D information with high temporal resolution and will be useful for the advanced analyses of specimens in scientific research and industry, machine vision in the form of holographic machine vision, and healthcare in our daily lives. Data availability Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the author upon reasonable request.

Conflict of interest
The authors declare no conflicts of interest. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/.