Resolution enhancement of digital holographic microscopy via synthetic aperture: a review

Digital holographic microscopy (DHM), which combines digital holography with optical microscopy, is a wide field, minimally invasive quantitative phase microscopy (QPM) approach for measuring the 3D shape or the inner structure of transparent and translucent samples. However, limited by diffraction, the spatial resolution of conventional DHM is relatively low and incompatible with a wide field of view (FOV) owing to the spatial bandwidth product (SBP) limit of the imaging systems. During the past decades, many efforts have been made to enhance the spatial resolution of DHM while preserving a large FOV by trading with unused degrees of freedom. Illumination modulation techniques, such as oblique illumination, structured illumination, and speckle illumination, can enhance the resolution by adding more high-frequency information to the recording system. Resolution enhancement is also achieved by extrapolation of a hologram or by synthesizing a larger hologram by scanning the sample, the camera, or inserting a diffraction grating between the sample and the camera. For on-chip DHM, spatial resolution is achieved using pixel super-resolution techniques. In this paper, we review various resolution enhancement approaches in DHM and discuss the advantages and disadvantages of these approaches. It is our hope that this review will contribute to advancements in DHM and its practical applications in many fields.


Introduction
Optical microscopy is a powerful tool for detecting subtle structures of samples, playing an irreplaceable role in many fields.However, traditional optical microscopy can only obtain the amplitude information of samples, and transparent samples, such as living cells cannot be investigated.Although fluorescence microscopy can selectively render and highlight the structures of interest by prior tagging with fluorescent markers, the phototoxicity and photobleaching of fluorescence tagging make it difficult to continuously observe living cells over a long time.Thus, there is an urgent need for label-free microscopic tools able to follow organelles in live cells.Digital holographic microscopy (DHM) [1][2][3][4][5][6] , which combines digital holography and microscopy, is a labelfree, quantitative phase microscopy approach.A typical setup for DHM is shown in Fig. 1a, in which a magnified object wave and a reference wave interfere with each other, and the generated off-axis hologram or phase-shifting holograms are recorded by a CCD camera.From the offaxis hologram (Fig. 1b), both the amplitude and phase images (Fig. 1c, d) of a sample can be obtained simultaneously.As an alternative, inline lensless DHM can help recover the phase information with an iterative algorithm without using an independent reference wave 7,8 .Refocusing of the reconstructed images can further be performed digitally using the obtained complex amplitude of the object wave 9,10 .In general, DHM can not only observe transparent samples with high endogenous contrast, but also quantitatively assess the thickness or refractive index distributions of these samples.Therefore, DHM has been widely applied in industrial inspection 11,12 , visualization of liquid/gas flows 13 , biomedical imaging 14 , etc.
The spatial resolution is of great importance for DHM because it determines the smallest structures that can be resolved.As illustrated in Fig. 2a, when two infinitely small points separated by a Rayleigh distance of 0.61λ/NA are imaged by a diffraction-limited imaging system, two Airy patterns are generated in the imaging plane, whose complex amplitudes are denoted as O 1 and O 2 , respectively.Here, λ and NA indicate the illumination wavelength and the sum numerical aperture (NA) of the imaging system (NA imag ) and illumination system (NA illum ), respectively.For incoherent illumination, the intensity of the image is linearly related to the intensity emitted from the sample, that is, |O 1 | 2 +|O 2 | 2 .The two patterns are φ φ φ considered to be resolved if the center of the intensity pattern of O 1 coincides with the first zero of the pattern of O 2 (Fig. 2b).For coherent imaging (e.g., DHM), a linear relationship holds between the input complex amplitude and the output complex amplitude.In this case, the intensity of the image corresponds to |O 1 +O 2 | 2 , which is dependent on the phase difference Δ between O 1 and O 2 .It is evident from Fig. 2c that for Δ = 0 (in-phase), the Airy patterns of O 1 and O 2 are considered unresolved.However, for Δ = π (anti-phase), the Airy patterns of O 1 and O 2 are well separated.To conclude, there is no easy criterion for evaluating the resolution of coherent microscopy (DHM), considering that the resolution depends on both the microscope and the phase of the sample.Nevertheless, we recommend the following: for evaluating the resolving power of optical microscopy.
Factor κ 1 is determined by the experimental parameters, such as the coherent noise level and the SNR of the detector, etc 3,15 .Herein, we assign κ 1 = 0.82 16,17 , for which the principal intensity maximum of pattern O 1 coincided with the first minimum of the in-phase O 2 .This suggests two options for enhancing the spatial resolution of the DHM.The first is the use of a shorter wavelength, for example, 193-nm 18 light was used to enhance the spatial resolution of the DHM.The other is to enlarge the NAs of both the illumination and the recording systems.In addition to the resolution, the field of view (FOV) is also a vital parameter of an imaging system.The ratio between these two parameters determines the spacebandwidth product (SBP) of the imaging system 19,20 .Often, a DHM system has a limited SBP, and consequently, there is an inherent trade-off between resolution and FOV.This means that a reconstructed image with a higher resolution has a smaller FOV.Just as an example, a standard 20 × /0.4 microscope lens with a resolution limit of 0.8 μm and a circular FOV of 1.1 mm in diameter, provides a SBP of ~7 megapixels 21 .However, microscopists always expect to have a higher SBP in the sense of an increasing resolution while maintaining FOV.There are many degrees of freedom, including spatial/temporal resolution, FOV, polarization, spectrum, etc., in an imaging system.We can convert or trade some unused degrees of freedom (e.g., time) for an increase in SBP.In the past few decades, computational microscopy methods have emerged, generating images with high resolution and a wide FOV by using modulated illumination or applying other physical manipulations during the imaging process.By synthesizing apertures using the object information obtained under different modulation/manipulation operations, the SPB limitation of conventional microscopes (including DHM) can be bypassed 22,23 .
This paper aims to review the resolution enhancement approaches of DHM, which are classified into three types: 1. Enlarging the NA of illumination via oblique illumination, structured illumination, and speckle illumination; 2. Enlarging the NA of the recording system via hologram extrapolation, hologram expansion, and pixel super-resolution.3. Artificial intelligence (AI) assistance resolution enhancement approaches for DHM.This paper will cover the basic principles, implementation schemes, and exemplary results of the different approaches.We conclude by providing a summary and future outlook of resolution-enhancement approaches for DHM.

Enlarging the NA of illumination
A microscopic imaging system is commonly introduced to DHM, which provides a magnified and high-resolution image of the sample.A plane wave is often used for illumination, and only the spatial frequencies diffracted by the sample up to ~ NA imag /λ can be transmitted through the limited aperture of the microscope lens (Fig. 3a, upper part), resulting in an upper limit for the spatial resolution in terms of 0.82λ/NA imag .Therefore, using oblique illumination, structured illumination, or speckle illumination to provide a NA illum is an effective method for further improving the resolution.

DHM with oblique illumination
Since the first implementations of oblique beam illumination in DHM in the 1970s [24][25][26][27] , many approaches have been extensively reported on for replicating a larger aperture  . When  sample is illuminated by an oblique wave, the high spatial frequency of the object wave in the opposite direction of the oblique illumination will be downshifted and, thus, will pass through the limited aperture of the imaging system (Fig. 3a, lower part).The downshifted frequencies are back-assembled to their original positions in the spectrum of the object, thus, synthesizing a wider spectrum than that of the NA-defined aperture 31 .The final resolution-enhanced image is obtained by an inverse Fourier transform (FT) of the synthesized spectrum.In comparison with conventional lens-based DHM with a resolution limit of 0.82λ/NA, DHM with oblique beam illumination has an enhanced spatial resolution, as described in Eq. 1.
Kuznetsova et al. and Schwarz et al. used oblique beam illumination to downshift the high-frequency components In-phase Anti-phase of the spectrum of the object, and introduced a conjugate carrier phase to upshift the spectrum content back to its proper region 28,50 .This process was repeated sequentially for both the x-and y-illumination directions, thus allowing for the transmission of extended frequency bandwidths of the 2D object, yielding a resolution gain with a factor of three.Mico et al. utilized a vertical-cavity surface-emitting laser (VCSEL) array to generate multi-directional oblique illumination of DHM and retained a resolution gain with a factor of five 47 .The use of scanning elements has also been reported as a technological improvement for providing more flexible oblique beam illumination 28,33,37,51 , which has also been validated 29,31,52 in comparison with the tilting of the illumination direction.For instance, Cheng et al. used a 2D Galvo-scanner to generate oblique illumination for DHM (Fig. 3b), achieving a two-fold isotropic resolution enhancement. 49Oblique beam illumination for resolution enhancement has also been implemented using a spatial light modulator (SLM) [53][54][55] or a fiber bunch [56][57][58][59][60] .Meanwhile, single-shot synthetic aperture DHM was realized by using an SLM to generate multiple non-coplanar illumination beams, and using coherence gating to avoid selfinterference between these non-coplanar beams 53 .Moreover, wavelength multiplexing techniques [34][35][36] were utilized to perform multiple oblique illuminations during a single exposure.Oblique beam illumination techniques have recently been implemented in a commercial upright Olympus microscope 39 .Notably, the maximum synthetic numerical aperture is limited by NA<1 in imaging systems with dry objectives, and further resolution improvements to λ/3.7 were demonstrated using evanescent waves 29 .In addition, axial rather than transversal resolution improvement has also been validated by SA generation 61 , and an application of the technique to edge processing was also reported 62 .It is interesting to mention that oblique illumination has also been applied to differential interference contrast (DIC) microscopy 63 and Zernike phase contrast microscopy 64 .
Recently, oblique illumination was also applied to obtain 3D refractive index (RI) tomographic images of transparent or translucent samples via optical diffraction tomography (ODT) 49,65 .ODT enables the probing of 3-D RI maps of a sample by recording the complex wavefront diffracted while rotating the sample 66,67 or varying illumination angles 68,69 .Kim 68 adapted DHM to ODT by using a 2D galvanometric mirror to generate different illumination directions and consequently, reconstructed a 3D RI distribution of the samples with high lateral and axial resolutions.Ozcan 70 presented an on-chip ODT scheme that enables the imaging of a large volume of approximately 15 mm 3 , with a spatial resolution of < 1 μm ×1 μm × 3 μm.In 2020, Wang et al. 71 reported a mechanical-scanning-free ODT configuration using self-accelerating Airy beams that are tilted along the beam path direction, providing a 3D volumetric view of the cells.Recently, Kus et al. 72 proposed a real-time ODT system for recording a complete set of projections in a single hologram in a multiplexing manner by using a microlens, with which the tomographic images of living cells and flow cytometry could be observed clearly.In addition to advances in ODT hardware, artificial intelligence (AI)-based reconstruction algorithms have been reported to reduce the recording time and improve the quality of the reconstructed images 73 .

DHM with structured illumination (SI)
Structured illumination microscopy (SIM) is a superresolution optical microscopy technique that has the merits of being fast, minimally invasive, and has no requirement for fluorescent labeling 74,75 .Compared with conventional wide-field (WF) microscopy, SIM provides two-fold spatial resolution enhancement by illuminating the sample with a periodic pattern and recording the generated moiré pattern.Because of the moiré effect, the high spatial frequency components of the sample, which are not accessible in conventional microscopy, are shifted into the detectable domain and thus can be observed.However, the application of SIM has been limited to fluorescent samples.In 2010, Mudassar and Hussain 60 applied it to nonfluorescent scattering imaging, for which fringe patterns were generated by the interference between the beams carried by two fibers.
A DHM with structured illumination generated by an electric modulation device has been proposed.The fringe patterns were generated by a spatial light modulator (SLM) 76 in 2013 and by a digital micromirror device (DMD) 77 in 2015, whereby fringes with different orientations and phase shifts were projected without mechanical motion.In the implementation, four binary phase gratings rotated by m×45° (Fig. 4a) and generated by an SLM or a DMD are projected onto the sample, so that the sample is illuminated sequentially by four sinusoidal fringe patterns.After passing through a telescope system comprised of the microscope objective MO and lens L, the object wave interferes with a tilted reference wave R, and the generated holograms are recorded by a CCD camera, as shown in Fig. 4b.We denote with Ψ mn the object wave generated by illuminating the sample with the grating as having the m-th orientation and n-th phase shifting.Thus, the recorded hologram can be described as I mn = |R+ Ψ mn | 2 .From this intensity, the wave Ψ mn can be reconstructed using standard reconstruction methods, as in off-axis DHM.Generally, Ψ mn can be decomposed into three components, A m, −1 , A m, 0 , and A m,1 corresponding to the −1 st , 0th, and +1 st diffraction orders of the illumination wave.When we assume that the phase shift increment for each shifting of the grating is α, Ψ mn can be written as: where γ −1 , γ 0 , and γ 1 denote the magnitudes of the diffraction orders.From Eq. 2, we can calculate the reconstructed object waves A m, −1 , A m,0 , and A m,1 from the different diffraction orders: Then, A m,-1 and A m,1 under each illumination direction are compensated for by the carrier phase due to the tilted propagation direction along the ±1 st diffraction orders, and synthesized together with A m,0 in the Fourier plane to yield the synthetic spectrum.Finally, a focused image with enhanced resolution was retrieved by the inverse FT of the synthetic spectrum.Assuming that the angular aperture of the imaging system is NA imag , the synthetic NA of the SI DHM is NA = NA imag +sinθ illum .Here, θ illum denotes the illumination angle of the +1 or −1 diffraction orders, and the highest value of the sinθ illum is, in turn, determined by the NA of the condenser lens.Fig. 4c validates the resolution enhancement of the structured illumination in comparison with on-axis illumination.Of note, the phase image at the bottom (obtained with structured illumination) has a better resolution compared with that on the top, and the two particles separated by 1.4 μm became Recently, structured illumination has been widely applied to different variations of DHM for the purpose of imaging transparent samples with resolution enhancement [76][77][78][79][80][81][82][83][84][85][86][87][88][89] .SI-based DHM was extended for resolution enhancement in the axial direction, aside from the lateral directions 90 .An iterative reconstruction algorithm, as well as a PCA-based algorithm, were employed to obtain resolution-enhanced images using structured illumination with unknown phase shifts or free of phase shifts [82][83][84]89 . In 221, an end-to-end deep-learningbased method, DL-SI-DHM, was proposed for improving the reconstruction efficiency and accuracy of SI-DHM 91 .In addition to synthetic-aperture phase imaging, Shin et al. in 2015 and Chowdhury et al. in 2017 utilized DMD-and SLM-based structured illumination to obtain 3D refractive index tomographic images of samples 77,92 .

DHM with speckle illumination
Speckle illumination has been known since the invention of the laser in 1960 93 .Speckle illumination often reduces the signal-to-noise ratio (SNR) in coherent imaging.Speckle fields can be understood as a combination of plane waves with various random illumination directions.These oblique plane waves shift the spectrum of the object in the reciprocal plane, allowing for access to additional spatial frequencies.Thus, speckles can be used to enhance the spatial resolution and reduce the coherent noise in DHM or other coherent imaging 37,[94][95][96][97] .5 shows the optical setup of DHM with a speckle illumination according to Ref. 95.The speckle field was generated by an SLM and projected using a telescopic system L 1 -MO 1 in the sample plane, where is the ith speckle illumination and refers to the complex amplitudes of the object wave when the sample was illuminated with . When interferes with a reference wave R, the hologram can be denoted as . The complex amplitude of the speckle field can be reconstructed from the hologram recorded in the absence of a sample.Often, the holographic resolutionenhanced image can be obtained by averaging multiple angle-dependent E-field images O ave = 96 .It is worth mentioning that the resolution enhancement capability of such an averaging operation will be lower when the speckle field has an unequal weight or power over the entire frequency spectrum.
To overcome this limitation, an iterative reconstruction method 95 was proposed for synthesizing a larger NA.The key idea of the iteration is to propagate the object wave between the image plane and the CCD plane, updating the speckle illumination in the image plane and the object wave intensity (with the recorded one) in the CCD plane.The iterative method enhances not only the resolution but also the SNR when the reconstructed object wavefronts from different speckle-based holograms are averaged.Utilizing SLM for speckle field illumination avoids mechanical movements and allows for a high repeatability in comparison with dynamic devices.Time-changing speckle illumination is similar to structured illumination.The concept in common is the encoding of an object with an a priori known high-resolution pattern that folds the high spatial frequencies into the low bandwidth range.The decoding patterns should be matched to the time-changing encoding patterns.
Similar to speckle illumination, a scattering medium can also be placed before the sample to enhance the spatial resolution and FOV 98,99 .Multiple scattering of the medium is a deterministic process that is described by a transmission matrix (TM) in transmission geometry.TMbased imaging has been introduced in DHM, yielding an improved resolution beyond what an objective-lens-based imaging system can achieve 99 .Similarly, a single multimode optical fiber was demonstrated for scanner-free and wide-field endoscopic imaging 96 .Beyond resolution enhancement, Baek et al. demonstrated in 2018 that a highresolution, long working distance, reference-free DHM can be performed by using a scattering layer to replace the conventional imaging lens 100 .By exploiting the randomness of a multiple-scattering layer, it allows for holographic imaging of a microscopic object without introducing a reference beam.The authors experimentally demonstrated a high NA (0.7) with a working distance of 13 mm.

Enlarging the NA of DHM recording systems
Compared to DHM with an imaging system, DHM without using an imaging lens (lens-free DHM) possesses three significant advantages.The first aspect is its compactness and low-cost because it does not employ an objective.The second is a large FOV.The third is the high compatibility and ease of integration with microfluidic and on-chip devices.In the following sections, we review the resolution enhancement of lens-free DHM via hologram extrapolation, hologram synthesis, and pixel superresolution techniques.

Hologram extrapolation
As a major type of lens-free DHM, lensless DHM can be traced back to an inline architecture proposed by Gabor 8 .In Gabor, in lensless DHM 8,101,102 (Fig. 6a) a sample is illuminated by a spherical wave.The wave diffracted by the sample interferes with the non-diffracted wave, which also acts as a reference wave.Direct reconstruction of the generated hologram suffers from overlapping of the DC term and the twin image.Iterative approaches 7,103,104 , and compressive sensing 105 have been invented to retrieve phase information from an oversampled diffraction pattern.The achievable spatial resolution is believed to be limited by the size of the hologram 8,104 : where λ is the wavelength, z is the distance between the object and the screen, N is the number of pixels, and Δ S is the pixel size of the detector.In 2013, resolution enhancement of lensless DHM was reported by extrapolation of the hologram 104, , and later, this method was applied to terahertz inline lensless holography 106,107 .The flowchart of the resolution enhancement routine is shown in Fig. 6b, and includes the following steps: (1) Initialize the object wave U(x S , y S ) in the hologram plane by taking the square root of the hologram intensity H 0 of N 0 ×N 0 and padding it with zeros to a matrix of N × N with N > N 0 (see Fig. 6a 1 ).In contrast, random noise can be added to the padded area.
(2) Propagate the padded wave U(x S , y S ) to the object plane by using the propagation integral 7    Once the above procedures have been repeated over 500 iterations, the complex amplitude of the object wave can be retrieved, which has a resolution of σ = λz/(NΔ S ) instead of λz/(N 0 Δ S ).It is clear that the image reconstructed with hologram extrapolation (Fig. 6a 4 ) has finer structures than that obtained without using self-extrapolation (Fig. 6a 3 ).Quantitatively, the resolution was enhanced from 5.9 μm to 1.8 μm by padding a hologram with 300 × 300 to 1000 × 1000 pixels 108 .The reason for the resolution enhancement by hologram extrapolation is that the hologram itself contains a series of wavelets constituting the wavefront, which can be fitted better through extrapolation and yield a larger effective hologram 104 .This hypothesis is confirmed by a closer look at the intensity distributions ((iv) in Fig. 6a 2 ) in the hologram plane.

Hologram synthesis
In addition to inline lensless DHM, DHM, which records the interference between an object wave and an independent reference wave can reconstruct the phase of a sample free of twin-image disturbances.Fig. 7a shows a standard lens-free DHM configuration using an independent reference wave 109,110 .Fig. 7b shows a modified lensless DHM configuration by the insertion of an SLM into the classical Gabor architecture 102 .The SLM is used to perform phase shifting of the DC term of the object wave.The complex amplitude distribution of the object wave passing through a sample can be recovered using a standard phase-shifting reconstruction algorithm.For the two types of lens-free DHM configurations (with and without an independent reference wave), the spatial resolution is limited by the size of the hologram, as shown in Eq. 4. In 2002, resolution enhancement of lens-free DHM was first demonstrated 111 by generating an expanded hologram with multiple sub-holograms.Each subhologram covers a different frequency spectrum of the input object.When the reconstructed image of each hologram is synthesized, an image with an enlarged FOV and an enhanced spatial resolution can be obtained 109,110 .Generally, hologram synthesis of lens-free DHM can be performed shifting the digital camera 109 or sample 110,112 , using oblique beam illuminations 113 , or inserting a diffraction grating between the object and digital camera 114,115 .The above-mentioned three strategies are equivalent and generate the same output, which is a larger synthetic hologram that contains a larger sample spectrum.
Figs. 7c−e show the experimental results of the lens-free DHM based on hologram synthesis.The synthesized spectrum is shown in Fig. 7c, and the resolution-enhanced image (Fig. 7e) can be obtained from the Fourier transform of the synthesized spectrum.It is clear that the SA image of the blood cells in Fig. 7e has much finer structures than that in the conventional image in Fig. 7d.Furthermore, it was found that the hologram synthesis enhanced the effective NA from 0.13 to 0.75.

Pixel super-resolution for on-chip DHM
The architecture of a standard on-chip DHM [116][117][118][119] is shown in Fig. 8a.A partially coherent light source is used for illumination, usually at > 2-3 cm (z 1 distance) above the sample.A semi-transparent sample is placed on top of an image sensor with a typical spacing of < 1 mm (z 2 distance).As a result, the sample casts an inline hologram, which is directly recorded by a CMOS or CCD image sensor, without using any imaging lenses.The magnification time of the on-chip DHM is defined as M = (z 1 + z 2 )/z 1 .For z 1 much larger than z 2 , the magnification is approximately 1.The hologram recorded in the on-chip DHM is an in-line hologram (Gabor).The direct reconstruction of object-related holographic information from an inline hologram is often obscured by the twin image and the DC term.Phase retrieval approaches using an iterative approach 7 or additional constraints, such as multiple measurements at different heights [120][121][122][123] or illumination angles 124 , can be employed to eliminate twin image noise.There are several factors that can limit the resolution of on-chip DHM, including diffraction, pixel size, image chip area, and coherence of the light source.In practice, the primary limitation of lens-free on-chip DHM resolution comes from the pixel size of the sensor (typically 1-2 μm).
To improve the resolution of on-chip DHM, pixel superresolution (Pixel SR) 125,126 can be adopted by shifting the hologram along the x-and y-directions in subpixel increments.The relative lateral shift of the hologram with respect to the sensor array can also be performed via shifting of the image sensor 127 , the illumination source 128 (Fig. 8a, b), or the sample 129 (Fig. 8c), respectively.A lowresolution hologram is captured at each location of the shifted grid.Then, using multiple LR holograms, a higherresolution hologram can be digitally synthesized by using a "shift-and-add" algorithm, in which the low-resolution (LR) holograms are upsampled, shifted, and digitally added.Alternatively, iterative methods can be used to upsample a high-resolution image from a series of lowresolution images.For this purpose, a cost function can be used for optimization 125 : This equation contains two operations, namely, norm and regulation, where i represents the recording times.y i is a low-resolution hologram, and x is the desired highresolution image.W i represents the digital processes of shifting and down-sampling of an image.This synthesis method is equivalent to recording holograms with smaller pixel sizes and higher spatial density detectors.The second part γ(x) is a regularization parameter to maintain/regulate the desired quality in the reconstructed image.By solving this optimization function, an image with a pixel size smaller than that of the digital camera can be obtained by synthesizing a series of sub-pixel-shifted LR images.A comparison of Fig. 8d, e shows that PSR can provide a more refined hologram with high-frequency fringes.PSR yields a high-resolution image (Fig. 8f) of a grating (line- width ~300 nm), whose resolution is equivalent to an effective NA~0.9, as shown in Fig. 8g 116 .Using the PSR techniques, sub-micrometer resolution over a wide FOV of 20-30 mm 2 can be obtained, providing gigapixel throughput with a simple, compact on-chip DHM design 130,131 .
Instead of physically moving the image sensor, the light source, or the sample, virtual pinhole scanning completed by deconvolving a hologram with the measured PSF is an alternative way to improve the resolution of an on-chip DHM with an extended light source 132 .PSR can also be performed by varying the propagation distances (i.e., z 2 ) 133 , or by varying the illumination wavelengths in small increments (e.g., 2-3 nm) 134 .To further speed up the subhologram recording process by varying the illumination wavelength, color LR holograms can be recorded, where the holograms of the red, reen, and blue illuminations are multiplexed using a sub-pixel shifted Bayer color filter array 119 .Considering that PSR can only enhance the effective NA of the on-chip DHM to 0.9, a synthetic aperture-based on-chip microscope with which the illumination angle is scanned across the surface of a dome (from −50° to 50°) was proposed, which further increased the effective NA to 1.4, achieving a 250-nm resolution at a wavelength of 700-nm across the very large FOV of 20.5 mm 2124 .
Theoretically, an effective NA of 1+n can be achieved in a lensless on-chip DHM using the PSR/SA approach, assuming that the medium between the source and the sample planes is air.Here, n denotes the refractive index of the medium that fills the space between the sample and sensor planes.However, none of these techniques can be considered to exceed the diffraction limit of light, as they are entirely based on propagating waves that result from coherent light-matter interactions.

Deep-learning based methods
Deep-learning 135,136 has been demonstrated as being a powerful tool for solving various inverse problems by training a network with a large quantity of paired images.Deep learning can map the relationship between the input and target output distributions without any prior knowledge of the imaging model.Deep learning has been widely used for holographic image reconstruction [137][138][139] , autofocusing 140,141 , and resolution enhancement 142,143 .
The deep-learning configuration for the resolution enhancement of DHM is shown in Fig. 9.By training a set of matched LR and high-resolution images, the relationships between them can be learned.Thus, a high resolution can be retrieved from one or multiple lowresolution images captured by a similar setup.The establishment of the mapping function between the highresolution output image and the LR input images is called the objective function and is expressed as follows: where R learn is the optimal solution of the weight parameter of the network, x and y are the training set, F 145 is the loss function for appropriate error metrics, and g(θ) is a regularization term for the parameter θ to avoid overfitting 146 .By continuously adjusting θ, the error between the output R θ (y n ) of the neural network and the target image x n is minimized.Therefore, as long as the establishment of the model is completed through data statistics, it can be applied to solve inverse problems in imaging.Using the network described in Eq. 6, deep learning has also been used to enhance the resolution of data-driven models for the following two systems: (1) lensless DHM and (2) lens-based DHM systems.
In the first case, a deep learning algorithm based on a generative adversarial network (GAN) must be used in an on-chip lensless DHM to achieve sub-pixel resolution 147 .In this method, the LR reconstructed images are used as inputs and the corresponding high-resolution images of different samples obtained by the "shift-and-add" approach are used as the labels for training the network.A welltrained framework can generate high-resolution images from unseen LR data without an iterative engine and prior knowledge of the system.In 2019, Liu et al. demonstrated that a resolution-improved image could be obtained from much fewer (e.g., 1 or 4 instead of 36 in conventional Pixel SR algorithm) low-resolution sub-pixel-shifted images of the same object by using a neural network, which significantly reduced both the number of hologram measurements and the computation time necessary for reconstruction 144 .
The network training process for the lens-based microscopic imaging system is shown in Fig. 8.The reconstructed images of the holograms obtained under the low-magnification objective were used as inputs, and the counterparts obtained under the high-magnification objective were used as labels in the network to generate an image with a resolution of the high-magnification objective, but with the same FOV as of the lowmagnification objective 144 .In this way, the diffraction limit defined by the NA of the objective and the SBP of the system can be broken, so that the holographic reconstruction image has a higher resolution and the reconstruction efficiency is improved as compared with the iterative process for each image.A deep-learning-based upscaling method was proposed for overcoming the tradeoff between a large FOV and a high resolution, yielding an enlarged hologram with a smaller effective pixel size (enhanced sample details), for which inline holograms are used as training material for the convolutional neural network 148 .In this method, the deep-learning method only uses a single image obtained with a low-NA objective for magnification, allowing detailed analysis of a sample with a high resolution and a large FOV.However, considering that LR-input images do not contain high-resolution components physically, the resolution enhancement capability (or the right guessing of unknown details) of the network is limited by the adequacy of the training data 149 .As a remedy, people tend to integrate deep-learning-based strategies and illumination-modulation techniques to enhance the resolution of DHM.For instance, in 2021, Meng et al. demonstrated that the deep-learning algorithm simplifies 91 not only the resolution-enhanced reconstruction process of SI-DHM but also minimizes the recording requirements 91 .
In general, for both lensless DHM and lens-based DHM, a deep-learning-based framework can enlarge the SBP of coherent imaging systems using image data and convolutional neural networks, and provides a rapid, noniterative method for solving inverse image reconstruction or enhancement problems in optics.Furthermore, it has a faster imaging efficiency, even with little prior knowledge 147 , and can reconstruct in real time 150 .This is because deep-learning methods use large-scale datasets to impose potential constraints on the resolution enhancement problem.A neural network has learned millions of weight parameters from a large amount of training data.These parameters can establish a high-dimensional nonlinear relationship between the input and output, which is difficult to express through simple formulas.Although classical principled algorithms are often outperformed by deep learning models, they also retain key advantages.First, classical algorithms inherently produce outputs that are consistent with their inputs because they rely exclusively on the underlying physics and principles formulated.Second, classic algorithms can be generalized for any valid measurement because they are not limited by the adequacy of the training data.Third, classic algorithms, in contrast to deep learning models, do not produce widely erratic outputs after minute tweaks to their inputs. 149.In light of the above analysis, it is our opinion that deep learning results cannot be blindly trusted, something certainly true of any singular piece of evidence.Incorporating physical models into deep neural networks provides a new idea for expanding the generality and reliability of imaging 151 , which should be a new trend in deep learning-based resolution enhancement.

Discussion
In this review, we summarize resolution enhancement approaches for digital holographic microscopy (DHM).Generally, by trading with other degrees of freedom, the spatial resolution is enhanced, and a large field of view (FOV) is preserved (Table 1).Resolution enhancement approaches can be classified into two categories, namely, lens-based DHM and lens-free DHM, based on whether an objective lens is used.In lens-based DHM, oblique illumination, structured illumination, and speckle illumination were used to enhance the spatial resolution of DHM.In sample-distant lens-free DHM, self-extrapolation of holograms can enhance the physical-aperture-limited spatial resolution.Furthermore, the resolution can be further enhanced by synthesizing a larger hologram by scanning the sample, camera, illumination, or inserting a diffraction grating between the sample and the camera.With a lens-free on-chip DHM, resolution enhancement can be obtained by pixel super-resolution and a synthetic aperture by shifting the illumination beam.Recently, deep learning has also been used to enhance the resolution in DHM.Being trained with plenty experimental datasets obtained under well-controlled imaging conditions, deep neural networks enable the enhancement of the spatial resolution or the SBP of DHM in a rapid, non-iterative manner.It is worth pointing out that the resolution enhancement approaches in DHM can surpass the resolution limited by the geometry (NA) of the imaging system in terms of 0.82λ/NA.However, they cannot surpass the physical diffraction limit λ(2NA) 152 in far-field imaging.
In the past decade, intensive efforts have been made to achieve coherent imaging resolution beyond the diffraction limit.There are two coherent imaging approaches that can achieve super-resolution capability.The first is syntheticaperture DHM using evanescent waves, which can exceed the diffraction limit in the near-field region.The second is super-oscillation illumination or super-oscillatory lensbased optical microscopy 153 .We envisage that there will be an increasing number of investigations concerning the physics of DHM resolution enhancement, and these investigations will widen the applications of DHM in different fields.

Fig. 1
Fig. 1 Principle of digital holographic microscopy (DHM).a Schematic setup of off-axis (left) and in-line DHM (right).b Exemplary hologram of off-axis DHM.c, d Amplitude and phase images of HeLa cells reconstructed from b. Scale bar in c is adapted with permission from Ref. 76 ©The Optical Society.

Fig. 2
Fig. 2 Resolution criteria in incoherent and coherent systems.a Intensity distribution of the Airy patterns for the complex amplitude distributions, O 1 and O 2 , obtained by imaging two infinitely small point sources through a diffraction-limited system; the superimposed intensities of O 1 and O 2 in incoherent microscopy b and coherent microscopy c.In-phase and anti-phase in c indicates that the cases O 1 and O 2 have a phase difference of 0 and π, respectively.

Fig. 3
Fig. 3 DHM with oblique illumination.a Schematic of synthetic aperture via oblique illumination in different orientations.b schematics of setup of off-axis DHM with oblique illumination.Fig. 3b is adopted from Ref. 49.

Fig. 4
Fig. 4 DHM with structured illumination.a Schematic of structured illumination with different orientations and phase shifts.b Schematic setup of DHM with structured illumination.c Experimental results for resolution enhancement.Reconstructed phase images using plane wave illumination (top) and structured illumination (bottom).b is adopted from Ref. 77. and c is adopted from Ref. 76.

( 3 )
In the object plane, a constraint concerning the finite size and positive absorption of the sample is applied to o(x o , y o ), resulting in (x o , y o ).

( 4 )
Propagate (x o , y o ) in the hologram plane and replace the amplitude of (x o , y o ) with the square root of the hologram intensity H0 within the central N 0 ×N 0 area (see Fig.6a2 ).

Fig. 6
Fig. 6 Resolution enhancement of lensless DHM by self-extrapolation of holograms.a Experimental setup and results.a 1 , hologram with zeropadding.a 2 , hologram after 100 iterations.a 3 and a 4 refer to the reconstructed image without and with self-extrapolation, respectively.b Illustration of iterative reconstruction of a hologram by self-extrapolation.The image is adopted from Ref. 104.

Fig. 7
Fig. 7 Resolution enhancement of DHM via synthesizing a larger hologram.a Synthetic aperture of lens-free DHM using an independent reference wave.b Synthetic aperture of lensless DHM 102 , reproduced with permission © IOP Publishing.c Synthesized frequency spectrum.d and e, Images of blood cells without and with applying synthetic aperture, respectively.The images in (c−e) are adapted from Ref. 113 ©The Optical Society.

Fig. 8
Fig. 8 Pixel super-resolution (PSR) for on-chip lensless DHM.Subpixel shifts in a digital hologram can be achieved through a source shifting, b an array of static light sources, e.g., fiber-coupled LEDs that are individually controllable, sensor shifting, or c sample shifting using e.g., a micro-fluidic flow.d An in-line hologram before PSR.e Pixel super-resolved hologram of the same object.f Reconstructed image of the grating with 338 nm line-width using PSR.g Microscope comparison of the same region of interest.The image is adopted from Ref. 118.

Fig. 9
Fig. 9 Schematics of the deep-learning based method for resolution enhancement of DHM.Images were taken from Ref. 144.

Table 1
Overview of Different Resolution enhancement techniques for DHM

•
Complex configuration due to the usage of an objective, suffers from objective-related aberration • Achievable resolution: • Resolution enhancement factor of 2~5, depending on NA illum /NA imag .No extra measurement is needed • Achievable resolution is λz/(NΔ S ) instead of λz/(N 0 Δ S ), when the hologram is effectively extrapolated from N 0 Δ S to NΔ S .Achievable resolution: δ=λz/L hologram • Effective NA up to 0.3, and FOV: up to 20×16 mm 2 110, 112, 113 Pixel super-resolution for on-chip DHM • Subpixel-shifts (e.g.,6×6) of hologram yields an NA of 0.9 • Oblique illumination further enhances NA to 1.4 and improves the SNR.Relies on large-scale data sets, time-consuming.• Reconstruction can be wrong if the training data is inadequate.