Implementation of free-space Fourier Ptychography with near maximum system numerical aperture

: Over the past decade, the research field of Fourier Ptychographic Microscopy (FPM) has seen numerous innovative developments that significantly expands its utility. Here, we report a high numerical aperture (NA) FPM implementation that incorporates some of these innovations to achieve a synthetic NA of 1.9 – close to the maximum possible synthetic NA of 2 for a free space FPM system. At this high synthetic NA, we experimentally found that it is vital to homogenize the illumination field in order to achieve the best resolution. Our FPM implementation has a full pitch resolution of 266 nm for 465 nm light, and depth of field of 3.6 µm. In comparison, a standard transmission microscope (incoherent) with close to maximum possible NA of 0.95 has a full pitch resolution of 318 nm for 465 nm light, and depth of field of 0.65 µm. While it is generally assumed that a free-space coherent imaging system and a free-space incoherent imaging system operating at their respective maximum NA should give comparable resolution, we experimentally find that an FPM system significantly outperforms its incoherent standard microscopy counterpart in resolution by a factor of 20%. Coupled with FPM’s substantially longer effective depth of field (5.5 times longer), our work indicates that, in the near-maximum NA operation regime, the FPM has significant resolution and depth of field advantages over incoherent standard microscopy.


Introduction
Fourier Ptychographic Microscopy (FPM) [1,2] is a computational microscopy method that combines the key concepts of synthetic aperture and phase retrieval, allowing the user to synthesize a series of low resolution microscopy images into a high resolution microscopy image. The low-resolution images are acquired through standard microscopy where the angle of the illumination light is varied for each of the raw images. In the angular spectrum space, the information of the raw images overlaps with each other. By exploiting the information redundancy in the overlapping regions, we can apply phase retrieval methods [1,[3][4][5][6][7] to reconstruct the missing phase information and then apply synthetic aperture concept to correctly merge the disparate raw data in the angular spectrum space into a contiguous wide spanning and accurate angular spectrum representation of the original microscopy sample. When transformed back into the spatial domain, the resulting image has improved resolution and field of view. By this means, FPM enables the user to generate images with superior resolution than what the native numerical aperture (NA) of the objective is supposed to be able to provide. As a reference point, the original FPM paper [1] showed that an objective of NA = 0.08 in FPM was able to generate images with resolution comparable to those generated with an objective of NA = 0.5 in a standard microscope.
In addition to improving resolution, the FPM's computational microscopy algorithmic approach allows the user to correct inherent physical system aberrations by using aberration correction functions [7]. This ability enabled a shift in standard microscopy design paradigm. Prior to FPM's invention, it was generally assumed that the correction of aberration has to be done in the physical world by building microscopes with as little aberrations as possible. FPM showed that an imperfect microscope with aberrations can still be used to generate optically 'perfect' (aberration-free) images by correcting the aberrations in the computational image processing [7]. This substantial advantage was leveraged in parallel microscopy implementations, where very compact but highly aberrative microscope arrays can be used to perform high-quality parallel imaging by using FPM to correct aberrations computationally [8].
A corollary consequence of FPM's phase recovery ability is that it can be applied to digitally refocus its rendered images as desired. This digital refocusing ability can substantially extend the effective depth of field (DoF) of the microscope several times beyond its native DoF dictated by the NA of the objective. In the original FPM paper, the DoF of a 2x objective (0.08NA) was extended from ∼80 µm to ∼300 µm [1].
The advantages of resolution improvement and extended DoF associated with the FPM method have been extensively studied over the past decade [7][8][9][10][11][12][13][14][15][16]. Of particular interest within the community is the improvement in resolution which is directly determined by the synthetic NA of FPM. According to the Fourier frequency model of FPM, the synthetic NA of FPM, denoted as NA syn , is expressed as Eq. (1): where NA obj is the NA of the objective lens and NA illu is the maximum illumination NA (the largest illumination angle  [13]. The respective synthetic NA attained by these prior works are, respectively, 1.5, 1.05, 1.33 (dry) and 1.6 (oil immersion). This paper reports our findings at combining some of the previously reported methods to implement an FPM system with a NA syn of 1.9 -close to the theoretical upper limit of 2. Our experiment is aimed at studying whether FPM can perform well at such high NA syn limit, and whether any non-obvious complications need to be addressed to achieve optimal performance. To accomplish this task, we implemented a high-NA system with an objective of NA = 0.95 and an LED quasi-dome with illumination NA = 0.95. On the processing side, we incorporated illumination correction circular edge detection (CED) [15] and simulated annealing (SA) algorithms [11,13] into the algorithm that generates high-quality FPM reconstruction. This high-NA FPM implementation has a sparrow-limit based, full pitch resolution of 266 nm with 465 nm incident light when measured on a Siemens star target.
In the following sections, we first introduce the experimental arrangement, then compare the resolution of our FPM with a standard high-NA microscope, and finally compare images of biological samples taken with both systems. We additionally report the measured DoF performances of the high-NA FPM system and then conclude by summarizing our findings. Figure 1(a) shows the schematic of the high-NA FPM arrangement. We implemented an inverted transmission FPM system for this experiment. A quasi-dome LED array with illumination NA of 0.98 [18,19] was placed about 50 mm above the focal plane of a 50x/0.95 objective (Olympus MPLAPON50X). A sample to be imaged can be inserted in between the LED array and the objective. A tube lens was inserted after the objective and imaged the LED light onto a CMOS camera (Hamamatsu C11440-42U30, 6.5 µm x 6.5 µm pixel pitch). The entire set up resulted in a 0.27 mm diameter field of view. During the data acquisition process, the sample was illuminated with 39 LEDs successively, resulting in an effective illumination NA of 0.95 and a total synthetic NA of 1.9. The center wavelengths of the full-color LEDs were 621 nm (red), 525 nm (green), and 465 nm (blue), and the average power of the red, green and blue LEDs were 1.6 mW, 3.5 mW and 3.4 mW respectively. Due to the large range of illumination angles involved, the collected images had a large span of intensities. Although the quasi-dome has increased the illumination efficiency at large illumination angles by its specially designed geometry, the incident light irradiance at high angles is still relatively low due to the grazing angle nature of the illumination. To accommodate the large dynamic range of captured image intensity, we adopted two procedures to avoid camera saturation. First, the camera exposure times were set separately, 0.3 s for low NAs (≤ 0.75) illuminations and 5 s for high NAs (>0.75) illuminations. Second, at high NA illumination, the raw data was recorded and averaged over three camera images. The total acquisition time for the entire sequence of raw images at one color channel was about 12 mins. The image processing procedure is summarized in Fig. 1(b). We performed illumination correction prior to image reconstruction, by using one raw data set (described in Section 2.2). This procedure accurately determines the illumination angle associated with each LED and provides us with an accurate determination of the Fourier frequency shifts in Fourier domain. We then employed embedded pupil function recovery (EPRY) [7] reconstruction to correct for residual aberrations and then output a high-NA FPM image. In our experiments, illumination correction was only performed once after the system has been set up. Subsequent FPM rendering all share the same corrected illumination data.

Illumination correction
Accurate determination of illumination angle and corresponding Fourier frequency shift induced by each oblique illuminating LED is essential for high quality FPM reconstruction. In a recent paper, researchers were able to determine the LED positions of a flat LED array based Fourier Ptychography system by representing the misalignment as rotation and translation of the array in a physics model [20]. While the approach has certain advantages in implementation and flexibility, we were unable to adopt it for our system due to the quasi-dome's more complicated geometry. Unlike flat LED array, whose simple geometry makes it straightforward to calculate Fourier frequency shifts, the LED quasi-dome has complicated LED distribution, hindering direct and accurate calculation of Fourier frequency shifts. To perform successful illumination correction, we first performed pre-iteration coarse position correction, followed by iterative fine intensity and position correction (see Fig. 2(a-b)). With only rough values of the Fourier frequency shifts (without any correction), FPM reconstruction would be inaccurate, and the resulting images would appear blurry and distorted, as showed in Fig. 2(c1).
We used circular edge detection (CED) [15] as our coarse position correction method (see Fig. 2(a)). As the maximum illumination NA matched the objective NA in our experiment, all raw images were bright field images. As showed in Fig. 2(a), the Fourier spectrum of a bright field image contains two centrosymmetric circles, for which CED has been showed in prior works [15] to be effective for coarse position correction. By jointly considering the first and second derivation of the Gaussian filtered spectrum [15], both of which indicate the circle edge, CED can estimate the position of the circle center. Figure 2(c1-c2) shows the FPM image improvement in a Siemens star image when CED was used for coarse position correction. However, we also note that while Fig. 2(c2) (CED corrected) is visibly sharper over Fig. 2(c1) (no correction), there are still significant spoke distortions (marked by yellow arrows) that indicate a need for further improvement.
For finer corrections, we next performed iterative fine intensity and position correction to further improve the FPM reconstruction quality. The fine correction was based on simulated annealing (SA) algorithm [11] and was embedded within the embedded pupil function recovery (EPRY) [7] FPM reconstruction process. Figure 2(e) is a schematic of the workflow. Within one iteration, for each raw image, intensity correction was first performed prior to the execution of the SA algorithm. The SA algorithm then performed fine correction of the Fourier frequency shift (see Fig. 2(b)). The FPM algorithm then updated the FPM image complete with phase information. During the fine correction, SA algorithm will consider Fourier frequency shift values close to current values and use rendered FPM image from previous iteration to generate a set of estimated oblique illumination images that correspond to the original oblique illumination raw image. If we have the correct Fourier frequency shift values, the estimated image should match raw image exactly. As such we can use the mean square error between estimated set and captured set as an indicator of whether the correct Fourier frequency shifts have been achieved. We continued to run this workflow until the mean square error was minimized. A corrected Siemens star image rendered through this process is shown in Fig. 2(c3). Compared to the prior two images, we can clearly see that it has the sharpest appearance and the least distortions. For our setup, the final corrected illumination map is showed in Fig. 2(d).

Resolution assessment and pupil correction
In this section, we assess the resolution performance of the high-NA FPM system and compare it with a standard transmission (incoherent) microscope imaging under 0.95 NA brightfield illumination. We selected these two system for comparison because they share the same pass band in Fourier frequency domain [21]. A Siemens star (Ready Optics, 2017_Pair) target was imaged under blue light (465 nm) with the imaging strategy described in Section 2.1 to experimentally determine the resolution. The Sparrow-limit criteria was adopted here and resolution was defined as the smallest periodicity at which all spokes of the Siemens star are observable (full-pitch resolution) [22][23][24]. This assessment is less prone to resolution mis-reporting and can be applied to compare resolution performance of different microscopy methods [23]. The Siemens star target can be used to examine a large range of spatial frequencies along all directions and verify the degree of isotropy of the imaging system. As such, it can prevent the latent risk of reporting higher Fourier frequency that can only be resolved along specific directions. Such risk would be an issue for targets such as USAF target [13,14] and two-slit target [17], as those targets generally allow resolution characterization in at most two directions. Siemens star target is also helpful in revealing aberrations, as the spokes will appear distorted or blurry when imaged by an aberrated imaging system. However, conventional resolution equations are only applicable to two-point targets and it is inappropriate to apply them directly to a Siemens star target. Thus, we performed simulation to find reliable resolution expressions for both systems with Siemens star as a target. For completeness, the specific process and results are reported in Supplement 1.
According to our simulation, a high-NA FPM system has a resolution (Sparrow-limit) determined by: With the same generalized simulation, we also found that the resolution (Sparrow-limit) of the standard transmission microscope is given by: where λ is the incident wavelength. Our imaging experiment with both the high-NA FPM and the standard transmission microscope yielded Siemens-star target images that are shown in Fig. 3(a1) and Fig. 3(b1). Normalized intensity plots along the smallest circles where all spokes are resolvable for high-NA FPM (green circles) and standard transmission microscope (blue circles) are included as well (Fig. 3(a2-3) and Fig. 3(b2-3)). We assess the resolvable spoke periodicity based on two criteria. First the spokes must be well spaced at the predicted periodicity. In Fig. 3(a2), (a3), (b2), (b3), we placed down evenly spaced vertical grid line at the nominal troughs of the spokes. If two or more spokes appear within any of the grid delineated intervals, we would reject the pattern as unresolved. Second, we require the individual spokes to be identifiable by Sparrow's criterion -in other words, there is an observable trough or minimum between the spokes. Figure 3(a2) shows that spoke periodicity of 266 nm is just resolvable by our high-NA FPM system, and thereby establishing its resolution to be 266 nm, which is 9% deviated from the theoretical resolution prediction. Figure 3(b3) shows that that spoke periodicity of 318 nm is just resolvable by the standard transmission system, and thereby establishing its resolution to be 318 nm, which is 15% deviated from the simulated resolution prediction.
The fact that high-NA FPM outperformed standard transmission microscope in resolution can be attributed to 3 aspects.
First, we note that the contrast of the inner Siemens star spokes is notably higher for the high-NA FPM systems compared to the standard transmission microscope (see Fig. 3(a1) and Siemens star imaged by standard transmission microscope (b1) and intensity plots along inner green (b2) and outer blue (b3) rings. Scale bar: 5 µm Fig. 3(b1)). This is because the high-NA FPM benefits from coherent imaging where high Fourier frequency components are less attenuated than that in incoherent imaging.
Second, the high-NA FPM system benefits from EPRY's ability to correct residual aberration in conventional objectives -an ability missing from standard transmission microscopy, which has been previously reported in literature [7].
Third, intensity correction was adopted when high-NA FPM was performed, compensating the inhomogeneity of different illumination angles and reducing weak-illumination-induced high Fourier frequency attenuation. To show that the impact of such corrections is significant for high-NA FPM, we ran a comparison experiment where we did not compensate for the inhomogeneity. Figure 3(a) and Fig. 4(a) show the FPM image with and without compensation, respectively. The resolution achieved in the uncompensated processing is 294 nm -11% poorer than for the compensated processing. We further note that this issue is particularly acute for high-NA FPM systems, as low-NA FPM systems do not have to illuminate the sample at large grazing angles.
In principle, this type of intensity correction can also be performed for standard transmission microscopy. The illumination source would have to be designed such that the illumination intensity is carefully scaled based on the illumination angle. In a standard Kohler illumination scheme, a well-designed attenuation mask inserted at the right Fourier plane may be able to accomplish this at the cost of significant net attenuation. For our current study, we can derive the results for an equivalent intensity correction for standard transmission microscope was performed by summing up the illumination-corrected (with method described in section 2.2) raw data together to generating a standard transmission microscope image. Intensity corrected results of standard transmission microscope are demonstrated in Fig. 4(b). The resulting resolution of standard transmission microscope was improved from 318 nm (15% deviation from simulation) to 300 nm (8% deviation from simulation). This clearly demonstrates the utility of illumination inhomogeneity compensation for both high-NA FPM and high-NA standard transmission microscopy.

Imaging biology samples
We performed color high-NA FPM imaging and color standard transmission microscope imaging on both a blood smear and a bone marrow smear (Precision for Medicine). We repeated the imaging procedure in Section 3.1 for each color channel and then synthesized RGB color channels together as color images. Figure 5 shows comparison images of the blood smear sample from the high-NA FPM system and the standard transmission microscope. In addition to the intensity images, the high-NA FPM system is also able to generate phase images (Fig. 5(d) and 5(h)) and provide an objective aberration characterization in the form of the pupil wavefront for different regions of interest (ROIs) (Fig. 5(e) and 5(i)) via the EPRY algorithm [7]. All reported phase images were acquired with blue light (465 nm) illumination. Consistent with our experimental findings in Section 3.1, the high-NA FPM reveals more detailed sample structures (indicated by arrows in Fig. 5(b2), 5(c2), 5(f2) and 5(g2)).
For completeness, the imaging results of a bone marrow smear sample are reported in the Supplement 1. The results showed similar advantages for the high-NA FPM system.

Depth of field measurement
We examined the depth of field (DoF) of both the high-NA FPM and standard transmission microscope experimentally. A Siemens star displaced at controlled distances (defocus distances) from the objective lens focal plane was imaged by both high-NA FPM and standard transmission microscope with blue light (465 nm). A 3-axis motorized stages (Thorlabs MLS203-1 for XY directions and Thorlabs MZS500-E for Z direction) controlled the defocus distances precisely. Digital refocusing [1,7,12] with light field initialization [25] was performed in high-NA FPM experiments. Resolutions under different defocus distances were determined with the same method as reported in Section 3.1. We varied the defocus distances from 0 µm to 5 µm, with 0.5 µm spacing. For both modalities, we calculated resolution deviations at each defocus distance from their corresponding in-focus resolution and applied interpolation with piecewise cubic Hermite interpolating polynomial (PCHIP) [26,27] method. Both measurement data points and interpolated curves are plotted in Fig. 6(a). For completeness, the exact resolution values, images of Siemens star under different defocus distances taken by both image systems and corresponding resolution plots are included in the Supplement 1.
The DoF of NA-matched standard transmission microscope, denoted as Dof tr , is given by where n is the refractive index, M is the magnification of objective lens and e is the camera pixel size. The DoF of high-NA FPM is denoted as Dof FPM in the following session. The half Dof FPM was estimated by finding the defocus distance where the high-NA FPM's resolution deviation equals that of the standard transmission microscope under half Dof tr . Interpolated resolution deviation curves indicated defocusing half Dof tr lead to 8.2% resolution deterioration for the standard transmission microscope. To accumulate the same amount of deterioration, the high-NA FPM system would have to be defocused by 1.8 µm. Thus, the DoF of our high-NA FPM setup was 3.6 µm (double of 1.8 µm) or 5.5 times longer than that of the standard transmission microscope. It is also worth mentioning that, compared to standard transmission microscope, high-NA FPM achieved better resolution at every defocus distance, and has a more gradual resolution deterioration. This observation further supports the idea that high-NA FPM has better DoF properties, as it has an extended DoF and less resolution deterioration when the defocus is severe.

Conclusion
We have developed a high-NA FPM system that combines both close-to-limit high-NA (0.95) objective lens and high-NA (0.95) illumination. A synthetic NA of 1.9 was achieved, resulting in 266 nm full-pitch resolution (Sparrow-criterion) with 465 nm illumination. The DoF of the high-NA FPM was 3.6 µm, 5.5 times larger than that of a standard transmission microscope.
FPM takes advantage of oblique-illuminated data and is thus able to perform imaging at a NA higher than that claimed by the objective lens. However, as FPM is a coherent imaging method, this does not necessarily mean that its resolution would be substantially higher than that of a standard transmission microscope operating with the same objective. From Fourier optics' perspective, the pass band in Fourier frequency domain of FPM (a coherent imaging) is determined by the summation of the objective lens NA and the illumination NA. While for an incoherent situation the pass band in Fourier frequency domain is determined by twice of the objective lens NA. Therefore, to achieve the same pass band in Fourier frequency domain, the synthetic NA of FPM must be twice as large as the objective lens NA. This means that the illumination NA must match the objective lens NA for FPM. When the illumination NA exceeds the objective NA, FPM can outperform standard transmission microscope in resolution as FPM's synthetic NA is now more than double the objective's NA. This ability was demonstrated in the original FPM paper where an FPM system operating with a NA = 0.08 objective was able to render images comparable in resolution to images acquired in a standard transmission microscope with an objective of NA = 0.5.
In our current experiment, exceeding the objective's NA is not feasible as the objective's NA is already close to unity. Nevertheless, when we operate our FPM with the illumination NA matched to the objective's NA, we can still observe a resolution improvement compared to a standard transmission microscope operating with the same objective (as reported in Section 3.1). These results are consistent with previous research reporting that FPM outperforms standard transmission microscope with comparable NAs [17] in resolution tests. In prior studies, the better FPM performance was attributed to the coherent imaging nature of FPM and FPM's ability to correct aberrations. For high synthetic NA situation, we found that the relative ease by which FPM setup can tailor and achieve illumination intensity homogeneity also contribute to FPM's ability to achieve better resolution than the standard transmission microscope. While standard transmission microscopy can only tailor illumination intensity at the hardware level, the FPM can correct for intensity heterogeneity during computation -by rescaling the data accordingly.
In conclusion, our study demonstrates that the experimental implementation of a high synthetic NA FPM can achieve resolution that is close to the theoretical predicted limit. In combination with the substantially longer effective depth of field (5.5 times longer) than a comparable standard transmission microscope, our study indicates that a well-designed and well-implemented high synthetic NA FPM can be a viable replacement for standard transmission microscope for high NA applications.