Optimization of sampling pattern and the design of Fourier ptychographic illuminator

: Fourier ptychography (FP) is a recently developed imaging approach that facilitates high-resolution imaging beyond the cutoff frequency of the employed optics. In the original FP approach, a periodic LED array is used for sample illumination, and therefore, the scanning pattern is a uniform grid in the Fourier space. Such a uniform sampling scheme leads to 3 major problems for FP, namely: 1) it requires a large number of raw images, 2) it introduces the raster grid artefacts in the reconstruction process, and 3) it requires a high-dynamic-range detector. Here, we investigate scanning sequences and sampling patterns to optimize the FP approach. For most biological samples, signal energy is concentrated at low-frequency region, and as such, we can perform non-uniform Fourier sampling in FP by considering the signal structure. In contrast, conventional ptychography perform uniform sampling over the entire real space. To implement the non-uniform Fourier sampling scheme in FP, we have designed and built an illuminator using LEDs mounted on a 3D-printed plastic case. The advantages of this illuminator are threefold in that: 1) it reduces the number of image acquisitions by at least 50% (68 raw images versus 137 in the original FP setup), 2) it departs from the translational symmetry of sampling to solve the raster grid artifact problem, and 3) it reduces the dynamic range of the captured images 6 fold. The results reported in this paper significantly shortened acquisition time and improved quality of FP reconstructions. It may provide new insights for developing Fourier ptychographic imaging platforms and find important applications in digital pathology. provide new insights for developing fast FP platforms and find important applications in digital pathology.


Introduction
Fourier ptychography (FP) is a recently developed imaging technique for wide-field, highresolution microscopy [1]. In brief, FP illuminates the sample using oblique incident angles and captures the corresponding low-resolution images. Sharing its roots with ptychography [2][3][4][5][6][7][8] and other phase retrieval methods [9][10][11], the Fourier ptychographic recovery process iteratively synthesizes the captured images in the Fourier domain to recover a high-pixelcount complex sample image [1,[12][13][14][15][16][17]. In particular, FP switches between the spatial and the Fourier domain. In the spatial domain, the captured images are used as the intensity constraint for the solution. In the Fourier domain, the confined pupil function of the objective lens is used as the support constraint for the solution. This pupil function constraint is digitally panned across Fourier space according to the illumination angle. By introducing a phase factor to model the pupil function, the FP approach is able to bypass the design constraints of a conventional microscope platform and digitally correct for aberrations associated with the employed optics [18][19][20]. Recently, we have also extended the FP recovery scheme for super-resolution fluorescence microscopy [21] and incoherent photography [22].
To better understand the motivation of this paper, it is helpful to compare and contrast ptychography and FP. Both ptychography and FP capture multiple intensity images of the sample and seek a complex solution that is consistent with these measurements. In ptychography, the sample is scanned in the spatial domain and diffraction patterns are captured at the Fourier domain (i.e., the far field) without using any lens. For FP, the spatial domain and the Fourier domain are swapped using a lens. Thus, the scanning process of FP is at the Fourier domain and the images are captured at the spatial domain. A critical consideration for both approaches is the scanning pattern, which includes the scanning sequence, overlap ratio between adjacent samples, and the overlap uniformity. For ptychography, different scanning patterns in the spatial domain have been demonstrated to achieve uniform overlap and good convergence, including the raster scanned mesh pattern with random offset [23], the concentrated-circles [24], and the Fermat spiral trajectory [25]. Unlike ptychography, the scanning pattern of FP is in the Fourier domain and the signal distribution may need to be taken into consideration [26]. For most biological samples, signal energy is concentrated at low-frequencies which makes it better to start the scanning sequence from the low-frequency regions where most of the energy is located. By doing so, the solution converges to the global minimum with fewer iterations. Furthermore, sampling overlap is applied non-uniformly at different regions to best capture signal distribution in the Fourier space [26]. In the original FP approach, a periodic LED array is placed at the far field for sample illumination, and therefore, the scan pattern is a periodic grid in the Fourier domain. Such a uniform sampling scheme leads to 3 major problems for FP, namely: 1) it requires a large number of raw images to be acquired, 2) it introduces raster grid artifacts [8] in the reconstruction process, and 3) it needs a high-dynamic-range detector.
In this paper, we investigate the scanning sequences and sampling patterns to optimize the FP approach. We will show that signal energy criteria can be used to determine the scanning sequence in the reconstruction process. Based on the energy distribution in Fourier domain, a non-uniform sampling scheme can be used to reduce the number of acquisitions. To this end, we have designed and built an illuminator using LEDs mounted on 3D-printed plastic rings. The advantages of this illuminator are threefold: 1) it reduces the number of image acquisitions by at least 50% (68 raw images versus 137 in the original FP setup), 2) it departs from the translational symmetry of sampling to solve the raster grid artifact problem, and 3) it reduces the dynamic range of the captured images 6 fold. The results reported in this paper significantly shortened the acquisition time and improved quality of FP reconstructions. It may provide new insights for the development of FP platforms and find important applications in digital pathology. This paper is structured as follows: in section 2, we will investigate the scanning sequence of the FP approach for fast solution convergence. In section 3, we will investigate the sampling pattern in the Fourier domain to reduce the number of acquisitions. In section 4, we will discuss the design of the Fourier ptychographic illuminator and demonstrate its advantages. Finally, we will summarize the results in section 5.

Choice of scanning sequence for Fourier ptychography: energy criteria
The choice of scanning sequence is important for fast solution convergence during the FP reconstruction process. Such a sequence defines the starting point of the optimization algorithm. If the starting point is not properly chosen, the iterative alternative projection algorithm [27] may stagnate at a local minimum instead of reaching the global minimum. In this section, we will answer the following question: If we have N raw images corresponding to different incident angles, what is the optimal sequence of these raw images for the FP reconstruction algorithm? For example, consider raw images I 1 , I 2 , and I 3 with each image corresponding to a different incident angle. In the FP reconstruction algorithm, we can update the sample estimate first with I 1 , then with I 2 , and lastly with I 3. We can also choose another updating sequence such as I 2 , I 1 , and I 3 . We note that in the experimental implementation of FP, the order of acquiring raw images is irrelevant to the algorithm (one can reorder the sequence after the data have been acquired). When processing the data post-acquisition, we need to be mindful of the sequence of raw images we use to update the sample estimate. However, implementing fast FP would require each captured raw image to be processed in real time, and in such a case the acquisition scanning sequence needs to be the same in the hardware, i.e., we need to light up the LED elements in the same optimal order.
Here, we will discuss three recovery sequences and compare the quality of the reconstructed images and convergence speeds. The three recovery sequences are 1) a random sequence, 2) the sequence ranked by the LED illumination numerical aperture (NA), and 3) the sequence ranked by the total energy of the raw image. For case 1, we will generate a random sequence of the captured images and use them to update the sample estimate. For case 2, we will update the sample estimate using the images from the smallest incident angle to largest incident angle. For case 3, we will reorder the captured images according to their total intensity values and use this to update the sample estimate. For each case, we will use the Fourier ptychographic algorithm for recovering the high-resolution images. The algorithm starts with a high-resolution spectrum estimate of the sample. Next, this sample spectrum estimate is sequentially updated with the intensity measurements. For each updating step, we select a small sub-region of the spectrum estimate, corresponding to one position of the circular aperture, and apply Fourier transformation to generate a new low-resolution target image. We then replace the target image's amplitude component with our measurement to form an updated, low-resolution target image. This image is then used to update its corresponding sub-region of the sample spectrum. The replace-and-update sequence is repeated for all intensity measurements, and we iterate through the above process several times until solution convergence [1,15]. Figure 1 shows the reconstructed images obtained from the three cases discussed above. We quantified the root-mean-square error (the difference between the FP reconstruction and the ground truth) for each of the results in Fig. 1(e). We can see that the image quality of the random order is worse when compared to the other two sequences and the corresponding RMS error is also much higher. This simulation study shows that, a carefully chosen recovery sequence is important for fast solution convergence. In the case of a random sequence, the solution may stagnate at a local minimum instead of the global minimum, as shown in Fig.  1(e). We also note that in the study shown in Fig. 1, the illumination-NA order and total energy order give similar results and convergence speeds. The reason can be explained as follows: the energy of sample image in the Fourier domain is concentrated at low-frequencies, and thus, the energy level decreases as the illumination NA increases. Therefore, these last two cases have the same image sequence. To study the difference between the illumination NA order and the total energy order, we need to consider a sample image with Fourier spectrum energy not concentrated at low frequencies. In Fig. 2, we consider the same ground truth image ( Fig. 1(a)) modulated by a sinusoidal pattern. In this case, the energy in the Fourier space is concentrated at two different positions, determined by the sinusoidal frequency. We repeat this simulation study with three different sequences: a random order (Fig. 2(a)), the illumination NA order (Fig. 2(b)), and the total energy order (Fig. 2(c)). The RMS error curves are plotted in Fig. 2(d) which clearly shows that the total energy order gives the best performance for solution convergence. During the iterative recovery process, raw images with high energy levels quickly guide the solution to the global minimum of the solution space. Therefore, considering the energy distribution of the sample image helps to quickly converge to the solution. This leads us to discuss nonlinear sampling patterns in the next section.

Non-uniform sampling pattern for Fourier ptychography
In the previous section, we have shown that the energy criteria can be used to optimally order the reconstruction sequence. Here, we will investigate various sampling patterns in the Fourier domain. As discussed in the previous section, we will account for the energy distribution of the sample and perform non-uniform sampling in the Fourier domain. In Fig. 3, we simulate various sampling patterns in the Fourier domain. We can define the Fourier overlapping percentage as the overlapping area (in the Fourier space) of two adjacent acquisitions, divided by the area of pupil function. Going from Fig. 3(a1) to (a5), the scanning patterns have a higher overlapping percentage at the center while the total number LED elements remains the same. In particular, Fig. 3(a2) shows a linear sampling pattern where the LED elements are uniformly distributed in Fourier space. The corresponding FP reconstructions are shown in Fig. 3(b1)-(b5). In Fig. 3(c), we plot the overlapping percentages as a function of illumination NA for the cases shown in Fig. 3(a1)-(a5). We further define the sampling density ratio as the overlapping percentage at the edge divided by that at the center. Using this definition, the sampling density ratios increase from 0.7 to 2.0, going from Fig.  3(a1)-(a5). The RMS errors corresponding to these sampling density ratios are shown in Fig.  3(d). We see that a higher sampling density at the center of the Fourier space helps to recover a better FP image. This result can be explained using the same logic as the discussion in section 2: Since the energy of the sample concentrated at low-frequencies, a higher sampling density at the low-frequency region helps the solution converging faster to the global minimum.
Another important consideration of the sampling pattern is the translational symmetry. In the implementation of FP, we often update the sample and the pupil function simultaneously [19,20]. By doing so, we can recover unknown pupil aberrations and get a better highresolution sample estimate. If the sampling pattern (the LED pattern) is a periodic pattern in the Fourier space, it leads to the raster grid pathology problem for the pupil function [8], i.e., the recovered pupil function becomes a mixture of the true aberrations and a periodic pattern. This corrupted pupil function then degrades the recovered FP image. We study this raster grid artefact problem in Fig. 4, where we compare two cases: a non-linear sampling pattern and a periodic pattern in the Fourier space, as shown in Fig. 4(a) and (b). The RMS errors are plotted as a function of the loop iteration in Fig. 4(c). Figure 4(d)-(g) show the recovered sample images and the pupil functions. We can see that with the periodic sampling pattern, the recovered image degrades with more iterations. The recovered pupil function also contains a periodic pattern, as shown in Fig. 4(f2) and (g2). On the other hand, with the nonlinear ring pattern shown in Fig. 4(a), the recovered FP image continues to converge with more iterations. The recovered pupil function does not contain a periodic pattern, either. To summarize, we have demonstrated how a non-uniform sampling pattern in the Fourier space is able to 1) improve the solution convergence if the sampling density is higher at highenergy regions, 2) break the translational symmetry and solve the raster grid artefact problem. In the next section, we will discuss an experimental implementation of the non-uniform sampling pattern using ring LEDs. In particular, we aim to achieve the sampling pattern shown in Fig. 4(a). This sampling pattern is designed under the restriction on the available LED ring elements. One can design other sampling patterns under a simple guideline: the spatial frequency overlapping ratio decreases from 40% at the center (bright field images) to ~15% at the edge (darkfield images).

Design of Fourier ptychographic illuminator
Based on the discussion in the previous section, we need to consider two design aspects of the Fourier ptychographic illuminator: 1) non-uniform sampling with higher density at the center, and 2) breaking the translational symmetry. Figure 5 shows our design of illuminator that achieves the sampling pattern shown in Fig. 4(a). To achieve a higher sampling density at the low-frequency region, we placed 4 LED elements and the first LED ring far away from the sample and use a mirror to direct the light to the sample. Other LED rings are positioned at larger incident angles and closer to the sample. We used a 3D printer (Makerbot) to create a plastic mount for LED rings, as shown in Fig. 5. A microcontroller was used to sequentially light up each LED element. Figure 5(a) shows the microscope setup (Olympus BX-43) with the reported illuminator. In our experiment, we used a 4X, 0.1 NA objective lens to acquire the raw image sequence and the final synthetic NA is about 0.55. Another design consideration of the FP illuminator is the dynamic range of captured raw images. By illuminating the sample from different incident angles, FP capture both brightfield and dark-field images. Bright-field images correspond to low illumination NA and darkfield images correspond to high illumination NA. The intensity levels of the dark-field images are orders of magnitude lower than those of bright-field images. Therefore, an optimal FP illuminator needs to compensate this effect by reducing the bright-field illumination intensity while increasing the relative dark-field illumination intensity. In the reported illuminator, the 4 LED elements and the first LED ring deliver bright-field images (Fig. 5(c)) and they were placed far away from the sample (the illumination NA of these elements are less than the collection NA of the objective lens). As such, we reduce the relative illumination intensity for bright-field compared with that of dark-field. Figure 6 shows the measured intensity of the LED elements as a function of illumination NA. This figure compares two cases, one for the uniform LED array (Fig. 6(a)) and the other for reported illuminator (Fig. 6(b)). From this comparison, we can see that the reported illuminator is able to reduce the dynamic range of FP raw image sequence 6 fold.  In Fig. 7, we compared the FP reconstructions of the LED matrix with the reported FP illuminator. For both cases, we used 68 LED elements for sample illumination and a 4X (0.1 NA) objective for image acquisition. The final synthetic NA for both cases is ~0.55. We can see that, the image quality of the FP reconstruction using the reported illuminator is much better than that of the LED matrix. The entire iterative reconstruction process using the reported illuminator can be found in Media 1. Compared to the LED matrix illuminator, the reported FP illuminator is able shorten the acquisition time by at least 50% (68 images versus 137 images [1]). In Fig. 8, we recovered the color images using R/G/B illuminations and compared them to the conventional high-NA microscope objective. Again, the reported illuminator is able to deliver much better image quality. As we have discussed in section 3, the reported FP illuminator is able to break the translational symmetry and solve the raster grid artifact problem that plagued the original FP approach. We performed an experiment to verify this solution. In Fig. 9(a), we used a 15 by 15 LED matrix for sample illumination and recover both the sample image and the pupil function. In this experiment, we put the LED matrix farther away from the sample to get enough sampling overlap at the center of the Fourier space (If we only use 68 LEDs as in Fig.  7, we cannot get a converged solution). From Fig. 9(a), we can clearly see the raster grid artefact problem where the pupil function is corrupted by the periodic artifact. In Fig. 9(b), on the other hand, we do not see raster grid artefact problem and the recovered image has a higher quality.

Conclusion
In summary, we have discussed optimizing the scanning sequence and sampling pattern for the FP approach. For most biological samples, signal energy is concentrated at lowfrequencies which makes it better to start the scanning sequence from the low-frequency regions where most of the energy is located. By doing so, the solution converges to the global minimum with fewer iterations. We also show that, a non-uniform sampling overlap leads to better FP reconstruction compared to the original LED array illuminator. To implement the non-uniform Fourier sampling scheme in FP, we have designed and built an illuminator using ring LEDs and mounted on 3D-printed plastic assembly. Comparing to the original LED array illuminator, the reported FP illuminator is able to reduce the number of image acquisitions by at least 50%. It also breaks the translational symmetry of sampling and solves the raster grid artefact problem. The sampling strategy discussed in this paper can also be implemented using a recently reported illumination engineering scheme with a low-cost liquid crystal display [28]. In this case, we do not need to worry about the size and the shape of the LED elements. One can simply place a transparent liquid crystal display at the back focal position of a condenser lens and set any open aperture on the display [28]. The results shown in this paper may provide new insights for developing fast FP platforms and find important applications in digital pathology.