Fast and robust misalignment correction of Fourier ptychographic microscopy

Fourier ptychographi cmicroscopy(FPM) is a newly developed computational imaging technique that can provide gigapixel images with both high resolution (HR) and wide field of view (FOV). However, the positional misalignment of the LED array induces a degradation of the reconstruction, especially in the regions away from the optical axis. In this paper, we propose a robust and fast method to correct the LED misalignment of FPM, termed as misalignment correction for FPM (mcFPM). Although different regions in the FOV have different sensitivity to the LED misalignment, the experimental results show that mcFPM is robust to eliminate the degradation in each region. Compared with the state-of-the-art methods, mcFPM is much faster.


Introduction
As we all know, almost all of the conventional microscope has a trade-off between its resolution and FOV. To solve this problem, a new computational imaging technique called FPM has been proposed [19,17]. In a typical FPM system, a programmable LED array is used instead of the conventional microscope's light source for providing angularly variant illumination. After capturing a sequence of low resolution (LR) images under different illumination angles, an iterative phase retrieval process [5,8,2] is used to stitch together those LR images in the Fourier space, and then an HR and high space-bandwidth product (SBP) complex field of the sample can be recovered. Compared with the conventional microscopy, the FPM can achieve HR, wide FOV and quantitative phase imaging [9]. Therefore, it has great potential in a variety of applications, such as biomedical medicine [15,6,3], characterizing unknown optical aberrations of lenses [1,10].
The FPM shares its roots of ptychography [11,4,12]. In the conventional ptychography, the mechanical scanning in the imaging process makes the position correction of the probe function essential. Similar to the conventional ptychography, the position misalignment of the LED array in the FPM is of great importance. Because the position of each LED determines the wavevector of the illumination, misalignment of it induces significant errors to the pupil function in the reconstruction process. In the conventional ptychography, a simulated annealing (SA) algorithm was adopted to correct the position errors of the probe function [7]. Similarly, to correct the LED misalignment in the FPM, the SA algorithm has been introduced into each sub-iteration of the reconstruction algorithm. An optimal shift of the pupil function in the Fourier domain can be obtained during this process. However, this method may lead to an algorithmic disorder of the LED array, resulting in the degradation of the reconstruction [16]. To avoid this problem, a position correction approach, named as pcFPM [13], has been proposed. It is based on the SA algorithm and a non-linear regression technique. In the pcFPM, a global position misalignment model of the LED array was introduced to ensure the corrected LEDs are positionally ordered. Although the pcFPM can effectively eliminate the LED misalignment, the additional process of non-linear regression increases the algorithm complexity and computer load. Actually, during the FPM reconstruction process, it usually requires to divide the captured images into many small segments. However, with these existing methods [16,13], we find that the global shift of the LED array corrected according to the central segment of the FOV may not work for all of the segments. Influenced by the system errors and lens aberration, it seems that different segment requires different global shift of the LED array. Although these methods can correct the global shifts of different segments respectively, they are heavily time consuming.
In this paper, we propose a simple and fast method to correct the global shift of the LED array for the whole FOV. Rather than correcting the shift errors of the pupil function in the Fourier domain, we introduce a global position misalignment model, and then directly correct the global shift of the LED array. The experimental results show the proposed mcFPM is robust and faster than the other methods. This paper is arranged as the following: In section 2, we will investigate the problem of LED misalignment in the FPM and propose our mcFPM. In section 3, we will present experimental results to verify the effectiveness of the proposed mcFPM. Finally, we will make a conclusion. in the imaging process, the LEDs are switched on sequentially, and the intensity images of the sample under different illumination angles are captured. Let A object (x, y) represent the complex amplitude of the sample, A output (x, y) represent the output complex amplitude of the sample, and h(x, y) represent the coherent point spread function. When the m th row and n th column LED in the array is on, this process can be modeled as [18]: where (k m,n x , k m,n y ) represents the wave-vector of the parallel light, and ⊗ the 2D convolution operator. In the Fourier domain, Eq. (1) can be written as where G object (k x , k y ) represents the object spectrum, G output (k x , k y ) the output spectrum of the microscope, and H(k x , k y ) the coherent transform function of the microscope. The image captured by the camera thus can be written as where F −1 represents the inverse Fourier transform. With a sequence of LR images captured under different illumination angles, an HR image of the sample can be reconstructed. The recovery process of the FPM follows the strategy of the phase retrieval technique [5,8]. The algorithm switches between the spatial and Fourier domains. In the spatial domain, the LR intensity measurements are used as the object constraints to ensure the solution convergence. In the Fourier domain, the confined coherent transfer function of the objective lens is imposed as the support constraint. After several iterations, both HR complex field of object and the pupil function will be obtained.

Global shift of the LED array in the FPM
In the FPM system setup, it requires that the central LED of the LED array is on the optical axis. In the pcFPM [13], four global positional factors (rotation, horizontal and vertical shifts) of the LED array were introduced. However, compared with the other factors, the horizontal shift is hard to eliminate by the hardware. Details will be shown in section 3. In this work, to simplify the position model, a global model with two factors is proposed, as shown in Fig.2. The two factors that can determine the position of each LED, are marked as ∆x, ∆y. The position of each LED then can be written as where (x m,n , y m,n ) represents the position of the LED at the m th row and n th column, and d the distance between two adjacent LED elements. The incident wave-vector (k m,n x , k m,n y ) for each segment can be written as [19] k m,n where (x o , y o ) is the central coordinate of the sample, λ is the central wavelength of the LED, and s is the distance between the sample and the LED array. From Eq. (5), we can find that the global shift of the LED array is a factor that determines the incident wave-vector. Equation (2) shows that the error of the incident wave-vector can cause a dislocation of the object spectrum. In other words, during the FPM reconstruction process, the global shift can induce a shift error to the pupil function in the Fourier domain. Figure 3 is a simulation example. Figure 3(c) and 3(d) show the reconstructed amplitude and phase profiles of the object with a global shift of the LED array. Compared with the original amplitude and phase profiles in Fig. 3(a) and 3(b), we can obviously observe the image degradation. This is similar to the reconstructed images in the regions away from the optical axis we observed in the experiments. The flowchart of the proposed mcFPM is shown in Fig. 4. Actually, the conventional FPM algorithm is a part of the proposed mcFPM, as the part within the yellow dashed rectangle in Fig. 4. It is sketched as follows [19,14]:

Proposed LED misalignment correction method
Step 1. Initialize the Fourier spectrum of the reconstructed HR object O j (k x , k y ) and the pupil function P j (k x , k y ).
Step 2. Generate an LR image corresponding to the m th row and n th column LED with the incident wave-vector of (k m,n x , k m,n y ) by the below equation where Ψ m,n j (k x , k y ) represents the Fourier spectrum of the LR image obtained by illuminating the sample by the m th row and n th column LED.
Step 3. Impose the intensity constraint with the captured images with where φ m,n j (x, y) and ψ m,n j (x, y) are the complex field of the LR images with and without the intensity constraint respectively, and At this time, the updated Fourier spectrum of the LR image is Step 4. Update the object and the pupil functions with where δ 1 and δ 2 are two regularization constants used to ensure numerical stability, which are set as δ 1 = 1, δ 2 = 1000 in this work, and ∆ 1 and ∆ 2 are defined as Step 5. Repeat steps 2-4 for all of the LEDs. The LED updating range is where R 1 is the number of the LEDs in each side of the LED array.
Step 6. Repeat steps 2-5 until the algorithm converges. In order to illustrate the mcFPM better, we first introduce the conventional SA method [7,16]. In the conventional SA method, it assumes that each LED has an independent shift. After step 2 of the above FPM algorithm, the SA module is added to search the deviation of the illumination wave vector ∆k m,n x , ∆k m,n y , and the corresponding pupil function with the cost function is defined as Because the conventional SA method assumes that the shifting value of each LED is independent, it does not provide any constraint on the LEDs' positions.
After several iterations in the SA, the LED position coordinate may become disordered in the algorithm, especially in the edge regions of the LED array [16]. Besides, for each sub iterations, the SA process is used for every LED position correction, which is heavily time consuming.
In the proposed mcFPM, rather than correcting the wave-vector (k m,n x , k m,n y ) of each LED in the Fourier domain, we directly correct the global shift of the LED array.
The global shift of the LED array in the mcFPM has been defined in Sec. 2.3. According to Eq. (4), the position of each LED with the global shift is (x m,n , y m,n ) = (md + ∆x, nd + ∆y). Considering the distance between two adjacent LEDs is d, the ranges of the global shifts ∆x and ∆y are set to [−d, d]. As Fig. 4 shows, in the mcFPM, the initial values of ∆x and ∆y are 0 mm respectively. According to Eq. (5), the updated incident wave vectors of (k m,n x , k m,n y ) is To where I m,n F P M (x, y, ∆x, ∆y) is the corresponding calculated intensity image using the conventional FPM algorithm (step 1-6) with a global shift. Compared with E 1 , E 2 is the summation of all LEDs within the bright field of the objective. Therefore the time that the mcFPM calls the optimization algorithm is much less than the conventional method.
Different from the FPM reconstruction process of the conventional SA method, only R 2 × R 2 central LR images are used in the mcFPM, since these LR images contain mainly information of the sample. The procedure of the FPM reconstruction process in the mcFPM only iterates 5 times (J = 5). After the fast FPM reconstruction process finishes (j = J), the cost function E 2 can be calculated according to Eq. (8).
To minimize the cost function E 2 , there are several searching methods, such as SA algorithm and genetic algorithm. In order to compare with the existing methods, we use the SA algorithm to search ∆x and ∆y in our mcFPM. This loop continues until the cost function reaches a minimal value. After correcting the global shift of the LED array, all LR images are used to reconstruct the HR images of the sample using the conventional FPM algorithm. In some cases, after correcting the global shift of the LED array using the mcFPM, one iteration of the conventional SA is used to correct the inherent local subtle position error of each LED. Finally, the degradation of the reconstructed HR amplitude and phase caused by the LED misalignment can be eliminated. shows how we install the LED array (the yellow box of (a)) on the microscope. (c) shows a method to adjust the central LED (the red box of (a)) along the optical axis (the z-axis in (b)).
The system setup in this work is shown in Fig. 5(a). We built an FPM system by replacing the light source of an Olympus IX 73 inverted microscope with a programmable LED array (32 × 32 LEDs, 4 mm spacing) controlled by an Arduino. The LEDs have a central wavelength of 629 nm and a bandwidth of 20 nm. All samples were imaged with a 4× 0.1 NA objective and a scientific complementary metal oxide semiconductor (sCMOS) camera (PCO. edge 4.2). The pixel size of the sCMOS camera is 6.5 µm. The distance between the sample and the LED array is 113.5 mm. In the experiment, the central 17 × 17 LEDs were switched on sequentially to capture 289 LR intensity images, in which only the LR images of the central 5 × 5 LEDs are within the bright field range of the objective. This means that R 1 = 17 and R 2 = 5 in this work.
Coarse adjustment for the position of the LED array was first performed before the image acquisition. After capturing all the LR images, the proposed algorithm was applied to realize fine correction in the image reconstruction process. As Fig. 5(b) shows, in order to avoid the position error caused by the rotation of the LED array, we used a set of rods and buckles to fix it on the microscope stage. Then, a level instrument was used to make the sample plane and the LED array plane parallel. Figure 5(c) shows the principle of the coarse adjustment to make the central LED on the optical axis (the z-axis in (b)). First, we turned on the central LED (the red box of Fig. 5(a)) without a sample placed on the stage. Then we adjusted the stage along the x-axis and y-axis to find four critical positions x 1 , x 2 , y 1 , y 2 respectively. The light spot is tangential to the image borders of the camera (the blue box of Fig. 5(a)) at these positions. Finally, we moved the stage to the coordinate of ((x 1 +x 2 )/2, (y 1 +y 2 )/2). After this process, the rotation effect is almost eliminated.

The reconstructions influenced by the global shift of the LED array
To observe the influence of the global shift in the FPM, we reconstructed the segments of the sample in different regions within the FOV. The FPM reconstruction algorithm we used in this study is adapted from the open source code published in Ref. [14]. Figure 6 shows the reconstructed results using the FPM algorithm without any position correction. Figure 6(a) is the captured original image of a young plant root sample. Figure 6(b)-6(g) are the reconstructed HR intensity images in the different regions of Fig. 6(a). From the reconstructed results, we can find that the central regions ((e) and (f)) are much better than that of the edge regions ((b) and (c)). In the edge regions, obvious stripes are clearly observable. Based on the phenomenon shown by Fig. 3, we speculate that different global shifts occur at different regions of the FOV. In our coarse adjustment, we carefully aligned the LED array and the camera so that the central LED was imaged to the center of the camera, therefore the influence of global shift near the center regions of the reconstructed image is small. But for the edge regions, aberration of the objective may also affect the reconstruction. This global shift can not be corrected by the FPM algorithm with a correct pupil function. To verify the feasibility of the proposed method, we compare the reconstructed results of different samples with and without LED position correction. Figure 7 shows the recovered images of two segments of a USAF resolution target. Figure 7(d) is the original captured image, fig. 7(a) and 7(e) show the enlargement of two different parts (128 × 128 pixels) of it. Figure 7(b) and 7(f) show the reconstructed HR intensity images using the FPM algorithm without any position correction. Figure 7(c) and 7(g) show the reconstructed results using the mcFPM. Compared with Fig. 7(b) and 7(f), Fig. 7(c) and 7(g) have much higher visible image quality.

Performance verifications of the proposed mcFPM
To test the robustness of the mcFPM in different segments of the FOV, we recovered an HR image with full FOV of a young plant root sample. The full FOV images (8112×8112 pixels) were reconstructed as Fig. 8 shows. Figure 8(a) shows the reconstructed images using the conventional FPM algorithm without any position corrections. The image quality in the center FOV is good while the stripes become obvious in the edge regions. Therefore, the ability of gigapixel imaging with the FPM is seriously decreased because of the regionally dependent global shift of the LED array. Figure 8(b) shows the reconstructed result with our proposed mcFPM. Compared with the reconstructed results without position correction, we can find that the stripes in the edge regions of the FOV are significantly eliminated, and the image quality is improved greatly as well. This clearly demonstrates the robustness of the proposed mcFPM. Furthermore, we compare the proposed mcFPM with the other existing techniques (conventional SA and pcFPM). The experimental LR images in the edge regions of the sample were chosen as the test data. The pixel sizes of the LR images and the reconstructed HR images are 128 × 128 and 512 × 512 respectively. The codes were run with MATLAB R2015b on a Windows 10 Enterprise Edition operation system (Inter i7-6700 CPU @3.40Ghz, 8 GB DDR4 memory). The SA algorithm terminates when the average change of the cost function is less than 10 −3 , or the number of iterations exceeds 100. Figure 9 shows the reconstructed results using different methods. The left images of each sub figure are the reconstructed amplitude profiles and the right ones are the phase profiles. Figure 9(a) shows stripes on both of the amplitude and phase. As shown from Fig. 9(b), conventional SA method can reconstruct a good HR amplitude, but the stripes in the phase still exist. Besides, it takes 246 s for one segment. The reason is that it calls SA algorithm once for each LED. As Fig. 9(c) shows, the pcFPM removes the stripes in both the amplitude and phase, however, it takes 728 s. Figure 9(d) shows the reconstructed results using the mcFPM. Compared with the previous methods, the mcFPM takes the least time to run, which is only 40 s. On the whole, the conventional SA method is time consuming, and breaks the physical constraint of the LED array. The pcFPM takes the physical constraint into account, but it takes more time because of the additional optimization process. Our method directly correct the global shift of the LED array, which is the most efficient among all the testing algorithms.

Conclusion
In the FPM, the misalignment limits its capability to realize gigapixel imaging. By analyzing the experimental data, we found that different regions of the FOV have different global LED position shifts. This produces observable stripes in the reconstructed HR images. To eliminate the global shift of the LED array, we have proposed the mcFPM algorithm. Rather than correcting the shift errors of the pupil function in the Fourier domain, we introduced a global position misalignment model of the LED array with two factors, and then directly corrected the global shift of the LED array. Experimental results have shown that the mcFPM performs robustly in different regions of the FOV. The experiments have shown the mcFPM is more efficiently than all of the state-of-the-art techniques.