Rapid Image Reconstruction of Structured Illumination Microscopy Directly in the Spatial Domain

Super-resolution structured illumination microscopy (SIM) routinely performs image reconstruction in the frequency domain using an approach termed frequency-domain reconstruction (FDR). Due to multiple Fourier transforms between the spatial and frequency domains, SIM suffers from low reconstruction speed, constraining its applications in real-time, dynamic imaging. To overcome this limitation, we developed a new method for SIM image reconstruction, termed spatial domain reconstruction (SDR). SDR is intrinsically simpler than FDR, does not require Fourier transforms and the theory predicts it to be a rapid image reconstruction method. Results show that SDR reconstructs a super-resolution image 7-fold faster than FDR, producing images that are equal to either FDR or the widely-used FairSIM. We provide a proof-of-principle using mobile fluorescent beads to demonstrate the utility of SDR in imaging moving objects. Consequently, replacement of the FDR approach with SDR significantly enhances SIM as the desired method for live-cell, instant super-resolution imaging. This means that SDR-SIM is a “What You See Is What You Get” approach to super-resolution imaging.


Introduction
In the conventional optical microscopy, using either wide-field or point-scanning illumination to detect scattering or fluorescence signals, the spatial resolution of the object's image is confined by the well-known Abbe diffraction limit. This has been a barrier for optical imaging for over a century. In the past two decades, a variety of methods have been developed to overcome this limitation by exploiting the nature of light, adopting new imaging contrast mechanisms, or utilizing the nonlinear interaction effects between photons and molecules. The representative approaches for far-field super-resolution (SR) methods so far include stimulated emission depletion microscopy (STED) based on directly compressing the point spread function [1], [2]; single-molecule localization microscopy (such as PALM or STORM) by precisely localizing enormous point-source positions [3], [4], and special patterned illumination microscopy (such as structured illumination microscopy, SIM) via extending the detected spectrum in the spatial frequency domain or, called k-space based on the Fourier transform theory [5], [6].
In such SIM systems, the basic workflow to generate a super-resolution image is built on the Fourier transform and spatial spectrum processing, which inevitably slows down the image reconstruction speed because of the complicated reconstruction process [23], [24]. Furthermore, improper treatments occurring in the frequency domain will cause unexpected artifacts in the recovered SR image [25]- [27]. Hence, one should pay careful attention to the frequency operation in the conventional, frequency-domain reconstruction (FDR) scheme [24].
To address the reconstruction speed and artifact problems, we introduce a new approach to super-resolution image reconstruction for SIM, named spatial domain reconstruction (SDR). The idea is inspired by the concept of a series expansion of a function in mathematics. That is, any function can be expressed by a linear superposition of a group of weighted complete orthogonal basis. In the SDR scheme, the SR image is attained by linear superposition of the patterned illuminated raw images with appropriately weighted coefficients that are derivable analytically. This concept appeared in the early works of Lucosz and So et al. [28], [29].
However, both of these studies were based on a theoretical model of structured illumination and were not rigorously tested experimentally.
In this paper, we present the processing routine of the SDR scheme to retrieve the SR image for SIM. The validity of the SDR approach is verified by both numerical simulation and its functionality demonstrated in different experiments. As there is no Fourier transformation and spectrum processing, the considerable burdens of the frequency operations like padding, filtering, and recombination in FDR are circumvented by the SDR method. Consequently, the simple data processing algorithm of SDR makes the image reconstruction speed significantly faster than that of FDR. As the proposed approach is fully compatible with conventional SIM systems, it can be applied to any linear SIM system, and conceivably can be further extended to non-linear SIM schemes to achieve theoretical, unlimited resolution as well.

Theory for SDR-SIM
In the linear SIM system, the image formed on the detector plane D(r) can be described as: where O(r), I(r), and H(r) represent the fluorescent emitter distribution of the object, the illumination intensity, and the point spread function (PSF) of the system, respectively. For simplicity without loss of generality, taking the one-dimensional case, in which the illumination field must be shifted several steps (indicated by the index j) towards one direction, (1) can be expressed in x-coordinate as followed: (2) where δ j is the shifting displacement in step j. The reconstructed SR image R SDR(x) is expected to be attained by linear superposition of the designed phase-shifted patterned illumination images D j (x) multiplied with each corresponding coefficient function c j (x), that is, This formula can be regarded as a new image formation equation if the following equation is satisfied: where P(x) is regarded as a new PSF of the system. Since any periodic function can be expanded into a series of orthogonal complete basis, it is solvable to the expressions of c j (x) when an appropriate target function T(x−x′) is chosen. Here, we express I(x'-δ j ) by the first order of orthogonal trigonometric basis (which means a sinusoidal fringe illumination), and assume the target function to be a normal cosine function, i.e., I x′ − δ j = I 0 1 + mcos 2πk 0 x′ − δ j + φ 0 , Here I 0 , m, k 0 , and φ 0 denote the mean intensity, modulation depth, spatial frequency, and initial phase of the illumination cosine fringe, respectively. In this circumstance, it can be proved that the least number of phase-shifting for the illumination fringe is three to solve out the coefficients c j (x), (i.e., j = 1, 2, 3), and further taking the three phase shifts to be 2πk 0 δ j = {0, −2π/3, 2π /3}, we can obtain: Dan et al. Page 4 Then, the reconstructed SR image and the new PSF can be attained: As indicated by (7b), the concept of the SDR approach is similar to PSF engineering, which improves the resolution by modulating and compressing the original PSF. As we know, the diffraction limit of a conventional light microscope is typically 200 nm. Assuming the PSF of a wide-field microscope is described by a Gaussian function, that is, H x = e −x 2 / 2σ 2 , where the full width at half maximum (FWHM) value of H(x) is equal to 200 nm. After modulating by a cosine function according to (7b), H(x) would be transformed into a new PSF, P(x), which is compressed narrower, as shown in Fig. 1(a). Here, the cosine function represents the distribution of structured illumination patterns. Its spatial frequency k 0 is also diffraction-limited that it is not able to beyond the system cut-off frequency in value of 1/200 nm-1. The FWHM value of P(x) curve will decreases with the decreasing of the period of the illumination pattern T (T = 1/k 0 ), as shown in Fig. 1(b). When T = 200 nm, i.e., the spatial frequency of the illumination pattern reaches the allowed maximum value, the FWHM value of P(x) approaches to its minimum value, approximately 90 nm here. When T = +∞, i.e., the case of uniform illumination, the FWHM value of P(x) curve goes to the diffraction limit value of 200 nm.
Theoretically, we can design any new PSF by selecting a suitable target function T(x) to modulate the original PSF, H(x). However, a more complex PSF generally requires a more complex structured illumination field I(x) and a greater number of raw images D j (x), because the illumination field needs to be expanded into more groups of orthogonal basis. The case of cosine illumination pattern and three equidistant steps phase-shifting is the simplest analytical solution.

Superiority of SDR-SIM
The SR image reconstruction workflows for FDR-and SDR-SIM are illustrated in Fig. 2. In the FDR scheme, nine raw images from three orientation illumination patterns have to be Fourier transformed to obtain their spectra. All the spectra are then separated, OTF compensated, and recombined to form an enlarged isotropic spectrum, which corresponds to an SR image in the spatial domain after an inverse FFT (iFFT). The multiple operations of FFT and iFFT severely consume the processing time. In contrast, the SDR scheme is performed in the spatial domain with three simple operations: image multiplication, summation, and deconvolution. In addition, the multiplied coefficient matrixes are only dependent on the parameters of the structured illumination light field, and these can be precalculated to save the processing time. A comparison of the two pathways for image reconstruction suggests that due to its simplicity, SDR should be significantly faster than FDR in obtaining a super-resolution image.  Fig. 3.
The results show that the mean value of execution time for SDR is 6.7∼7.2 fold faster than that of FDR for all four image sizes. The mean value of the execution time for SDR without deconvolution is about 42∼64 times faster than that of FDR. Further, the superiority of SDR is apparent when the image size becomes larger (Fig. 3b). The increase in the processing speed of SDR is due to the concise processing workflow.

Experimental Verification
To demonstrate the effectiveness of SDR-SIM in producing super-resolution images at highspeed, the experiment was implemented in a home-built SIM system, illustrated in Fig. 4. First, we used 40 nm-diameter polystyrene fluorescence beads (Ex.565 nm/Em.580 nm, Thermo Fisher Scientific Inc., USA) as the sample to calibrate the spatial resolution of the system. Fig. 5 shows the measurement results for the wide-field, FDR-and SDR-SIM. To qualify the spatial resolution, we select 20 individual beads to plot their intensity profiles and fitted with Gaussian function to get the FWHMs, as shown in Fig. 5(d). The averaged FWHM values for wide-field, FDR-and SDR-SIM are 217±11 nm, 109±5 nm, and 107±4 nm, respectively. It can be seen that the resolution reconstructed by the FDR-and SDR-SIM are identical, and nearly double of the wide-field. We can also clearly see in the zoom-in region of the field in Fig. 5(a) that two beads are too closed to be resolved. But in the field of FDR-and SDR-SIM, they are separated distinctly. From Fig. 5(e), this can be presented by the intensity profiles.
Next, a biological specimen was used to evaluate the performance of the system. The specimen is a slide of bovine pulmonary artery endothelial cells (BPAE, Thermo Fisher Scientific Inc., USA) with the mitochondria labeled by MitoTracker Red CMXRos (Ex.579 nm/Em.599 nm). The imaging results of wide-field, FDR-and SDR-SIM are presented in Fig. 6(a)-6(c), respectively. To make a clear comparison, we select a region of interest (ROI) in the field to zoom in, as shown in Fig. 6(d)-6(f). Two lines of adjacent mitochondria are discerned by the FDR-and SDR-SIM, whereas they are blurred in the wide-field image. This can also be distinguished in the plots of the intensity profiles along the marked lines, which shows that SDR is slightly better at discerning fine details than FDR ( Fig. 6(g)).
To demonstrate the merit of SDR-SIM in imaging speed, we observe the Brownian motion of 100 nm fluorescence beads suspended in a mixture of water and glycerin at a ratio of 1:50:450. The mixture is pipetted onto a slide and covered by a 170μm-thickness coverslip and sealed with nail polish. The Brownian motion of the beads is monitored and recorded by the SIM system in real-time. Each frame of the super-resolution image is reconstructed by computing nine raw images with the SDR method as described above. The raw image requisition speed is only limited by the frame rate and the sensitivity of the sCMOS camera because the refreshing rate of the fringes generation by the ferroelectric liquid crystal SLM is higher (4 kHz) than the camera.
The raw images were captured by the camera at speeds up to 100 fps at an image size of 2048 × 2048 pixels. The processing time of the SDR algorithm is 0.1s to output one frame of SR image from 9 raw images. Under these imaging conditions, the final maximum imaging speed (including data acquisition, processing, and display) of SDR-SIM is estimated to be 1/ (0.01s × 9 + 0.1s) ≈ 5.2 fps. In practice, the longer exposure time of the camera owing to the weak signal will slow down the frame rate of the camera. In the demonstration of Brownian motion of 100 nm fluorescence beads, the exposure time of 3 ms and readout time of 10 ms of the camera is required to guarantee an adequate SNR of the image. In this case, the output SR video (see Visualization 1) for the dynamic beads is attained running at the frame rate of 4.6 fps, that is, 1/(0.013s × 9 + 0.1s). In sharp contrast, FDR-SIM is unable to reconstruct and display super-resolution frames instantly.
To show the merit of SDR-SIM against the wide-field microscope in spatial resolution in a dynamic scenario, we selected a region of interest in the full field of the video to track the motion behavior of two adjacent beads. Fig. 7 presents five time-sequential frames in an interval of 217 ms from the videos of the wide-field and SDR-SIM, respectively. As expected, the SDR-SIM produces clearer and sharper images of the beads compared to widefield. Importantly, the separation process of the two adjacent beads (in the zoom-in boxes) from the overlapping to the separated state observed at t = 4.557s can only be shown by SDR-SIM.
We note that the SR video frame rate of SDR-SIM can be boosted by selecting the camera's ROI working mode (only parts of pixels are exposed and readout) to reduce the image size and the field-of-view, depending on the observing requirement. Thus, at the 512 × 512 pixels working mode, the readout time of the camera is reduced to 2.5 ms. The SDR processing time can also be decreased to 16 ms. Then, the SR video-rate would be 1/(0.0055s × 9 + 0.016s) ≈ 15 fps.

Comparison of SDR-SIM With FairSIM
FairSIM, a broadly used program for reconstructing super-resolution images of SIM, is based on the principle of FDR [31]. To determine how SDR-SIM compares to FairSIM, we performed a comparison using a USAF resolution chart (Fig. 8). To be fair, we included FDR-SIM as well.
As the deconvolution process plays an important role in the resolution enhancement for structured illumination reconstructions, we employed the Wiener filtering-based deconvolution method in each approach. A comparison of the zoomed-in regions of each image in Fig. 8 demonstrates that SDR-SIM produces identical images to that produced by either FairSIM or FDR-SIM. This is further evident in the profile plot that the SDR-SIM can recover the same details as the FDR-SIM and FairSIM.
When the 2048 × 2048 image reconstruction times for each reconstruction method are compared, the superiority of SDR is realized. The reconstruction time for SDR is 0.7 s. In contrast, the reconstruction time FDR-SIM is 7-fold slower, taking 5.1s, and FairSIM is 23fold slower taking 16s to produce the same image. Thus, SDR-SIM produces SR images significantly faster than currently available methods, and the image quality produced by SDR is identical to that of the widely-used FairSIM.

Conclusion
The primary conclusion of this study is that a novel structured illumination image reconstruction algorithm we call spatial domain reconstruction (SDR) produces superresolution images of high quality and 7 to 23-fold more rapidly than the currently available methods. SDR achieves its goals using a far simpler approach to image reconstruction. The significantly enhanced image production speed and image quality produced by SDR is the result of two components of the scheme. First, image reconstruction is performed in the spatial domain thereby eliminating the need for Fourier transforms which slow the reconstruction process and can produce imaging artifacts. Second, the SDR SR image is attained by linear superposition of the patterned illuminated raw images with appropriately weighted coefficients that are derivable analytically. The method is elegant, simple, produces images rapidly, and is therefore ideally suited to live-cell imaging.  The SDR workflow is intrinsically simple compared to the complex, multi-step FDR approach. Top, the five steps of the FDR workflow to attain a super-resolution image. Bottom, the straightforward, 3-step SDR workflow. Schematic of the home-built SIM system. HWP: half-wave plate, ZV-HWP: zero-order vortex half-wave plate, SLM: spatial light modulator.