High-speed super-resolution imaging with compressive imaging-based structured illumination microscopy

: Structured illumination microscopy (SIM) has been widely applied to investigating fine structures of biological samples by breaking the optical diffraction limitation. So far, video-rate imaging has been obtained in SIM, but the imaging speed was still limited due to the reconstruction of a super-solution image through multi-sampling, which hindered the applications in high-speed biomedical imaging. To overcome this limitation, here we develop compressive imaging-based structured illumination microscopy (CISIM) by synergizing SIM and compressive sensing (CS). Compared with conventional SIM, CISIM can greatly improve the super-resolution imaging speed by extracting multiple super-resolution images from one compressed image. Based on CISIM, we successfully reconstruct the super-resolution images in biological dynamics, and analyze the effect factors of image reconstruction quality, which verify the feasibility of CISIM. CISIM paves a way for high-speed super-resolution imaging, which may bring technological breakthroughs and significant applications in biomedical imaging.


Introduction
Optical microscopy, as an indispensable tool, has promoted the developments of biology by turning human insight into the microscopic scale to observe subcellular structures and biochemical functions. However, suffering from the optical diffraction limitation, the spatial resolution of optical microscopy can only reach about 200 nm, which makes it difficult to acquire the fine structures within organelles. Driven by the urgent demand for visualizing the structural details of sub-organelles, various super-resolution microscopy techniques have been developed to break the optical diffraction limitation. For example, stimulated emission depletion microscopy (STED) [1,2] incorporates a stimulated emission depletion of fluorophores into the scanned confocal microscopy, and minimizes the size of a scanned focal point by depleting the fluorescence in some specific regions of the sample while leaving a center focal spot active to emit the fluorescence, which has achieved the spatial resolution of ∼20 nm in biological samples [3] and ∼6 nm in nitrogen-vacancy centers [4]. However, the scanning-based data acquisition slows down the imaging speed of STED. Besides, the high-power laser is used to induce stimulated emission depletion, which could also lead to phototoxicity and photo-bleaching. Single-molecule down-sampling and structured patterns and collect the fluorescence signal with a photomultiplier tube [29]. Due to the employment of the point detector and the switching modulator, the imaging speed of this CS-based framework is even slower than that of conventional SIM in a large field of view. In contrast, CISIM can obtain multiple frames of images in each exposure, and each frame will be simultaneously modulated by a structured pattern in SIM and random codes in CI. By CS-based image reconstruction algorithm, sequential super-resolution images can be recovered from each recorded image. That is, CISIM can extract multiple super-resolution images from a recorded image, which is different from conventional SIM that recovers a super-resolution image from multiple recorded images. In this way, the imaging speed of SIM can be significantly improved, surpassing the physical imaging speed of a camera. According to our simulations, CISIM can improve the imaging speed by two orders of magnitude compared with conventional SIM.

Results
To verify the feasibility of CISIM and seek the optimal parameters of system design, we simulated the optical sampling and image reconstruction of CISIM. These simulations were performed in Matlab (2019b) environment using a computer with a CPU of Intel Core i9-10920X @3.5 GHz and 256GB RAM. In the simulations, the fluorescence wavelength was set as 580 nm, the magnification of the objective lens was 60× and its numerical aperture was 0.8, and the pixel size of the camera was set as 6.5 µm. We also evaluated the image reconstruction quality of CISIM with block matching and 3D filtering denoiser (CISIM-BM3D) under the noiseless and noisy conditions, respectively. Moreover, we analyzed the effect of image compression ratio (ICR) on the image reconstruction quality of CISIM, here ICR is defined as the number of super-resolution images that can be obtained in one exposure.
The image reconstruction results by CISIM and conventional SIM are first compared, here a test chart is used as the ground truth, which contains the "East China Normal University" logo picture and surrounds striped patterns with different orientations and spatial frequencies. The ground truth image, wide-field measured image, OpenSIM [30] reconstructed image and CISIM reconstructed image are shown in Fig. 1, together with the corresponding distribution in the frequency domain by Fourier transform and the details of some striped patterns labeled with red square. Here, the wide-field measured image is calculated from the convolution of the ground truth image with the PSF of the optical system. The OpenSIM reconstructed image is obtained from nine structured illumination images, while the CISIM reconstructed image is obtained from one compressed image that contains nine structured illumination and spatial encoding images. As shown in Fig. 1, both OpenSIM and CISIM can recover the detail of high-frequency information that is missed in the wild-field measured image. Interestingly, the image reconstruction quality by CISIM seems even higher than that by OpenSIM, which is easy to be seen in the spatial frequency spectrum, where the frequency components beyond the SIM-extended bound are also recovered. This phenomenon is mainly due to the sparse prior utilized in CISIM but not in OpenSIM. The pattern of the test chart is simple and monotonous, and thus it can be compressed sparsely in some transform domains, such as the 3D transform domain of BM3D. Although the high-frequency components beyond the SIM-extended bound are not physically collected, the sparsity of the test chart in some transform domains makes it possible to recover more high-frequency information from the compressed sampling result via sparse prior constraints. Actually, by adding the sparse prior constraint to the merging operation of nine images in conventional SIM, the super-resolution image with the high-frequency components beyond SIM-extended bound can also be achieved, such as TV-SIM [31]. However, it should be emphasized that CISIM can acquire multiple super-resolution images from one recorded image, while conventional SIM only can obtain one super-resolution image from nine recorded images. A very important application of CISIM is biomedical imaging. To show the super-resolution results of CISIM in biological samples, two dynamics, which are the meiosis I in a living insect spermatocyte [32] and the movement of mitochondria in a COS-7 cell [25], were measured based on the data acquisition of CISIM, and then were recovered by the image reconstruction of CISIM. Here, the ICR was set as 1, that is, one super-resolution image was obtained from one recorded image. The ground truth image, wide-field measured image and CISIM reconstructed image are shown in Fig. 2 for the two biological dynamics, together with the intensity profile of the selected region labeled with red line. As can be seen in Fig. 2(A4) and Fig. 2(B4), the dense spindle fibers, which are difficult to be distinguished in wide-field imaging, can be clearly recognized in CISIM, here the intensity distribution of the CISIM reconstructed image is almost the same as that of the ground truth image. To quantitatively analyze the image reconstruction performance of CISIM, the peak signal-noise ratio (PSNR) and structural similarity (SSIM) were also calculated. The PSNR values were 35.77 and 42.81 dB for the spermatocyte and mitochondria, and the SSIM values were 0.895 and 0.994, respectively. Compared with the wild-field measured image with the PNSR values of 29.46 and 27.01 dB and the SSIM values of 0.778 and 0.918 for the spermatocyte and mitochondria, both the PSNR and SSIM values of the CISIM reconstructed image were greatly improved. Obviously, CISIM can get very high image reconstruction quality of biological samples, which will have a very important application prospect in biological imaging.
Because of the multi-frame super-resolution reconstruction ability of CISIM from one recorded image, we further studied the relationship of ICR and the image reconstruction quality of CISIM. In this simulation, a clip with 16 images for the meiosis I in a living insect spermatocyte was used as the target dynamic scene. To simplify, one representative image by CISIM reconstruction under different ICRs is shown in Fig. 3(A), together with the intensity profile of the selected region labeled with red line. Here, only the central area is enlarged to display for more intuitive observation, and the intensity profiles of the corresponding selected regions in both the ground truth image and wild-field measured image are also provided for comparison. It is obvious that the image reconstruction quality decreases with the increase of ICR. Similarly, the PSNR and SSIM values for the CISIM reconstructed images were also calculated and given in Fig. 3(B) and Fig. 3(C), together with those of the wild-field measured images. Both the PSNR and SSIM values monotonously decrease when ICR increases. The decrease of image reconstruction quality can be understood based on statistical signal processing and convex optimization theory. On the one hand, considering that the bit depth of the compressed image is limited, the weight of each super-resolution image in the compressed image decreases as ICR increases. In other words, higher ICR results in more information loss in the compressive sampling of CISIM. On the other hand, more super-resolution images need to be recovered from one compressed image for higher ICR, and thus the condition number for the inverse problem of CISIM is larger, which makes it more ill-conditioned and leads to more calculation error in the image reconstruction. That is to say, higher ICR will decrease the accuracy of information decoding and image reconstruction. Here, only when both the PSNR and SSIM values of the CISIM reconstructed image exceed those of the wild-field measured image, the super-resolution imaging is considered to be obtained. As shown in Fig. 3(B) and Fig. 3(C), when ICR is increased to 13, the PSNR and SSIM values of the CISIM reconstructed image are, respectively, 27.17 dB and 0.747, both the two values are larger than those of the wild-field measured image with 26.27 dB and 0.741, which still can demonstrate the ability of super-resolution image reconstruction. Compared with conventional SIM that needs nine images to recover a super-resolution image, CISIM can greatly improve the imaging speed by two orders of magnitude. Considering the imaging speed of SIM with tens of frames per second, CISIM can obtain an imaging speed of thousands of frames per second. Of course, the critical threshold of ICR depends on the complexity of the structure and dynamics of biological samples. Usually, simpler structure and dynamics of biological samples can obtain higher critical threshold of ICR and so get higher super-resolution imaging speed.
Considering the physical noise in the actual CISIM system, we further exploited the effect of additive white Gaussian noise (AWGN) on the image reconstruction quality of CISIM. Similarly, the dynamics of the meiosis I in a living insect spermatocyte are compressed to generate one image based on the data acquisition of CISIM, here the ICR is also set to 1. The CISIM reconstructed images under different noise levels are shown in Figs. 4(A-F), together with the intensity profile of the selected region labeled with red line, and the PSNR and SSIM values for the CISIM reconstructed images and wild-field measured images are also calculated and  given in Fig. 4(G) and Fig. 4(H). The noise level is indicated by a noise standard derivation σ. To intuitively compare, the intensity profiles of the corresponding selected regions in the ground truth image and wild-field measured image are also given in Figs. 4(A-F). As can be seen, with the increase of σ, the CISIM reconstructed image shows blur and artifacts due to the noise effect, and both the PSNR and SSIM values decrease. However, compared with the wild-field measured image, the CISIM reconstructed image can get higher PSNR and SSIM values and clearer intensity profile of the selected region. That is, a small amount of noise does not affect the ability of CISIM to improve the image reconstruction quality. However, the larger the noise, the worse the super-resolution image reconstruction effect. Moreover, the denoiser module has been utilized in the image reconstruction of CISIM, but the noise still cannot be eliminated, which greatly depresses the image reconstruction accuracy of CISIM. In the experiment, the optical system of CISIM should be carefully calibrated to maximize the signal-noise ratio of the compressed image, and thus the super-resolution images can be well reconstructed.

Discussion
As shown above, CISIM provides a novel strategy to greatly improve the super-resolution imaging speed by combining SIM and CS, and the computational algorithm plays an important role, which determines the final image reconstruction quality. Here, we utilized CISIM-BM3D to recover the original dynamics. As a widely used denoiser, BM3D has high performance and good robustness. To obtain a higher image reconstruction quality and more resistance to noise, the denoisers with sparser constrain are easy to be embedded into the PnP framework in the CISIM reconstruction algorithm. For example, the weighted nuclear norm minimization (WNNM) [33,34], and even some deep-learning-based image denoisers, such as FFDNet [35] and CBDNet [36]. Besides, the temporal continuity can also be utilized to further improve the image reconstruction quality, such as VBM3D [37] and FastDVDnet [38]. Moreover, the framework of CISIM reconstruction is based on the spatial domain operation, which has the feature of high computation speed, but it is vulnerable to complex physical noise, and therefore such a framework puts forward strict requirements for the calibration of the optical sampling system. Thus, the frequency domain-based framework can be further exploited to increase the image reconstruction robustness for CISIM. In addition, deep-learning-based end-to-end algorithms can also be developed to improve the image reconstruction quality and computational efficiency of CISIM. In the future research, these computational imaging algorithms will be further explored. It should be pointed out that, compared with traditional SIM, CISIM can acquire higher super-resolution imaging speed, but it is at the cost of lower super-resolution ability due to the lower photon efficiency.
In actual experiment, the CISIM strategy is also feasible, and the experimental scheme is designed in Fig. 5. A continuous-wave laser is expanded with a beam expander (BE), and then hits a high-speed digital micromirror device (DMD1), which is used to generate strip patterns of different directions and phase-shift steps. A spatial filter (SF) is used to select the ±1st order diffraction beams for the generation of cosine fringes in three orientations via interference. A nearby vortex half-wave plate (VHP) is utilized to passively adjust the polarization of beams in different directions for the high modulation contrast. After being relayed by a 4f system that consists of a lens pair (L2 and L3), the beams are focused with an objective lens to interfere with the sample to form the structured illumination. The fluorescence signal generated by the structured light illumination is collected with the objective lens, reflected by a dichroic mirror (DM), relayed by a tube lens (TL), and then projected onto another digital micromirror device (DMD2) for spatially encoding by a 4f system consisting of a lens pair (L4 and L5). Finally, the encoded fluorescence signal is collected by the same 4f system, reflected by a beam splitter (BS) and detected by a camera. Here, DMD1 and DMD2 are synchronously triggered, and the camera is also synchronized with DMD1 and DMD2, but the working frequency of the camera is much lower than that of DMD1 and DMD2 to obtain the compressed image. This designed scheme in Fig. 5 can provide a reference for future CISIM experiment. However, some points need to be considered in the experiment of CISIM. Firstly, the micromirror array of DMD has the effect of a reflection blazed grating, and thus the angle between the incident light field and DMD should be calibrated carefully to maximize the diffraction efficiency. Secondly, since the encoding plane is not perpendicular to the propagation direction of the light field, the final camera should also be placed with a tilt angle by the Scheimpflug principle [39] to collect a clear image, which can avoid the blur caused by the limited depth of the light field. Thirdly, the pixel size of spatial codes should be enough small to avoid the resolution loss due to the compressive sampling. In a word, the optical components of CISIM should be precisely adjusted to guarantee the compressed image acquirement with high quality, and thus the dynamics can be successfully recovered with the super-resolution images. Another limitation of CISIM is that the collected fluorescence photon efficiency will be reduced due to the additional optical elements, such as beam splitter and DMD, which will greatly affect the super-resolution image reconstruction quality. Additionally, a higher excitation power is needed compared with conventional SIM, which may induce phototoxicity and photo-bleaching. A feasible solution is to utilize complementary encoding strategy to reduce the loss of fluorescence photons [40].
These key technologies that need to be noticed in CISIM experiment are very helpful to guide the subsequent experimental studies.
In summary, we have developed a novel CISIM to achieve super-resolution imaging for highspeed dynamics. By synergizing SIM and CI, CISIM could acquire multiple super-resolution images from a single exposure. We have verified the feasibility of CISIM by reconstructing the super-resolution images of the meiosis I in a living insect spermatocyte and the mitochondrial movement in a COS-7 cell and analyzing the effects of image compression ratio and physical noise on the image reconstruction quality. Compared with conventional SIM, CISIM could improve the super-resolution imaging speed by two orders of magnitude, which has important applications in the super-resolution imaging of various high-speed biological dynamics, such as neuron firing [41], organelle dynamics during mitosis [42] and laser disruption of subcellular organelles [43]. Additionally, the CI strategy can also be extended to three-dimensional (3D) SIM [44], the super-resolution imaging speed of 3D-CISIM can be greatly improved by capturing the super-resolution cross-section in each exposure. Given the high-speed and super-resolution imaging ability of CISIM, it is expected to provide a new technical method for the study of nanoscale microsecond dynamics in physics, chemistry, and biology.

Methods
In conventional SIM, nine recorded images with different illumination patterns are usually required to extract a super-resolution image. Therein, three images with phase-shifted illumination patterns along a single orientation are used to uncouple spatial frequency components, and three different directions are used to provide isotropic resolution enhancement. However, because of multiple sampling, there is some information redundancy in the acquired images of SIM, which is reflected in the repetition of zero-order low-frequency information and the overlap of one-order high-frequency information among the three directions in the spatial frequency spectrum. When a microscopic dynamic process is continuously imaged using SIM, the information redundancy will further increase due to the temporal correlation and continuity of the dynamic scene. Given the abundant information redundancy, CS can be employed in SIM to improve its imaging speed. In this work, CISIM is using CS theory to break the imaging speed barrier of SIM. The flow chart of CISIM is shown in Fig. 6, which is composed of two parts: data acquisition and image reconstruction. The data acquisition mainly involves structured illumination, spatial encoding, and compressed sampling. The image reconstruction mainly employs an iterative search algorithm based on CS theory. In data acquisition, as shown in top row of Fig. 6, the illumination light with nine structured patterns is periodically projected onto the fluorescently-labeled sample with a high-speed dynamics d(x, y, t). Nine structured patterns are assigned in three directions and three phase-shift steps p(x, y, n). The number of images included in the dynamics is denoted by T. In this way, a series of structured fluorescent images with the number of 9 T is generated, which contain the super-resolution information in different directions. After passing through an objective lens, the acquired stripped images s(x, y, t, n) can be considered as a convolution of the structured fluorescent images with the point spread function (PSF) h(x, y) of the objective lens according to the optical diffraction, which can be written as In conventional SIM, the stripped images s(x, y, t, n) are directly recorded by a camera to reconstruct the super-resolution images. In contrast, in CISIM, these striped images are further modulated by pseudorandom temporal-spatial encoding c(x, y, t, n). It should be pointed out that the temporal-spatial encoding is synchronized with structured pattern illumination. Finally, all the fluorescent photons will be accumulated into a compressed image m(x, y) by a camera with a single exposure, which can be expressed as For simplification, the sampling process of CISIM can be rewritten with matrix form and is given by In Eq. (3), m is the matrix of a compressed image m(x, y), d is the dynamic spatial distribution of fluorophores d(x, y, t), P represents the excitation operator with structured illumination patterns p(x, y, n), H represents the convolution operator with PSF h(x, y) in the optical microscopy system, C represents the encoding operator with the temporal-spatial modulation c(x, y, t, n), I represents the integration operator with the camera. The overall operator of CISIM can be simplified as A.
The details of the data acquisition in CISIM can be seen in Note 1 of Supplementary Materials. In image reconstruction, it is worth noting that a simple concatenation of the reconstruction algorithms for CACTI and SIM does not work for CISIM, although it seems that the optical system of CISIM is just a combination of SIM and CACTI. The basic reason is that some high-frequency information will be treated as noise and abandoned in CACTI reconstruction, which actually contains the necessary super-resolution details. In the image reconstruction, it is difficult to recover a super-resolution dynamic scene from a CISIM measurement due to the difference in data processing between the reconstruction algorithms for SIM and CACTI. To solve the inverse problem for decoding the objective information of the dynamic scene from the compressed image, an integrated image reconstruction algorithm is specially designed for CISIM. This image reconstruction algorithm is an iterative searching algorithm based on a Plug-and-Play (PnP) framework [45,46], and its flow chart is shown in bottom row of Fig. 6. A random objective dynamic scene is set as the initialized iterative variable in the searching algorithm. Firstly, the current expectation of the compressed image for the iterative variable is estimated based on the optical transfer process of CISIM, which can be measured in the system calibration. Secondly, the loss between the current expectation and the measured value of CISIM is calculated. Thirdly, the iterative variable is automatically optimized by the generalized projection method based on above loss value and loss function. Fourthly, the result from the gradient descent searching is filtered with a specified image denoiser that satisfies some kind of sparse constraints. Finally, the discriminator will judge the convergence status of the algorithm. If the residual between the last iterative variable and the new one is below the threshold value, the iterative algorithm will stop searching and output the reconstruction result. Otherwise, the algorithm will continue the next iterative searching. Benefiting from the modularization of the PnP framework, image/video denoisers with different priors can be flexibly switched in this image reconstruction algorithm for CISIM, such as the classical image denoiser block-matching 3D filtering (BM3D) [47,48].
Based on CS theory, the inverse problem of the image reconstruction in CISIM can be described mathematically as d = argmin d ||Φd|| 1 , subject to Ad = m, where Φ is the transformation operator of a sparse domain, and ||·|| 1 denotes the L1 norm. Considering the maximum likelihood estimation (MLE), the above-mentioned optimization problem can be solved iteratively using alternative steps with alternating direction method of multipliers (ADMM) [49] and is given as where v and u are auxiliary intermediate variables of ADMM, ρ is a scale factor of ADMM, Denoiser Φ is the denoiser with sparse constraint. The details of the image reconstruction algorithm in CISIM is given in Note 2 of Supplementary Materials.