Speed limits of structured illumination microscopy

A theoretical framework for widefield structured illumination microscopy (SIM) reconstruction from fewer than the commonly used nine raw frame acquisitions is introduced and applied in silico and in vitro. The proposed scheme avoids the recording of redundant spatial frequency components, which was necessary in previous SIM algorithms. This allows for gentler superresolution imaging at faster speeds. A doubling of frame rates is possible solely via changes in the computational reconstruction procedure. Furthermore, we explore numerically the effect of the sample movement on the reconstruction quality and the number of raw frames recordable. Our results show that there exists a limit above which deconvolution microscopy becomes superior to SIM.

Structured illumination microscopy (SIM) [1] stands out among the optical superresolution techniques due to its modest illumination intensities combined with high acquisition frame rates.Furthermore, it is compatible with the full gamut of chemical and biological dyes, which makes it one of the most versatile of all nanoscopies.Widefield SIM can image at almost real-time speeds reaching 11 Hz with a resolution approaching 100 nm [2].The more light intense confocal SIM method can perform at over 100 Hz, but at a lower resolution of about 150 nm [3].Although the resolution improvement in widefield and confocal SIM modalities exploits the same physical principle [4], their implementations and reconstruction procedures differ substantially [5][6][7].Here we focus on widefield SIM, as illumination intensities are inherently low, which renders it the most gentle and useful superresolution technique for live subcellular imaging.The same applies for longer-term imaging of living systems: SIM makes better use of the fluorophores' photon budget compared to alternative superresolution methods [2], and imaging over hundreds of time-points is possible without damaging the specimen through phototoxicity or fluorophore bleaching.This is especially true when combined with selective plane illumination microscopy (SPIM) [8].In general, SIM increases resolution by encoding high spatial frequencies of the sample in beat patterns.These are transmittable through widefield microscopes, but need to be un-mixed from the raw data in order to increase resolution.Conventional 2D-SIM requires three images to be acquired with translated illumination patterns to un-mix spatial frequencies along a single orientation.According to the well-known SIM image formation equation, the sample s is modulated by a sinusoidal excitation pattern, featuring spatial frequency k o and phase ϕ n .Imaging through the objective blurs the image via convolution, ⊗ , with the point spread function h to yield the final recorded image i.The sample coordinates are x.Equation (1) does not include any magnification.Using the convolution theorem, it can be shown that the structured illumination causes shifting of spectral sample information into the passband of the objective, which is subsequently overlaid with other components of the sample spectrum.For different phases ϕ n this mixing can be expressed as a linear equation system in Fourier space, linking the actual sample spectrum to the recorded raw data spectra [9]: Here I n are the raw data spectra, S are the shifted frequency components of the sample spectrum, and H is the optical transfer function (OTF).The OTF has a limited support, a circle in Fourier space around the origin, which suppresses all spatial frequencies k larger than the cutoff frequency k c .When combining the sample spectrum with the OTF in a single vector S, Eq. ( 2) can be written as I MS, where I contains the raw data spectra; M is the mixing matrix.Hence, a matrix inversion yields the "superresolved" components for SIM.To provide isotropic resolution enhancement, this process is performed three times with rotated illumination patterns, which yields a total of nine raw images per a superresolved SIM image [1].Reducing the number of raw frames in SIM images has therefore been an active field of research, providing an avenue to increase the acquisition speed and to reduce phototoxic effects and bleaching [10][11][12].In a conventional SIM, one mixing matrix per orientation is used, which produces the low-frequency component S 0 three times [gray circles, Fig. 1(a)].Including all orientations in the same mixing matrix allows a unique reconstruction already from only seven structured illumination (SI) images [see Fig. 1(b)].In what follows, we will use the notation #SIM to denote the number # of raw frames required for particular SIM reconstruction algorithms.
For example, 7SIM works with just 2 SI images per orientation and a single widefield image [gray circle, Fig. 1(b)].Redundancies also exist between SI images of the same orientation.In fact, each SI frequency component contains just a certain amount of "superresolution" information; the rest is information from shifted widefield spectra.A simplification of this observation is illustrated Fig. 1(c).The left-hand side shows the sample spectrum represented with the same color coding as before.One sees considerable spectral overlap between widefield (gray) and superresolution (colored) spectral components.To illustrate how redundancies arise in overlapping regions, we sketch raw image spectra in rectangular, rather than circular format.This can be exploited to obtain a solvable equation system for each orientation without requirements for phase stepping during the acquisition procedure and 4 image acquisitions are enough to obtain superresolution (4SIM = 3SI + WF).A rigorous analysis shows that this argument holds also for circular OTFs.To see this, let us first consider the complete SIM spectrum to be decomposed into the 66 unique spectral segments (see left-hand side of Fig. 2).Sixty-six unknowns have to be determined.Each SI spectrum provides 18 equations consisting of three unknowns, representing unique spectral segments (for example, segments [6,52,65] in Fig. 2).The widefield image provides 18 equations with direct access to 18 unknowns, shown as grayed out areas in the top of Fig. 2, while the three SI images (one per orientation) consist of superimposed frequency information, which must be unmixed.Each of the overlay spectra contain 18 regions, i.e., a total of 3 × 18 56 equations, where each equation has three unknowns from the whole set of 66 unknowns.
Therefore, in 4SIM, a total of 72 equations is available to retrieve the 66 unknown segments, representing an overdetermined equation system, which can be inverted uniquely.Note that all equations are mutually independent, as can be proven by calculating the rank of the 66 × 72 mixing matrix M. The number of raw images can be further reduced by leaving out the widefield component, which we term 3SIM.In this case, however, the system is now underdetermined, as only 56 equations are available for 66 unknowns, and M is no longer uniquely invertible.However, using a maximum likelihood approach, we can approximate the optimal separation of segments to reconstruct SIM images from only three SI images to achieve similar resolution as in 9SIM, but with a three-fold raw data reduction: Here i o are the acquired raw images and e o are excitation patterns for orientations o.This minimization problem can be solved using a joint Richardson-Lucy (jRL) deconvolution algorithm, as described in previous publications [13,14].In short, the algorithm minimizes the difference between the real raw data and artificial raw data, which is generated from an estimate of a superresolved SIM image.We have implemented the algorithm for  jRL-SIM reconstruction and make it available in an open source software package (see Appendix A).We note that an iterative regularized reconstruction procedure is advantageous for all SIM methods, but particularly for 4SIM, as the full rank mixing matrix M has a high condition number and, thus, is strongly affected by noise, which can lead to artifacts.The patterns e o can be retrieved conventionally from a 9SIM stack [1].To verify the 3SIM concept, we performed in silico imaging of a known ground truth structure consisting of 25 beads of 80 nm diameter each, distributed over a 2 × 2 μm 2 field of view.(The full simulation parameters are listed in Appendix B.) Figure 3(a) shows ground truth, simulated widefield, and deconvolved widefield images, as well as the 3SIM and 9SIM reconstructions.Fifty jRL iterations were used per a deconvolved image.As shown, the bead sizes apparent from the 3SIM and 9SIM reconstructions are similar to those of the deconvolved image, which is expected as all three approaches were implementations of jRL.jRL is known to enhance point-like features.However, a closer inspection shows performance differences: for adjoining beads [see, for example, the inlays in Fig. 3(a)], a two-point below the Sparrow limit is only achieved by the SIM modalities, but not by deconvolution alone.This becomes even clearer on analysis of respective Fourier transforms.Figure 3(b) shows a plot of the radially averaged spectra.The hard edge of the simulated beads produces circular ripples in the ground truth spectrum, which can only be restored via SIM superresolution [arrow head in Fig. 3(b)]; pure deconvolution boosts high-frequency content, but fails to reconstruct the first ripple.Encouraged by these results, we imaged fluorescent beads (Tetraspeck, Thermo Fisher Scientific) of a nominal diameter of 100 nm on a custom-built SIM system [15].The microscope was equipped with a 1.49 NA objective (Olympus), and the SI pattern frequency was set to 70% of the maximal spatial frequency of the 488 nm excitation light.An sCMOS camera (ORCA Flash v4.0, Hamamatsu) was used, with each pixel representing 65 nm in the sample plane.Figure 4 compares an acquired widefield image and its deconvolved version to both 3SIM and 9SIM.To enable quantitative comparison, all four images were generated from the same data set: nine raw SI images were first acquired and summed to form a widefield image, which was subsequently deconvolved.The 3SIM image was reconstructed from three out of the nine raw frames, the 9SIM image from all nine raw frames.jRL was used in all cases.Again, the reconstructed bead sizes of the deconvolved image are similar to both SIM reconstructions.Real resolution improvements can be observed in the enlarged panels of neighboring beads.Only the 3SIM and 9SIM algorithms lead to point separation beyond the widefield Sparrow limit, although slight distortions are apparent in the 3SIM example.
To quantify the performance of #SIM on dynamic samples, we simulated a moving target.(The full simulation parameters are listed in Appendix B.) The target featured a double-line structure spaced 180 nm apart and moving from left to the right at constant velocity v.After each full acquisition sequence, the velocity was increased (Fig. 5), and the simulation repeated.Widefield and SIM imaging were simulated at a 100 Hz raw frame rate, and the same line profiles was taken for every sample velocity.Figure 5 contrasts these "velocity-dependent" line profiles, starting at 0 nm/ms at the top and finishing at 5 nm/ms at the bottom.Shown are the ground truth, widefield, and deconvolved widefield cases, as well as four different #SIM implementations.The simulation was repeated 20 times for each parameter setting, and the results were averaged.Already for a stationary target the double-line feature remains unresolved in the widefield image, while the deconvolution recovers it according to the Sparrow criterion.All SIM versions resolve the feature clearly.As the velocity increases, a sample shift becomes significant as the number of raw frames increases, and the SIM modalities begin to fail at various points.We find that 9SIM reaches its limits at target velocities of about 1.5 nm/ms for the sample modeled.This is close to the speed at which, for example, membrane-bound vesicles traffic along microtubules [16].The presented results provide experimental and numerical validation for the proposed SIM framework.They demonstrate that SIM imaging with only three raw frames is possible.
Faster events can only be imaged reliably in SIM reconstructions from fewer raw frames.For sample velocities above 3 nm/ms, deconvolution microscopy (implementing energy conservation, positivity, support constraints, etc.) turns out to be superior to SIM and, therefore, should be the method of choice.We also find that the standard deviation of the reconstructions in the SIM modalities increases with increasing target velocity (not shown).Note that although the acquisition speed is tripled, 3SIM can only provide a robust reconstruction performance up to frame rates of about twice that of 9SIM.Generally, 3SIM offers potential for new combinations of SIM with light sheet microscopy [17] over those demonstrated  so far [8].As optical sectioning is intrinsic to SPIM, the raw frame signal-to-noise ratio (SNR) is improved, which favors 3SIM.An extension of the #SIM principle to 3D-SIM is also thinkable.On the other hand, the fact that 3SIM requires the minimal number of raw images to achieve superresolution makes excellent use of the photon budget, allowing for benign imaging conditions.In addition, 3SIM eliminates the need for pattern shifting, significantly simplifying the complexity of SIM-SPIM hybrid systems.On the downside, imaging samples with a low SNR remain a challenge for SIM.Furthermore, the overall illumination is nonuniform, which results in spatially dependent photobleaching and locally varying SNR.Potential software solutions to this problem include stricter regularization with total variation approaches [18], using the of mutual information and Kullback-Leibler divergence as an image proximity metric [19], and the penalization of frequencies associated with illumination frequency spikes [20].Furthermore, we found that beyond a certain sample speed, deconvolution outperforms SIM methods.Hence, within the parameter space between speed, gentleness, and resolution, we conclude that 3SIM has its place in the center.The image recording was simulated at 100 Hz.The line profiles through the target are plotted for different v.At velocities up to 1 nm/ms, all SIM modalities were able to resolve the lines clearly.The images of faster moving targets deteriorated as the underlying structure varied significantly during the acquisition sequence.The modalities for which fewer raw images are required permit imaging of faster moving samples.Above 3 nm/ms, the optimal modality is deconvolution microscopy for the target shown.

Fig. 1 .
Fig. 1.Principle of SIM reconstruction from (a) 9 or (b) 7 images.In the top right-hand corner, the widefield support is shown.Colors are used to indicate different orientations of illumination patterns.Using all orientations concurrently avoids redundancies in the reconstruction process (see the text).(c) 4SIM: here each SI frequency component is decomposed into "superresolution" information and widefield information.Segmenting each SI spectrum into multiple components yields additional equations.Adding a widefield image permits unambiguous SIM reconstruction (see the text).

Fig. 2 .
Fig.2.SIM spectrum can be divided into 66 segments (left).SIM imaging yields spectra that contain spectral information overlaps.For example, SI 3 shows a superposition of segments[6,52,65], which are highlighted as an example.Hence, each SI spectrum gives rise to 18 equations totaling to 3 × 18 56 equations.This equation system is underdetermined as there are 66 distinct spectral elements in the problem, but a maximum likelihood estimation can nevertheless yield resolution improvements comparable to 9SIM.If a widefield spectrum (WF) is recorded and added to the three SI spectra, a unique reconstruction is possible (illustrated by the dotted line).The segmentation is illustrated in further detail in Visualization 1.

Fig. 3 .
Fig. 3. (a) Simulations of widefield imaging and SIM with 25 beads of 80 nm diameter.The widefield image was deconvolved using jRL to allow comparison with 3SIM and 9SIM data.The simulation parameters are listed in the supplementary information.The line profiles show that both 3SIM and 9SIM achieve similar resolution and are beyond the Sparrow limit, while the widefield and deconvolved versions are not.(b) Radially averaged spectra of (a) demonstrate that only 3SIM and 9SIM offer true superresolution and restore frequencies of the ground truth spectrum up to twice the Abbe limit.The scale bars are 0.5 μm.

Fig. 4 .
Fig. 4. Experimental comparison if different imaging modes.For all methods, nine raw SI images were used as the basis for processing.(a) Widefield image was generated by summing all raw images.(b) Widefield image deconvolved with the jRL algorithm.(c) 3SIM image was reconstructed from three of the nine SI raw images.(a)-(d) Inlay panels show two neighboring beads.Both 3SIM and 9SIM distinguish the beads and, thus, achieve resolution beyond the Sparrow limit.The scale bars are 1 μm and 100 nm in the enlarged panels.

Fig. 5 .
Fig. 5. Double-line target with 180 nm spacing generated in silico, as shown in the upper left-hand corner, was modeled to move in a direction perpendicular to the line pattern at different velocities v.The image recording was simulated at 100 Hz.The line profiles through the target are plotted for different v.At velocities up to 1 nm/ms, all SIM modalities were able to resolve the lines clearly.The images of faster moving targets deteriorated as the underlying structure varied significantly during the acquisition sequence.The modalities for which fewer raw images are required permit imaging of faster moving samples.Above 3 nm/ms, the optimal modality is deconvolution microscopy for the target shown.