Resolution doubling using confocal microscopy via analogy with structured illumination microscopy

Structured illumination microscopy (SIM) is a super-resolution fluorescence microscopy with a 2-fold higher lateral resolution than conventional wide-field fluorescence (WF) microscopy. Confocal fluorescence (CF) microscopy has approximately the same optical cutoff frequency as SIM; however, the maximum theoretical increase in lateral resolution over that of WF is 1.4-fold with an infinitesimal pinhole diameter. Quantitative comparisons based on an analytical imaging formula revealed that modulation transfer functions (MTFs) of SIM reconstructed images before postprocessing are nearly identical to those of CF images recorded with an infinitesimal pinhole diameter. Here, we propose a new method using an adequate pinhole diameter combined with the use of an apodized Fourier inverse filter to increase the lateral resolution of CF images to as much as that SIM images without significant noise degradation in practice. Furthermore, the proposed method does not require a posteriori parameterization and has reproducibility. This approach can be easily applied to conventional laser scanning CF, spinning disk CF, and multiphoton microscopies.


Introduction
Fluorescence microscopy has contributed significantly to the elucidation of molecular-level mechanisms within living cells by imaging specific fluorescently tagged proteins in biological specimens with high temporal and spatial resolutions. However, conventional wide-field fluorescence (WF) microscopy cannot image structures smaller than 200 nm because of the optical imaging diffraction limit, as first described by Abbe. 1) However, since the early 2000s, various superresolution (SR) fluorescence microscopy techniques that can resolve distances smaller than the diffraction limit have been developed. 2) SR techniques in fluorescence microscopy include stimulated emission depletion (STED) microscopy 3) and pointillist techniques, 4,5) but structured illumination microscopy (SIM) [6][7][8] is considered to be the most compatible with biological applications. This is because SIM requires no special conditions for fluorescent dyes and staining protocols, SIM has the highest acquisition frame rate among SR methods, and its imaging area is sufficiently large for whole-cell imaging. 9,10) SIM illuminates an object with patterned excitation light so that fluorescence images of the object are modulated [Figs. 1(a) and 1(b)]. Two types of image detection are used with this technique: image recording without masking, as in WF detection (WF-SIM), and the application of an aperture mask to an image, as in confocal fluorescence (CF) detection (CF-SIM). In WF-SIM, the excitation pattern is sinusoidal, and the recorded image of the fluorescence emission distribution from the object is modulated. That is, the recorded image contains spatial-frequency-shifted components of the object distribution. Recording several modulated images while changing the position of the patterned excitation light enables the computational separation of each spatial-frequency-shifted component. A Wiener filter is applied to the spatial-frequency-shifted components to reduce white noise. Then, the components are shifted back to their original positions in the Fourier space to be demodulated image components. The demodulated image components are summed using a generalized Wiener filter to form a reconstructed SR image. 11) The lateral resolution of WF-SIM is 2-fold that of WF microscopy, specifically, 100-130 nm. In CF-SIM, an image scanning microscopy (ISM) [12][13][14] technique is incorporated to increase the lateral resolution to 2-fold that of WF microscopy. ISM was originally devised as an SR technique for laser scanning CF microscopy, which uses an imaging detector instead of a single-channel detector. Confocal images from the imaging detector channels are separately position-corrected according to the detector channel position and merged to generate an SR image. There are ISM variants that implemented this position correction optically such as optical photon reassignment microscopy (OPRA) 15) and re-scan confocal microscopy (RCM). 16) In CF-SIM, the excitation pattern consists of multifocal spots, and an image of the fluorescence emission from each focal spot is reduced locally by half either digitally such as multifocal SIM (MSIM) 17) and confocal spinning-disk ISM (CSD-ISM) 18) or optically such as instant SIM (ISIM) 19) and spinning-disk confocal microscopy using optical photon reassignment (SD-OPR). 20) These CF-SIM techniques are hereinafter referred to as MSIM(ISIM). CF microscopy [ Fig. 1(c)] was once categorized as a type of SR microscopy because the lateral cutoff frequency is 2-fold that of WF microscopy. 21) The theoretical increase in the lateral resolution of CF microscopy compared with that of WF microscopy is 1.4-fold when the confocal pinhole aperture is infinitesimal, 22) which is called ideal CF. However, because priority is given to brightness in practice, the confocal pinhole aperture is usually set to approximately the diameter of the Airy disk (the central bright circular region of a diffraction limited spot pattern inside the first dark ring), or one Airy unit, which reduces the lateral resolution to a level comparable to that of WF microscopy.
The increase in the lateral resolution of SIM compared with that of WF microscopy has been attributed to the increase in lateral cutoff frequency. However, a 2-fold increase in lateral cutoff frequency does not always yield a 2-fold improvement in lateral resolution. Although the lateral cutoff frequency of CF microscopy is also 2-fold that of WF microscopy [ Fig. 1(d)], the increase in lateral resolution is no more than 1.4-fold higher than that of WF microscopy. To explain this inconsistency, a quantitative analysis of SIM image formation is essential.
The first purpose of this study is to conduct a quantitative comparison of the imaging properties of SIM and CF microscopy. Although CF microscopy has been suggested to be a special case of SIM, 14) an analytical explanation of this categorization has not been provided. The second purpose of this study is to determine whether CF microscopy can be regarded as SR microscopy with 2-fold the lateral resolution of WF microscopy. First, the imaging formula of SIM is derived; this is followed by numerical simulations of SIM image properties and parameter optimization of CF for SR regarding both the lateral resolution and the signal-to-noise ratio (SNR). The findings are then discussed. Some experimental results are given (Fig. S6 in the online supplementary data at http://stacks.iop.org/JJAP/55/082501/ mmedia).

Fundamental analytical expressions of SIM and CF
Here, a one-dimensional (1D) structured illumination model is assumed for simplicity. The structured pattern is assumed to be a function of the spatial coordinate x. However, expansion to a multidimensional structured illumination model is straightforward. For derivation, see the online supplementary data at http://stacks.iop.org/JJAP/55/082501/mmedia.
In SIM, the excitation intensity distribution in the object space E SI (x) can be expressed as a Fourier expansion series with a finite number of Fourier coefficients {a m } and Here, j is an imaginary unit, M is an integer, and k p is defined on the basis of the fundamental period of the structured illumination pattern p, where For the evaluation of the raw signal energy distribution in the Fourier space, the Wiener filter on the spatial-frequencyshifted image components is neglected here and a raw demodulated image of WF-SIM I WF-SIM is introduced by summing demodulated image components without weighting. The reconstructed SR image of WF-SIM can be obtained by applying a generalized Wiener filter to I WF-SIM when noise is neglected.
I WF-SIM (x) can be written as a convolution of the object distribution O(x) and the point spread function (PSF) of WF-SIM P WF-SIM (x), which is a product of E SI (x) and the PSF of the fluorescence wavelength P FL (x).
Here, x denotes the convolution operator in the x coordinate.
In comparison, the CF microscopy image I CF (x) obtained with an excitation intensity distribution in the object space E CF (x), the confocal aperture A PH (x), and P FL (x) can be described by Therefore, a raw reconstructed image of WF-SIM has fundamentally the same mathematical formula as that for an ideal CF image, i.e., A PH (x) is the Dirac delta function of x, because both PSFs are products of the excitation intensity distribution in the object space and the PSF of the imaging system for a fluorescence wavelength. Therefore, WF-SIM can be regarded as computational CF microscopy, or CF microscopy can be categorized as SIM.
For a CF-SIM technique such as MSIM and ISIM, we can regard E SI (x) as a form of an array of PSFs of the excitation wavelength P EX (x) by the period p. Thus, Incorporating the ISM technique, the recorded CF-SIM image I CF-SIM (x) becomes Here, A PH (x) is the aperture function of each confocal pinhole aperture.

Numerical calculations
Using the formulae derived above, the 2D distributions of PSFs and MTFs in the focal plane are numerically calculated for quantitative comparison. An MTF is an absolute value of an optical transfer function (OTF), which is a Fourier transform of a PSF. 3D distributions of PSFs and MTFs are also calculated for reference (Figs. S1-S4 in the online supplementary data at http://stacks.iop.org/JJAP/55/082501/mmedia).
3D-SIM 23) is selected as the WF-SIM here because it is most widely used among WF-SIM techniques. 3D-SIM uses a threebeam interference of the excitation beam inside the object space to generate a 3D structured illumination pattern. Typical 3D-SIM records three orientations of the structured illumination pattern every 120°. Here, the first orientation of the pattern rotation is defined as the x-axis. There are five Fourier components for each orientation of the structured illumination pattern; therefore, the number of spatial-frequency-shifted components in a detected image is also five. Separation and demodulation procedures are performed for each orientation by acquiring five images at different illumination pattern positions. The demodulated images with different orientations are then summed to obtain a roughly isotropic SR image.
The PSFs and MTFs are calculated for 3D-SIM and MSIM(ISIM) [Figs. 2(a), 2(b), 2(d), and 2(e)]. The spatial coordinates are normalized by the HWHM of WF PSF so that the resolution improvement over a WF image can be seen directly. The wavenumber coordinates k r are normalized by the cutoff wavenumber of WF MTF in the xy-plane k r,c with Here, N is the numerical aperture of the objective lens and λ FL is the fluorescence wavelength. The optical parameters for the numerical calculations are N = 1.35 for the numerical aperture of the objective lens and 1.4 for the refractive index of the object medium. The excitation wavelength is approximated by λ FL for simplicity. The distance between the +1 and −1 order diffraction beams of WF-SIM is assumed to be 90% of the pupil diameter of the objective lens ( Fig. S2a in the online supplementary data at http://stacks.iop.org/JJAP/55/ 082501/mmedia). The diameter of each confocal pinhole aperture of MSIM(ISIM) is assumed to be one Airy unit. As a standard measure, the curve for ideal CF CF β=0 is shown in each plot, where β is the ratio of the pinhole aperture diameter to one Airy unit. Sectional plots on the xand y-axes are shown for 3D-SIM because PSF and MTF are not circularly symmetrical. MTFs are shown in logarithmic scale.
The PSFs of 3D-SIM and MSIM(ISIM) in the focal plane are almost identical to that of ideal CF. That is, the increases in lateral resolution compared with that of WF are approximately 1.5-and 1.4-fold in the two cases, respectively. To obtain the final 2-fold increase in lateral resolution compared with that of WF, postprocessing, such as using a generalized Wiener filter 23) or a dipping Fourier filter, 24) is applied to 3D-SIM, and either iterative deconvolution 17,19) or Fourier filtering 18) is applied to MSIM(ISIM).
The results of the numerical simulations above demonstrate that both WF-SIM and CF-SIM techniques require postprocessing to achieve final 2-fold improvements in lateral resolution compared with that of WF. Before postprocessing, the lateral resolution is identical to that of ideal CF microscopy ( β = 0). In particular, because the MTFs themselves of MSIM and ISIM are nearly identical to that of ideal CF microscopy, the application of a similar postprocessing to CF images would probably increase the lateral resolution 2-fold. However, ideal CF microscopy using an infinitesimal pinhole  diameter blocks all signals from the object and is therefore impractical. In the next section, the imaging properties of CF microscopy with a finite pinhole diameter are investigated to obtain a practical solution.

Aperture optimization and postprocessing of CF images
PSFs and MTFs in the focal plane are calculated for CF microscopy with β = 0, 0.5, and 1 [Figs. 2(c) and 2(f)]. The WF case is shown for reference. As β decreases, the closer the PSFs and MTFs become to those of ideal CF microscopy. However, image brightness decreases as β decreases.
To determine the optimum β, first, the PSF intensity integration around the pinhole edge, E ed ( β ) = βP FL (1.22 βr c ), is calculated [ Fig. 3(a)]. Here r c = 2π=k r,c . Because the confocal image properties including super-resolution information can be regarded as results of diffraction at the pinhole edge. The peak position of E ed ( β ) lies in the β = 0.3-0.5 range.
Second, to evaluate the absolute intensity in the high-frequency region (1 < k r < 2), MTFs multiplied by the encircled energy of a focused signal within the pinhole aperture E en ( β ) is calculated [ Fig. 3(b)].
The absolute intensity in the high-frequency region is maximal when β is in the range of 0.3-0.5. On the other hand, E en ( β ) and, consequently, the signal intensity from the focal plane becomes approximately half the conventional CF ( β = 1) or CF-SIM value [ Fig. 3(a)]. However, because the blurred noise intensity from outside the focus region is proportional to the area of the pinhole aperture, or β 2 , the ratio of blurred noise to the focused signal, B r ( β ) = β 2 =E en ( β ), decreases by approximately one half as β decreases from 1 to 0.5 but does not change much in the β = 0-0.5 range [ Fig. 3(a)].
Third, the optical sectioning ability, I z -curve, is checked ( Fig. S5 in the online supplementary data at http://stacks. iop.org/JJAP/55/082501/mmedia). I z -curve is the defocus intensity distribution of the image of a thin flat object and given by where a tilde (~) on top of the function symbol indicates the Fourier transform of the function and F k z ½Á Á Á denotes the Fourier transform operation along the k z coordinate. In the CF case, dy A PH ðx; yÞ½E CF ðx; y; ÀzÞ xy P FL ðx; y; zÞ: The FWHM of I z -curve FWHM z does not change much between β = 0 and 0.5 [ Fig. 3(a)]. Taking these aspects into account, the optimal β lies between 0.3 and 0.5. Here β = 0.5 is selected in the following calculation. Next, postprocessing to achieve a 2-fold increase in the lateral resolution compared with that of the WF image is considered. Since data from an image-taking device is 2D in a focal plane, postprocessing is conveniently defined here as 2D in the x-y plane and is applied to each image obtained to increase the lateral resolution while maintaining the I z -curve property. Formulae are defined here as 2D in a focal plane.
The Wiener filter derives a solution that minimizes the statistical estimation of mean square error assuming that random white noise with an SNR. Therefore, application of the generalized Wiener filter to CF images is first considered for the estimation of the required SNR for CF images. Setting the target SR PSF P SR , the generalized Wiener filter F W for a CF image, the Wiener filtered CF image I W , and its point spread function P W are defined by F W ðkÞ P SR ðkÞP Ã CF ðkÞ=½jP CF ðkÞj 2 þ " 2 ; I W ðkÞ ¼Ĩ CF ðkÞF W ðkÞ ÕðkÞP W ðkÞ; Here, ε is a small, empirically adjusted parameter, which is usually defined as the inverse of the SNR of the input image. Since the desired SR image has 2-fold the WF lateral resolution,P SR ðkÞ is set to the WF OTF of 2-fold the cutoff frequencyP FL ðk=2Þ. 13) The generalized Wiener filter acts as a low-passed inverse Fourier filter [ Fig. 4(a)]. There is an ε-dependent actual cutoff frequency above which the MTF value drops steeply [ Fig. 4(c)]. The larger the ε, the lower the actual cutoff frequency becomes. However, the FWHMs of the resultant PSFs are less sensitive to ε than the actual cutoff frequency for ε values less than 0.03. The increases in resolution over that of WF are 1.57-fold for ε = 0.03, 1.70-fold for ε = 0.01, and 1.81-fold for ε = 0.003 [ Fig. 4(e)]. Therefore, if we expect the resolution increase over that of WF to be 1.7-fold or higher, an SNR of 100 or larger is preferred for CF images.
When recently developed highly sensitive photodetectors such as gallium arsenide phosphide (GaAsP) photomultiplier tubes are used, photon noise dominates the SNR of CF microscopy images. Since the occurrence of fluorescence emission is a random event that obeys a Poisson distribution, the SNR is roughly the square root of the detected photon number. For laser scanning CF microscopy whose photon capacity for each pixel is 10 3 , the SNR of a single fluo-  rescence image is expected to be 30 at most for a single exposure. To obtain an SNR of 100, frame averaging or pixel binning is required. In the following, a CF image is assumed to have an SNR of 100. The steep decrease in the intensity of the generalized Wiener filter above the actual cutoff frequency is a consequence of minimizing the statistical estimation of mean square error. However, it loses information above the actual cutoff frequency. For ε = 0.01, the generalized Wiener filter has an actual cutoff frequency at 1.6 above which information in CF images is lost. To save information in the higherfrequency region without amplifying noise too much, an apodized Fourier inverse filter having the same maximum intensity as the generalized Wiener filter (ε = 0.01) and a higher actual cutoff frequency is next investigated.
The Fourier inverse filter F inv (k) can be expressed as This simple formula causes F inv (k) to diverge for k near the cutoff frequency of CF. A practical measure is the application of an apodized Fourier inverse filter, F apo (k), which is defined in relation to the maximum magnitude of the filter, F inv,max , and an apodization mask, M apo (k).
F apo ðkÞ min½F inv ðkÞ; F inv,max M apo ðkÞ ð13Þ F inv,max limits the magnification of F apo to a certain value to avoid excessive noise amplification. The optimized value can be determined by the maximum value of generalized Wiener filter for SNR of input CF images. The noise amplification is expected to be substantially the same as that of a generalized Wiener filter of the same maximum value. M apo can be of any shape, which smoothly decays the magnification near the cutoff frequency of a CF image (k r = 2) to avoid ringing artefacts while maintaining the resolution. Since the filtered CF image I apo is given by the product of I CF and F apo in the Fourier space, the PSF of the filtered CF image P apo is given byĨ apo ðkÞ ¼Ĩ CF ðkÞF apo ðkÞ ÕðkÞP apo ðkÞ; P apo ðkÞ ¼P CF ðkÞF apo ðkÞ: ð14Þ Calculations were conducted in the β = 0.5 case [Figs. 4(b), 4(d), and 4(f)]. F inv,max is set to 7.6, which is the maximum value of F W,ε=0.01 . M apo (k) has not been optimized but intuitively defined here as Although F apo is not analytic at points where F inv = F inv,max and k r = 2, the effects of these anomalies are practically negligible because the MTF values at those points are quite low and actual noise drowns them out. The increase in the resolution of I apo over WF is 1.74-fold, which is slightly higher than that of I W with F W,ε=0.01 [ Fig. 4(f)]. Image simulation was conducted using a thin Siemens star chart object model [Figs. 5(a)-5(f)]. The lateral resolution limits of the filtered images were approximately half (yellow solid circles) that of the WF image (yellow broken circles). Experiments with a biological sample were also performed to show the actual resolution increase by an apodized Fourier inverse filter on a CF β=0.5 image (see Supplementary Methods and Fig. S6 in the online supplementary data at http://stacks.iop.org/JJAP/55/082501/mmedia).

Discussion
Apodized Fourier inverse filters have certain advantages over generalized Wiener filters when the OTF of an input image is well known. An apodized Fourier inverse filter can be defined on the basis of optical properties of the system, which include the excitation and fluorescence wavelengths, the numerical aperture of the objective lens, magnification, and pixel size. F inv,max can be fixed by assuming that the photon noise levels of input images are within a certain range by adjusting excitation power or exposure time to earn a sufficient number of photons. Then, the freedom from a posteriori parameterization renders the results deterministic and reproducible. The results of the apodized Fourier inverse filtering on the confocal image can be regarded as images with an OTF P apo ðkÞ, as in Eq. (14). Therefore, additional deconvolution is possible with this virtual OTF to specifically reduce noise  amplified by the apodized Fourier inverse filter. On the other hand, the actual cutoff frequency of the Wiener filter varies by P CF and P SR even if ε is kept constant. Therefore, P W and the final resolution vary by each optical setup and additional deconvolution becomes complicated. Resolution after deconvolution should be examined carefully. For example, MLE deconvolution estimates the object function from a noisy image using statistical likelihood. The Richardson-Lucy (RL) algorithm is widely used for fluorescence microscopy images, in which Poisson noise is dominant. The RL algorithm works well for deblurring but does not improve lateral resolution. Images of independent lines or spots become very sharp, but images of close lines or spots occasionally spread and overlap. 25) Therefore, images after MLE deconvolution are nonlinear to object intensity and measuring the lateral resolution on the basis of single line width or single spot size is not reasonable.
Raw demodulated images of 3D-SIM have higher MTF values in the super-resolution-frequency region (1 < k r < 2) than CF images [ Fig. 2(d)]; therefore, CF images seem at a disadvantage in terms of resolution increase by postprocessing when the SNR of an input image is lower than the preferred value. Since photon noise has a constant intensity distribution in the Fourier space, a signal in the superresolution-frequency region buried by photon noise level cannot be salvaged by filtering. However, for thick objects, the optical sectioning property of CF images may have a higher SNR than that of a raw demodulated image of 3D-SIM, which is degraded by the out-of-focus background.
When the SNR of an input CF image is enough, the utilization of an apodized Fourier inverse filter, which can be set so that the magnitude does not fall off so rapidly above an actual cutoff frequency of a generalized Wiener filter, is expected to compensate for this disadvantage to some extent. Optimizing M apo remains to be carried out.
For the same reason, applying an apodized Fourier inverse filter on a raw demodulated image of 3D-SIM may give better results than applying a generalized Wiener filter, which is a future subject. The proposed method can also be applied to spinning disk CF microscopy. 26) The pinhole diameters of conventional Nipkow disks are designed to be approximately one Airy unit for the same reason as that for conventional CF microscopy. Extender optics between the objective lens and the Nipkow disk may be used to reduce the effective pinhole diameter to obtain the desired β. Spinning disk CF microscopy uses cooled CCD or scientific CMOS detectors, whose photon capacity for each pixel is 10 4 or more; therefore, an SNR of 100 can be obtained by a single exposure.
Apodized Fourier inverse filters can also be used for multiphoton microscopy because the imaging formula is similar to that of CF microscopy. For example, the formula of the PSF of two-photon microscopy P 2P (x), 27) has essentially the same form as that of ideal CF microscopy, disregarding the difference in wavelength between excitation and fluorescence. Therefore, for two-photon microscopy, hardware modification is unnecessary, and the increase in lateral resolution over that of conventional systems is up to 1.4-fold.

Conclusions
Analytical formulae for raw demodulated images of WF-SIM and CF-SIM have been derived to show their mathematical resemblance to that of CF microscopy: their PSFs are the product of the illumination intensity distribution and the PSF of imaging optics. Quantitative comparisons based on the formula revealed that the MTFs of SIM reconstructed images before postprocessing are nearly identical to the MTFs of CF images recorded with an infinitesimal pinhole diameter.
Using an adequate pinhole diameter combined with an apodized Fourier inverse filter, the lateral resolution of CF images can be increased to be as much as that of SIM images without significant degradation of noise. Furthermore, this method does not require a posteriori parameterization and provides reproducibility of the final resolution. This approach can easily be applied to conventional laser scanning CF, spinning disk CF, and multiphoton microscopies with minor modifications.