Introduction

Super-resolution (SR) microscopy enables biological researchers to see nanoscale images of intracellular structures. Single-molecule localization microscopy (SMLM), such as PALM/STORM (Betzig et al. 2006; Hell 2007; Rust et al. 2006), uses photocontrollable fluorescent proteins to label target proteins and image a small number of single molecules that are randomly activated at different times. And then the central position of a single fluorescent molecule is precisely located by a Gaussian fitting algorithm. Next, a final image with diffraction-unlimited resolution is reconstructed if a sufficient number of single molecules are obtained. This SMLM technology can achieve a spatial resolution of approximately 20–30 nm. However, this method requires thousands of image frames to reconstruct SR structures, and it is difficult to meet the researchers’ requirements for temporal resolution.

To achieve higher temporal resolution, a faster signal acquisition with a sufficiently high signal-to-noise ratio is important. Increasing the exposure frequency (i.e., decreasing the exposure time) provides faster signal acquisition, but for back-illuminated electron multiplying charge-coupled devices (EMCCDs), the read speed is currently limited to 70 frames per second, limiting the temporal resolution of SR techniques that use multiple sequential frames to reconstruct a diffraction-unlimited image, especially SMLM microscopy. Compared with EMCCD cameras, the newly developed sCMOS camera with the ability to detect single molecules has the advantages of a large field of view, fast readout speed, etc., which can significantly increase the data acquisition rate and improve the temporal resolution. However, unlike an EMCCD, an sCMOS camera has a unique single-pixel-dependent readout noise, and the random noise appearing in each single frame at different time points will be incorrectly fitted to the fluorescence signal of a single molecule. Since an SMLM image is a superposition of a single molecule localized in a large number of single-frame (tens of thousands to several tens of thousands) images, the superposition of randomly occurring noise in a single-frame image generates a pseudostructure in the final SR image.

The unique noise of the sCMOS camera limits its application in fast SMLM imaging. Therefore, there is an urgent need for noise calibration and for the development of a corresponding single-molecule localization algorithm. Huang et al. developed an algorithm to calibrate the independent readout noise by addressing three important characteristics (Huang et al. 2013): (1) Remove single-pixel-dependent offsets; (2) correct single-pixel-dependent gain values; and (3) remove the systematic fluctuation noise of each pixel. This method first calculates the brightness offset and the fluctuation level of each pixel by acquiring the dark image of the camera and then calculates the gain per pixel by illuminating the camera with different intensities. Then, these three parameters are used to calibrate the actual fluorescence image. This method is a good way to distinguish the source of readout noise and to calibrate and eliminate the single-pixel-dependent systematic readout noise of the camera. However, Huang et al. assumed Gaussian noise model of readout noise for all pixels on the sensor, which is based on the data obtained when the camera is dark, may not reflect the underlying fluctuation pattern for each pixel on the camera at different time points during the actual sampling, because during the actual sampling the fluctuation level of each pixel at a certain time point varies greatly. In SMLM experiments, these fluctuating noises can easily be treated as signals by single-molecule detection algorithms, resulting in artifacts in the reconstruction results.

The Hessian matrix is a square matrix composed of second-order partial derivatives of multivariate functions and is often used in the field of boundary detection and denoising (Lefkimmiatis et al. 2013; Sun et al. 2015). In 2018, Huang et al. developed a Hessian-SIM method (Huang et al. 2018). The Hessian penalty matrix was used to remove the scattering noise after SIM reconstruction, and the SR images with minimal artifacts were obtained. In this paper, we develop the new technique Hessian-SMLM, an SMLM imaging technology for rapid imaging using an sCMOS camera and combining the Hessian and Huang’s sCMOS calibrating algorithms to effectively remove sCMOS pixel-dependent readout noise. The new method can significantly reduce the reconstruction artifacts due to noise when used for actin PALM imaging.

Overview of Hessian-SMLM for SR imaging

First, we collected the image when the sCMOS camera was in the dark or illuminated by a series of white lights with different intensities and obtained the calibration parameters of the sCMOS camera: offset and gain. Then, a fluorescently labeled biological sample was imaged by the sCMOS camera. The corresponding pixel offset obtained previously was subtracted from each pixel in the acquired single-molecule data, and the fluorescence of each pixel was then divided by the gain. A Hessian denoising algorithm was then used to remove the fluctuation noise caused by variance. Next, the noise-corrected single-molecule image was used for single-molecule extraction. Last, the precise single-molecule position was obtained by Gaussian fitting, and the SR image was reconstructed. The specific process of Hessian-SMLM is shown in Fig. 1.

Fig. 1
figure 1

Diagram of Hessian-SMLM SR imaging. First, single-pixel level characterization of sCMOS is obtained, and the single-molecule data of each pixel are calibrated by the gain and offset values of the sCMOS camera. Then, the calibrated data are treated with the Hessian algorithm to remove the noise caused by variance. In the localization step, the images are segmented and fitted by Gaussian fitting to obtain the final SR image

Full size image

Acquisition of offset and gain of sCMOS camera

Offset is a constant luminance term in the readout process of sCMOS that is added in advance to avoid the negative luminance caused by the readout noise in the sCMOS camera. By taking a series of dark images with an sCMOS camera and calculating the average value of each pixel, the offset of each pixel can be measured. First, we turned off the autocalibration mode of the Prime 95B sCMOS camera (Photometrics, Tucson, AZ USA) and recorded 60,000 frames of dark images (Fig. 2A). The offset of each pixel in the sCMOS camera can be derived from the following statistical analysis:

$$Offset_{ij} = \frac{1}{N}\sum\limits_{s = 1}^{N} {pix_{ij}^{s} } ,$$
(1)

where N is the total number of dark frames and \({pix_{ij}^{s} }\) is the photon number at frame s for pixel ij.

Fig. 2
figure 2

sCMOS characterization at the single-pixel level. A To measure the offset and variance of sCMOS, 60,000 dark frames were used. B A series of image sequences were recorded at different average intensity levels to measure the mean values and variances. C The gain for each pixel can then be calculated with the help of the previously obtained variance and mean value. The red dashed line is the variance at different mean intensities of one pixel of the sCMOS camera, and the solid line shows the linear fit result

Full size image

Because the gain of each pixel of the sCMOS camera is different, the amplification of the fluorescence signal is different for each pixel during actual imaging, so it is necessary to measure the gain of each pixel for calibration. To calculate the gain for each pixel, we first took a series of dark images and calculated the variance as follows:

$$Var_{ij} = \frac{1}{N}\sum\limits_{s = 1}^{N} {\left( {pix_{ij}^{s} - Offset_{ij} } \right)^{2} } .$$
(2)

Then, we uniformly illuminated the camera with white light of different light intensities ranging from ~20 to 200 photons, which were evenly divided into 15 groups from low to high according to the light intensity. In total, 20,000 frames of images were captured for each group (Fig. 2B). The variance and mean value of brightness under different light intensities were then obtained, and linear fitting by the least-squares method was performed, as in Eq. 3 (Fig. 2C).

$$\hat{g}_{ij} = \arg \;\hbox{min} \sum\limits_{k = 1}^{N} {\left( {\left( {\text{var}_{ij}^{k} - \text{var}_{ij} } \right) - g_{ij} \left( {Mean_{ij}^{k} - Offset_{ij} } \right)} \right)^{2} } .$$
(3)

Hessian algorithm removes variance noise in single-molecule images

For Prime 95B sCMOS imaging, the pixel size is 110 nm/pixel for a 100× magnification microscope. Due to the optical diffraction limit, the full width at half maximum (FWHM) of a diffraction spot is greater than 250 nm, so when a single molecule on the sample is imaged through an optical system, the molecule occupies at least two pixels on the camera, which means that spatial correlation exists for two pixels. At the same time, if the speed of single-molecule blinking is lower than the camera’s readout speed, there should also be correlations across time series for the same pixel. However, there should be no correlation of random Gaussian noise, especially sCMOS pixel-dependent noise, between different pixels and/or across time series for the same pixel. Therefore, we refer to a reference (Huang et al. 2018) and introduce the Hessian penalty to remove the temporally and spatially random noise. The optimization function is as follows:

$$H\left( {r,t} \right) = \arg \;\hbox{min} \left[ {m\left\| {H\left( {r,t} \right) - D\left( {r,t} \right)} \right\|_{2}^{2} \,+\, R_{{\rm {Hessain}}} \left( {H\left( {r,t} \right)} \right)} \right],$$
(4)

where D is the raw data collected by the camera; H is the optimized data; m is the relative weight between the first term and the Hessian penalty; and RHessian is the Hessian penalty (Lefkimmiatis et al. 2012), which is expressed by Eq. 5,

$$R_{\rm Hession} \left( {H\left( {r,t} \right)} \right) = \int_{\varOmega } {\left\| {\left( {\begin{array}{*{20}c} {H_{xx} \left( {r,t} \right)} & {H_{xy} \left( {r,t} \right)} & {\sqrt \sigma H_{xt} \left( {r,t} \right)} \\ {H_{yx} \left( {r,t} \right)} & {H_{yy} \left( {r,t} \right)} & {\sqrt \sigma H_{yt} \left( {r,t} \right)} \\ {\sqrt \sigma H_{tx} \left( {r,t} \right)} & {\sqrt \sigma H_{ty} \left( {r,t} \right)} & {\sigma H_{tt} \left( {r,t} \right)} \\ \end{array} } \right)} \right\|_{1} {\text{d}}r{\text{d}}t},$$
(5)

where r is the position of each pixel; Ω represents the integral area that contains all pixels within the image H; and Hxy is the second-order partial derivative of H versus xy; σ is a parameter that was introduced to enforce the continuity of structures along the time axis.

We thus finally obtain the optimal noise-removed single-molecule data by calculating the minimum value of Eq. 4 from the single-molecule data, and use these data for single-molecule extraction and reconstruction as previous reference (Olivo-Marin 2002; Smith et al. 2010).

Cell culture

U2OS cells were cultured using complete MCMM (McCoy’s 5A Medium Modified, Gibco) supplemented with 10% inactivated fetal bovine serum (FBS, Gibco) in a 37 °C incubator with 5% CO2. The construct of LifeAct-mEos3.2 for labeling actin structures and the Hessian-SMLM SR imaging were described in Zhang et al. (2012). The constructs were transfected using Lipofectamine™ 2000 Transfection Reagent (Invitrogen, USA). After a 48-h transfection, the cells were then fixed by 4% (w/v) paraformaldehyde (NOVON Scientific, Pleasanton, CA USA) and 0.2% glutaraldehyde (Electron Microscopy Sciences, Hatfield, PA USA) and imaged by a custom built PALM microscope.

Simulation of Hessian-SMLM imaging

To verify the Hessian-SMLM’s ability to remove variance noise, we simulated single-molecule data containing fluctuating noise and performed PALM imaging simulations. We simulated and generated the original data shown in Fig. 3B and used the Thunder STORM imaging plug-in to generate single-molecule images (Ovesný et al. 2014). Simulated wave noise was added (Fig. 3A) so that we obtained simulated single-molecule data containing fluctuating noise. Then, the PALM single-molecule localization algorithm was used to reconstruct the image with (Fig. 3D) and without (Fig. 3C) Hessian denoising methods. As shown in Fig. 3C and D, the resulting reconstruction without the Hessian denoising method contains more artifacts, while Hessian denoising efficiently removed the artifacts caused by the fluctuating noise.

Fig. 3
figure 3

Simulation of single emitters on parallel lines based on the shown variance map with localization using single-molecule localization microscopy with and without the Hessian algorithm. A Simulated variance map. B Raw data used to simulate the single-molecule image. C, D Simulated single-molecule image reconstructed without (C) and with (D) the Hessian algorithm. The green box denotes the enlarged section shown in the inset

Full size image

Hessian-SMLM imaging of actin

To verify the effectiveness of Hessian-SMLM in SR imaging of biological samples, we transfected LifeAct-mEos3.2 in U2OS cells and acquired single-molecule data using the Prime 95B sCMOS camera. We treated the single-molecule data to reconstruct an SR image using different algorithms separately as follows: (1) Single-molecule localization algorithm of PALM; (2) the sCMOS-calibrated single-molecule localization algorithm by Huang et al.; and (3) the Hessian-SMLM algorithm in the current study.

We found that when using the Prime 95B sCMOS camera to acquire single-molecule data, there is substantial noise in the variance map of the fluorescence signal. Most of the noise is completely colocalized with the variance map of the dark camera (Fig. 4C), suggesting that the noise comes mainly from the variance when the camera is dark. Moreover, when the single-molecule data collected by the sCMOS camera are directly used in the PALM single-molecule localization algorithm, the reconstruction results contain more artifacts (Fig. 4D, indicated by boxes). Obviously, these artifacts come from the variance of the sCMOS camera because they are exactly the same noise points in the variance map when the camera is dark (Fig. 4A). Using Huang et al.’s sCMOS correction method to correct the variance, gain, and offset greatly improved the reconstruction results and significantly reduced the artifacts due to single-pixel fluctuations. However, for some areas, especially those where the variance is smaller, the calibration effect of this method is not ideal (Fig. 4E). One possible reason is that the variance we obtained with the dark camera can only represent the statistical fluctuation level of each pixel during the sampling process. In the actual sampling, the averaged variance of each pixel from the dark sCMOS camera does not accurately characterize the true fluctuation level of each frame at different time points. However, as shown in Fig. 4F, Hessian-SMLM can efficiently remove the variance noise in each frame produced by the fluctuation at different time points, and the artifacts generated in the reconstruction result are significantly reduced.

Fig. 4
figure 4

Hessian-SMLM imaging of actin with three SMLM algorithms. A Readout variance map of the sCMOS camera. B Standard deviation (STD) projection image of single-molecule data. C Merged image of the variance map (A) and the STD projection image (B). Green pixel-level readout variance of the dark camera; Red STD image. D–F Reconstructed super-resolution images of actin analyzed using the conventional algorithm (D), sCMOS camera-specific algorithm (E), and Hessian-SMLM algorithm (F). The green box denotes the enlarged section shown in the inset. Color scales, bottom to top: 0–65 ADU2, where ADU means analog-to-digital units (A), minimum-to-maximum signal (B), same upper bound chosen for best visualization for D–F. Scale bars 2 μm

Full size image

Next, we further compared the effect of the above three methods on a LifeAct-mEos3.2 reconstruction with a low exposure time and laser intensity. The low exposure time and laser intensity significantly reduced the signal-to-noise ratio of the single-molecule data. The structure directly reconstructed with the PALM localization algorithm had substantial noise and many artifactual structures (Fig. 5B). Using the sCMOS calibration method removed some of the noise (Fig. 5C), the effect was not as good as the effect when the signal-to-noise ratio was high (Fig. 4E). However, with the Hessian-SMLM method, the reconstruction artifacts were basically removed, and the structural continuity of actin was much better than the continuity with the other two methods (Fig. 5D compared to Fig. 5B and C, and enlarged regions shown in Fig. 5E). We chose four pixels with different variance and measured the pixel intensity before and after Hessian algorithm (Fig. 5F). The fluctuating value of the four pixels decreased significantly after Hessian algorithm (Fig. 5F, blue and red solid lines), while the mean pixel values are almost the same before and after Hessian algorithm (Fig. 5F, blue and red dash lines). Therefore, Hessian-SMLM is able to efficiently remove the variance noise of each frame produced by the fluctuation at different time points, and reduce the artifacts.

Fig. 5
figure 5

Hessian-SMLM significantly removes the noise of sCMOS and artifacts in SMLM images even when the signal-to-noise ratio is low. A Readout variance map of the sCMOS camera. B–D Reconstructed super-resolution images of actins analyzed using the conventional algorithm (B), sCMOS algorithm (C), and Hessian-SMLM algorithm (D). E Zoomed in regions (i–iv) from (A–D) show actin structures by the PALM localization algorithm, the sCMOS calibration method, and Hessian-SMLM algorithm. F The pixel intensity traces of the selected pixels were from cropped regions (i–iv) over 50 frames. Color scales, bottom to top: 0–50 ADU2 (A); same upper bound chosen for best visualization for B–D. Scale bars 2 μm (B–D), 0.2 μm (E)

Full size image

Discussion

Although sCMOS cameras are inferior to EMCCDs for single-molecule imaging, sCMOS cameras have the advantages of fast speed and large field of view and have potential applications in live-cell SMLM imaging. However, the unique single-pixel-dependent readout noise of sCMOS cameras severely limits their application in live SMLM, especially when the signal-to-noise ratio of a single-molecule signal is low. For SMLM, there are at least two situations that produce low signal-to-noise ratios for single-molecule signals. The first situation is the imaging of fixed cells by SMLM when the signal-to-noise ratio becomes worse as the sampling time increases. The other one is live SMLM. For live SMLM, on the one hand, the exposure time must be decreased to increase the sampling frequency and thus improve the temporal resolution. On the other hand, it is necessary to reduce the irradiation light intensity to reduce phototoxicity. However, reducing the exposure time and reducing the intensity of the illumination light significantly reduce the signal-to-noise ratio of the fluorescence signal. Under these conditions, the camera’s readout noise will be particularly notable. In particular, the camera’s variance noise will be Gaussian fitted as a fake single-molecule signal, which will produce false structures or artifacts in the final reconstructed SR image. In this study, our new Hessian-SMLM SR imaging method can well remove the single-pixel-dependent readout noise and effectively reduce the artifacts caused by the readout noise of the sCMOS camera. This method is especially suitable for SMLM imaging when the signal-to-noise ratio is low and has greater advantages over other SMLM imaging methods under conditions of low exposure time and laser intensity. Therefore, Hessian-SMLM is potentially more suitable than the other methods for live-cell SMLM imaging.