Easy Two-Photon Image-Scanning Microscopy with Spad Array and Blind Image Reconstruction

Two-photon excitation (2PE) microscopy is the imaging modality of choice, when one desires to work with thick biological samples, possibly in-vivo. However, the resolution in two-photon microscopy is poor, below confocal microscopy, and the lack of an optical pinhole becomes apparent in complex samples as reduced quality of optical sectioning. Here, we propose a straightforward implementation of 2PE image scanning microscopy (2PE-ISM) that, by leveraging our recently introduced ISM platform – based on a new single-photon avalanche diode array detector – coupled with a novel blind image reconstruction method, is shown to improve the optical resolution, as well as the overall image quality in various test samples. Most importantly, our 2PE-ISM implementation requires no calibration or other input from the user – it works like any old and familiar two-photon system, but simply produces higher resolution images (in real-time). Making the complexity disappear, in our view, is the biggest novelty here, and the key for making 2PE-ISM mainstream.

. Description of the 2PE-ISM system. In a) a schematic of the 2PE-ISM system is shown. The ISM part was built as an extension to a regular confocal microscope with several laser lines and detectors. Only three additional lenses (L3-L5) and a dichroic mirror (DM*2) are needed to convert a confocal/2PE system into a ISM super-resolution microscope. The dichroic mirrors DM*1-2 can be reconfigured to enable ISM imaging with the visible excitation lines. In b) an illustration of the pixel matrix of the SPAD array is shown. The pixels are numbered from 0-24, starting from the upper left corner. At the end of a scan, an image can be associated with each pixel position of the array. Each image is shifted with respect to the central detector element (12) image. In c) the adaptive pixel reassignment workflow is illustrated. First our iterative image registration method is applied to align all the individual images with the image from the central detector element (12). After registration, the images are added together to form the regular pixel reassignment result. The actual (Real) shifts are quite different from those suggested by theory (Theoretical), as shown by the two scatter plots. In order to take full advantage of the additional bandwidth of the ISM, as shown in d), a blind Wiener filter is applied on the reassignment result; the PSF is estimated from the ISM reassignment result image with FRC.
As already discussed for all-optical methods (26), proper alignment and fusion of the sub-images is essential 47 for optimal functioning of an ISM microscope. A big advantage of our ISM implementation is that all the 48 raw data is automatically recorded, without speed concerns and large data overhead that hampered the early 49 camera-based ISM implementations. In (27) we proposed an adaptive ISM reconstruction method, in which 50 image shifts are estimated directly from the data, by Fourier domain phase-correlation based method. For 51 the 2PE-ISM, we implemented a more robust, completely blind ISM reconstruction method, which requires no 52 calibration measurements or any prior knowledge of the sample or microscope configuration. We show how 53 our algorithm allows improving the resolution -typically up-to the theoretical limit -as well as the optical 54 sectioning in a variety of samples. We also propose a simple algorithm that makes it possible to construct the 55 ISM result image in real-time, pixel-by-pixel for live visualisation. 56 The simple optical configuration, the novel image reconstruction methods and the real-time reassignment in 57 our view are rather important steps towards the larger adoption of ISM, as they entirely hide its complexity; 58 ideally the end user would just see a regular confocal or 2PE microscope, with improved resolution. 59 Results 60 ISM image reconstruction. ISM image reconstruction consists of two tasks: image registration and image 61 fusion. Image registration is used to align the 25 sub-images (Fig. 1b)) into a common coordinate system, 62 whereas image fusion is used to combine the registered sub-images into a single result image. 63 A new iterative image registration method is proposed, based on software originally developed for Tomo-64 graphic STED microscopy image reconstruction (28). In image registration one tries to find a spatial transfor-65 mation that aligns the details in two images as closely as possible. In iterative image registration, as illustrated 66 in (Fig. 1c)) the search for the optimal spatial transformation is considered an optimisation problem: one image, 67 called fixed image, is used as a reference, while a second image, called moving image, is translated until the 68 details in the two images match. The images are considered to match when a chosen similarity metric reaches 69 its maximum value. In our ISM registration implementation, the central detector image is always used as the 70 fixed image and the 24 other images are sequentially registered with it.

71
After the shifts for all the 24 images have been calculated, images can be added together to produce the 72 adaptive pixel reassignment ISM (APR-ISM). However, both in ISM and SIM some form of deconvolution 73 or frequency domain filtering (re-weighting) is commonly used in order to maximise the effective resolution 74 (e.g. (3,16,18-20)). To this end, as illustrated in (Fig. 1d) Wiener filtering could be alternatively combined, as is done in SIM (3, 30), but we preferred to keep the two 79 tasks separated here, e.g. in order to enable straightforward sub-pixel registration as well as the possibility 80 to re-sample the resulting image into a different pixel size with respect to the raw data, and to support more 81 complex (deformable) spatial transforms that may become useful at a later date. In addition to the APR-ISM b+ reconstruction method, we also propose a simple algorithm that allows constructing the ISM result image point-83 by-point during data-acquisition, similarly to regular confocal or 2PE microscopes (Note S 1). This is possible 84 because our SPAD array detector has no frame rate, and thus every single photon can in real time be assigned 85 to its correct spatial location.

86
Benchmark measurements using test samples. In order to get an idea of the performance of our 2PE-  In (Fig. 2 a)) four images of the nanoparticle sample are compared: a regular two-photon image (2PE), a  (Fig. 2 a)). Interestingly, while the shifts clearly form a grid, as could be expected, already with a simple sample 99 like beads, there are clearly some aberrations present and the grid is also somewhat tilted. The pixel no. 24 in 100 the SPAD array used for the 2PE experiments is more noisy than the others, which appears to compromise the 101 image registration with the sparse beads sample. As shown in the FRC measures in (Fig. 2 a)), the ISM pixel

109
With the fixed cell sample, as demonstrated in (Fig. 2 b), the APR-ISM resolution is a factor of ∼ 1.6 superior 110 to the regular 2PE image, and blind Wiener filtering is able to recover ∼ 2.5 fold resolution gain. It is worth 111 noticing however that the resolution scale here is different than with the nanoparticles (2PE: 372 nm, APR-ISM: is shown to produce good results with the HeLa cell image, with similar level of details that were observed in 119 ( Fig.2 b)). A strong signal-to-noise ratio (SNR) gain is observed with respect to the 2PE ph. image, which 120 helps to underline how, already with simple test samples such as the two shown here, introducing an optical 121 pinhole (2PE ph.) has a strongly adverse effect on the SNR in 2PE microscopy. While the features clearly get In any case, it is quite evident from ( Fig. S 3 c)) that there is no "optimal" reassignment factor that would work 136 with all images, as the effective image shifts strongly vary from image-to-image. Our APR-ISM reconstruction 137 clearly produces the highest quality results, as it makes no assumptions of the reassignment factor, but instead 138 estimates the image shifts directly from the data.

139
Diving into a hazy brain slice. A multi-photon microscope arguably is most at home with thick samples 140 that are hard or impossible to image with regular fluorescence microscopy methods, and thus we decided to 141 test our 2PE-ISM in such samples. A partially optically-cleared mouse brain that expresses YFP in neurons, 142 was imaged at different penetration depths, up-to the maximum working distance of the microscope objective, 143 ∼ 140 µm. As shown in (Fig. 3 a)) the 2PE-ISM is able to maintain rich details and good contrast all the way 144 up to the maximum working distance. APR-ISM b+ practically provides a constant image quality across the 145 imaging depths, by maintaining a spatial resolution of ∼ 170nm, whereas the resolution in regular 2PE images 146 degrades from 500 nm to 570 nm as a function of depth, as can be expected (see FRC measurements in (Fig. 3   147 b)); this translates to a resolution improvement of factor three to four. The resolution improvement with the Looking at the ISM shift vectors in (Fig. 3 c)), our adaptive blind pixel reassignment appears to be doing much 153 more than simply introducing a tiny virtual pinhole. As discussed in (27), the light distribution on the array will become visible as changes in the image shifts. In fact, it is actually possible to identify optical effects, 156 such as coma (45 µm) and astigmatism (140 µm) in the ISM shift scatter plots in (Fig. 3 c)), when comparing 157 the results to the ideal rectangular grid. Typically such effects are corrected during imaging with adaptive 158 optics (20), but our adaptive blind ISM image reconstruction appears to enable correcting some aberrations 159 at the post-processing stage as well. As shown in (Fig. S 3), the distribution of the lengths of the ISM shift 160 vectors is also clearly less linear than with the Hela cell/Beads samples, which is also indicative of presence of 161 aberrations; however, the phenomenon is much more evident in the 2D scatter plot of the ISM shifts.

163
In this paper, we introduced a 2PE-ISM system based on a SPAD array detector that, compared to current  Because all the raw data is saved in our 2PE-ISM implementation, it is possible to treat stacks as 25 individual 182 3D images, rather than a series of 2D images, as is the case in the all-optical techniques. This seems to be 183 rather important, as based on a simple preliminary experiment with the Beads sample, the axial information 184 is important and the axial shifts appear to be rather large (Fig. S 6). The results are especially interesting as 185 there appears to be a strong resolution gain in the axial direction as well. In order to reach any conclusions 186 however, further investigation is required.

187
The final thoughts as well as several examples in this paper, help to underline the importance of computation 188 in modern microscopy. In wide-field microscopy, similar approaches have been in regular use already for a while 189 -e.g. axial sectioning based on deconvolution, SIM, localization based super resolution -but confocal and 2PE think that ISM with its massive information content, such as our 2PE-ISM, can put an end to that.

193
The Custom Microscope Setup. Our 2PE-ISM system was implemented as a modification to a confocal 194 microscope, which itself is a modified Abberior Instruments RESOLFT system (Abberior Instruments, Ger-195 many). As shown in Figure 1, the femtosecond two-photon excitation laser (Chameleon Ti:Sapphire, Coherent) functions of the microscope system and to preview and process the imaging results.

210
The ISM reconstruction method. The ISM image reconstruction is a two step process. First, all the sub-211 images (array pixels 1-25) need to be registered. Second, the registered sub-images need to be fused to produce 212 a single result image.

213
In iterative image registration one image, called fixed image, is used as a reference, while a second image, called

246
After all the 25 images have been registered, they can be added together to for the regular APR-ISM pixel 247 reassignment result. In order to take full advantage of ISM however, one has to still remove the blurring effect 248 of the PSF, which in this case is that of the ISM reassignment result, as explained in context of SIM image 249 reconstruction in (30). To this end, we applied our FRC based blind Wiener filter on the reassignment result.

250
The filter first estimates the PSF directly from the image with an FRC measurement, after which a classical 251 Wiener filter is applied; the FRC cut-off value is used as a full-width-half-maximum (FWHM) value for a 252 Gaussian PSF. Please refer to (29) for a detailed description.

253
The ISM image reconstruction workflow described above is mainly intended for the post-processing stage.

254
The pixel reassignment can, however, also be performed in real-time, pixel-by-pixel, just as in a regular confocal 255 or two-photon microscope (Note S 1).

256
Resolution measurement. The resolution measurements shown in this work, were calculated with our one-    deconvolution. The RL algorithm produces sharper looking results with strongly improved contrast, but quantitatively, as shown by FRC measurements in c), the resolution in the two images is the same. It may thus be beneficial to use RL or other iterative algorithm to produce the crispiest looking results, but as shown in b) it takes about 50 iterations for the RL algorithm to reach the same resolution scale the the Wiener filter is able to produce with a single division. Thus, when speed is an issue, Wiener filter provides a much superior solution (of course RL can be significantly accelerated with GPU if necessary). Scale bar 4 µm. Real-time pixel reassignment is shown to dramatically boost the SNR in the HeLa cell image, when compared to 2PE image with a pinhole. This makes it possible to achieve the resolution gain (and optical sectioning capability) in practice that the small pinhole can in theory provide. In the scatter plot ISM shifts discretized to multiples of the pixel size (Note S 1) are compared to the actual registration results. The discretized shifts are used by the real-time pixel reassignment algorithm, because no re-sampling is performed, but simple array indexing is used instead. While the rounding does produce small errors, the observed image quality still remains high. Scale bar 4 µm. Comparing theoretical and data-based ISM shifts. In a) scatter plots of the registration results (ISM shifts) for all the images of the main article are shown. In b) the lengths of the data-based shifts (from a)) compared with theoretical, optical geometry based values (Fig. S 4). The theoretical shifts are shown as distances on the detector plane, whereas the real distances are on the object plane -therefore the slope of the linear model fit roughly reflects the total magnification of the microscope. The theoretical and real values increase rather linearly with the thin test samples, but the differences increase in the brain sample images. As shown in c), in all cases, the theoretical shifts (green triangles) on a 2D plane never even closely match the real ones (blue circles), as the various optical effects are not accounted for. . The SPAD array is a 5x5 square detector grid with 75 µm detector pitch. For every pixel (detector element), absolute distance from the center (pixel 12) can be calculated with the multipliers shown in a). The length of the theoretical ISM shifts on the detector plane (theor abs ), can be obtained by further multiplying the pixel offsets with the reassignment factor (0.5 here). Plotting the theoretical shifts as a function of length of the real ISM shifts on the object plane (real abs = x 2 + y 2 , for all (x,y) pairs in b)), allows us to estimate the effective magnification in the microscope from the slope of the linear model fit in c). Theoretical and calibrated ISM shifts were generated by first creating a 5x5 meshgrid d), and then, in order to have the shifts in physical units (µm) on the object plane, multiplying the grid coordinates by pitch·reassignment factor magnification in e) and pitch·reassignment factor slope in f), where slope denotes the slope of the linear model fit in c). ISM reconstructions are shown for the HeLa cell image with adaptive, theoretical and calibrated shifts; the theoretical and calibrated shifts were obtained, as illustrated in (Fig. S 4). The adaptive ISM reconstruction produces clearly superior results. Scale bar 4 µm.