Temporal resolution doubling in fluorescence light-sheet microscopy via a hue-encoded shutter and regularization

Studying dynamic biological processes, such as heart development and function in zebrafish embryos, often relies on multi-channel fluorescence labeling to distinguish multiple anatomical features, yet also demands high frame rates to capture rapid cell motions. Although a recently proposedmethod for imaging dynamic samples in transmission or reflection allows to conveniently switch between color imaging or boosting the frame rate by use of spectrally-encoded, temporallymodulated illumination sequences and a hue-encoded shutter (hue-encode shutter method, HESM), the technique is not applicable directly in fluorescence microscopy, where the emitted light spectrum is mostly independent of the excitation wavelength. In this paper, we extend HESM by using samples labeled with multiple fluorophores, whose emission signal can either be used to distinguish multiple anatomical features when imaged in multi-channel mode or, if the fluorophores are co-localized in a dynamic tissue, to increase the frame rate via HESM. We detail the necessary steps to implement this method in a two-color light-sheet microscope to image the beating heart of a zebrafish embryo. Specifically, we propose an adapted laser modulation scheme for illumination, we identify caveats in choosing a suitable multi-color fluorophore labeling strategy, and derive an `1-regularized reconstruction technique that is sufficiently robust to handle the low signal-to-noise ratio and labeling inhomogeneities in the fluorescence images at hand. Using the case of a beating heart in a zebrafish embryo, we experimentally show an increase in the frame rate by a factor two while preserving the ability to image static features labeled in distinct channels, thereby demonstrating the applicability of HESM to fluorescence. With a suitable illumination setup and fluorescent labeling, the method could generalize to other applications where flexibility between multiple channel and high-speed fluorescence imaging is desirable. For fluorophores that are not co-localized, the imaging system is similar to a conventional light sheet microscope. © 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement


Introduction
Fluorescence microscopy is a tool used in many biological studies. It is possible to have fluorescence light emitted by a sample from specific biological features such as nuclei, membranes, or selected molecules [1]. In addition to the need for distinctly labeling anatomical, cellular, or subcellular features, many live biological processes are highly dynamic and a high temporal resolution in imaging is required to study them [2]. Although a recently proposed method for imaging dynamic sample in transmission or reflection allows to conveniently switch between color imaging or boosting the frame rate by use of a spectrally encoded illumination and a hue-encoded shutter method (HESM), the technique is not applicable directly in fluorescence The sample (x) contains co-localized fluorophores (e.g. green and red fluorescent proteins) that respond to the excitation by emitting fluorescent light. The sample may also contain static regions that can be labeled with a single fluorophore type. The color sensor of the camera (CAM) captures the light emitted over the entire exposure time through the detection objective (DO). (b) Acquired (raw) multi-channel images (c) Our proposed method converts the color frames to multiple monochrome frames at a higher frame rate (here, by a factor 2) in a region of interest that contains dynamic features. The color information can be preserved in static regions outside of the region of interest. microscopy, where the emitted light spectrum is mostly independent of the excitation wavelength.
In this paper, we propose to extend the hue-encoded shutter method presented in [3] to fluorescence microscopy and to apply it to light-sheet microscopy [4]. The method in [3] increases the flexibility of microscopes by allowing to rapidly switch between color imaging and fast imaging, or to do both simultaneously, within regions of interest. Briefly, that method uses a multi-color LED illumination source that is synchronized to a color camera. The scene (which is assumed to be monochromatic) is illuminated with a temporally-varying hue at a sub-frame rate frequency. The individual color frames are then processed to convert the color channels into multiple (monochrome) time frames. The monochrome requirement of [3] cannot be translated directly to fluorescence microscopy, as the spectral signature of the illumination is lost in the excitation-emission process at the heart of fluorescence. To overcome this limitation, we propose to use multiple co-localized fluorophore species with different emission spectra. The overview of the proposed method is presented in Figure 1.
Improving temporal or spatial resolution of microscopes has been tackled by many different approaches. We briefly review some that are of particular interest or use related concepts.
Several temporal super-resolution methods directly capitalize on strong prior knowledge of the signal itself, for example that sample motion is repetitive [5,6] or that the signal is sparse in some basis [7][8][9]. Other approaches do not require any prior knowledge on the signal. Bub et al. [10] used a modified camera with individually addressable pixels to stagger exposure times. This method fits well for fluorescence microscopy, but requires a modified camera, which may not be widely available. Wagnert et al. [11] presented a method to instantaneously acquire full volumes of isotropic resolution thanks to two light-field imagers. While limited to small fields of view and having a lower spatial resolution than light-sheet microscopy, their method shows great potential as whole volumes can be acquired at camera frame rate, despite requiring a dedicated setup.
Our method takes advantage of temporally patterned illumination. With illumination long known to be an essential factor to a microscope's performance (as shown by Köhler [12]), the field of structured illumination microscopy has demonstrated the possibility of improving spatial resolution beyond the limits of the acquisition device [13][14][15][16][17][18]. These methods mostly target spatial resolution improvements, often at the cost of temporal resolution, since they require multiple modulated images be merged into a higher spatial resolution image. Methods that specifically seek to improve temporal resolution by taking advantage of fast modulable illumination sources include the coded illumination method by Gorthi et al. [19], who perform motion deblurring for fluorescence microscopy using the fluttered shutter principle [20]. This approach is limited to linear motion deblurring. Yet other approaches, such as that of Dillavou et al. in [21] assume that the imaged sample is binary (either black or white) and use the dynamical range of a camera to drastically increase its frame rate. This strong assumption on the signal might, however, limit applicability to fluorescence microscopy.
Our method allows us to do either color imaging or fast imaging in fluorescence microscopy without the limitations of the aforementioned methods as it does not require multiple or other custom modified cameras, neither do we require the signal to be sparse, repetitive, binary, or following any particular (linear) motion.
The main contributions of the present paper are (i) the adaptation of the method in [3] to fluorescence microscopy; (ii) an improved temporal super-resolution reconstruction algorithm to compensate for slight model mismatches; (iii) demonstration of the applicability to light-sheet microscopy for biological applications.
The paper is organized as follows. In Section 2, we present the signal and imaging models and detail the assumptions on the image acquisition and the signal and derive the algorithm to perform temporal super-resolution. In Section 3, we demonstrate our method in practice by performing frame-rate doubling in light-sheet microscopy of the beating heart of a double-labeled, 2 days post-fertilization old zebrafish embryo. We finally discuss our experiments and conclude in Section 4.

Methods
Before we delve into the details of our proposed method, we briefly recall assumptions of the HESM method developed in [3] and the limitations that prevent it from being directly applicable to fluorescence microscopy. In [3], the light captured by the camera (Fig. 2 A, B) was assumed to be either reflected by the imaged scene or transmitted through a sample. This implied that the light spectrum measured by the camera was that of the illumination light source modulated by the spectral sample response. To follow a monochromatic object with higher temporal resolution, temporal variations could be encoded spectrally in the illumination pattern by using light sources of different hues. In the case of fluorescence microscopy with a single type of fluorophore ( Fig. 2 C, D), while the emitted light (which results from the fluorophores' relaxation following their excitation by the incoming light) has an overall intensity that depends on the excitation light via the molecules' spectral absorption properties, its spectrum only depends on the emission properties of the fluorophore but not on the spectrum of the excitation. Hence, the spectrum captured by the camera does no longer depend on the light source used, which precludes the possibility of using HESM in a way similar to the brightfield case. Our proposed way to extend HESM to fluorescence involves the use of multiple co-localized fluorophores (Fig. 2 E, F). Although each fluorescent species' individual emission spectrum is independent of the illumination spectrum, the fluorophores' combined emission spectrum depends on the spectral overlap of the illumination  [3] assumes a bright field microscopy setup, where the illumination light is transmitted through (or reflected by) a sample then captured by an RGB camera. The transmitted (or reflected) spectrum directly depends on the combination of the illumination spectrum and the sample's spectral transmittance. HESM relies on using a temporally-variable spectrum in the illumination, which can be decoded following capture with an RGB camera. (Middle row) If using a single fluorophore and two different illumination sources (C and D), the emission from the sample only changes in overall intensity, but the shape of the emission spectrum remains unchanged. Hence, temporal modulation of the illumination spectrum would not univocally modulate the spectral shape of the emission and HESM could therefore not be used as-is to improve the frame-rate. (Bottom row) In this paper, we propose to use two light sources E and F and a sample labeled with two co-localized fluorophore species such that the combined emitted light spectrum is directly dependent on temporal modulation patterns of the illumination spectrum, making the use of HESM to improve the frame rate possible.
with the absorption curve of each fluorophore. An added challenge for the fluorescence case is that fluorophores typically have broad and overlapping emission spectra, which makes it more difficult to isolate their contributions in distinct RGB channels as could be done with narrow-band illuminations in the case of transmission HESM. This overlap, therefore, requires an unmixing step.

Imaging model
The fluorescence imaging setup we consider is made of a color camera with C channels and L lasers of different wavelengths (Fig. 1). We further consider a dynamic sample x(t) labeled with F different, co-localized fluorophore species. We write the intensity measured in the c th channel of a single pixel, during the k th acquisition at time t k , as: where E is the exposure time of the camera, s (t) the time-varying intensity function of the th laser, b f , is the absorption cross section of the f th fluorophore to the th laser, γ c, f is the' sensitivity of the c th color channel to the f th fluorophore emission spectrum and d c is an offset from the sensor's electronics. Similarly to the model in [3], we assume that the temporal signal of interest lies in the space spanned by shifted B-splines of degree 0 , i.e.: where Q is the super-resolution factor (i.e. x(t) takes Q different values over the time of an exposure time E) and β 0 (t) is the indicator function over the interval [− 1 2 , 1 2 ]. We can rewrite Equation (1) as: where t k,i is the index for the values of x[·] and s [·] at time t k + E Q i. We adopt a matrix notation to write Eq. (3) for the full system: where Γ ∈ R LCQ×Q expresses both the response of the fluorophores to the excitation light (b f , ) and the sensitivity of the camera's sensor to the emitted fluorescent light (γ c, f ), S ∈ R C×LCQ is the matrix built by computing the integral in Eq. (3) and by shifted zero-padded vectors of time-functions for each laser, is the offset vector [22]. As a concrete example (which is the case we will consider in our experiments) we set C = 3, L = 2, and Q = 2. We choose to set the illumination temporal functions of the L = 2 lights as s 1 = [1, 0] and s 2 = [0, 1]. That is, each light source will be switched on for half of the exposure time, consecutively. In this setting, the matrix S is written as S = S 1 S 2 and (see [3] for further details): We also framed in green, respectively blue, the part that corresponds to the green, respectively blue, laser in the Γ matrix which expresses the efficiency of each fluorophore to the lasers combined with the sensitivity of the camera to the fluorescence emission spectra.
is the matrix expressing the first light source's temporal activation pattern. Similarly, is the matrix expressing the second light source's temporal activation pattern. We recall that with Γ , = 1, 2, the spectral mixing matrix for the th light source: where I 2 is the identity matrix of size 2 × 2. We recall that γ c, f is the spectral impact of the f th fluorophore on the c th channel, and that b f , is the response of the f th fluorophore to the th light. Thus, (γ 2,1 b 1,1 + γ 2,2 b 2,1 ) is the compound contributions from both fluorophores reacting to the first illumination source as captured by the camera's second channel. Figure 3 further illustrates these quantities and displays the values corresponding to the experimental setting in Section 3.
for every acquired frame, where d 0 and Γ were obtained via calibration. We solved Eq. (8) for each pixel and each acquired frame independently, without any dependency between consecutive frames. While this approach performs well as long as the hue is homogeneous over the imaged scene, we observed that fluorophore labeling can be inhomogeneous, as illustrated in Figure 5.
To make the method in [3] more robust to model mismatches, we propose to add a regularization term over time which introduces a dependency over consecutive frames (temporal dependency). Given the model presented in Eq. (4), the vector d 0 and matrix Γ obtained through calibration [3] and a sequence of M acquired color pixels y = y 0 , y 1 , . . . , y M−1 ∈ R MC×1 , we want to reconstruct the temporal high-resolution signal x = x 0 , x 1 , . . . , x M−1 ∈ R MQ×1 with a temporal super-resolution factor of Q. We propose to solve where T ∈ R M Q×M Q is a first-order derivative matrix, λ ∈ R + is a free parameter and d ∈ R MC×1 is d 0 stacked M times. The matrix A ∈ R MC×MQ is a block-diagonal matrix built by repeating SΓ along the diagonal M times. The regularization is known as Total Variation [24] and promotes piecewise-constant solutions, which is natural in time series of images as it preserves sharp edges. Equation (9) has the generalized LASSO form [25] and can be solved with the Alternating Direction Method of Multipliers (ADMM) [23,26], an efficient choice among other possible methods [25,27,28]. The ADMM considers a problem equivalent to that in Eq. (9), and builds the augmented Lagrangian defined as where ρ > 0 is a penalty parameter, u is a primal variable (as well as x) and z is a dual variable, also called Lagrangian multiplier. The ADMM algorithm consists of iterative steps in which the gradient of the augmented Lagrangian is set to zero for the primal variables [23]. At each iteration k, we alternate between solving a least-squares problem with fixed regularization weights and updating the regularization parameters (Lagrange multipliers). The global regularization weight λ, which balances the relative importance of the data and regularisation terms, is fixed during the entire process (we set its value empirically but it could be set using an L-curve method [29,30]). The iterations stop when a maximal number of iterations k max has been reached or when the difference between consecutive solutions for x k is lower than a given value ∈ R + . Algorithm 1 provides the corresponding pseudo-code in our case.

Experiments
We used our implementation of a light sheet microscope (based on OpenSPIM [31]), with two lasers (Stradus, Vortran Laser Technology) of wavelengths 488nm and 561nm to generate the excitation illumination light sheet. A four-dimensional stage (USB 4D-STAGE, Picard) held the sample. The detection axis consisted of a 20×/0.5 water immersion microscopy lens (UMPLFLN 20×W, Olympus) coupled to a 180 mm tube-lens (U-TLU-1-2, Olympus), terminated by a 1.2 mega pixels CMOS color camera equipped with an RGGB Bayer filter (DCC3240C, Thorlabs Instruments). The lasers and camera were electronically synchronized using a micro-controller (Arduino Due, Arduino, Italy). The micro-controller monitored the flash output of the camera and controlled the lasers using their ON/OFF electrical input, similarly to the electronics scheme provided in [32].
We conducted experiments on more than one fish sample, and there were differences in fluorescent proteins expression between different samples. In order to have acceptable SNR in the acquired images, we adjusted the exposure time of the camera, hence different frame-rates in the following experiments.

Temporal super-resolution light-sheet imaging of the beating heart of a zebrafish
To illustrate the potential of our method, we tested whether it was applicable to imaging the beating heart of a zebrafish embryo. To that end, we chose zebrafish that co-express ubiquitous cytoplasmic green fluorescent protein, EGFP, and red fluorescent protein, mCherry [33] (two co-localized fluorophores, F = 2). We embedded the embryos on a glass bottom dish in 1% low melting agarose and acquired 30 frames (M = 30) with our RGB camera (C = 3); setting the super-resolution factor to Q = 2, the two lasers (L = 2) were controlled so that they were each switched on for half of the exposure time (consecutively), so that we had s 1 [i] = [1, 0] and s 2 [i] = [0, 1], i ∈ {0, 1}. Figure 4 shows two consecutively acquired frames ( Fig. 4(a),(e)), all color channels of the acquisitions (Fig. 4(b)-(d), (g)-(h)), and the corresponding four reconstructed frames ( Fig. 4(i)-(l)). Note that, as discussed in Fig. 3, the blue layer ( Fig. 4(d),(h)) contains very little energy. Our method was therefore capable of doubling the frame rate. Here, we had F = 2, which limited the frame rate increase to Q = 2, but with additional fluorophore species and acquisition channels, higher Q could be considered.
The matrix Γ and the vector d were obtained with the calibration procedure in [3], using the code available online [32]. The area used for calibration is shown in Figure 5(d). Figure 3 illustrates how the matrix A = SΓ is built. We observe that the green laser has a high impact (first value, 0.7) on the red channel of the camera, which is due to the green laser mostly exciting the red fluorophores and the ensuing red fluorescent light will mostly be captured on the red channel of the camera. Similarly, the blue laser has a high impact on the green channel of the camera (value of 0.8), as it mostly excites the green fluorophores. Finally, note that none of the lasers really impact the blue channel, since there are no blue fluorophores. The values displayed on Figure 3 are the ones obtained after calibration and were used for the following experiments. (i)-(l) reconstructed frames using Algorithm 1 with λ = 0.001. The super-resolution factor is 2, hence for two acquired frames, we observe 4 reconstructed frames. Note that there are 60 milliseconds between the two acquired frames (a) and (e) and 30 milliseconds between the super-resolved region in frames (i)-(l), framed in a white rectangle. Outside of this region, our method allows keeping the color information. On (j), we marked two regions with drastically different colors, green ( ) and orange (♣). Standard imaging, in the presence of a single fluorophore species, would only produce frames (c) and (g), respectively (b) and (f) using samples labeled with only green fluorescent proteins, respectively only red fluorescent proteins. With Fluo-HESM, there are twice the number of these frames. This is also visible in Visualization 4, which shows a comparison between standard light-sheet data and reconstructions from our method in 3D+time. To highlight the link between this data and the schematic of Figure 1, (a) and (e) on this figure correspond to Fig. 1 (b) and (i)-(l) to Fig. 1 (c). See Visualization 1 and Visualization 2 for the full movie. Fish orientation indicated: anterior (A), posterior (P), right (R) and left (L). Scale bar: 100 µm.

Homogeneity of the fluorescent co-expression ratio
Our method requires that the imaged sample shows consistent fluorescent co-expression. We set out to make sure that the (static) calibration area and the (dynamic) heart area have indeed homogeneous and matching green and red fluorescent protein expression levels. To that end, we computed the red over green ratio (R/G) for each pixel in the images we acquired to compute the calibration, displayed in Fig. 5(b). We then computed the R/G histograms over various areas to compare them (Fig. 5(c'), (d') and (e')). Figure 5 shows an acquired color image where the lasers were temporally controlled as explained in Figure 3. The ventricle (V), atrium (At) and pericardium (p) are outlined. In the pericardium in Fig. 5(a), some patches appear green, while the heart appears orange. This is better seen in Fig. 5(b), comparing (c), (d) and (e) and their corresponding histograms on (c'), (d') and (e'). Inhomogeneities in the co-expression of red and green fluorescent proteins are visible over the sample (for example Fig. 5(e) and (e') compared to Fig. 5(d) and (d')). Given this expression variability (which is common from individual to individual or within the sample), it is thus important to choose a motionless calibration area (Fig. 5(d) and (d')) that exhibits similar R/G ratio than that of the dynamic phenomenon of interest, like the beating heart (Fig. 5(c) and (c')). Even with a careful choice, slight differences in the histogram remain (compare (c') and (d')). This justifies the use of an algorithm that is sufficiently robust to mismatches than a plain least-squares method without regularization. We further investigated this aspect in Section 3.4.

Combined fast and color imaging on the beating heart of a zebrafish
We applied Algorithm 1 only in the region comprising the beating heart. In this way, we could maintain the color information in static areas and do fast imaging where needed. Since we require the imaged sample (in this case, the heart) to be homogeneously labeled for temporal superresolution, after Algorithm 1 we can virtually assign the (single) hue back to the reconstructed images. Figure 6 shows two frames where we have temporal super-resolution reconstructed frames only in a region of interest (Fig. 6(c)) and where the acquired hue is preserved in static areas ( Fig. 6(a) and (b)). Fig. 6. Fluo-HESM is robust to fluorescence labeling inhomogeneities thanks to the temporal 1 regularization. (a)-(b) Composite color images and temporal super-resolution reconstructions. The dynamic area of the scene (c), with the beating heart, has been manually selected and inside of (c), the frame rate is twice that of outside. This illustrates the ability to choose between fast single hue imaging and slow color imaging. (d) Reconstructions at the location shown on (c) with a heart symbol ( ) using least-squares (without regularization, LSTSQ) or ADMM. The TV regularization we used in Eq. (9) favors piecewise constant reconstructions and preserves sharp edges. (e) Reconstructions at the location shown on (c) with a club symbol (♣) using LSTSQ or ADMM. At this location, there is little motion of the sample, and the LSTSQ reconstructions oscillate a lot, while the ADMM reconstructions show little motion, justifying the use of ADMM over LSTSQ. Within (c), we assigned a hue to the reconstructed images, that we measured at calibration. Scale bar: 100µm. See visualization 2 and additional Visualization 3 on a different sample.

ADMM compared to least-squares without regularization
As observed in Section 3.2 and Figure 5, there can be inhomogeneities in the co-expression ratios of the fluorescent proteins over the sample. We investigated whether the algorithm we introduced in Section 2 (ADMM) is more robust to these variations than the least-squares method (LSTSQ) without regularization of [3]. To that end, we compared reconstructions obtained using either ADMM or LSTSQ with the same input data. Figure 6(d) shows a comparison of both reconstructions at the same location, where the heart beats. The cardiac period is visible in both plots yet the ADMM reconstruction is smoother with sharp transitions maintained, unlike the LSTSQ reconstruction. This behavior is to be expected with the Total-Variation regularization. In Figure 6(e), we compared ADMM and LSTSQ reconstruction at a location where the imaged sample is mostly static. There the reconstructions is expected to be nearly constant, which is indeed the case for the ADMM reconstruction, while the LSTSQ reconstructions oscillates, most likely because of a slight image model mismatch.
For the ADMM reconstruction, the processing time for 45 time-points on a single pixel (i.e. 90 reconstructed time-points) was ≈ 1.7 milliseconds on a MacBook Pro 2018 (2.9 GHz 6-Core Intel Core i9, 32 Gb RAM). The code is in standard Python 3. We believe that with GPU programming, separating the processing of each individual pixel, and some tweaking, the code could run in real-time.

Impact on the reconstruction of 3D volumes of the beating heart
Light-sheet microscopy is a well-established tool for developmental studies thanks to its high spatio-temporal resolution and its low level of induced optical damage to the sample [34]. Furthermore, it has good sectioning capabilities, which we took advantage of to produce a dynamic 3D+time volume series (XYZ + time). We acquired movies of optical sections at different depths within the sample, applied our temporal super-resolution method to each individual movie, then temporally registered consecutive (in z) movies for the heartbeat to be synchronous in all slices. For the temporal registration, we used a dynamic programming method [35]. The method finds the best warping function of a temporal series onto another one by maximizing the sum of normalized mutual information between all frames of both time-series, which has already been shown to be well-suited for cardiac images [36][37][38]. Visualization 4 shows a 3D+time rendering that highlights the raw data where motion is encoded within the color channels and the doubled frame rate reconstruction.

Roadmap to higher super-resolution factors
In this paper, we presented a method to double the frame-rate of imaging in fluorescence microscopy. According to our model, the temporal super-resolution factor is determined by the rank and conditioning number of matrix A in Eq. (9). The rank is upper-bounded by both C, F, and L (respectively the number of color channels, the number of co-localized fluorophores and the number of lasers at our disposal), so to have a super-resolution factor of, say, Q = 3, we would need at least three color channels, three co-localized fluorophores and three lasers.
Also, the conditioning number of the matrix A provides a means to predict the quality of the reconstructions, as we have illustrated in [3]. If the chosen fluorophores have too close light emission spectra, this conditioning number will increase, hence lowering the quality of the reconstruction. In [1], Shaner et al. provide insights on how to chose fluorescent proteins and optical filters for maximal spectral separation, which improves the conditioning of our system. A recommended setup of [1] would have so little spectral cross-talk that we could have Q = 3 with close to a perfect conditioning number, hence good quality reconstructions at three times the frame-rate. This setup recommends using three bandpass detection filters centered at 480 nm (bandpass width 40 nm), 575 nm (bandpass width 25 nm) and 675 nm (bandpass width 130 nm) and the following three co-localized fluorophores: Cerulean (center wavelength of emission 475 nm), mOrange (center wavelength of emission 562 nm) and mPlum (center wavelength of emission 649 nm).

Discussion and Conclusion
In this paper, we investigated whether a hue-encoded shutter method (HESM) to improve the temporal resolution [3] could be applied to fluorescence microscopy. We showed that despite the physical differences between fluorescence (where the illumination spectrum cannot be inferred from the emission spectrum) and light absorption or reflection properties (which directly modulates the illumination spectrum to produce the transmitted or reflected light), HESM can still be used in fluorescence microscopy, provided one can use temporally modulated illumination, a color camera, and samples labeled with co-localized fluorophores in the dynamic regions where a frame rate boost is desired. We showed imaging of the beating heart of a zebrafish labeled with co-localized red and green fluorescent proteins at twice the acquisition frame rate on a light-sheet microscope, with the ability to preserve the hue-information in static areas. Although we experimentally showed an increase of the frame rate by a factor of 2, our method is general enough that it should allow for higher factors if additional co-localized fluorescent species, lasers, and color channels are available as discussed in Section 3.6.
Our use of a low SNR RGB camera leads to the performance of the current implementation of our method to be below that of the highest frame-rates achievable with dedicated high-sensitivity cameras (100-200 fps) built into some light-sheet microscope implementations, we foresee that the use of alternative imaging components (light splitting and multiple high-speed cameras) could lead to even higher frame rates. Also, alternating between the excitation of multiple populations of fluorophores might be beneficial as it allows for additional recovery time between excitations or lower excitation intensities at each wavelength. However, since this effect will likely depend on the specificities of each use case (in vivo imaging, labeling procedure, choice and proximity of fluorophores) further studies will be required to fully characterize the benefits of using multiple populations of fluorophores over simply increasing the concentration of a single one (when such a concentration increase is biologically feasible).
A key element that we introduced in this paper for the HESM approach to be sufficiently robust in practice, is the temporal regularization, which can cope with sample labeling inhomogeneities and low signal quality inherent to fluorescence imaging, as shown in Figure 6 (d). We further observed that two aspects are particularly important when selecting a fluorescent sample for HESM: (1) the co-localized fluorescent labels should be as homogeneous as possible in the dynamic regions and (2) a static area with a labeling ratio sufficiently similar to the dynamic region is required to calibrate the system. This means that a sample where only moving cells are labeled would not be suitable for our method, as the calibration could not be performed.
In Section 3.3, we illustrated the versatility afforded by our method (either slow color imaging or fast monochrome imaging), by applying it to a region of interest surrounding the beating heart of a zebrafish, while preserving the acquired color images outside of the region. Furthermore, the added possibility of using fluorescence opens the use of HESM to sectioning methods that rely on fluorescence (such as light-sheet microscopy). We illustrated the benefits of a doubled frame rate in the reconstruction of the beating heart in four dimensions (3D + t).
Our method is generic enough that it could potentially speed-up other fluorescence microscopy methods, provided samples with co-localized fluorophores can be generated and multiple lasers can be modulated in time at high-speed, in synchrony with the color camera.