Temporal super-resolution microscopy using a hue-encoded shutter

: Limited time-resolution in microscopy is an obstacle to many biological studies. Despite recent advances in hardware, digital cameras have limited operation modes that constrain frame-rate, integration time, and color sensing patterns. In this paper, we propose an approach to extend the temporal resolution of a conventional digital color camera by leveraging a multi-color illumination source. Our method allows for the imaging of single-hue objects at an increased frame-rate by trading spectral for temporal information (while retaining the ability to measure base hue). It also allows rapid switching to standard RGB acquisition. We evaluated the feasibility and performance of our method via experiments with mobile resolution targets. We observed a time-resolution increase by a factor 2.8 with a three-fold increase in temporal sampling rate. We further illustrate the use of our method to image the beating heart of a zebraﬁsh larva, allowing the display of color or fast grayscale images. Our method is particularly well-suited to extend the capabilities of imaging systems where the ﬂexibility of rapidly switching between high frame rate and color imaging are necessary.


Introduction
Many biological processes are highly dynamic and a low time-resolution in microscopy seriously limits their study [1].Several recent developments in both illumination and detection technology allow pushing towards higher frame rates.Light emitting diodes (LEDs), which are bright yet emit little heat, are both cost-effective and reliable [2].Several open-source projects have made building custom microscopes increasingly accessible [2][3][4] and facilitated hardware control outside of standard operating modes [5,6].Despite these developments, many imaging setups remain constrained by the achievable frame-rate, integration time, and color sensing patterns, because digital cameras have limited operation modes.
In this paper, we propose to extend the temporal resolution of a conventional digital color camera (whose frames can be externally-triggered) by leveraging a multi-color LED illumination source and computational post-processing.Our method assumes that the observed object is of a single hue (such as obtained by use of a single stain or dye) and embeds time information into each acquired frame by spectrally encoding temporal light patterns that are then collected by a color camera.Following acquisition, the images undergo an unmixing procedure that increases the frame-rate and effective temporal resolution.Our approach is related or combines approaches leveraged for other imaging methods, which we briefly review below.
To increase the (temporal and spatial) resolution beyond what cameras can offer directly, several computational approaches have been proposed, often relying on multiple simultaneous observations of a signal, which are then fused to reconstruct a high-resolution version of the signal of interest [7][8][9][10].Despite the resolution gains, these methods require availability of multiple cameras, which can be hard to integrate in a standard microscopy setup, or which may not be compatible in low photon count situations.Other methods make assumptions on the signal structure itself, for instance assuming the signal has a sparse representation in a function basis [11][12][13] or relying on the repeatable nature of the imaged motion [14,15].Alternative approaches, which require no prior assumptions include a method by Bub et al. [16], who proposed using a modified camera, whose pixels have staggered exposure times, which allows for a flexible tradeoff between time and spatial resolution.This method offers great possibilities for microscopy, yet with the drawback that it requires a modified camera, requiring low-level hardware control.
Controlled illumination is a core aspect of any optical microscope's performance, as demonstrated by Köhler over a century ago [17].In particular, structured illumination has been proposed as a way to access high-frequency components of the object via multiple modulations [18][19][20][21][22].The modulated signals are combined computationally, and numerical methods have focused on aspects such as taking into account experimental artefact [23], performing structured illumination without precise knowledge of the projected pattern [24], or lowering the number of required images [25].Our proposed method leverages ideas from structured illumination, albeit in the temporal domain.Our method also relies on unmixing spectrally encoded signals, which bears similarities with multi-spectral unmixing in fluorescence microscopy [26].
In order to improve temporal resolution and reduce motion blur, several methods have been proposed that take advantage of the availability of controllable illumination sources driven by rapid controllers that work in synchrony with the camera, followed by computational post-processing.For example, Staudt et al. [27] used short light pulses (stroboscopy) to reduce motion blur (while remaining limited by the camera framerate) when imaging the beating heart.Gorthi et al. [28] proposed a method for linear motion deblurring for fluorescence microscopy based on the fluttered shutter principle [29] by using a pseudo-random temporal illumination sequence, allowing to reduce motion blur by a factor of 50.This improvement is, however, only possible in the case of linear motions, which are common in cytometry, yet might not be applicable to more general biological motions.
Other active illumination methods have also been proposed in fields other than microscopy.Shiba et al. in [30] used an active illumination to project six dense dot patterns during an image acquisition duration and recover both depth and speed of elements in the imaged scene through computation.Rangarajan et al. in [31] presented active computational imaging methods to do spatial super-resolution as well as depth-estimation, via the projection of space-varying illumination patterns.
The main contributions of the present paper are: 1. a procedure to encode temporal details by illuminating each frame with colored light patterns; 2. a reconstruction method to achieve temporal superresolution based on the encoded measurements; 3. the characterization of our method's performance and its robustness both on synthetic and experimental data.
The paper is organized as follows: in Section 2 we present the signal and imaging models and detail the assumptions on the image acquisition and the signal.In Section 3, we derive our superresolution method and present a color calibration procedure to adjust the free parameters of our method and retrieve base hues.In Section 4 we characterize our method in terms of resolution gain and robustness and demonstrate its applicability for imaging biological samples in microscopy in Section 5. We discuss these results in Section 6 before concluding in Section 7.

Imaging model
We consider an imaging system consisting of L co-located illumination light sources and a color camera with C color channels.We assume that the camera has a global shutter, meaning that each pixel collects light over the same, fixed interval of time.We further consider that each light source has a fixed spectrum while the overall intensity can be varied over the duration of the camera shutter time.The timing of the illumination is linked to the camera.The imaged scene is assumed to be of a single hue and the optical parameters are assumed to be constant over the field of view.Figure 1 schematically depicts an example arrangement for three illumination sources and a color camera with a Bayer pattern.
Our method operates on each pixel and each frame independently.We can therefore proceed with the derivation of our method by considering a single color pixel, denoted by the vector Y = (Y 1 , . . ., Y C ) , whose C color components can be modeled as: (1) where x(t), t ∈ [0, E) is the imaged time signal (which we wish to recover) at the location in the scene corresponding to the pixel, E is the exposure duration, s (t) ∈ R + 0 is the intensity function of the th active light over time, D c ∈ R + 0 is an electronic bias for channel c and γ ,c ∈ R + 0 is the spectral impact of the th light source on channel c.Within the duration of one movie frame, we model the imaged signal x(t) as a piecewise constant signal: with Q an integer number of steps over the exposure time, x[i], i = 1, . . ., Q, the values of x(t) at each step, and the causal B-spline of degree 0 (box function).Given this model for the signal x(t), Eq. ( 1) can be rewritten as: with the average light intensity S [i] in the i th sub-frame interval defined as: With these notations, we can rewrite Eq. ( 5) in matrix form: vector, and S Q contains the time coefficients of the L lights: with: The matrix Γ is built as: with: where I Q is the identity matrix of size Q × Q and 0 m×n a matrix with m rows and n columns of zeros (for clarity, we have indicated the dimensions of certain matrices as subscripts in a similar fashion).

Temporal super-resolution
The super-resolution problem is equivalent to retrieving the signal x from a single color pixel Y by solving Eq. (7).When the number of channels C is at least equal to the super-resolution factor Q, we propose to obtain approximate solutions in the least-squares sense by solving the minimization problem (under the assumption that the data is corrupted by additive white noise) When Q ≤ C and Q ≤ L, this minimization problem can be solved efficiently with a number of numerical methods (e.g.see Chapter 5.3 in [32], p. 236).

Determination of the system spectral mixing coefficients and electronics offsets
In order to retrieve x in Eq. ( 12), given the measured color pixel Y and the user-controlled illumination pattern S Q , the coefficients in matrix Γ Q and in the bias vector D must be known beforehand.We propose to determine these coefficients via a calibration procedure in which we image a static scene with a series of fixed illumination patterns that combine contributions from one or several LEDs.The static scene is illuminated with P static intensity combinations of the LEDs.These patterns are fully specified by the operator, who can choose which lights to turn on or off and who can manually select an area, comprising M pixels, on which to calibrate the system.We first consider a single pixel with a given illumination pattern and set Q = 1 in Eq. ( 7) to obtain: which we rearrange as: where I C is the identity matrix of size C × C. Then we combine similar equations for M pixels and P illumination patterns to form the full calibration matrix: where the x m are the intensity of the m th pixel of all M static calibration pixels, S p 1 is the p th calibration illumination pattern and Y m,p is the measurement vector on pixel m for illumination pattern p.With this setup, all involved quantities in Y cal and A cal are known, either measured or user-imposed.Note that the expression in Eq. ( 14) involves Γ 1 , rather than Γ Q , yet even if the dimensions and structure of Γ Q depend on Q, its free parameters, the γ c, , are independent of Q, which allows their inference from Γ 1 .
Given these definitions and measurements, we solve for Γ 1 and D in Eq. ( 14) to minimize the 1 -norm cost: We find the solution to this cost minimization problem by using an Iteratively Reweighted Least-Squares (IRLS) method [33].IRLS proceeds by solving, at each iteration, a weighted least-square problem: where MP ) is a diagonal weighting matrix, whose entries w (t+1) k are updated at each iteration t + 1 [34,35]: The weights are initialized with w (0) k = 1 and (0) = 1.We follow an acceleration method similar to that proposed by Chartrand and Yin [34], where the variable damping factor (t) is divided by 10 each time the relative change of the 1 -norm of the residual is smaller than √ (t) /100, until the residual converges or (t) reaches a set minimum value (10 −6 ).Once convergence is attained, we retrieve the values of Γ 1 and D from u (t final ) .In practice, solutions we obtained with this approach were identical to those obtained by use of an exact linear programing method (CPLEX [36]).We favored our implementation for its simplicity and the possibility to make it available [37].
Although a similar approach could be used for minimizing Eq. ( 12) in order to retrieve the data, we found that for our applications, the least-squares approach, which is direct rather than iterative, is sufficient.The 1 norm is more robust to mismatches between the affine response model and the actual measurements.Mismatches may be due to, for example, low photon count (in dark regions), saturated pixels, or a nonlinear detector response curve.Since good calibration has a strong influence on the reconstruction quality and can be carried out offline, we favored the 1 -norm in Eq. ( 16) over least-squares, despite it being slower to mimimize.We note that other robust norms, for which efficient algorithms exist, could be used.

Base-hue recovery and hue-dependent model-selection for non-gray samples
Although our method trades spectral information to gain temporal resolution, we can leverage our use of a color camera to collect the hue of the imaged sample during the calibration procedure of Section 3.2 (using white illumination), and assign the measured and normalized RGB triplet to build a color pixel x [i](R G B) from the monochromatic, temporally super-resolved reconstruction x obtained with our method described in Section 3.1.
Furthermore, if the scene to be imaged is made of moving objects of any one hue among N possible hues, we can recover super-resolved images as follows.We first calibrate the system according to Sec. 3.2 for each one of the possible hues (indexed by n = 0, . . ., N − 1), hence obtaining N parameter sets Γ (n) Q , D (n) and base hue triplets (R (n) , G (n) , B (n) ).After acquiring images of a moving object (whose type or hue index n is unknown), we apply our temporal super-resolution method using each model Γ (n)  Q , D (n) in turn (e.g. on a manually selected region of interest (ROI)).We then evaluate the quality of the reconstructions x ,(n) = as a measure of smoothness.The rationale behind this criterion is that only correct model parameters will reduce flicker in regions and time-intervals where the scene is static (which we assume are present in the scene), hence decreasing R (n) .

Hardware and parameters setup
We implemented the illumination with commonly available and cost effective hardware.We assembled a light source using a 6-LED chip (SLS Lighting RGBWA+UV, Aliexpress, China).The LEDs have hues red (λ ≈ 620nm), green (λ ≈ 525nm), blue (λ ≈ 465nm), amber (λ ≈ 595nm), white (broad spectrum via fluorescence), and ultra-violet (λ ≈ 395nm).We drove the LED via a micro-controller (Arduino Uno, Arduino, Italy), which we programmed to generate the illumination time-pattern shown on Fig. 1(b), individually controlling each color.For the LED-camera synchronization, the micro-controller monitored the flash trigger output of the camera.Whenever the trigger signal transitions from low to high state, the micro-controller starts the time-sequence of the LEDs for the frame about to be recorded.The LEDs were directly powered by the controller's outputs, without additional power amplification of the signal.
We used a CMOS color camera (Thorlabs DCC3240C, Thorlabs, Germany) with 1280 × 1024 pixels, each with a standard RGGB-Bayer filter pattern (C = 3).We used this camera both for imaging macroscopic objects, in which case we used a 12mm focal length camera objective (Navitar NMV-12M1, HR F1.4/12mm), and for microscopic samples, in which case we attached the camera to the camera port of a custom-built wide-field transmission microscope consisting of a 20× Olympus water dipping lens (Olympus Plan Fluorite UMPLFLN 20xW) combined with a 180mm tube lens (Olympus U-TLU-1-2).
We either used the LED source as-is, when illuminating macroscopic scenes, or placed it into the illumination port of the microscope, which we adjusted for Köhler illumination (transmission).
In all experiments presented here, we adjusted the exposure of our camera to E = 60 milliseconds, the target over-sampling factor to Q = 3, and three LEDs per experiment, hence For the validation experiment in Section 4.2 and the beating heart data acquisition of Section 5.2, we used the red, green, and blue LEDs.For the robustness characterization experiment in Section 4.3, we used all available LEDs alternatively, by set of three.In all of these experiments, the illumination code sequences, S [i], i ∈ {0, 1, 2} corresponding to Eq. ( 6) were: with s 1 (t) the time-function of the first, s 2 (t) the second, and s 3 (t) the third LED, respectively.In Section 4.4, we investigate various illumination sequences that are specified in Table 2.
For all experiments, to calibrate D and Γ 1 , we acquired P = 30 images (≈ 3 calibration images per channel and per LED) of a static binary patterned sample, each with one of P different combinations of LEDs that were turned on or off (see Section 3.2).

Resolution improvement characterization
To quantitate the resolution improvement achievable by our method, we moved a test target (USAF resolution pattern) printed on a white cardboard paper and imaged it either: (i) with steady white light illumination; (ii) with strobed white light (one 20 ms pulse per frame); and (iii) with our proposed HESM method, followed by reconstruction.
In order to replicate the same motion in each case and thereby to allow for direct comparison, we used a robotic arm (Baxter, RethinkRobotics, Boston, MA, USA) to carry out the motion.
Under constant white light illumination (Fig. 2(a)), the resolution bars of the test target are blurred since the shutter remains open while the test target moves.With a single white light pulse per frame (Fig. 2(b)) the bars are sharp but only one image per camera frame is available.Using our method (Fig. 2(c)) we observe both sharp bars (comparable to what can be obtained with the strobed white light) and an increase in the frame-rate by a factor of three.We determined the finest resolvable resolution bar triplet in both the images obtained under white light illumination (0.25 line pairs/mm) and with our proposed HESM method (0.707 line pairs/mm).This corresponds to a 2.8-fold improvement in lateral resolution, which directly results from the improvement in temporal resolution of the same factor, given that the motion of the resolution target was uniform.

Characterization of robustness with choice of illumination hues
Our method allows, in principle, for an arbitrary choice of LED wavelength spectra for the different LEDs.In practice, however, selecting appropriate wavelengths for the LEDs given the type of imaged sample is essential to ensure the stability of the reconstruction.To illustrate this point, we explored different combinations of colors, chosen among the 6 individually addressable LEDs in our illumination head: red (R), green (G), blue (B), amber (A), white (W), Fig. 2. Imaging a moving sample with (a, f) a constant white light, (b, g) a 20ms white pulse and (c, d, e, h, i, j) our proposed method (see Visualization 1).The zoom on the element 1 of the group −2 of the USAF-grid (close-up in a, b, c) shows that all three methods can resolve it.It is the limit for the constant illumination.This element is 0.625 mm wide.The detailed views on the whole group −1 (f, g, h) show that the stroboscopic illumination and our method (g, h) are able to resolve up to element 4.This corresponds to a resolution improvement factor of 2.8.Moreover, with our method operating at the same frame-rate, we have six reconstructed frames (c, d, e, h, i, j) while with the two other methods we have two acquired frames (a, b, f, g), thus we improved the frame-rate by a factor of 3. and ultra-violet (UV).Specifically, we repeated the experiment of the moving target using our proposed imaging method with the following color combinations: R-G-B, A-G-UV, B-UV-W, or R-UV-B (each turned on in sequence).In order to characterize the robustness of the imaging system in each case, we calibrated the system then we calculated the conditioning number κ(A) (see Chapter 4.4 in [38], p.82) of the obtained system matrix A = S Q Γ Q : where σ max (A) and σ min (A) are the highest and lowest eigen-values of the matrix A. Table 1 gives the condition number κ for the 4 combinations of LEDs that we tested.See Visualization 2 for the corresponding videos.We observed that whenever the system matrix was poorly conditioned, whose likely cause we attribute to overlapping spectra of two (or three) lights in a given combination (e.g.blue and UV LEDs in the B-UW-W combination), the reconstruction was noisy and flickering.We think that there is crosstalk in the lights signal contribution, which translates into a poorly conditioned system matrix.We observed sharp reconstructions with little noise for the color combinations R-G-B and A-G-UV.The reconstructions with the two other color combinations B-UV-W and R-UV-B flickered and showed amplified noise.proposed imaging method with the following color combinations: R-G-B, A-G-UV, B-UV-W, or R-UV-B (each turned on in sequence).In order to characterize the robustness of the imaging system in each case, we calibrated the system then we calculated the conditioning number κ(A) (see Chapter 4.4 in [38], p.82) of the obtained system matrix A = S Q Γ Q : where σ max (A) and σ min (A) are the highest and lowest eigen-values of the matrix A.
Table 1 gives the condition number κ for the 4 combinations of LEDs that we tested.See Visualization 2 for the corresponding videos.We observed that whenever the system matrix was poorly conditioned, whose likely cause we attribute to overlapping spectra of two (or three) lights in a given combination (e.g.blue and UV LEDs in the B-UW-W combination), the reconstruction was noisy and flickering.We think that there is crosstalk in the lights signal contribution, which translates into a poorly conditioned system matrix.We observed sharp reconstructions with little noise for the color combinations R-G-B and A-G-UV.The reconstructions with the two other color combinations B-UV-W and R-UV-B flickered and showed amplified noise.

The conditioning number of the system matrix depends on the illumination functions
We next investigated the influence of the illumination functions on the quality of the reconstructions.
To that end, we performed an experiment similar to that in Section 4.3 but keeping a single set of LEDs (R-G-B) to image the same repeating motion, while varying the illumination functions.
We then compared the condition number of the system matrix corresponding to each illumination pattern with the quality of the reconstruction.Table 2 shows the illumination intensities of the LEDs in the sub-frame time intervals for the four different cases with the corresponding condition number of the system matrix.We observed good reconstructions when the system was well-conditioned.A comparative video is provided (Suppl.Video 2).

Model selection applied to two samples
To demonstrate our method's ability to recover both the hue of an object and a temporally super-resolved sequence, we imaged two paper cards, one whose hue was white and the other off-white.In both cases, our method could retrieve both a temporally-super-resolved image sequences as well as assign RGB values for the base-color, directly from the raw images (Figure 3  b,c).

The conditioning number of the system matrix depends on the illumination functions
We next investigated the influence of the illumination functions on the quality of the reconstructions.
To that end, we performed an experiment similar to that in Section 4.3 but keeping a single set of LEDs (R-G-B) to image the same repeating motion, while varying the illumination functions.
We then compared the condition number of the system matrix corresponding to each illumination pattern with the quality of the reconstruction.Table 2 shows the illumination intensities of the LEDs in the sub-frame time intervals for the four different cases with the corresponding condition number of the system matrix.We observed good reconstructions when the system was well-conditioned.A comparative video is provided (Visualization 2).

Model selection applied to two samples
To demonstrate our method's ability to recover both the hue of an object and a temporally super-resolved sequence, we imaged two paper cards, one whose hue was white and the other off-white.In both cases, our method could retrieve both a temporally-super-resolved image sequences as well as assign RGB values for the base-color, directly from the raw images (Fig. 3(b,  c)).

Fast imaging of the beating heart
To illustrate the applicability of our HESM method for biological microscopy, we imaged the beating heart of a live 4 days post fertilization (dpf) old zebrafish larva mounted in agarose gel, with a wide-field microscope under transmitted illumination.Zebrafish (wild-type AB zebrafish strain (Zebrafish International Resource Center) were raised under standard laboratory conditions (14/10 hour light/dark cycle, fish water of the system (ZEBTEC Techniplast Aquatic Solution) at 26.5 • C temperature, 500 µs conductivity, and pH 7.3) in a facility approved by the Veterinary Service of the State of Valais (Switzerland).Fertilized eggs were collected and the embryos raised at 29 • C in standard E3 medium in an incubator (Termaks B8054), supplemented by 0.003% 1-phenyl 2-thiourea (PTU) from 24 hours post fertilization (hpf) to prevent pigmentation.For imaging, we embedded 4 dpf larvae, anesthetized with 0.1% tricaine (ethyl 3-aminobenzoate methanesulfonate salt, Sigma), in low melting agarose.
Following raw image acquisition (see hardware and parameter setup, above), we selected an ROI over which we applied our method, keeping the rest of the images in color.Figure 4 shows a single frame of Visualization 5, where color imaging allows clearly visualizing cells within the heart wall of both the atrium and ventricle, which are blurred in the color images.Visualization 5 first shows only color imaging of the beating heart, then it shows reconstructions from our method within an ROI and finally a comparison side by side of standard color imaging and our method, both on the same ROI.Our method therefore offers the flexibility of either using RGB or fast monochrome imaging.

Discussion
The hardware implementation of our method has only few hard design requirements.In particular, frame acquisition and illumination must be in synchrony, which is straightforward to implement provided the camera has a trigger output.Given variability in hardware clocks and data transfer, independently running the illumination and acquisition systems results in rapid asynchrony and departure from the acquisition model (making it difficult to invert the system matrix).Beyond synchronization, given the frame rate of our camera (60 frames per second), neither the clock-time of our micro-controller (16 MHz) nor the rise time of our LEDs (20 ns) appeared to be limiting our method.
In order to calibrate our system as described in Section 3.2, it is necessary to manually select an ROI.We observed that the best results were obtained when calibration ROIs were chosen among a wide range of intensities.Our method requires a static sample (or scene) for calibration, ideally with a variety of intensities.Since acquiring calibration images is fast, many images of identical regions can be acquired fairly rapidly, which allows to limit the influence of noise, in particular in darker regions.We empirically found that acquiring P = 30 images gave good calibration results and used that number for all experiments in Section 4.
The results in Section 4.2 show a time-resolution improvement of a factor √ 8 ≈ 2.8 with a temporal sampling improvement by a factor 3. This factor depends on the number of channels available and using additional channels (e.g. through wavelength splitting and use of multiple cameras) higher resolution factors might be achievable.Less motion blur can be obtained by shortening the illumination pulses yet without improvement of the frame-rate, at the cost of a higher peak-intensity (which is sometimes undesirable in microscopy) and at the risk of producing aliasing.Longer-duration pulses are less prone to aliasing and might help preserve live samples as their peak intensity can be lower for a given camera integration time.
Although the model presented in Section 2. is non-specific regarding the precise illumination functions, the discrete formulation in Eq. ( 7) reveals that our model becomes ill-posed should the matrix S Q Γ Q not be full rank, i.e. if rank(S Q Γ Q ) < Q.This provides us with a tool for verifying that a proposed illumination function does not lead to an ill-posed system.For example, should data from a particular channel be missing, Q should be lowered such as to have Q ≤ C and the matrices S Q adapted accordingly.Similarly, the matrix Γ Q should be well-conditioned, which depends on the sample itself, the color-filters of the camera, and the spectrum of the lighting.The spectrum absorbed and reflected by the sample and then acquired by the camera should form a matrix of rank Q, when calibrating with the procedure in Section 3.2.With the condition number κ (Section 4.3), we have a means of predicting the quality of reconstructions with our method given the sample, the LEDs, and the chosen camera.The results in Section 4.4 suggest that a condition number above 10 should be avoided, in order to guarantee a good reconstruction.For a given matrix Γ Q (specified by the sample, LED, and detection spectra) the matrix S Q containing the temporal patterns could be optimized for S Q Γ Q to have maximal rank, an NP-hard problem which we have not pursued here We investigated the impact of the illumination hues in Section 4.3.Since the camera captures transmitted or reflected light, the same conclusions apply to changing the hue of the sample, rather than the illumination.
Simply using more lights would not necessarily produce a more stable system matrix.Instead, chasing the proper combination of lights does.A simple case for choosing the number of LEDs, channels, and super-resolution factor, is to set L = C = Q (of course one must still ensure that the choice of lights and illumination functions produces well-conditioned matrices, as discussed above).
In practice, temporal flickering may remain in the reconstructed videos, depending on how well the system's matrix S Q Γ Q matches dynamic conditions.Calibration and experimental conditions may differ, for example, because calibration is carried out in regions different than those imaged.Furthermore, LED rise and fall times during calibration and imaging may differ as the LEDs' electronic current drivers can be frequency-dependent For single-hue objects, the possibility of recovering the base hue simultaneously to reconstructing temporally super-resolved image sequences (as shown in Section 3.3) is particularly appealing as this advantage comes without requiring an increase in the bandwidth of the system.This capability is preserved even in the case of multiple objects with different hues, via the model selection scheme we proposed in Section 3.3.Furthermore, for applications in microscopy, where often only a single camera can be mounted, making rapid switching to a different camera or view-port unfeasible, our method brings clear practical advantages: (i) the same camera can be used both for color and fast imaging as demonstrated in Section 5.2 and (ii) the motion blur can be reduced when acquiring stacks in continuous scanning mode.While the former may be particularly attractive for building versatile imaging systems, the latter may be particularly relevant for applications that require fast inspection, such as for screening or flow cytometry.We also foresee that the improved frame-rates and temporal resolution could be beneficial for object tracking applications, a point that we may investigate in the future.
Since the wavelength of the photons emitted by a fluorophore are independent of the excitation wavelength, our method would not be applicable on samples labeled with a single fluorophore.However, our method could be applied for imaging structures simultaneously co-labeled with two or more fluorophores: while the individual emission spectra shape would remain unchanged (except for scaling), as their combined emission intensity (and therefore the resulting combined spectrum) will vary with the relative excitation intensities of the illumination sources our method could, in principle, provide similar benefits to fluorescence imaging.
We provide the code, data, and instructions to reproduce results in this paper [37].

Conclusion
We introduced a general computational imaging method to carry out temporal super-resolution with a color camera and a set of multi-spectral active illumination sources.Each frame includes multiple copies of the signal at various times, encoded in the hue of the image.The computational procedure retrieves a high time-resolution signal, along with the base-hue, under the assumption that the imaged sample has a single color.We showed a direct method to characterize the robustness of the method, depending on the sensing and illumination spectra, as well as the base-hue of the imaged sample.We experimentally showed a temporal resolution improvement of a factor 2.8 combined with a three-fold increase of the frame-rate.We illustrated our method with an application exhibiting both color imaging and fast grayscale (on a chosen ROI) of the beating heart, showing its applicability to bio-microscopy.

Fig. 1 .
Fig. 1.(a) Acquisition setup.The moving sample is imaged with three active light sources s i (t).The projection of the scene to the camera is shown with x(t).The color (Bayer) filter makes each pixel sensitive to a specific spectrum that is independent of the light sources.Each light source has its own time function, capturing the sample at different times and encoding this information in different spectra which is then captured by the color sensor in the hue domain.(b) Example of possible temporal functions for the three light sources.(c) Real example of acquired data with the depicted system of (a).(d)-(e) Close-ups to the real acquired data, the Bayer filter is visible.(f) Reconstruction of three grayscale frames from the acquisition shown in (c).

Table 2 .
Condition number κ with various time functions.R 1 , R 2 , R 3 are the values of the red LED respectively at the first, second and third time-steps of the whole exposure time (see Visualization 3).

Fig. 3 .
Fig. 3.When any of several objects with different, but known, hues enters the field of view, the system matrix adapted to the object can be automatically selected.(a) Color image of a static scene, with two kind of papers illuminated by a white light.The gray areas show the calibration ROIs.(b) Each sample has a corresponding calibrated set of parameters Γ and D as well as the sample hue.(c) Data acquisition of a dynamic scene with the active illumination.(d) Reconstruction with model selection as explained in Section 3.3.(e, f) Two reconstructions with our method after model selection, using RGB LEDs and reconstructing the hue of the samples from the raw data acquired with our method (see Visualization 4).Scalebar: 5 cm.

Fig. 4 .
Fig. 4. Flexible color and fast grayscale imaging of the beating heart in a 4 days post fertilization zebrafish larva.(a) Single frame of an RGB color movie, with (b) ROI with reconstructed grayscale (no hue was measured beforehand, gray reconstruction) movie at threefold increased frame-rate.See Visualization 5 for the full movie.Anatomical features visible include the ventricle (v), the atrium (at), the bulbus arteriosis (BA) and the pericardium (p).Orientation is indicated as V: ventral, D: dorsal, A: anterior, P: posterior.Scalebar: 100µm.

Table 1 :
Condition number κ depending on the LEDs used (see Visualization 2).