Hadamard-transform fluorescence-lifetime imaging

We discuss a Hadamard-transform-based fluorescence-lifetimeimaging (HT-FLI) technique for fluorescence-lifetime-imaging microscopy (FLIM). The HT-FLI uses a Fourier-transform phase-modulation fluorometer (FT-PMF) for fluorescence-lifetime measurements, where the modulation frequency of the excitation light is swept linearly in frequency from zero to a specific maximum during a fixed duration of time. Thereafter, fluorescence lifetimes are derived through Fourier transforms for the fluorescence and reference waveforms. The FT-PMF enables the analysis of multi-component samples simultaneously. HT imaging uses electronic exchange of HT illumination mask patterns, and a high-speed, highsensitivity photomultiplier, to eliminate frame-rate issues that accompany two-dimensional image detectors. 2016 Optical Society of America OCIS codes: (300.6280) Spectroscopy, fluorescence and luminescence; (180.2520) Fluorescence microscopy; (170.3650) Lifetime-based sensing. References and links 1. J. R. Lakowicz, Principles of fluorescence spectroscopy 3rd edition (Springer, 2006). 2. J. A. Lavitt, D. R. Matthews, S. M. Ameer-beg, and K. Suhling, “Fluorescence lifetime and polarizationresolved imaging in cell biology,” Curr. Opin. Biotechnol. 20, 28-36 (2009). 3. H. Wallrabe, and A. Periasamy, “Imaging protein molecules using FRET and FLIM microscopy,” Curr. Opin. Biotechnol. 16, 19-27 (2005). 4. T. W. J. Gadella Jr., T. M. Jovin, and R. M. Clegg, “Fluorescence lifetime imaging microscopy (FLIM): Spatial resolution of microstructures on the nanosecond time scale,” Biophys. Chem. 48, 221-239 (1993). 5. X. F. Wang, T. Uchida, and S. Minami, “A fluorescence lifetime distribution measurement system based on phase-resolved detection using an image dissector tube,” Appl. Spectrosc. 43, 840-845 (1989). 6. I. Bugiel, K. Koonig, and H. Wabnitz, “Investigation of cells by fluorescence laser scanning microscopy with subnanosecond time resolution,” Lasers Life Sci. 3 (1), 47-53 (1989). 7. A. Squire, and P. I. H. Bastiaens, “Three dimensional image restoration in fluorescence lifetime imaging microscopy,” J. Microsc. 193 (1), 36-49 (1999). 8. X. F. Wang, T. Uchida, D. M. Coleman, and S. Minami, “A two-dimensional fluorescence lifetime imaging system using a gated image intensifier,” Appl. Spectrosc. 45(3), 360-366 (1991). 9. D. Elson, J. Requejo-Isidro, I. Munro, F. Reavell, J. Siegel, K. Suhling, P. Tadrous, R. Benninger, P. Lanigan, J. McGinty, C. Talbot, B. Treanor, S. Webb, A. Sandison, A. Wallace, D. Davis, J. Lever, M. Neil, D. Phillips, G. Stamp, and P. French, “Time-domain fluorescence lifetime imaging applied to biological tissue,” Photochem. Photobiol. Sci. 3(8), 795-801 (2004). 10. J. R. Lakowicz, H. Szmacinski, K Nowaczyk, K. W. Berndt, and M. L. Johnson, "Fluorescence lifetime imaging," Anal. Biochem. 202, 316-330 (1992). 11. J. R. Lakowicz, and K. W. Berndt, "Lifetime-selective fluorescence imaging using an rf phase-sensitive camera," Rev. Sci. Instrum. 62(7), 1727-1734 (1991). 12. T. Iwata, H. Shibata, and T. Araki, "Construction of a Fourier-transform phase-modulation fluorometer," Meas. Sci. Technol. 16(11), 2351 (2005). 13. M. Harwit and N. L. Sloane, Hadamard transform optics (Academic Press, 1979). 14. R. D. Swift, R. B. Wattson, J. A. Decker, R. Paganetti, and M. Harwit, "Hadamard transform imager and imaging spectrometer," Appl. Opt. 15, 1595-1609 (1976). 15. Q. S. Hanley, P. J. Verveer, and T. M. Jovin, "Spectral imaging in a programmable array microscope by Hadamard transform fluorescence spectroscopy," Appl. Spectrosc. 53(1), 1-10 (1999). 16. K. Hassler, T. Anhut, and T. Lasser, "Time-resolved Hadamard fluorescence imaging," Appl. Opt. 44, 75647572 (2005). 17. D. Magde, R. Wong, and P. G. Seybold, "Fluorescence quantum yields and their relation to lifetimes of rhodamine 6G and fluorescein in nine solvents: Improved absolute standards for quantum yields," Photochem. Photobiol. 75(4), 327-334 (2002). 18. D. Magde, G. E. Rojas, and P. G. Seybold, "Solvent dependence of the fluorescence lifetimes of xanthene dyes," Photochem. Photobiol. 70(5), 737-744 (1999). 19. T. Mizuno, S. Nakao, Y. Mizutani, and T. Iwata, "Photon-counting 1.0 GHz-phase-modulation fluorometer," Rev. Sci. Instrum. 86(4), 043110 (2015). 20. P. Harms, J. Sipior, N. Ram, G. M. Carter, and G. Rao, "Low cost phase-modulation measurements of nanosecond fluorescence lifetimes using a lock-in amplifier," Rev. Sci. Instrum. 70(2), 1535-1539 (1999).


Introduction
Fluorescence lifetimes are critical parameters for discriminating between two samples that have similar excitation and emission spectra.Information concerning molecular dynamics is also obtainable from lifetimes [1,2].Thus, fluorescence-lifetime-imaging microscopy (FLIM) is widely used in biochemistry, biophysics, and analytical chemistry, and for observing living cells labeled by fluorescent probes [3,4].
FLIMs are categorized into two groups in terms of light detection.(i) A scanning FLIM uses a single-channel detector, such as a photomultiplier tube (PMT) or an avalanche photodiode; (ii) a wide-field FLIM uses a gated image intensifier and a two-dimensional (2-D) imaging sensor, such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor camera.In case (i), the sample stage or the focused excitation source is scanned mechanically, [5,6] while the fluorescence is detected by the single-channel detector.Weak fluorescence can be detected with a high spatial resolution; and with a confocal microscope, three-dimensional depth resolution can be obtained [7].However, mechanical scanning can be slow, making it difficult to image biological samples that rapidly photobleach.High-speed scanning with a galvanometer mirror device requires careful attention to mechanical vibrations.In case (ii), wide-field FLIM allows the whole imaging area to be acquired at the same time and the fluorescence can be time-resolved with the gated intensifier.However, the sensitivity of the imaging detector is usually lower than that of the singlechannel detector.
In FLIM, there are two methods for sample excitation: pulsed or sinusoidally modulated.With pulsed light, the fluorescence decay can be monitored by delaying the gate with respect to the excitation.However, the time resolution is determined by the pulse width of the gate, which is at least 5.0 ns.In addition, it takes a long time to acquire the whole decay waveform, and the signal-gathering efficiency is extremely low.One way to overcome this problem is to use several time gates for each excitation [8].However, this is difficult for multi-component samples because the fluorescence decays need to be single exponentials.Furthermore, the repetition rate of the excitation is limited by the frame rate of 2-D detectors.High-speed cameras are expensive [9] and are limited by their image size in pixels.
Phase modulation is an alternative FLIM technique that uses a sinusoidally modulated light source.Fluorescence lifetimes are estimated from the phase difference and/or the modulation ratio between reference and fluorescence waveforms.One efficient method is to prepare four time gates during one modulation period, each phase-shifted in turn by π/2 [10].Another uses a mode-locked laser and a gated, intensified CCD camera [11] to acquire "lifetime-selective" fluorescence images.The phase-modulation method is less damaging to biological samples than pulsed excitation.However, for multi-component samples, several modulation frequencies must be used.(In fact, plural modulation frequencies should be used to verify single-component samples.)Finally, it is difficult to detect short-lived fluorescent molecules that have low quantum efficiencies.
To address the above problems, we have developed a Hadamard-transform fluorescencelifetime imaging (HT-FLI) technique.It is a combination of a previously reported Fouriertransform phase-modulation fluorometer (FT-PMF) [12], and Hadamard-transform imaging (HTI) [13][14][15].The use of HTI for fluorescence-lifetime measurements has been proposed [16]; however, the emphasis was on multiplexing advantages.Although the signal-to-noise ratio (SNR) and the signal-gathering efficiency are increased, the question remains whether the multiplex advantage is retained when a quantum photon detector such as a PMT is used.Here, we combined the two methods with the goal of solving the sensitivity and time resolution problems.An electronically driven cyclic HT excitation pattern solves the low frame-rate problem of 2-D detectors [13].HT-FLI enables us to obtain fluorescence-lifetime images for every component in a sample.Although such an image could be obtained with a conventional FLIM, it is cumbersome for multi-component samples, and often results in averaged fluorescence-lifetime mapping [4][5][6][7][8][9][10].
The primary aim here is to discuss the HT-FLI concept, and demonstrate a proof-ofprinciple prototype.For convenience, the prototype has an 88-pixel, laboratory-made, lightemitting diode (LED) array as an excitation source, and an inline optical arrangement for fluorescence detection.Therefore, spatial resolution is determined by the 4.0-mm physical interval between neighboring LEDs.Because of the 50-MHz upper limit of the LED modulation frequency, the lower limit for fluorescence lifetimes is a few nanoseconds.Because those specifications are insufficient for a practical FLIM, we present guidelines for future FLIM construction.

FT-PMF and HTI
The HT-FLI system consists of the FT-PMF and the HTI units, which are described here.The former is used for the measurement of the fluorescence lifetime and the latter for the imaging.The combined HT-FLI system will be explained in the next section.

FT-PMF
FT-PMF has been discussed in detail elsewhere [12].Briefly, it is a phase-modulation fluorometer, where the modulation frequency of the excitation source is swept (chirped) linearly from f min to f max during a fixed time T, as shown in Fig. 1.Then, the excitation waveform has the amplitude and phase distribution in the frequency domain as depicted in the figure .The amplitude and phase of the resulting fluorescence waveform varies with the modulation frequency.Therefore, from Fourier transforms of the excitation (reference) and fluorescence waveforms, the modulation ratio , of the two waveforms as a function of the modulation frequency f yield the impulse response function of the fluorescent sample.A is the amplitude and  is the phase, and the subscripts em and ex denote emission and excitation, respectively.Data processing used in conventional phase modulation yields the fluorescence lifetimes.If necessary, we can obtain a fluorescent decay waveform by setting 0 min  f [12].FT-PMF can also be used for multi-component samples.For a data-sampling frequency f s , the number of data points L is given by s Tf L  , and the frequency resolution is given by  we assign the value 1 to light transmission and 0 to light blockage.Then, the elements of the first row and the first column of M become zero.The remaining elements are also replaced so that 1 and -1 in ' H become 0 and 1, respectively.Therefore, M is still cyclic.The original cyclic pattern in ' H is generated by quadratic residue construction described in [13].

HTI unit
. This is possible because all elements of the first row and the first column in M are zero, and the value of

Block diagram of the HT-FLI system
Figure 3 shows a schematic of the HT-FHI system.The excitation light is a laboratory-aligned n n , 525-nm LED array (OSTG1608C1A, OptoSupply), which is sequentially turned on and off by the illumination mask patterns.In our prototype,

 n
. The interval between neighboring LEDs was 4.0 mm and the whole pattern is imaged 1:1 on the sample plane by lens L1.Fluorescence emitted from the sample is focused by lens L2 onto a photomultiplier tube (PMT; R7400U, Hamamatsu Photonics) after passing through a long-wavelength pass filter LPF in a collinear, afocal configuration.The afocal system prevents non-uniformity of the illumination pattern at the cost of fluorescence intensity on the PMT.We acquire ) 64 ( 2   n N sets of time-series data sequentially and synchronized with every exchange of the illumination mask pattern. The timing diagram for acquisition of N sets of time-series data is shown in Fig. 3(a-c).  .We thus obtain a pair of k N  real and imaginary spectral matrices.Then, by performing the inverse HT procedure, we get real-and imaginary-image column vectors for the k frequencies, including zero frequency, from which we reconstruct k pairs of n n real and imaginary images, as shown in Fig. 3(e).If we obtain reference images in advance by a similar procedure, we can calculate the n n modulation ratio m and phase  images for the k frequencies, as shown in Fig.

3(f).
For a multi-component sample, the numerical fitting procedure for each pixel [e.g., point P in Fig. 3(f)] yields the modulation ratio and fluorescence lifetimes by the normal phasemodulation procedure.We can thereby derive "shading" images for the corresponding fluorescence lifetimes, where a shading image is essentially the same as the "lifetimeselective" fluorescence image [11].For example, if there are two components, the fluorescence decay waveform is represented by ) exp( ) exp( .Application of such procedures to all pixels yields 1 a and 2 a images.These are shading images that correspond to fluorescence lifetimes of 1  and 2  , respectively.We set the error for the lifetimes at % 0 .In the above data processing procedure, we picked up k frequencies before performing inverse HTs, which saves calculation time.The linearity of DFT allows such a procedure.Also, it should be noted that the shading images of the fluorescence lifetimes cannot be obtained until HT-FLI is employed.

LED array driver
Figure 4 shows a schematic of an n n LED array driver, which consists of a 64-bit cyclic shift register (CSR; Toshiba TC74HS194  16, 70 MHz maximum frequency) for cyclic shifting of the Hadamard illumination mask pattern, a clock pulse function generator for the shift operation, an initial Hadamard pattern-setting circuit, a chirped sinusoidal waveform generator (arbitrary waveform generator; Tektronix AWG520, 500 MHz maximum frequency; for every exchange of the pattern.The master clock for the pattern exchange is a function generator (WF1965, NF Corporation).The driver used for double modulation of each LED is a high-speed NAND gate (Toshiba Semiconductors TC74AC00FT, 25-mA/gate maximum sink current).Although the NAND gate changes the sinusoidal waveform to a two-valued rectangular waveform, it does not affect the entire operation.Each LED is driven by the synthesized wave at a 6.4-mA peak current.The timing diagram (A)-(E) in Fig. 4 displays the operation.

Preparation of fluorescent samples
To evaluate the HT-FLI system, we measured fluorescence lifetimes of two single-component samples: (i) μM 10 rhodamine 6G in ethanol (R6G); and (ii) μM 10 rhodamine B in ethanol (RB).The samples were contained in two separate 10×10×45-mm quartz cells aligned sideby-side, as shown in Fig. 5(a).This arrangement mimics a single-component sample with a fluorescence lifetime that varies spatially.We also prepared mixed solutions of R6G and RB with volume ratios of (iii) 3:1, (vi) 1:1, and (v) 1:3, respectively, that were aligned as shown in Fig. 5(b).Because of the 1.25-mm-thick quartz walls, there are two 2.5-mm-thick nonfluorescent barrier areas.

Results
Figure 6 shows results for the two-cell sample illustrated in Fig. 5(a).Figure 6(a) is a dc (f=0) fluorescence image obtained with the HT-FLI, which is the same as that obtained from the conventional HTI.Two columns in the center correspond to the air space and the cell walls.Also, as described above, the (1,1) image element has no information.Fig. 6(b,c) show the modulation ratio m and the phase difference  as a function of the modulation frequency f, calculated at positions X and Y, respectively, as marked in Fig. 6(a).Numerical fitting yields fluorescence lifetimes  1 =4.2 ns and  2 =2.5 ns for R6G and RB in ethanol, respectively.Figure 6(d) shows the initial amplitude ratio a 2 /a 1 for cells (i) and (ii), calculated for each pixel assuming single-exponential fluorescence decays.From the image in Fig. 6(a), we can derive an averaged fluorescence shading image for a 1 and a 2 , as shown in Fig. 6(e,f), respectively, corresponding to 1  =4.2  0.21 ns and 2  =2.5  0.13 ns.Region (i) in Fig. 6(e) shows the fluorescence-lifetime image for R6G, while region (ii) has no signal.Region (ii) in Fig. 6(f) shows the fluorescence-lifetime image for RB, while region (i) has no signal.The degree of shading in the two images indicates the spatial fluctuation of the initial amplitude.
The two averaged lifetimes were in good agreement with that reported in literatures [17,18].Figure 7 shows the results for the multi-component cells shown in Fig. 5(b).Figure 7(a) shows the dc (f=0) fluorescence intensity image, while Fig. 7(b-d) show modulation ratios and phase differences as a function of the modulation frequency at positions X, Y, and Z marked in Fig. 7(a), respectively.From the frequency response curves, we can derive  1 and 2 ; the initial a 2 /a 1 is derived from numerical fitting.Figure 7(e) shows the a 2 /a 1 image, while Fig. 7(f,g) show the a 1 and the a 2 shading images, respectively, corresponding to 1  =4.2  0.21 ns and 2  =2.5  0.13 ns.The intensity plots for the individual a images shown in Fig. 7(e,f) are relative in that we can compare the degree of the shading directly for each pixel in the same image as well as in the other image.The number of fluorescent components was determined by minimizing the residuals between the measured and the fitted curve at every pixel.
To construct the FLIM from the HT-FLI system and a fluorescence microscope, we replace the LED array with a combination of a digital light processing (DLP) projector and a laser diode (LD).For example, if we use the commercially available DLP Discovery 4100 (Texas Instruments), which has a 1024  768 array of 13.6-m-pitch mirrors, and a  100 fluorescence microscope, diffraction-limited spatial resolution may be realized.High-speed measurements, one cycle measurement time T c =30.7 s explained in Fig. 4(A), might be attainable as well, which might result in the total measurement time around 0.5 s for 128  128 pixel, while T c =4.0 s and the total measurement time 256 s for 8  8 pixel in the present system.If the LED is replaced by a LD, the maximum modulation frequency would exceed 1.0 GHz [19].This would allow much faster fluorescence lifetimes (tens of picoseconds) to be measured.The present prototype system resulted in a few nanoseconds for the maximum modulation frequency of 50 MHz of the LED.
The maximum number of pixels resolved in the HT-FLI is determined by both the optical and detection systems.To acquire an n n pixel image with l-bit ordinate resolution, the detection system should have )) 2 / ( log ( 22 n l q   bit resolution, with the corresponding dynamic range as a maximum.For example, when n=64 and l=6, q=17.To increase n for a given q, one should conduct an accumulation procedure p times at the cost of the measurement time.This secures p 2 log bits in a random noise situation.Other options include heterodyne light detection for the enlargement of the detection dynamic range [20] or photon counting for measurements of weak fluorescence [19].shading image for 1  =4.20.21ns, and (g) a2 shading image for 2  =2.50.13 ns.

Conclusions
We have discussed a Hadamard-transform fluorescence-lifetime imaging (HT-FLI) technique and demonstrated a prototype in proof-of-principle experiments.The HT-FLI technique combines HT imaging with a Fourier-transform phase-modulation fluorometer.It has advantages in detection sensitivity, resolution times, and measurement times over conventional FLIs that use 2-D gated image detectors.For mixed standard solutions of rhodamine 6G and rhodamine B in ethanol, the HT-PMF acquired shading images for every fluorescence lifetime.We also briefly discussed guidelines for the construction of a fluorescence-lifetime-imaging microscope.

Figure 2 (
a) shows the working principle of the HTI system, which consists of a light source, N Hadamard masks, a sample object G, and a point detector.Light fluxes passed through the individual Hadamard masks and the sample object are gathered sequentially by the point detector, resulting in N time-series data y.In the HTI unit, we used a structured illumination light source made from an n n LED array that is a simple extension of the one-dimensional HTI procedure described below.As shown in Fig. 2(b), we divide the observation (imaging) area into n n pixels, from which we make an ( 1  N ) column vector g (where 2 n N  ) that represents the sample object G.By multiplying an N N  mask matrix M with g , 2(c)].M can be derived from an N N  Hadamard matrix H as follows.All elements in the first row and the first column of H are set to unity and the remainder is set to a matrix B , where all of the elements are unity, we can make an N in Fig. 2(d), elements of M consist of 1 and 0, where pattern for the n n LED array is generated from the N N  mask matrix M .As shown in Fig.2(d), we sequentially take n sets of values from the first row of M , where each set has n elements, and create an n n square matrix.This is the first illumination mask pattern.The second n n pattern is obtained from the second row of M by a similar procedure.Henceforth, the N th n n illumination mask pattern is obtained from the N th row of M .The N illumination mask patterns can still work if the 0 and 1 entries in M are reversed.Also, the first illumination mask pattern becomes a "black" pattern because entries in the first row of M are zero.Therefore, the first element of the observation vector y is zero: 0 1  y .In addition, the (1,1)th element of the reconstructed object image is always zero because entries in the first column of M are zero.

Fig. 2 .
Fig. 2. (a) Schematic of the HTI system, (b) generation of an N×1 object column vector g from an n×n object matrix G, where N=n 2 , (c) an N×1 observation vector y is obtained by multiplying an N×N mask matrix M with g: y=Mg, and (d) generation of n illumination mask pattern n×n matrices from M.
the illumination mask.The total time required to acquire the image is therefore μs) 256 ( NT , which is a higher frame rate than that of a normal 2-D detector.As shown in Fig. 3(c), we get ) 64 ( N sets of time-series fluorescence waveforms, with amplitudes and phases depending on the fluorescence lifetimes.Each fluorescence waveform is a sum of waveforms generated from data are captured with a digital oscilloscope (Tektronix TDS5054B: 10 6 data points; 400-ps sampling interval; 12-bit resolution) and processed by LabVIEW (National Instruments) software.As shown in Fig. 3(d), we get N pairs of real and imaginary a image.If necessary, to enhance SNR, we acquire p accumulations of the time-series data.

Fig. 3 .
Fig. 3. Schematic of HT-FLI; (a) cyclic Hadamard mask patterns; (b) chirped excitation waveforms; (c) fluorescence waveforms; (d) N pairs of real and imaginary spectra; (e) k(<<N) pairs of real and imaginary images obtained from (d) by an inverse Hadamard transform; (f) amplitude ratio m and phase difference  images calculated from (e); (g) shading image corresponding to each fluorescence lifetime.
transistor array.As described above, the LED array is turned on/off according to the cyclic Hadamard illumination mask pattern with an

Fig. 4 .
Fig. 4. Schematic of an n×n LED array driver (upper part).CSR; cyclic shift register used to generate Hadamard illumination mask patterns.AWG; arbitrary waveform generator for chirped sinusoidal waveform generation and its operating timing diagram (lower part); (A) master clock for pattern exchange.(B) settling time Td for preparing Hadamard illumination mask patterns and dwell time T for frequency chirping, (C) start pulse for the chirped waveform, (D) frequency-chirped excitation waveform, and (E) resulting fluorescence waveform.

Fig. 6 .
Fig. 6.(a) A dc (f=0) fluorescence image obtained with HT-FLI for the two-cell sample shown in Fig. 5(a).(b) Amplitude ratio m and phase difference  as a function of the modulation frequency f derived at a position X marked in (a), and (c) at a position Y marked in (a).(d) Initial amplitude ratio image a2/a1 calculated for each pixel.(e) a1 shading image for 1  =4.20.21ns. (f) a2 shading image for 2  =2.50.13 ns.

Fig. 7 .
Fig. 7. (a) A dc (f = 0) fluorescence intensity image obtained from HT-FLI for the multi-component cells in Fig. 5(b).(b)Amplitude ratio m and phase difference  as a function of the modulation frequency f derived at a position X,