Semi-autonomous real-time programmable fluorescence lifetime segmentation with a digital micromirror device

: Time-correlated single-photon counting (TCSPC) is the gold standard for performing lifetime spectroscopy in biological assays. Traditional fluorescence lifetime imaging (FLIM) using laser scanning microscopes are inherently slow due to point scanning all pixels in the field-of-view. Wide-field implementations of TCSPC spectroscopy using microchannel plates benefit from particularly fast acquisition times at the expense of temporal resolution, and are fundamentally limited by photon counting rates. Here, we introduce programmable lifetime imaging (PLI), combining the advantages of wide-field imaging using total internal reflection excitation with state-of-the-art TCSPC detector technology for accurate lifetime determination in an object-oriented manner using a digital micromirror device (DMD). The fluorescent emission is projected onto the DMD to facilitate the sequential segmentation of fluorescence from individual objects in the field-of-view, allowing for both image acquisition and fluorescence lifetime determination of the assay. The sensitivity of PLI is demonstrated by manually segmenting fluorescence from fixed cell assays. We also demonstrate an automated implementation of PLI, using a camera as a feedback mechanism to segment fluorescence produced by emitting objects of interest in the imaging field-of-view, highlighting the advantages of measurement only in areas where valuable information exists. As a result, PLI is able to reduce acquisition time of fluorescence


Introduction
Fluorescence lifetime spectroscopy is a widely used tool for extracting functional information from biological assays [1][2][3][4]. The fluorescence lifetime, τ , describes the time a fluorophore spends in the excited state. It is strongly dependent on the properties of the local microenvironment. Thus, fluorescence lifetime measurement is a common technique used in the elucidation of the underlying characteristics of a complex system. Amongst the most interesting insights such a tool can provide in biology is the extent of protein-protein interactions [5][6][7].
Applications of fluorescence lifetime imaging (FLIM) for spatio-temporal elucidation of physical, chemical or biological phenomena is extensive [6,[8][9][10]. FLIM implementations are broadly divided into frequency-and time-domain implementations, under either scanning [11] or wide-field [12,13] imaging modalities. Wide-field imaging methodologies with timedomain lifetime spectroscopy capabilities can suffer from the disadvantage of out-of-focus contributions in the resulting FLIM data which negatively impact the accuracy of the lifetime measurements [13][14][15]. Therefore, a well-defined illumination plane is highly desirable. Strategies circumventing this problem have been demonstrated using total internal reflection fluorescence (TIRF) imaging [16,17], as well as confocal sectioning using a Nipkov spinning disk [18] or structured illumination known as programmable array microscopy (PAM) [19][20][21][22][23][24]. The key advantages of PAM compared to traditional laser scanning confocal methods include: the omission of moving parts in the system; fast data acquisition speeds; and system flexibility afforded by the incorporation of programmable spatial light modulators (SLM), particularly if compatible with random access scanning. In addition to confocal sectioning, other applications of PAM include multiplexed fluorescence recovery after photo-bleaching (FRAP) [23], controlled light exposure microscopy (CLEM) [25], spectral imaging [26,27], as well as structured illumination microscopy (SIM) [28,29]. Therefore, the application of a PAM system depends highly on the patterns displayed on the associated SLM display. Fluorescence lifetime imaging (FLIM) in a PAM context has been performed extensively using frequency domain measurements [13,23,24,30]. Despite lifetime spectroscopy in the frequency domain being intrinsically fast and relatively inexpensive, it suffers from complex analysis requirements for multi-exponential decays, is sensitive to poor imaging signal-tonoise ratio's (SNR) and is prone to imaging artefacts [8,10,31,32].
The gold-standard methodology for lifetime spectroscopy is time-correlated single-photon counting (TCSPC), due to its single photon sensitivity [33], high SNR [34] and exploitation of well-defined Poissonian statistics describing photon arrival times [35]. Traditional laser scanning approaches benefit from the utilization of highly sensitive point detectors such as hybrid photomultiplier tubes (hPMT) which have superior temporal accuracy relative to traditional photomultiplier tubes (PMT) or avalanche photo diodes (APD), due to their large detector active area, low dark count contribution, high quantum efficiency, and measurement of fluorescent decays that are free of afterpulsing effects [36]. TCSPC works on the principle of the repeated measurement of photon arrival times relative to some reference. Typically, this reference is provided by a pulsed laser source. The difference in time between the detection of a reference pulse and the arrival time of a single photon of fluorescence is known as the microtime. Exponential fitting models can be applied to microtime histograms, representing the characteristic fluorescence decay of the fluorophore, to extract the measured lifetime at every pixel. This significantly simplifies the data analysis for more complex decay characteristics compared to frequency domain measurements [37]. TCSPC utilizing these superior point detection systems are often considered to be intrinsically incompatible with wide-field illumination methodologies, since the spatial information describing the structural detail of the assay will be lost in the absence of a suitable scanning mechanism. Previous attempts to overcome this limitation was demonstrated using a digital micromirror device (DMD) [38] to raster scan illumination across a fluorescent phantom [39]. Of note from this study, is that due to the architecture of the DMD display, it behaves in analogy to a blazed grating when the pixels have a tilt applied to them [40]. Therefore, a significant amount of the coherent illumination power will be distributed in higher order reflections that cannot be captured by low numerical aperture optics, thus significantly reducing the irradiance at the sample. The authors circumvented this by scanning "super-pixels" comprising 16 × 16 individual DMD pixels to make TCSPC measurements. However, the spatial resolution of their final FLIM data is a significantly degraded as a result. This would make the study of more structurally complex assays particularly challenging and quantitatively unreliable. It is also interesting to note that significant time is spent performing TCSPC measurements in areas of the sample where there is no information of value (i.e background). Importantly, this is also true for all point scanning systems studying particularly sparse assays on a pixel-bypixel basis across the full field-of-view (FOV). This can make data acquisition of such samples particularly cumbersome and inefficient. More recent developments using multiplexed excitation with beamlet arrays coupled to single photon avalanche diode (SPAD) arrays have been successful at significantly reducing data acquisition times [41][42][43]. However, this technique does not fundamentally resolve the issue of performing TCSPC measurements where there is no data. Furthermore, one may consider using SPAD arrays as a TCSPC camera for assays imaged under a wide-field modality [44]. However, the improved temporal resolution obtained through the use of such technology in comparison to microchannel plates or time-gated implementations [45], is also at the significant expense of spatial resolution. Furthermore, the low fill factor of the pixels will significantly impact the sensitivity of the system under wide-field illumination.
As a further contribution to the PAM class of microscopy technology, we have designed a system that allows for fluorescence spectroscopy under wide-field illumination, attempting to address the previously described limitations. We term this technology Programmable Lifetime Imaging (PLI). We use DMD technology, whose display is located conjugate to the image plane, to segment fluorescence obtained under TIRF illumination in an object-oriented manner. Importantly, object-oriented segmentation in this way avoids sampling regions of the FOV in which there is no information of value, significantly improving the efficiency and speed of data collection. Each object that is addressed using this mechanism has its fluorescence selected at the DMD and projected onto a hPMT for accurate TCSPC measurements. This technology is demonstrated using both manual and automated segmentation of the fluorescence emission. Automated implementation of PLI includes retention of the localisation information, allowing the spatial information of the assay to be reconstructed for wide-field FLIM image representation. We envisage such technology to be particularly suited for imaging sparse data sets, such as single molecule assays, and FLIM specifically at the plasma membrane.

Optical Setup
Two different optical configurations of PLI were prototyped. The first comprised only offthe-shelf optics [Thorlabs Ltd., UK], as well as utilizing a pulsed supercontinuum laser source (SC400-4-20 [Fianium, UK]), providing 400 < ex λ < 2000 nm excitation, with a pulse duration of 6 ps at a repetition rate of 20 MHz (Fig. 1a). The excitation was passed through an interference filter to select 480 < ex λ < 490 nm, which was then free-space coupled into an optical fibre with ~30-40% efficiency. The output of the optical fibre was then mounted on a moveable stage allowing the transition between epi-fluorescent and TIRF illumination. The excitation then passed through a tube lens and a FITC-compatible dichroic filter and focussed onto the back aperture of a 100 × 1.47 N.A. objective [Leica Microsystems, UK] mounted on a microscope platform (DMi8 manual [Leica Microsystems, UK]). The emission was  collected and passed through a telescope performing a 1.5 × magnification on the image, before being projected onto the DMD (DLP Discovery 4100 development kit (D4100) [Texas Instruments, USA]) display. The DMD comprises a 1024 × 768 array of micro-mirrors on a complementary metal-oxide semiconductor (CMOS) substrate. Each pixel can be programmatically addressed to incur a ± 12° tilt across the diagonal, relative to the incident illumination, allowing the optical path to be split in two directions; one for the imaging side of the system, and another for the TCSPC acquisition. To keep the optical paths parallel with the optical table, the DMD was rotated by 45° along the axis normal to its surface to compensate for the tilt orientation of its pixels. The imaging side of the system comprised an additional set of relay lenses producing a 1 × magnification, with an aperture located in the Fourier plane to select the 0th order diffraction off the DMD display. This had the effect of sharpening the image at the camera (QuantEM: 512SC [Photometrics, USA]). On the TCSPC side of the system, the image was (de)magnified by 0.2 × via two consecutive telescopes, with a custom square aperture placed at the intermediate image plane to attenuate fluorescent emission reflecting from the periphery of the DMD display, before being projected into the hPMT.
To address imaging limitations associated with the DMD display, a second configuration of the PLI system incorporated optics which allowed for the inclusion of a Schiefspiegler relay [Cairn Research Ltd., UK], in accordance with the specifications described elsewhere [46] (Fig. 1b). Other significant changes included, the removal of the optical fibre providing the excitation to prevent any patch cable associated losses in excitation intensity, the microscope platform was upgraded (Ti-E Eclipse [Nikon Instruments, UK]) to provide precise control of the , x y and z position of the sample, a 100 × 1.49 NA objective [Nikon Instruments, UK] was incorporated, and the consecutive telescopes on the spectroscopy side of the system provides a 0.375 × magnification of the emission before entering the hPMT.

Microscope Control Software
All subsequently described software was written in 32bit LabVIEW 2011 [National Instruments, UK] and executed on a computer running Windows XP SP3 with an Intel Core 2 Duo E4600 CPU operating at 2.6 GHz containing 4 GB RAM.
The DMD controller board (D4100) via USB2.0, connected to a computer by USB2.0 interface, was controlled using an application program interface (API) [Vialux GmbH, Germany]. The DMD display is segmented into 16 blocks encompassing 48 rows of pixels. Data is typically loaded onto the CMOS substrate in a block-wise manner and the state of the pixels is changed following a "reset" clocking pulse. There are four different block-clocking modes which are typically implemented when using DMD technologies, including single block mode, dual block mode, quad block mode and global mode. However, a particularly useful feature of the D4100 in our case is its compatibility with random row addressing, in which only the rows containing the pixels that are required to have their state changed have data loaded onto them before being reset.
For simplicity, the DMD software was written such that a square region of pixels can be addressed at any time, which we refer to as a "super-pixel". It should be noted that, in principle, the DMD can display any binary or 8bit greyscale image it is provided with. The centre of the n × n super-pixel corresponds to the pixel coordinate the DMD is supplied with. A 1D Boolean array, with elements being either "0" or "1", is then generated describing the pixels that require their state changed across the top row of the super-pixel. This is then encoded as an unsigned 8bit numerical array, which is subsequently duplicated across all the required rows to describe the entire super-pixel data. This data is then sent to the DMD for display. This super-pixel is responsible for redirecting the fluorescence emission away from the camera, and towards the hPMT for TCSPC measurements.
The effect the size of the super-pixel has on the speed at which the DMD can display them was quantified by measuring the execution time of displaying super-pixels with various sizes using programmatic timing functionality. For each super-pixel size, 2000 timing measurements were made.
Custom software using an API [Becker & Hickl GmbH, Germany] to control the SPC-830 was written and incorporated into the microscope control software. Trigger signals were sent from the D4100 general purpose in-out ( Due to the legacy operating system required for compatibility of the DMD drivers, it was acknowledged that perfect synchronicity is particularly challenging and that residual latency is likely to be present. To quantify this latency, three super-pixels of different sizes were sequentially displayed on the DMD under constant illumination. Each super-pixel was associated with a trigger signal across a known pin on the GPIO socket. The variation in size of the applied super-pixels also corresponded to measurable differences of photon flux measured at the photodetector, allowing for highly accurate determination of the time the super-pixel changed on the DMD display. The time a trigger is detected at the SPC-830 is also encoded in the photon data. Efficient control of the DMD can be facilitated by using the camera as a feedback mechanism to indicate where in the imaged FOV objects of interest are located. For this, careful pixel-to-pixel correspondence between the DMD and camera image planes was required. Previous methods demonstrating this process include the phase shifting Moiré method [47], and using Affine transforms [28]. In our case several super-pixels were generated on the DMD display with known locations. The camera then takes an image of these displayed super-pixels and calculates each centre-of-mass (COM). This repeats until all DMD pixels seen by the camera have had super-pixels applied to them, resulting in a raster scanning effect. The COM information populated a 3D look-up-table (LUT) of dimensions 512 × 512 × 2, where the x and y dimension sizes correspond to the size of the camera detector, and the two elements along z store the corresponding x and y location on the DMD. This allows for direct indexing of the corresponding DMD coordinates using the localisations found in the camera pixel space.
The microscope control software for automated data acquisition executes the following steps ( Fig. 2): (1) an image of the sample is acquired by the camera. (2) The image is processed allowing the identification of the COM coordinates of all the objects located in the calibrated image field. (3) Each coordinate is sequentially used to generate the super-pixel data, which is then sent to the DMD and displayed. (4) TCSPC measurements from that object's fluorescent emission are performed. Steps 3 and 4 repeats iteratively until all the identified objects are measured, after which the process repeats from step 1 for as many frames the user has specified to acquire. The COM for each object, single photon events, and camera image are streamed to disk throughout the data acquisition cycle to prevent RAM overflow on the host PC. Each coordinate in the list is extracted and transformed to correspond to the location on the DMD display. The super-pixel data is then generated, loaded and displayed on the DMD to redirect the fluorescence away from the camera and towards the hPMT. This process repeats for all identified objects. (e) When all objects have been scanned, the next frame is acquired. This entire process repeats until the desired number of frames of the sample have been collected.

Post-Processing Software
We developed further post-processing software to extract the photon arrival time-stamps acquired and represent them as histograms for elucidation of the fluorescence lifetime. Segmenting the individual measurements using trigger signals allows the photons recorded for a single object, over several frames, to be extracted and binned together to maximise the number of photons available for fitting against, improving the accuracy of the fluorescence lifetime measurement. This restructured raw data is then reformatted for compatibility with the pre-existing fluorescence lifetime analysis software, TRI2 [48]. The photon data is represented as an unsigned 32bit integer, in which the first 12bits encodes the microtime information associated with the measurement. For each measurement (in other words, for every super-pixel displayed on the DMD), this information is extracted and used to generate a microtime histogram which can subsequently be fit using exponential fitting models to extract the fluorescence lifetime. With knowledge of the location at which each measurement was made, a FLIM image akin to those extracted from traditional laser scanning techniques can be generated by embedding the associated microtime histogram in an image cytometry standard (ICS) file [49]. It is important to note that since the data is acquired autonomously over several frames, COM calculations are distributed over several pixels (dependent of SNR) for the same object when represented as an ICS format. As a result, the photon information associated with each localisation is then distributed across these pixels, despite them all being associated with the same object. To mitigate the extent of error arrising as a result of low photon counts when determining the lifetime of each object, we implemented an "object-binning" routine ( Fig. 3), which identifies the cluster of localisations made during the acquisition cycle for each object. The COM for each cluster of localisations made for each Fig. 3. Schematic illustration of the object-binning algorithm. (a) A scatter plot representing all the localisations made through an acquisition cycle is generated. (b) In most cases, the localisations associated with a single object are clustered around the true localisation of the object. (c) The object-binning algorithm identifies the extent of the localisation cluster (yellow box), and measure the COM (green cross) associated with that cluster. (d) It then assigns the location of that object with the COM measurement. Note that this does represent the true localisation of the object; it is in analogy to the average location, with single pixel accuracy, as seen by the camera over the entire acquisition. All the associated photon data acquired for that object is then binned into this single pixel. object is then extracted. This allows all the photon data for each object to be binned into a single pixel, increasing the number of photons available for fitting and improving the accuracy of the lifetime extraction.

Cell Culture
Transfection media consisting of: 0.4 µg of DNA encoding Lifeact-eGFP, or FRET standards containing eGFP as a donor separated from a mRFP1 acceptor molecule by X amino acids (aa) with an N-terminal CaaX motif (eGFP-X aa-mRFP1-CaaX), at a concentration of 1 µg µL −1 ; 3.2 µL of enhancer [Qiagen, Germany]; and 50 µL of DNA-condensation buffer. The transfection media was vortexed for 10 s and incubated at room temperature for 5 min. After, 5 µL effectene [Qiagen, Germany] was added to the transfection media and vortexed for a further 30 s, before incubating at room temperature for 15 min, completing the transfection reagent.
HeLa cells were grown in DMEM growth media. The growth medium was discarded and the cells were washed with PBS. 600 µL of growth media at 37° was added to the transfection reagent and added directly to the cells. They then incubated for 4 -8 hr at 37°. The cells were then washed twice with growth media at 37°, before being allowed to incubate overnight in fresh growth media at 37°. The media was then discarded and the cells were washed in PBS, before 250 µL of trypsin was added, which were allowed to incubate at room temperature for 5 min at 37°. 10 mL of fresh growth media at 37° was then added to the cells, before 500 µL were plated in an 8-well imaging chamber [Ibidi GmbH, Germany]. The media of the plated cells was removed before 4% PFA was added. The cells in the fixing media were incubated at room temperature for 15 min. The PFA was then removed and the cells were washed several times using PBS. The cells were then permeabilized for 10 min with 0.1% triton diluted in PBS at room temperature. The cells were then treated with 1 mg mL −1 sodium borohydride diluted in PBS, before an extensive washing with PBS. The cells were then mounted in 200 µL of PBS.

Fluorescence Microsphere Preparation
Fluorescent microspheres of diameter 0.100 ± 0.001 µm [Thermo Fisher Scientific, UK] were diluted in 1:1600 v/v in PBS and pipetted onto a coverslip imaging chamber [Ibidi GmbH, Germany] and left to incubate at room temperature for at least 30 mins to allow them to settle on the sample surface.

Manual Imaging Using PLI
Samples were imaged under TIRF illumination and fluorescence measured at the camera with a 150 × multiplicative gain and a 500 ms integration time. Super-pixels comprising 10 × 10 individual DMD pixels were applied to perform TCSPC measurements. After manual application of the super-pixel in the desired location, TCSPC measurements were conducted in first-in-first-out (FIFO) mode with a 250 ms acquisition time using proprietary control software (SPCM [Becker & Hickl, Germany]).

Automated Imaging Using PLI
Samples were imaged under TIRF illumination and fluorescence measured at the camera with a 150 × multiplicative gain and a 100 ms integration time. The DMD was configured to apply super-pixels comprising 9 × 9 individual DMD pixels, with a 50 ms dwell time. Objects were probed across a 200 frame acquisition, and TCSPC measurements were conducted continuously in FIFO mode using our custom PLI control software for the entire duration of the experiment. We first quantified how the random row addressing feature of the D4100 effects the speed at which super-pixels can be displayed on the DMD, given the number of rows that they occupy. Measuring the time taken for the software to apply super-pixels on the DMD for a variety of super-pixel sizes showed that there is a linear decrease of approximately 0.53 ± 0.04 Hz per additional row which the super-pixel occupies (Fig. 4).

Synchronization of the DMD with the TCSPC Electronics
Next, we quantified the residual latency that exists between the identification of the start of a new measurement on both the DMD and SPC-830, represented respectively by the sending and receipt of a trigger signal associated with the display of a particular super-pixel (Figs. 5a-5c). It can be seen that the measured photon flux shows a step increase in magnitude as the super-pixel increases in size ( Fig. 5d and Fig. 5e). Comparison between the time each rising edge of each step occurs with the time the associated trigger is received by the SCP-830 shows that there remains approximately 234.5 ± 30.5 µs latency between the detection of a trigger signal at the SPC-830. Observed outliers to the peak of the Gaussian fit can be ascribed to operating system interrupts and limitations associated with the USB2.0 bandwidth, as well as operation of other devices present on the same USB host controller.

Lifetime Spectroscopy under Evanescent Field Illumination
We first demonstrate the TCPSC ability of PLI (system depicted in Fig. 1a) by performing lifetime spectroscopy on fixed HeLa cells transiently expressing Lifeact-eGFP (Fig. 6). Three super-pixel locations were chosen at visible stress fibres. A mono-exponential Levenberg-Marquardt fitting model was applied to the measured microtime histogram to extract the lifetime; the average lifetime was found to be τ = 2.57 ± 0.002 ns.  We then assessed the ability of the PLI to distinguish two different lifetimes. Fixed HeLa cells transiently expressing FRET standards of the form eGFP-X aa-mRFP1-CaaX [50] (Fig.  7), whereby the donor molecule eGFP is separated from the acceptor mRFP1 by an amino acid linker of length X . Therefore, high FRET efficiency is expected with a linker of length 7aa, resulting in a decrease in the fluorescence lifetime of the eGFP compared to the low FRET construct with a linker length of 32aa. These constructs were previously shown to be distinguishable by the effect the proximity of the acceptor has on the measured lifetime of eGFP [50,51]. The N-terminal CaaX motif ensure expression primarily at the plasma membrane of the cell [52]. The average fluorescence lifetime extracted for each construct was 7aa τ = 2.01 ± 0.14 ns and 32aa τ = 2.26 ± 0.05 ns (N = 4). The increased standard deviation in the lifetime measurement for the 7aa linker construct compared to the 32aa linker construct can be ascribed to the error associated with the reduction in the number of photons measured over a fixed acquisition time. Given an eGFP lifetime of τ ≈2.6 ns (measured separately), this corresponds to FRET efficiencies of 23% and 13% for 7 and 32 amino acid FRET constructs respectively, which is in reasonable agreement with other studies [50,51]. We demonstrate the automated implementation of PLI (system depicted in Fig. 1b) using diffraction limited fluorescent microspheres. Several microspheres were observed in the FOV with a sparse distribution (Fig. 8a). A binary scatter plot representation of all the localisations made throughout the acquisition cycle allows the object-binning algorithm to identify the centre of all localisation clusters for single pixel representation of the microspheres (Fig. 8b). The generated ICS file was then analysed in TRI2 using a mono-exponential Levenberg-Marquardt fitting model to extract the lifetime information. The average measured lifetime for these microspheres was measured to be τ = 4.78 ± 0.79 ns (Fig. 8c) associated with an average goodness-of fit measure of 2 r χ = 1.11 ± 0.15 (Fig. 6d), indicating the suitability of the fitting model with this data set. It can be observed that throughout this particular acquisition, mechanical lateral drift manifests itself in the localisation scatter plot image (Fig.  8b). The benefit of the automated data acquisition is that the super-pixel applications follow such drift, making any impact of the acquired TCSPC data negligible. The single pixel representation of the imaged objects can also be considered a spatial average of each objects location, given the associated drift although, in principle full correction to the experimental origin could be performed. Particularly in the case where the assay has a sparse distribution of emitters, the data acquired over hundreds, or even thousands of frames, may be required for statistical significance in the lifetime measurements. As a result, it may also be useful to represent all the lifetime measurements from each object from several acquisitions as single image array (Fig. 9a-c). This allows the visualization of any global trends in the assay, or time-dependent variations that may inform on data quality or the presence of outliers. Across 10 independent acquisitions, the average number of measured photons from each bead was measured to be I = 1164.3 ± 220.5 counts (Fig. 9a and Fig. 9d), the average measured lifetime was τ = 4.41 ± 0.45 ns (Fig. 9b and Fig. 9e) and an average goodness-of-fit measure of 2 r χ = 1.02 ± 0.15 ( Fig. 9c and Fig. 9f), consistent with measurements made from a single acquisition. To verify that there is no intensity-dependent trend to the data or the presence of errors used in the fitting model for lifetime extraction, an intensity vs lifetime histogram was generated (Fig. 10). It can be seen that the distribution of events shows no clear correlation with intensity, indicating that an adequate number of photons were acquired for lifetime extraction.

Discussion
We first quantified the performance of the key operating aspects of our PLI imaging technique. It was demonstrated that the more rows the super-pixel occupies, the lower the achievable application rate of the system. We also quantified the residual latency that exists between the receipt of triggers at the SCP-830. The minimum required photon counts for accurate lifetime determination for a mono-exponential decay is 1000 measured counts. To prevent pile up at the photodetector, no more than 1 photon should be measured every 5 µs, which corresponds to 1% of the reference repetition rate. Therefore, a latency of 234.5 µs represents a 4.7% error in the origin of the measured photons associated with a particular object. This can be further improved with utilization of GPU optimization and/or real-time processing.
Next, we demonstrate the ability of our PLI imaging system to perform TCSPC lifetime spectroscopy under evanescent field illumination. For the first time, we demonstrate the advantages of using object-oriented lifetime detection rather than pixel-by-pixel scanning, as a method to improve the efficiency of the data acquisition and consequently data acquisition. By comparison, PLI operating in an automated scanning mode was able to acquire a data set with 26 objects (Fig. 8a) in 2.628 seconds per frame on average (including all the image processing and data transfer), whereas a point scanning system with a pixel dwell time matching our super-pixel dwell time would require 15.810 seconds per frame, regardless of the number of objects of interest that are present in the FOV. The acquisition speed of PLI is strongly dependent on the number of objects N , in the FOV that require measurement; imaging time will increase approximately linearly with N . For the experimental setup shown, this equates to PLI outperforming a laser scanning system with an assay consisting of up to 156 visible objects per frame. The maximum speed of PLI is limited by the USB2.0 control interface of the DMD. The total image processing time during runtime is approximately 25 ms, and it takes approximately 8 ms to generate, send and load the superpixel data to the DMD. Therefore, the maximum super-pixel application rate is currently approximately 30 Hz. This could be reduced with improved processing power, real-time operating system capabilities, and more modern interface technologies such as USB3.0 which are now available.  Fig. 6. (d -f) Histograms of the measured intensity, lifetime, and goodness-of-fit from panels a -c, respectively. Limitations associated with the system described in Fig. 1a primarily originate from the structure of the DMD display. The DMD forms a 2D grating causing a significant dispersive effect on the image, requiring an aperture at the Fourier plane to select the 0th order diffraction only. This improved the image quality significantly at the cost of achievable SNR. A number of different optical arrangements have been utilized that have been designed specifically to correct aberrations generated from the DMD display, for example: projection systems utilize high NA optics such as prisms [53]; customized condenser optics [54]; an Offner relay optical configuration [22]; and more recently a Schiefspiegler relay optical configuration [46,55]. The Schiefspiegler relay is a particularly attractive solution, since it projects the object to infinity, allowing for greater flexibility for subsequent optical transformations, if required.
An additional limitation associated with PLI more generally is the constant illumination of the sample while only a single object is being measured using TCSPC. This causes unnecessary photobleaching/photodamage to the assay. This could be circumvented by performing only the localisation step under wide-field illumination, followed by subsequent sequential targeted confocal excitation of the observed objects, mediated via illumination off the DMD. This would consequently allow for much longer measurement dwell times on each object while protecting the remainder of the assay. This may be particularly important for single molecule imaging, where SNR considerations become increasingly important, given their limited photon budget.
Imaging technologies such as PLI could be particularly useful of number applications requiring lifetime contrast where sparse fields of view are evident, such as imaging vesicles or single molecules. The advantage of a TIRF imaging modality is the sub-diffraction limited axial illumination depth and relatively low power densities at the sample in comparison with point scanning systems. It is also worth noting that TIRF is difficult to implement for a scanning system [56]. The ability to incorporate high performance photon detection systems also ensures optimum data quality. We anticipate that our PLI methodology will enhance the diversity of information that can be extracted from biological assays, providing more detailed insights into the underlying mechanisms governing cellular processes.