2.1.

Endomicroscopy System

A schematic of the proposed imaging system is shown in Fig. 1. The output beam from a CW solid-state laser diode (Vortran Stradus, $\lambda =488\mathrm{nm}$, ${\mathrm{TEM}}_{00}$) is expanded by a $2.5\times$ telescope to a beam diameter of $\sim 4\mathrm{mm}$. Then it is projected onto the SLM via a polarizing beam splitter (Thorlabs, PBS 251). The SLM incorporates a phase-only ferroelectric liquid crystal on silicon device (Forth Dimension Displays, QXGA-3DM) with $2048\times 1536\text{pixels}$ and $8.2\text{-}\mu \mathrm{m}$ pixel pitch. The SLM operates in a binary imaging mode, where pixels in the ON state reflect light with its polarization state changed and pixels in the OFF state reflect the light without changing the polarization. With the help of the beam splitter, the SLM operates in an amplitude-modulation mode when its pixels are in ON and OFF states. An arbitrary amplitude pattern can be applied on the beam by computer control via a software (MetroCon from Forth Dimension Displays). In this experiment, the SLM toggles between two illumination patterns, grid and uniform. The grid pattern has a user-defined pitch that can be controlled by the MetroCon software.

Fig. 1

Schematic of virtual slit line-scanning endomicroscopy with HiLo-illumination.

A half-wave plate is placed between the SLM and the beamsplitter to rotate the polarization state of the light for maximal light modulation in the grid pattern, similar to the design proposed in Ref. 25. Further, a linear polarizer is applied after the beamsplitter to remove the unwanted polarization light. The modulated beam is reflected off a galvanometer scanning mirror (Thorlabs, GVS001), then focused by a cylindrical lens ($f=50\mathrm{mm}$) and an achromatic doublet ($f=50\mathrm{mm}$). The laser line is then reflected by a dichroic mirror (Thorlabs, MD498) onto a $10\times$ microscope objective (Thorlabs, $\mathrm{RMS}10\times$) which focuses it on the proximal face of a fiber bundle.

The imaging fiber bundle transfers the laser line to the tissue and collects emitted fluorescence. For this study, the Cellvizio Gastroflex UHD probe (Mauna Kea Technologies, France) is used which has 30,000 cores, with a $600\text{-}\mu \mathrm{m}$ useful image diameter and a $2.9\text{-}\mu \mathrm{m}$ interspacing. The distal end of the fiber bundle is equipped with a miniaturized micro-objective with a $2.5\times$ demagnification, yielding a field of view of $240\mu \mathrm{m}$ and an imaged core separation in the sample of $1.16\mu \mathrm{m}$. The miniaturized micro-objective has a nominal working distance of $\sim 50\mu \mathrm{m}$ and a numerical aperture of 0.8.

The fluorescence emission is isolated with a notch filter (Thorlabs NF488-15) and an emission filter (Thorlabs FEL0500) and imaged onto a CMOS camera (Point Grey Flea 3, pixel size $3.63\mu \mathrm{m}$) via an achromatic doublet ($f=50\mathrm{mm}$). The camera provides 440 pixels across the fiber bundle, or $\sim 2.25\text{pixels}$ per core spacing, which is sufficient to allow individual cores to be visualized in the images. The camera is operated in a rolling shutter mode at a full frame rate of 120 Hz for the experiments reported below. The rolling shutter of the CMOS camera offers a versatile electrically controllable detection slit. The slit width, which corresponds to the active rows on the camera sensor, can be changed by varying the exposure time of the camera. If the virtual slit width is reduced sufficiently, most of the out-of-focus light can be rejected. The scanning mirror needs to be synchronized with the rolling shutter to retain most beam light while rejecting scattered photons along the direction of beam scanning. The synchronization details have been explained in our previous papers.^12,26 The physical slit width in a unit of micrometers, as projected onto the proximal face of the bundle, can be calculated by the multiplication of the exposure time, the camera line rate, the pixel size, and the magnification factor between the camera and the bundle.¹²

Hybrid illumination across the fiber bundle is achieved by sequentially projecting uniform and grid illumination patterns which are generated using the SLM. Since the maximum frame rate of the SLM is about 1 kHz, the image acquisition rate of the proposed system is limited by the camera frame rate, which is 120 fps. A custom designed Labview code is developed to acquire the endomicroscopy images and processed offline to reconstruct a HiLo image using the following steps. First, a circular window is applied on each frame of the data to crop the edges of the bundle. Two adjacent frames with uniform and grid illuminations are processed by the HiLo ImageJ plugin developed by Lim et al.¹⁸ for producing one modulated image frame. The theoretical background of the HiLo microscopy has been described in detail in Refs. 1718.–19. Basically, both of the uniform illuminated image $I_u(\overrightarrow{\rho })$ and the structured illuminated image $I_s(\overrightarrow{\rho })$ contains the in-focus and out-of-focus components, where $\overrightarrow{\rho }=\{x,y\}$ is the lateral position vector.¹⁹ In-focus high-spatial frequency components $I_{\text{high}}(\overrightarrow{\rho })$ are obtained by applying a high-pass filter HP to $I_u(\overrightarrow{\rho })$ directly:

For the structured illuminated image $I_s(\overrightarrow{\rho })$, only in-focus information is modulated by the grid pattern due to rapid attenuation of pattern in the out-of-focus area. Thus normalized $I_s(\overrightarrow{\rho })$ and $I_u(\overrightarrow{\rho })$ can be subtracted to calculate the grid contrast $C(\overrightarrow{\rho })$. $C(\overrightarrow{\rho })$ can be used to weight the in-focus portion of $I_u(\overrightarrow{\rho })$. Then in-focus low-spatial frequency components $I_{\text{low}}(\overrightarrow{\rho })$ are obtained by applying a low-pass filter LP to the weighted $I_u(\overrightarrow{\rho })$:

The final optically sectioned image $I_{\mathrm{HiLo}}(\overrightarrow{\rho })$ is simply the addition of the high- and low-spatial frequency components with a scaling factor to ensure a seamless fusion:

3.1.

Optical Sectioning

A homogeneous fluorescence reference slide (green, AG2273-G, Agar Scientific) is imaged first to test the structure modulation ability of the system. Representative images of uniform and structure illuminated beams on the fluorescence reference slide are shown in Figs. 2(a) and 2(b). The measured laser line width at the reference slide is around $2.7\mu \mathrm{m}$. The illumination beams are swiped across the reference slide at the perpendicular direction to form uniform and structure illuminated images. Figures 3(a) and 3(b) show raw structure illuminated images of the reference slide imaged at different slit widths. The slit width is set to its maximum of $120\mu \mathrm{m}$ in Fig. 3(a) and $6\mu \mathrm{m}$ in Fig. 3(b). Though smaller slit width is possible, $6\mu \mathrm{m}$ is chosen to minimize the trade-off between confocal performance and signal strength of the grid pattern. As have been demonstrated in Ref. 12, out-of-focus background largely remains at $120\mu \mathrm{m}$ slit width and imaging at this slit width can be considered as the widefield slit detection. The out-of-focus background is largely reduced in $6\mu \mathrm{m}$ and imaging at this slit width can be considered as the confocal slit detection. By comparing Figs. 3(a) and 3(b), the contrast ratio between the ON and OFF pixels is evidently improved from 0.6 dB at the widefield slit detection to 2.5 dB at the confocal slit detection, demonstrating an increased structure modulation with the confocal slit detection.

Fig. 2

(a) Uniform and (b) grid illumination beams on a homogeneous fluorescence reference slide.

Fig. 3

Grid illuminated images of the fluorescence reference slide at (a) widefield detection and (b) confocal detection.

To further evaluate the line-scan HiLo endomicroscopy, a sample comprising lens paper stained with 0.02% acriflavine solution is imaged. The resulting images illustrated in Fig. 4 are imaged at different slit widths of 120, 18, and $6\mu \mathrm{m}$. Figures 4(a)–4(c) show raw images illuminated with a uniform pattern. As the slit width is decreased, optical sectioning of the endomicroscopy system gradually increases, in which out-of-focus blur becomes weaker. However, even with the confocal slit detection, there is still some background left as shown in Fig. 4(c), demonstrating the need to further reduce the background light. Figures 4(d)–4(f) show raw images illuminated with a grid pattern at different slit widths. As have been demonstrated in the uniform fluorescent plane, the contrast between the ON and OFF pixels is improved as the slit width decreases. Figures 4(g)–4(i) represent combined HiLo images using uniform and structure illuminated images at the corresponding slit width. A clear reduction in fluorescence background is observed for the combined HiLo images at all the three slit widths and the optical sectioning increases as the slit width becomes smaller. By comparing the nine images in Fig. 4, the best optical sectioning performance is achieved when both the HiLo technique and the confocal detection are applied.

Fig. 4

Endomicroscopy images of stained lens tissue paper. Uniform illumination with a slit width of (a) $120\mu \mathrm{m}$, (b) $18\mu \mathrm{m}$, and (c) $6\mu \mathrm{m}$. Grid illumination with a slit width of (c) $120\mu \mathrm{m}$, (d) $18\mu \mathrm{m}$, and (e) $6\mu \mathrm{m}$. Combined HiLo image with a slit width of (e) $120\mu \mathrm{m}$, (f) $18\mu \mathrm{m}$, and (g) $6\mu \mathrm{m}$. (j) Intensity plots along the lines indicated in (a), (c), (g), and (i) (the raw images are Gaussian filtered and normalized before all plots), showing the out-of-focus background at peak A and the in-focus signal at peak B. Scale bar: $50\mu \mathrm{m}$.

The optical sectioning performance of the system is compared for four different cases: widefield, widefield HiLo, confocal, and confocal HiLo. As have been explained, the uniform illuminated images at 120- and $6\text{-}\mu \mathrm{m}$ slit widths are considered to be the widefield and confocal slit detections. The widefield HiLo and confocal HiLo results are combined from the uniform and structure illuminated images at the corresponding slit width. To determine the optimal spatial frequency of grid patterns, four different grid patterns with periods of 40, 30, 16, and $12\mu \mathrm{m}$ (measured at the sample plane) are tested. Average intensities of small regions in the dark (OFF) and bright (ON) areas are calculated. It is found that the contrast ratio between the ON and OFF pixels is 3.56 dB for $40\text{-}\mu \mathrm{m}$ period, 2.33 dB for $30\text{-}\mu \mathrm{m}$ period, 1.47 dB for $16\text{-}\mu \mathrm{m}$ period, and 0.53 dB for $12\text{-}\mu \mathrm{m}$ period, respectively. Based on these results, a grid pattern with $16\text{-}\mu \mathrm{m}$ period is chosen for the remaining experiments in this study as it provided a reasonable comprise between grid pattern contrast and optical sectioning capacity. Despite a smaller grid period leads to stronger axial sectioning in theory, it also leads in practice to a decreased grid pattern contrast in the sample.¹⁰ Figures 4(a) and 4(c) represent the widefield and confocal images without using the HiLo, whereas Figs. 4(g) and 4(i) represent the corresponding widefield and confocal results with the HiLo technique. Intensity profiles along the lines indicated in these images are plotted in Fig. 4(j), where peak A indicates the out-of-focus background while peak B is the in-focus signal. The intensities of the in-focus signal in all cases are very similar at peak B. When comparing the intensities at peak A, it is obvious that the confocal HiLo gives the lowest value while the widefield has the highest value, the confocal only or the HiLo only has a value in between. To further quantify the effect of contrast enhancement, the mean pixel values from $50\times 50\mathrm{pixel}$ regions of signal and background areas are used to calculate the signal-to-background ratio (SBR). It is found that the SBR is 1.82 dB in Fig. 4(a), 3.33 dB in Fig. 4(c), 3.56 dB in Fig. 4(g), and 7.59 dB in Fig. 4(i). Thus both the HiLo technique and the confocal slit detection can reduce the background while the largest background reduction happens when both the confocal and the HiLo are applied. Noise of the imaging results in four different cases is also compared by calculating the standard deviation of intensities over $50\times 50\mathrm{pixel}$ areas. It is found that the noise of confocal HiLo images is 1.08 times of confocal images while the noise of widefield HiLo images is 1.32 times of widefield images. Thus confocal imaging detection also reduces the amplification of noise in the HiLo process.

To further compare the optical sectioning performance for different configurations, a steup of a fluorescence reference slide mounted on a motorized translation stage is used. The fiber bundle tip is initially placed in contact with the reference slide and then moved away at a speed of $60\mu \mathrm{m}/\mathrm{s}$ up to a range of $150\mu \mathrm{m}$. The acquired image is normalized before taking the mean pixel value from a $50\times 50\mathrm{pixel}$ region of interest as the intensity at that depth position. Axial sectioning profiles are compared by plotting normalized intensities at different configurations as a function of the stage movement, as shown in Fig. 5. The sectioning profiles are only plotted from the peak value (focus position of the micro-objective, $50\mu \mathrm{m}$ from the fiber bundle tip) to a $100\text{-}\mu \mathrm{m}$ farther position to show near contact-based imaging. From Fig. 5, the half-width at half-maximum (HWHM) of the sectioning profile is calculated as $7.5\mu \mathrm{m}$ for the confocal HiLo as opposed to $14\mu \mathrm{m}$ for the confocal, $32\mu \mathrm{m}$ for the widefield HiLo and $97\mu \mathrm{m}$ for the widefield. The results demonstrated that the optical sectioning ability of the HiLo-confocal endomicroscopy is largely improved when compared to other configurations. Despite using different grid period, the $7.5\text{-}\mu \mathrm{m}$ HWHM of the proposed confocal HiLo endomicroscopy is also evidently reduced compared with a previous illustration of HiLo endomicroscopy ($15\text{-}\mu \mathrm{m}$ HWHM).¹⁷

Fig. 5

Optical sectioning performance at different configurations. Normalized intensities are plotted as a function of the depth movement to show axial sectioning profiles at widefield (purple), widefield HiLo (yellow), confocal (red), and confocal HiLo (blue).

3.2.

Imaging of Biological Samples

The optical sectioning performance can degrade in biomedical tissue due to its increased level of scattering than a reference target. In order to demonstrate the optical sectioning performance of the proposed system in a biologically relevant example, perirenal fat and renal cortex tissues from a porcine kidney are imaged. Imaging of perirenal fat and renal cortex plays an important role in staging for renal cell carcinoma which is the most common primary malignancy of the kidney.²⁷ The kidney is stained with 0.02% acriflavine solution for 1 min and then washed by saline. The thickness of the kidney tissue used in this study is $>1\mathrm{cm}$ and the light power delivered to the sample is around 0.5 mW. Figure 6 shows images of perirenal fat and renal cortex performed without and with the use of HiLo technique at the confocal slit width of $6\mu \mathrm{m}$. All images have been scaled so that the maximum pixel value is 255. In Figs. 6(a) and 6(d), there is still some background left since only confocal detection is used to reject out-of-focus background from the thick tissue sample. When the HiLo technique is applied combined with the confocal detection, the out-of-focus background is significantly reduced and tissue features become more apparent as shown in Figs. 6(b) and 6(e). To quantify the improvement, Fig. 6(c) shows intensity plots along the horizontal lines in Figs. 6(a) and 6(b) and Fig. 6(f) shows intensity plots along the lines in Figs. 6(d) and 6(e). Compared to only use confocal slit detection, the background intensity between the features can be reduced to half when both HiLo and confocal techniques are used. The SBR is also calculated in Fig. 6 using the same method as in Fig. 4 to further quantify the contrast enhancement. It is observed that the SBR increases from 6.6 dB in Fig. 6(a) to 14 dB in Fig. 6(b) and the ratio increases from 4.8 dB in Fig. 6(d) to 10.6 dB in Fig. 6(e), demonstrating an average twofold contrast enhancement in confocal HiLo images as compared to that of confocal-only images.

Fig. 6

Endomicroscopy imaging of porcine kidney tissue. The slit width is set to $6\mu \mathrm{m}$ in both cases for confocal detection. Perirenal fat (a) without and (b) with the use of HiLo. (c) Intensity plot along the lines indicated in (a) and (b) (blue line: confocal without HiLo and red line: confocal with HiLo). Renal cortex (d) without and (e) with the use of HiLo. (f) Intensity plot along the lines indicated in (d) and (e).