Image scanning lensless fiber-bundle endomicroscopy

Fiber-based confocal endomicroscopy has shown great promise for minimally-invasive deep-tissue imaging. Despite its advantages, confocal fiber-bundle endoscopy inherently suffers from undersampling due to the spacing between fiber cores, and low collection efficiency when the target is not in proximity to the distal fiber facet. Here, we demonstrate an adaptation of image-scanning microscopy (ISM) to lensless fiber bundle endoscopy, doubling the spatial sampling frequency and significantly improving collection efficiency. Our approach only requires replacing the confocal detector with a camera. It improves the spatial resolution for targets placed at a distance from the fiber tip, and addresses the fundamental challenge of aliasing/pixelization artifacts.


Introduction
Flexible fiber-based micro-endoscopes are an effective tool for minimally-invasive deep-tissue imaging.They offer micron-scale resolution with miniature probes having diameters of a few hundred microns, making them suitable for in-vivo imaging of neurons and cells [1][2][3].One of the popular implementations for miniature micro-endoscopes relies on multi-core fiber bundles (also known as coherent fiber bundles), where each fiber core serves as one imaging pixel.Conventional imaging using fiber bundles is performed either by placing the bare distal fiber tip in contact or close distance to the target, or by adding a gradient-index (GRIN) lens to the distal tip [4,5].Both solutions suffer from significant undersampling due to the low fill factor caused by the large spacing between fiber cores, compared to the core size [2] and out-of-focus background.
A common approach to improve the resolution and optical cross-sectioning of fiber-bundle endoscopes is through confocal imaging [4,[6][7][8][9].In confocal fiber-bundle-based endoscopy the illumination is scanned over each of the fiber cores, and only the light collected from the illumination core is used to form the image.Thus, each single core serves as an effective confocal pinhole, rejecting out-of-focus light and improving the spatial resolution.Importantly, the implementation of confocal fiber-bundle endoscope does not require the addition of any distal elements since the scan is performed at the proximal fiber end.
Over the years, many techniques have been suggested and demonstrated to improve lensless fluorescence fiber-bundle endomicroscopy.These methods include structured-illumination [10,11], retrieving light-field 3D information from the modal excitation within each core [12], optical fluctuation imaging [13], and recently, advanced wavefront-shaping and image reconstruction approaches [14][15][16][17][18][19].However, except for the advanced recent approaches where the fiber bundle effectively serves as the imaging pupil [14][15][16][17][18][19], all approaches suffer from aliasing/pixelization artifacts due to the undersampling by the fiber cores at the image plane.In addition, lensless confocal fiber-bundle endoscopy suffers from low collection efficiency when the target is not in close proximity to the distal facet.Unfortunately, the low effective fill factor of coherent fiber bundles, which leads to image undersampling, is an inherent design limitation required to minimize the cross-talk between neighboring cores [2].
Here, we demonstrate that both the collection efficiency and spatial sampling frequency can be significantly improved for targets placed at a distance from a lensless fiber bundle by adapting the principles of image scanning microscopy (ISM) [20] (also known as pixel-reassignment microscopy [21,22]) to lensless fiber-bundle endomicroscopy.
ISM has emerged as a microscopy technique that improves the signal-to-noise and resolution of confocal microscopes.ISM utilizes the same raster-scanned illumination as in confocal microscopy but replaces the small detection pinhole with a larger pixelated detector array, allowing the use of all signal photons without loss of resolution.In ISM, the resolution is maintained or improved compared to confocal microscopy by reassigning the detected light from each detector (pixel) to the midpoint between the detected and illumination positions.The final ISM image is the sum of the reassigned signals.Mathematically, the pixel-reassignment process is equivalent to spatially scaling by half (shrinking) each raw image from the detector array (Fig. 1), and summing all scaled raw images.Recently, ISM has been used to improve collection efficiency and resolution in confocal Raman microscopy [23], two-photon microscopy [24], scanning laser ophthalmoscopy [25], transmission-matrix based imaging in multi-mode fibers (MMF) [26], and quantum microscopy [27,28].
Here, we combine the raster-scanned illumination of conventional confocal endoscopy [6] with camera-based detection to implement ISM in a bundle-based lensless microendoscope.Our straightforward approach tackles two main drawbacks of conventional confocal fiber-bundle endoscopy.First, we collect fluorescence signals through all fiber-bundle cores rather than a single core.This proves beneficial for targets placed at a distance from the bundle tip, as the collection efficiency is greatly increased.Second, ISM doubles the spatial sampling frequency of the fiber bundle due to the spatial scaling of the pixel-reassignment process (Fig. 1), addressing the fundamental problem of aliasing and pixelization artifacts of targets located at close distances to the fiber.

Principle
The principle of the image-scanning fiber endoscopy is schematically shown in Fig. 1.Our approach is based on a conventional confocal endoscopy configuration (Fig. 1a).A focused beam scans the fiber cores at the proximal tip.The same fiber bundle collects the fluorescence signals.
The proximal facet is imaged on a camera through a dichroic beam-splitter (DM) and a band-pass filter (F).At each scan position, the camera captures the fluorescence intensity distribution over the proximal facet (Fig. 1b).The sampling grid of the bundle cores is marked by white X's in Fig. 1(b-d).The camera images allow performing both confocal imaging and ISM.For confocal endoscopy only the collected signal from the illumination core is integrated (Fig. 1c).For ISM, the entire camera image is scaled by a factor of 0.5 [22] and saved (Fig. 1d).This reassignment procedure creates 'virtual' cores at a new detection grid, with a spatial sampling frequency of 2   that is two times larger than the original cores grid (Fig. 1b).The final ISM image is the summation of all reassigned images, having a denser spatial sampling grid and a higher collection efficiency.
Quantitatively, the point spread function (PSF) of confocal imaging, ℎ   (), is the product of the illumination and the detection PSFs [29].
Where r is the transverse spatial coordinate, ℎ  is the illumination PSF, and ℎ  is the detection PSF.When both illumination and detection have similar Gaussian profiles, confocal imaging results in a  (2) times narrower effective Gaussian PSF than each of these PSFs.
In the case of ISM, the effective PSF, ℎ   is the convolution of the illumination and detection PSFs, scaled by a factor of half [20].In the case of identical Gaussian profiles for illumination and detection PSFs, the final effective PSF of ISM is identical to the confocal.This result assumes an adequately Nyquist sampling of the collected signals.While this is conventionally attained in a conventional microscopy setup, it is not the case in bundle-based endoscopy.As a result, the doubling of the spatial sampling frequency in fiber-bundle ISM improves the final optical transfer function (OTF, the Fourier transform of the PSF), even in the case of identical detection and illumination PSFs since it avoids aliasing of spatial frequencies in the range [   /2   ] (see Fig. 2).

Numerical study of the optical transfer function
To study the expected improvement in the normalized OTF due to the increase in spatial sampling frequency of ISM fiber-bundle endoscopy, we numerically simulated the resulting images of a point target placed at distances of 60m and 150m for ISM and confocal fiber bundle endoscopy, and compared to the ideal confocal image sampled at a spatial frequency ∼ 6-times that of the fiber bundle (i.e., without relevant aliasing).The OTF for each case was calculated by Fourier transforming the simulated image.The results of this study are presented in Fig. 2. For each distance, we studied the results when the point target is placed directly in front of a fiber core (Fig. 2a,c) and when the point target is placed between two neighboring cores (Fig. 2b,d).1d, and the folding frequencies (dashed and dotted vertical lines, respectively).This allows correct retrieval of ×2 higher spatial frequency content in the ideal unsampled confocal OTF (dashed green).(c-d) (c-d) Same as (a-b) for an object distance of 150m.At larger object distances, the doubled sampling frequency of ISM endoscopy has a negligible impact due to the lower frequency content of the defocused object image.Nonetheless, the gain in signal collection efficiency of ISM improves the reconstructed images (see Fig. 3 and Supplementary Fig. S2).
As expected, the 1-D cross-sections of the OTFs in all cases show that aliased spatial frequencies, which result from undersampling, are present in the confocal images at frequencies higher than   /2 (blue curves and arrows).In contrast, in the ISM images, the aliased frequencies are only present at frequencies above   (red curves).The doubling of the spatial sampling frequency is important at close distances (Fig. 2a,b) since the captured image contains higher frequency content than at larger distances (Fig. 2c,d), where the effective defocus blur due to the propagation low-pass filters the higher frequencies content.Importantly, as we show below (Fig. 3, Supplementary Fig. S2), ISM endoscopy also improves resolution at longer distances due to the significantly improved collection efficiency and SNR.

Experimental results
Our experimental setup is schematically depicted in Fig. 1a, and the experimental results are summarized in Fig. 3.In all experiments, a 532nm CW laser (Oxxius LCX-532S) illuminates a pair of scanning mirrors (Thorlabs GVS012), which are imaged on the back focal plane of an objective lens (OL, 20×, 0.4 NA, Olympus) using a 4-f system.The galvanometric mirrors raster-scan the focused beam on the proximal facet of the MCF bundle (Schott P/N 1563385), having an effective imaging area of 0.45 diameter, containing ∼ 5, 600 cores, with an average core diameter of 4.1m and an average core-to-core distance of 6.5m.The fluorescence signal is filtered by a dichroic mirror (DM) and band-pass filters (F, Semrock BrightLine FF01-576/10-25, and Thorlabs FEL0550), and the descanned image is formed on an sCMOS camera (Andor Zyla 4.2).The target object is composed of Nile Red fluorescent beads (Spherotech), averaging 6m in diameter, dispersed on a cover glass.A motorized actuator (Thorlabs Z825B) controls the distance between the fiber and the sample.
The ISM endoscopic image is formed in an identical fashion to conventional ISM image formation [20].Specifically, each captured descanned frame is rescaled by a factor of 1/2 (Fig. 1d), and its center is shifted to the illumination position.The scaled and shifted frames are summed to form the final ISM image.

Confocal Endoscopy Z obj =150𝜇𝑚
Z obj =150 To demonstrate the spatial resolution and SNR improvements of ISM over confocal microendoscopy, we image the target at different distances from the fiber distal facet to the target.First, we image the sample at a contact distance (   = 0m) to establish the ground truth for the experimental measurements (Fig. 3a).Next, we position the target at a distance of 60m, where we observe an improvement in spatial resolution for the ISM-endoscopy image over the confocal image (Fig. 3b,c), due to the higher sampling frequency (Fig. 2a-b, Fig. 1d).A cross-section of the three images (Fig. 3d) shows the improvement in resolution, resolving the beads that compose the target at this cross-section, as validated in a transmission microscope image of the sample (Fig. 3e).At a larger distance of 150m (Fig. 3f,g), the ISM image displays a significant improvement in SNR over the confocal image due to the higher collection efficiency, leading to a resolution improvement that does not originate from the higher spatial sampling frequency (Fig. 2c,d, supplementary Fig. S2).Fig. 3i displays the experimental improvement in collection efficiency (black dots) measured by averaging the fiber cores that image sufficiently bright beads as a function of target distance, together with numerically simulated results.The full information on the numerical simulation is given in Supplementary Section 1.
All displayed ISM, and confocal images are Fourier-interpolated using a Hann filter to eliminate the cores pixelization.Since the ISM image has a ×2 denser grid of 'virtual' cores (Fig. 1d), the bandwidth of the Hann filter, set by the inverse of the core-to-core pitch, is doubled in the ISM images, compared to the filter used in the confocal image.

Discussion
In this work, we demonstrated a lensless micro-endoscopy approach that adapts the principles of image-scanning microscopy (ISM) to fiber-bundle endomicroscopy.Our proposed configuration utilizes off-the-shelf components and is based on a conventional confocal fiber-bundle endoscope with a camera replacing the confocal detector.By implementing ISM-endoscopy, we demonstrate an improved collection efficiency of up to a factor of approximately 18 at an object distance of 150m, and an enhanced spatial resolution compared to confocal endoscopy.
While the improvement of spatial resolution and collection efficiency in ISM has been extensively studied [20,22], we highlight an unreported additional advantage of ISM for spatially undersampled systems such as MCF bundles: doubling of the effective spatial sampling frequency, which doubles the accessible spatial frequency bandwidth of the imaging system.
ISM endoscopy can be applied to any fiber bundle endoscope, and the collection and resolution enhancements may vary according to the MCF parameters and object distances.This is primarily due to different fiber bundles having different numerical apertures (NA), which affect the distances in which the resolution improvement due to doubling the sampling frequency and improvement in collection efficiency are obtained.Importantly, the optimal number of reassigned neighboring cores depends on the object distance and fiber NA, where a larger number of cores collect a useful signal at a larger object distance, and higher NA [30].Combining ISM MCF-based endoscopy with advanced techniques that rely on wavefront shaping for focus-scanning and wavefront correction of fiber distortion [19,26] may be interesting for future studies.
A drawback of employing a camera rather than a single detector in the imaging system is a potentially slower image acquisition speed.At larger object distances, it may be interesting to study sparse and/or compressive measurements.
Lastly, we note that our ISM-based endoscope may enable label-free reflection endomicroscopy, which is challenging to perform in confocal detection.This is because, in reflection endomicroscopy, the illumination and detection wavelengths are identical.Detecting the weak reflected target signals in the presence of relatively-strong background reflections from the MCF facets is a great challenge [31,32].However, applying ISM while digitally masking the illumination core allows the reconstruction of target images from the signals collected in the neighboring unilluminated cores.

Fig. 1 .
Fig. 1.Image scanning lensless fiber-bundle fluorescence endomicroscopy.(a) setup: At the proximal tip of a multi-core fiber (MCF) a focused laser sequentially illuminates the MCF cores one by one using a galvo-scanner.The excitation light from each core illuminates a target object placed at a small distance from the distal tip (top inset).The fluorescence signal is collected by both the illuminating core and the neighboring cores (bottom inset).The collected fluorescence pattern for each illumination is imaged on an sCMOS camera through a dichroic mirror (DM) and a bandpass filter (F).(b) A simulated single camera image for an object placed at a distance of 60m from the distal tip, where the center core is illuminated.A confocal image can be generated by recording the signal only from the illuminating core, ignoring the signals from the neighboring cores (c).White 'x's mark the cores positions, i.e. the sampling grid.(d) An ISM image is formed by resizing the full camera image (b) by a factor of 0.5, and summing all captured frames.Thus, utilizing all detected photons and creating a ×2 denser sampling grid with new 'virtual' cores.

Fig. 2 .
Fig.2.Simulated OTFs comparisons between confocal endoscopy (blue), ISM endoscopy (red), and confocal imaging without sampling cores (dashed green) (a,b) OTFs crosssection for an object placed at a distance of m from the distal tip: due to the spatial sampling of the MCF cores, endoscopic OTFs suffer from aliasing artifacts exhibited by the folding of high spatial frequencies content (arrows), compared to ideal confocal imaging.ISM-endoscopy (red) doubles the effective spatial sampling frequency from   of confocal endoscopy (blue) to 2   (see Fig.1d, and the folding frequencies (dashed and dotted vertical lines, respectively).This allows correct retrieval of ×2 higher spatial frequency content in the ideal unsampled confocal OTF (dashed green).(c-d) (c-d) Same as (a-b) for an object distance of 150m.At larger object distances, the doubled sampling frequency of ISM endoscopy has a negligible impact due to the lower frequency content of the defocused object image.Nonetheless, the gain in signal collection efficiency of ISM improves the reconstructed images (see Fig.3and Supplementary Fig.S2).

Fig. 3 .
Fig. 3. Experimental results comparing ISM and confocal endoscopy images of fluorescent beads.(a) Reference 'contact' image of the fluorescence target when placed in contact with the distal fiber facet (   = 0).(b) Confocal image of the same target when placed at a distance of    = 60m from the distal tip.(c) ISM image from the same camera frames taken in (b) exhibiting improvements in resolution and SNR.(d) cross-sections of (a-c) over the dashed white line in (a) show the improvement in resolution of ISM endoscopy (red) over confocal endoscopy (blue).(e) Reference transmission microscope image of the target.(f,g) Same as (b,c) when the target is placed at a distance of    = 150m from the distal tip.At such a large distance, the improvement in SNR of ISM endoscopy due to the higher collection efficiency is significant.(i) Experimental (black dots) and numerical (red trace) plots of the improvement in the signal collection as a function of object distance.The improvement is significant at larger distances due to the larger defocus blur in the raw image (Fig.1b).Scale bars: 50m.