Ptychographic lensless coherent endomicroscopy through a flexible fiber bundle

Conventional fiber-bundle-based endoscopes allow minimally invasive imaging through flexible multi-core fiber (MCF) bundles by placing a miniature lens at the distal tip and using each core as an imaging pixel. In recent years, lensless imaging through MCFs was made possible by correcting the core-to-core phase distortions pre-measured in a calibration procedure. However, temporally varying wavefront distortions, for instance, due to dynamic fiber bending, pose a challenge for such approaches. Here, we demonstrate a coherent lensless imaging technique based on intensity-only measurements insensitive to core-to-core phase distortions. We leverage a ptychographic reconstruction algorithm to retrieve the phase and amplitude profiles of reflective objects placed at a distance from the fiber tip, using as input a set of diffracted intensity patterns reflected from the object when the illumination is scanned over the MCF cores. Our approach thus utilizes an acquisition process equivalent to confocal microendoscopy, only replacing the single detector with a camera.


Introduction
Flexible fiber micro-endoscopes allow microscopic imaging deep inside tissues beyond conventional microscopes' absorption and scattering limits.Recent devices based on flexible optical fibers allow imaging with micrometer resolution, making them suitable for imaging neurons and cells in various contrast mechanisms [1][2][3].To generate an image, fiber-based endoscopes traditionally required bulky optical elements, such as lenses [4] and/or mechanical scanners [5] to be placed at the distal fiber end, increasing their footprint and the potential tissue damage.
While lensless endoscopy is a desired goal, the main hurdle in lensless coherent imaging through flexible fibers is the dynamic wavefront distortions due to phase randomization between the different transverse modes.In recent years, lensless fiber endoscopes utilizing bare multi-mode fibers (MMF) or multi-core fibers (MCF, also referred to as coherent fiber-bundles) have been demonstrated via wavefront-shaping techniques [6,7].In these works, the inherent wavefront distortions are corrected physically (using a spatial light modulator, SLM) or computationally.Initially, the wavefront distortions have been measured prior to the imaging experiments [8][9][10][11][12][13][14][15][16][17], and the correction was effective as long as the fibers remained static after the wavefront distortion measurement (or calibration) step.
However, as these calibration-based approaches cannot handle the temporally varying wavefront distortions of flexible fiber bending, more recent developments have allowed calibration-free lensless endoscopy based on either iterative wavefront optimization of a nonlinear signal or image metric [18,19], the addition of a distal mirror [20], computational reconstruction using multiple holographically measured fields [21][22][23], or phase-retrieval of simple incoherent targets [24,25].
Here, we demonstrate that coherent phase-sensitive lensless imaging through MCFs can be achieved by simple intensity measurements, without correcting the core-to-core phase distortions, by employing ptychographic phase retrieval.Specifically, we implement a conventional rasterscanned illumination in epidetection geometry as performed in confocal-endoscopy [4,26] to record the diffraction intensity patterns from targets placed at a small distance from the MCF (Fig. 1).This set of diffraction patterns, measured at the proximal facet of the fiber, is used as the input of a standard ptychographic reconstruction algorithm [27,28], which recovers the object amplitude and phase profile.The approach is agnostic to the MCF phase distortions as the MCF is used in detection only as an intensity relay, and by providing the illumination to the object through a single (scanned) core.The true phases of the diffraction patterns are computationally retrieved in the ptychographic reconstruction process.The principle of ptychographic endoscopy is depicted in Fig. 1.Our approach is based on the insight that for a target placed at a small distance from the fiber, conventional raster-scanned illumination of the fiber cores (Fig. 1a inset), results in scanning a small illumination spot over the target area (Fig. 1b).The reflected light intensity pattern measured at the proximal MCF facet for each illumination position is the diffraction pattern of each illuminated area on the target, which overlaps with the neighboring illuminated areas (Fig. 1b).These measurements are thus equivalent to the measured diffraction patterns of ptychographic coherent-diffractive imaging (CDI) [29], where a small illumination 'probe' is sequentially scanned over the target.

Principle
The well-established ptychographic phase-retrieval algorithms [30] can thus be utilized for reconstructing the complex-valued target from this set of lensless, intensity-only measurements (Fig. 1c).
The process of ptychographic endoscopy thus consists of two steps: 1) Measurement step (Fig. 1a,b): In every acquisition step, the illumination is focused on a single core, as shown in the inset of Fig. 1a.This effectively illuminates a small area on the object, defined by the diffraction of the excited mode in the single core.The reflected light from the illuminated object area propagates back to the distal MCF facet (Fig. 1b), and the resulting diffraction pattern is collected by the MCF and recorded in its proximal end.Scanning the illumination over the MCF cores results in overlapping illuminated areas across the object.2) Reconstruction step (Fig. 1c): The camera images of the diffraction patterns from each of the illuminated cores (Fig. 1c, left) are fed into a ptychographic reconstruction algorithm.In addition to the recorded intensity patterns, an initial estimation of the illumination (the ptychographic 'probe') and the distance from the fiber to the target are all fed as inputs to the reconstruction algorithm.The standard ptychographic iterative reconstruction algorithm searches for the complex-valued object (and exact probe function) that will best fit the entire set of measured diffraction intensity patterns, i.e., it retrieves the amplitude and phase of the target object (Fig. 1c, right) [30].

Forward model and reconstruction algorithm
Here, we present a straightforward mathematical model of the forward process and the reconstruction algorithm.Consider a monochromatic coherent illumination given by the complex-valued electric field distribution at the MCF distal facet,   (r), where r is the transverse position along the MCF facet, and  is an index marking the illuminated core.The illumination function at the object plane (the 'probe' function) is given by the free-space propagation of   (r) to the target plane: Here, P denotes the free-space angular-spectrum propagation over a distance , the distance between the distal facet of the MCF and the target.As an initial guess of the illumination function, we consider the free-space propagation of the fundamental fiber mode (LP01) of the -th illuminated core from the distal facet of the MCF the target plane.At the target plane, this probe can be approximated by a 2D Gaussian amplitude and a parabolic phase, arises from free-space propagation.
The unknown target object is modeled by a 2D amplitude and phase complex-valued reflectivity profile denoted by  (r).The reflected field at the object plane in the -th illumination is given by the product of the illumination (probe) beam and the object reflectivity:   (r) =  (r) •   (r).Further simplification can be made by the approximation of a shift-invariant probe function, which perfectly holds for identical MCF cores illuminated at the fundamental modes: Where   is the position shift for the -th illumination core center to its de-scanned position.The propagation of the reflected field at the target plane,   (r), back to the distal facet of the MCF is modeled by the same free-space propagation, giving the complex amplitude of the reflected field at the MCF distal facet:   (r) = P  [  (r)].This back-reflected field is coupled into the MCF cores, each with a limited collection NA (∼   = 0.35 in our experiments).Due to the spacing between fiber cores (∼ 4.1 in our experiments) being relatively large compared to the Nyquist criteria (∼ /4 ), the field that is measured at the proximal side by the camera is sparsely sampled.Thus, the intensity measured at the proximal facet by the camera is modeled by: (i) filtering the field at the distal facet,   (r) by a Gaussian filter having a bandwidth of   • 2/ in the angular spectrum domain, (ii) taking the intensity of this 'proximal' field: (iii) summing the power of this intensity distribution at each core, and assigning this value as the measured intensity in the core position:    (r) ≈     (r) *    (r), where    (r) is the core shape, approximated by the average circular core profile.
To reconstruct the complex reflectivity of the object  (r) from the intensity patterns measured by the camera, we follow the well-established reconstruction scheme of the ptychographic iterative engine (PIE) [27] family, where the propagation between the object plane and the detection plane is given by angular spectrum propagation for a near-field/Fresnel ptychographic reconstruction [31][32][33][34].At each iteration, the algorithm uses the camera measurements one by one to update the current object and probe estimates such that the probed object diffraction intensity patterns minimize the error with the measured data in a manner similar to hybrid input-output (HIO) conventional phase retrieval [35].Ptychographic reconstruction offers a more robust reconstruction performance over conventional "keyhole" phase retrieval due to its utilization of joint information from overlapping object parts.The best reconstruction results were obtained by using the momentum ptychography iterative engine (mPIE) version [27], and the Semi-implicit relaxed Douglas-Rachford algorithm (sDR) whose source codes are available [28], which we adapt to near-field/Fresnel ptychographic scheme using angular spectrum propagation instead of the far-field propagation.

Experimental results
Our experimental setup is schematically depicted in Fig. 1a, with the experimental results summarized in Fig. 2 and Fig. 3.In all experiments, a diode laser (Finesse Pure, Laser Quantum) with a wavelength of  = 532  is focused and reflected by a two-axis galvanometer mirror (GM) onto the proximal facet of a one-meter-long MCF (Fujikura, FIGH-10-350S), having ∼ 8, 350 cores, each with an average core diameter of ∼ 2 , and an average core-to-core distance of 4.1  and a total diameter of 350  [21,36].
As the target objects, we used reflective test targets (Ready Optics Siemens Star, Edmund Positive 1951 USAF Target) placed in a water-filled tank at known distances ranging from 100 to 700 from the distal facet.During each acquisition step, a single core is illuminated while the neighboring cores of the same fiber collect the back-reflected signals.The illumination beam's numerical aperture (NA) is chosen so that at each illuminated core, the illumination mainly excites the fundamental fiber mode (LP01).This ensures that the illumination probe at the object is similar throughout the scan.To take into account the possible variation of the probe beam from the ideal fundamental fiber mode, we allow the reconstruction algorithm to correct the initial probe guess during the iterations [37].While core-to-core variations in excitation and core shape are present [36], allowing the algorithm to update the probe separately at each illumination position did not improve the reconstruction quality.
During the acquisition, the sCMOS camera (pco.edge 4.2) captures an image of the optical intensity at the MCF proximal facet, magnified by a factor of ∼ 37.As mentioned above, the MCF cores undersample this intensity distribution due to the inherently large core-to-core distance compared to the core diameter.To accommodate this undersampling, we have used as the input to the algorithm for each illumination an intensity distribution that is an interpolation of the raw measured cores intensities.
Sample reconstructions of the different targets are presented in Figures 2-3.Fig. 2 presents the reconstruction of a Siemens star resolution target and a USAF target at distances of 200 and 700, respectively.For these reconstructions, a positivity constraint was used in the object plane.We study the effect of different object-plane constraints in Supplementary Section 3. Several raw measured camera frames are given in Fig. 2a,e, demonstrating that while the diffracted patterns at close distances show some similarity with the illuminated object parts (Fig. 2a), the reflected patterns at the longer distance from the sample (Fig. 2e) are diffracted over a large area, with an average intensity that is significantly lower than the reflection from the illuminated core facet.We use 2, 000 − 3, 000 frames of the proximal facet as an input to the Fresnel Ptychographic algorithm to reconstruct the objects (Fig. 2b, f).The objects can be reconstructed from fewer illuminations, as we study in Supplementary Section 2. While the Ptychographic reconstructed images clearly show the fine details of the targets, conventional widefield images taken through the same MCF (by summing the measured patterns) are unable to resolve any object features (Fig. 2c, g), due to the diffraction from free-space propagation back to the fiber distal facet.We note that the relatively strong reflections from the illuminated core (Fig. 2e) do not allow confocal-reflection endoscopy.We have digitally filtered the illuminated cores from the raw measured images to mitigate these strong reflections.
Reference contact measurements of the two targets were taken by bringing the tip of the fiber bundle into contact with each target and illuminating the targets from the back using a light-emitting diode (Thorlabs M530L3).These reference images are shown in Fig. 2d,h, and represent the best images that can be achieved by conventional MCF endoscopy.However, unlike conventional MCF-based micro-endoscopes where each fiber core acts as an imaging pixel, limiting the spatial resolution to twice the core-to-core pitch (Fig. 3a), advanced holographic [20,21] and wavefront-shaping imaging approaches [38] can theoretically offer diffraction-limited resolution and unpixelated field reconstruction (Fig. 3b).This feature is also provided by our ptychographic endoscopy approach, as can be already observed by comparing the contact reference images (Fig. 2d,h) and the ptychographic reconstructions (Fig, 2b,f).To further study and demonstrate this feature, we provide in Fig. 3 two azimuthal cross-sections through the Siemens star target at radii of 20 and 50, corresponding to periodicity frequencies of 3.2 and 8, respectively (Fig. 3c).Our ptychographic reconstruction improves even over conventional contact measurements.It can distinguish line pairs at a 3.2 and 8 periodicity, while the conventional contact measurement can only separate spokes with an 8 periodicity.

Numerical study of complex (amplitude and phase) targets
To further investigate the potential of our method, which can, in principle, recover the phase profile of unlabeled specimens, we numerically simulate different phase and amplitude targets using the same experimental parameters.The results of this study are presented in Fig. 4.
The simulated object has an amplitude profile given by the 'cameraman' image (Fig. 4a), and a phase profile that is an orthophoto of West Concord (Fig. 4b).The simulated object is placed at a distance of 200 from the distal facet of the fiber.841 cores of the simulated fiber are illuminated sequentially in a Fermat spiral, known to improve the uniformity of illumination overlap in ptychography [39].A widefield summation, which sums over the  = 841 detected intensity patterns on the camera, fails to resolve any phase or amplitude information (Fig. 4c).However, the ptychographic reconstruction (Fig. 4d) recovers an estimate of the target's amplitude and phase.However, we find that strong (0 − 2) sharp phase modulation results in mixing phase and amplitude information in the reconstructed image.Such artifacts are not present in ideal Fresnel Ptychography (Fig. 4f).Thus, their presence can be attributed either to the illumination probe being a soft-edge Gaussian with a parabolic phase rather than a top-hat flat-phase probe or to the undersampling of the detected intensity patterns by the MCF cores.
To study the source for these artifacts, we simulate a theoretical endoscopic ptychographic setup that utilizes the same illumination probe but avoids undersampling of the intensity patterns by sampling the diffracted patterns with an ideal camera with a pixel-to-pixel pitch of 0.4.While this configuration demands major changes, enlarging the device's footprint and requiring angled illumination and detection, we use it to study artifact origins in our suggested setup.The result of this simulation (Fig. 4e) shows significantly improved reconstruction, suggesting that the MCF undersampling is a major source of the ptychographic reconstruction artifacts.For completeness, we simulate a conventional Fresnel ptychography measurement and reconstruction of the same target using a top-hat flat-phase probe measured by the same camera (Fig. 4f).The improved reconstruction fidelity of these measurements suggests that the probe shape also affects the ptychographic reconstruction.We plot the field correlation between the object and each reconstruction to assess the reconstruction quality quantitatively.For clarity, the illumination and detection scheme changes are schematically depicted in a transmission geometry equivalent to unfolding the reflection setup.(d) Reconstruction using our suggested lensless ptychographic endoscopy is given from  = 841 different core illuminations.The detection scheme we suggest is insufficient for a clear reconstruction of the simulated target but shows a significant improvement over the image given in (c).(e) Suggested modification of our setup, switching the detector of the back-reflected intensity patterns to a camera with a high fill factor and higher numerical aperture.(f) Comparison to a conventional ptychography setup [30], where an aperture is placed directly against the specimen at each illumination, and the diffraction pattern is captured with a camera.The field correlation coefficient between the simulated object field and each field reconstruction is presented for all simulation results.

Discussion
We have introduced a simple lensless micro-endoscopic approach for coherent reflection imaging, utilizing readily available multi-core fibers.We adopt the image acquisition approach of standard confocal laser microscopy but replace the confocal detection scheme with a camera-based detection and reconstruct the complex-valued object reflectivity by a ptychographic phaseretrieval.For imaging fluorescence objects, we have recently proposed a simple reconstruction scheme that employs the principles of image scanning microscopy (ISM) [40].ISM improves the resolution of conventional MCF-based endoscopes but can not reach the significant resolution improvement of the ptychographic reconstruction, theoretically offering a diffraction-limited resolution.
Since the diffracted patterns span a large area on the MCF for sufficiently far objects, the ptychographic algorithm does not necessarily require scanning the illumination over all MCF cores.In our experiments, the reconstructions were obtained by illuminating approximately only 10 − 30% of the fiber cores (see Supplementary Section 2).Interesting potential in improving the acquisition speed may be offered by multiplexed measurements [41][42][43].
Our study indicates that the main limitation of the ptychographic endoscopy is in the reconstruction phase and amplitude objects, given the spatial under-sampling of the commercial MCF used.This limitation can be overcome by using higher fill factor fibers or other detector arrays or by changing the reconstruction framework to account for potential priors in the imaged objects.To this end, deep-learning-based techniques may prove useful, as demonstrated in ptychography [44] and in imaging through diffuser [45].
Interestingly, while we have considered only planar reflecting objects here, ptychographic reconstruction can provide a three-dimensional reconstruction of thick samples [46][47][48].

Fig. 1 .
Fig. 1.Ptychographic lensless fiber-bundle endomicroscopy.(a) The optical setup: A 2-axis galvanometer scans the illumination across the proximal facet of a multi-core fiber (MCF), illuminating each core one by one.An sCMOS camera that images the proximal facet records the diffraction patterns back-scattered from a target object placed at a small distance from the MCF distal tip for each illuminated core (inset).(b) At the -th core illumination, a different area in the target object is illuminated, effectively realizing the scanning 'probe' of conventional ptychography,   (r).(c) Sample numerically simulated diffraction intensity patterns recorded for different illuminated cores (left) and the resulting reconstructed target intensity profile (right).Scale bars: 25.

Fig. 2 .
Fig. 2. Experimental imaging of high-resolution targets at different distances from the fiber tip.(a) a few camera images of the reflected diffraction intensity of a target located 200  from the fiber tip illuminated from a single core, looking at the proximal fiber facet.(b) From the  = 3000 intensity patterns, an experimental reconstruction of a Siemens star resolution target is obtained by Fresnel ptychography processing of the raw frames.(c) An image is formed by summing the detected diffraction images in (a), estimating an image given by a wide field illumination.(d) Direct contact measurement of the target, back-illuminated by an incoherent illumination.Insets in (b-d) show magnified images of the target center, demonstrating resolution improvement over widefield summation and contact images.(e-h) Same as (a-d) for the USAF resolution target 700  from fiber tip and  = 2000 intensity patterns.Scale bars: 25.

Fig. 3 .
Fig. 3. Experimental resolution demonstration using a Siemens star target with 40 spoke and space pairs from 150  width.(a) Direct contact measurement with incoherent back illumination featuring a high-resolution spoke zoom-in.(b) Our lensless ptychographic endoscopy reconstruction of the target, placed 200 away from the fiber tip, with a matching zoom-in.(c) Amplitude plots along the circumference of the red circle in (a) and the blue circle in (b) at 3.2 and 8 periodicity, respectively, demonstrate our method resolving pairs that are indistinguishable in contact measurements at 3.2  periodicity.Scale bars: 25

Fig. 4 .
Fig. 4. Numerical investigation of complex (Amplitude and Phase) targets.(a-b) An object with complex amplitude and phase patterns is placed at 200  from the fiber tip.(c) Widefield summation of the target cannot recover the amplitude or phase of the target.(d)Reconstruction using our suggested lensless ptychographic endoscopy is given from  = 841 different core illuminations.The detection scheme we suggest is insufficient for a clear reconstruction of the simulated target but shows a significant improvement over the image given in (c).(e) Suggested modification of our setup, switching the detector of the back-reflected intensity patterns to a camera with a high fill factor and higher numerical aperture.(f) Comparison to a conventional ptychography setup[30], where an aperture is placed directly against the specimen at each illumination, and the diffraction pattern is captured with a camera.The field correlation coefficient between the simulated object field and each field reconstruction is presented for all simulation results.