View images with unprecedented resolution in integral microscopy

Integral microscopy is a novel technique that allows the simultaneous capture of multiple perspective images of microscopic samples. This feature is achieved at the cost of a significant reduction of the spatial resolution. In fact, it is assumed that in the best cases the resolution is reduced by a factor that is not smaller than ten, what poses a hard drawback to the utility of the technique. However, to the best of our knowledge, this resolution limitation has never been researched rigorously. For this reason, the aim of this paper is to explore the real limitations in resolution of integral microscopy and to obtain optically, without the need of any image-processing algorithm, perspective images with the best resolution ever achieved in integral microscopy. This result opens a wide range of new possibilities of using integral microscopy in any imaging application were micron resolution is required. © 2018 Optical Society of America under the terms of the OSA Open Access Publishing Agreement


Introduction
Integral imaging [1] permits the recording of the spatial and the angular information of 3D scenes under incoherent illumination.In recent years, the interest in this technology has experienced a spectacular growth, and has been renamed in other ways, such as lightfield technology [2], or plenoptics technology [3].This technology can be utilized, for example, to compute depth maps [4], to remove occlusions [5] or to perform tracking of micrometer-sized specimens [6].Biomedical applications are also remarkable, such as the use of plenoptic technology in otoscopy [7], ophthalmology [8], endoscopy [9], and for deep-tissue inspection [10,11].
An application of integral imaging that deserves special attention is 3D microscopy [12].There exist three different modes for applying the Lippmann concept to microscopy.The first mode was proposed first by Levoy et al. [13] and is based on inserting a microlens array (MLA) just at the image plane of the host microscope.The integral microscope mounted according to this operation mode will be named here as IMic.The main advantage of this IMic is that it provides a high number of perspective images.The main drawback is that the spatial resolution of the views is determined by the competence between the MLA pitch, and the size of the point-spread function (PSF) of the host microscope.As a result, up to now IMic has demonstrated capacity for obtaining views with a spatial resolution that in the best case is about 10 times worse than that of the host microscope [14,15].
The second mode, denoted as integral-imaging microscopy is based on the so-called plenoptic 2.0 concept [16].In this case, the MLA is inserted at an intermediate plane within the image space of the host microscope.The main advantage of this design is that it is possible to obtain views with a resolution that is more close to that of the host microscope [17,18].The main drawback is that the number of views is small and therefore the 3D effect is strongly limited.In the third mode, the MLA is inserted at the Fourier plane of the host microscope [19][20][21].As in the second mode, this microscope provides views with good resolution, but at the cost of reducing the number of views.The main advantage of this design is the longer depth of field of views images.
Focusing our attention in the IMic mode, it is remarkable that in the state of art it is assumed that it can work accurately only when the PSF of the host microscope is significantly smaller than the MLA pitch.This has limited strongly the resolution of the views provided by IMic [13][14][15].Note however that in the past, no systematic study has been performed in order to find which is the acceptable ratio between the host PSF and the MLA pitch.Thus, no research has been focused in exploring which is the real resolution frontier of IMic; i.e., how much can the resolution of view images to the host resolution approach?In this context, the aim of this paper is to obtain perspective views with the best resolution ever achieved in integral microscopy.To get this aim we design an ad hoc experiment that permits to explore and find such frontier.As results of this experiment we demonstrate that integral microscopy can provide perspective images with resolution close to the one provided by 2D conventional microscopy.

Basic theory of IMic
To build an IMic, the microscope objective (MO) and the tube lens (TL) of the host microscope are placed in telecentric configuration.Then, a MLA is placed at the image plane, where the camera sensor was originally located, and the latter is shifted by a distance that is equal to the focal length of the MLA, ML f (see Fig. 1).With this configuration, an IMic records an integral image that is comprised by a set of microimages that carry the 3D information of the sample.The number of microimages is equal to the number of usable microlenses in the array.To avoid any overlapping between the recorded microimages, the numerical aperture in the image space of the MO, NA', and the numerical aperture of the microlenses, NA ML , should be equal.This condition can be written as ML ' , where M = f TL /f MO is the lateral magnification of the host microscope.From the integral image, it is easy to compute, by a pixel-mapping procedure, the collection of orthographic view images [14].Using as the input such perspective images, regular refocusing algorithms [22,23] are utilized for computing the wanted 3D information of the studied sample.Note that in the refocusing process it is possible to gain some resolution by digital processing techniques [24].However, since such digital procedures can be applied to any collection of view images, the key point here is to obtain optically the view images with the best possible resolution.
The spatial resolution limit of the computed perspective views is the result of the competence between two factors [20,21].One factor is due to diffraction and the other is due to the sensor pixilation.This competence can be expressed as where p is the pitch of the MLA, and stands for the resolution limit as defined by Abbe at the objet plane, being 0 λ the emission wavelength .
At this moment, it is convenient to define the dimensionless parameter hst ' , where ρ' hst = M ρ hst is radius of the host PSF, but evaluated at the image plane.This parameter, named here as the coupling parameter, accounts for the number of microlenses covered by the image host PSF.Now Eq. ( 2) is rewritten as The depth of field of the view images can be calculated, also in terms of the coupling parameter [20], as ( ) where

Exploring the validity of resolution formula
IMic is a hybrid technique in which we can distinguish two very different stages.The first stage, the capture of microimages, is a purely optical process.As result, the information held by the microimages is strongly influenced by optical experimental factors, such as aberrations, vignetting or diffraction.The second stage is the calculation of the perspective images from the microimages.This is a purely computational procedure in which it is assumed that each pixel holds the spatio-angular information of a single light ray that passes through the center of the microlens.In the state of art of IMic it is stated that this assumption is acceptable provided that the size of the image host PSF, ' hst ρ , is much smaller that the MLA pitch [15,24,25].In other words, the state of art assumes that distinguishable perspective views with the resolution predicted by Eq. ( 5) can be obtained only when η 0.2 < .Thus, one should expect that higher values of η would give rise to a strong conflict between the wave nature of microscopic imaging and the para-geometric nature of IMic computations.As result of this conflict, the higher the value of η , the smaller the number of distinguishable perspectives.Thus, it is commonly assumed that, it has no sense to build an integral microscope with η 0.2 > .For this reason, no high-resolution IMic has ever been built.Nevertheless, a remarkable fact is that for values of η 0.2 > there is no any experimental research that explores the capacity of IMic for providing distinguishable perspectives with the resolution predicted by Eq. ( 5) and the expected parallax.Thus, and in order to fill this gap, we consider necessary to perform a rigorous and systematic experimental study in which many IMic with η 0.2 > are implemented and the perspective images are analyzed.To this end, an IMic like the one sketched in Fig. 2 was set up.The host microscope was composed by a 50x, NA = 0.55 MO.The parameters of the MLA were p = 110 μm and NA ML = 0.010 (MLA #12-1192-106-000 from SUSS MicroOptics).A digital camera Canon EOS 450D assembled with a macro objective 1:1, was the selected camera sensor.To modify continuously the value of η, one should have a large collection of MLAs with different pitch but same NA ML .Other possibility, much more feasible, is to use tube lenses with different values of f TL , and to insert an iris diaphragm at the aperture stop.However, since the aperture stop is not easily accessible, a telecentric relay is needed.In our case the relay is composed by two converging doublets of focal lengths f 1 = 150 mm and f 2 = 75 mm.In our study, we used tube lenses with TL f ranging from 100 mm to 300 mm .The diaphragm diameter, D φ , was set from 0.50 mm to 2.20 mm.This leaded to values of η ranging from 0.25 to 1.67.In the first experiment, we used as an object an USAF 1951 resolution target, which was placed at the focal plane of the IMic, and was illuminated with the white light proceeding from a fiber bundle.We captured the set of microimages for six different values of η.In Fig. 3 we show the results of the experiment for the particular case of η = 0.25.In particular, we illustrate what is the aspect of the microimages captured optically with the IMic, and of the view images computed from them.We show, as well, the procedure for evaluation of the resolution limit.Specifically, for determining if one element of the USAF target is resolved we integrate the intensity profile along the element and check for the existence of an intensity dip of at least 25%.
The experimental results are summarized in Table 1.The keys for the correct understanding of the Table are the following: In the first column we write the focal length of the tube lens.This value affects the effective magnification of the host microscope, which can be calculated as where M = 50 is the nominal magnification of the MO, which is calculated assuming a tube lens with focal length of 200 mm.
In the second column we write the diameter of the diaphragm.The value φ D = 2.20 mm corresponds to the case in which the diaphragm is totally open and therefore the system aperture is limited by the MO aperture stop.In the last three rows it was necessary the use of small diameters of the diaphragm.
In the third column we write the effective NA of the system.When the diaphragm is open the NA eff does not change.However, as the diaphragm is set smaller and smaller, the effective NA decreases proportionally.
In the fourth column we have written the host resolution limit, which is calculated with Eq. ( 3).In column fifth we write the size of the PSF at the image plane.It is calculated as ρ' hst = M eff ρ hst .In the sixth column we write the value of the coupling parameter η.
Finally, in the seventh and eight columns we have written the theoretical, Eq. ( 5), and measured value of the resolution limit of the view images.

Theoretical
Measured From Table 1 we confirm that for low values of η the spatial resolution of the calculated view images is much worse than the host resolution.However, as the value of increases, the difference between the two values of resolution is shrunk.For values of η close to, 2.0 the two resolutions are comparable.But the most important outcome is that the obtained resolutions are the ones predicted by Eq. ( 5).Thus we can confirm that the validity of such equation is not restricted to values of η < 0.2, but at a much wider range.

Confirming the existence of distinguishable perspective views
In the previous Section, we have demonstrated that IMic can produce view images whose resolution is not restricted to View hst 10 > ρ ρ .However, our work is not finished yet, since we still need to demonstrate that the high-resolution view images have the expected parallax and can be combined to produce high-quality refocused images.
To demonstrate this, we performed a new microscopy experiment, but using now as the object a 3D microscopic sample stained with a fluorescent ink.As the sample we used regular cotton, which from microscopic point of view, is an almost hollow specimen with large depth range and composed by long fibers with thin structure.The cotton was stained with fluorescent ink proceeding from a marking pen, and illuminated with the light proceeding from a laser of wavelength exc 532 nm = λ .The emission wavelength was around 0 600 nm = λ .Thus, a chromatic filter ( c 550 nm = λ ) was used to reject the laser light.For this experiment we used the IMic setup corresponding to 0.50 = η .Note that we could no use for this experiment higher values of η, because they require (for the MLA available in our Lab.) for the insertion of a diaphragm that reduces the effective NA of the system and therefore the captured parallax.Anyway, even working with this value of η the IMic can produce view images with, by far, the best resolution ( View 2.  Additionally, we have prepared Visualization 3 and Visualization 4 with the images obtained with η = 0.25.Note that in this case resolution is worse than with η = 0.50, but the DoF is longer, as expected from Eq. (6).

Conclusions
In summary, in this letter we have researched for the spatial resolution capability of IMic.
Contrarily to what it is assumed in the IMic community, we have demonstrated that IMic can produce view images with high resolution, and that these view images have the expected parallax and the capacity of depth refocusing.Apart from the proof-of-concept experiment, we have implemented an IMic that has produced view images with the best resolution ever achieved in IMic.
It is remarkable that in the past some interesting research was addressed to design new digital processing algorithms, see for example [24], for improving the resolution of refocused images calculated from low-resolution view images.Although the design or use of such algorithms is not the aim of our research, naturally this kind of computational tools could be applied as well to the high-resolution views obtained with IMic operating with high η.

Fig. 2 .
Fig. 2. Scheme of the experimental setup for determining the validity of resolution and DoF formulae.

Fig. 3 .
Fig. 3. Results of the experiment for η = 0.25.(a) Microimages recorded by the camera sensor; (b) View images computed from the microimages; (c) and (d) show intensity profiles along resolved (Element 7.6) and non-resolved (Element 8.1) elements, respectively, of the central view image.
2 m = ρ μ ) ever achieved in integral microscopy.The captured microimages are shown in panel (b) of Fig. 4. From the microimages we calculated 21x21 view images.However, as shown in panel (c) only 225 among them are free of vignetting, and therefore useful as input for the refocusing algorithm.In panel (d) we show the central view image.With some of the view images we composed a video, Visualization 1, to demonstrate the horizontal and vertical parallax.In integral microscopy, and also in any other imaging technique based on the Lippmann concept, the capacity of refocusing comes from the existence of many different views of the same 3D object.So the capacity of refocusing is fully preserved.To demonstrate this in Visualization 2 we show a movie showing the depth refocusing calculated from the view images.