Resolution improvements in integral microscopy with Fourier plane recording

Integral microscopes (IMic) have been recently developed in order to capture the spatial and the angular information of 3D microscopic samples with a single exposure. Computational post-processing of this information permits to carry out a 3D reconstruction of the sample. By applying conventional algorithms, both depth and also view reconstructions are possible. However, the main drawback of IMic is that the resolution of the reconstructed images is low and axially heterogeneous. In this paper, we propose a new configuration of the IMic by placing the lens array not at the image plane, but at the pupil (or Fourier) plane of the microscope objective. With this novel system, the spatial resolution is increased by factor 1.4, and the depth of field is substantially enlarged. Our experiments show the feasibility of the proposed method. ©2016 Optical Society of America OCIS codes: (110.6880) Three-dimensional image acquisition; (180.6900) Three-dimensional microscopy; (100.6890) Three-dimensional image processing. References and links 1. J. Huisken and D. Y. R. Stainier, “Selective plane illumination microscopy techniques in developmental biology,” Development 136(12), 1963–1975 (2009). 2. E. H. K. Stelzer, K. Greger, and E. G. Reynaud, Light Sheet Based Fluorescence Microscopy: Principles and Practice (Wiley-Blackwell, 2014). 3. M. A. A. Neil, R. Juskaitis, and T. Wilson, “Method of obtaining optical sectioning by using structured light in a conventional microscope,” Opt. Lett. 22(24), 1905–1907 (1997). 4. J. B. Pawley, Handbook of Biological Confocal Microscopy, 3rd ed. (Plenum, 2006). 5. E. Sánchez-Ortiga, A. Doblas, G. Saavedra, M. Martínez-Corral, and J. Garcia-Sucerquia, “Off-axis digital holographic microscopy: practical design parameters for operating at diffraction limit,” Appl. Opt. 53(10), 2058– 2066 (2014). 6. G. Lippmann, “Epreuves reversibles donnant la sensation du relief,” J. Phys. 7, 821–825 (1908). 7. H. E. Ives, “Optical properties of a Lippmann lenticulated sheet,” J. Opt. Soc. Am. 21(3), 171–176 (1931). 8. H. Arimoto and B. Javidi, “Integral 3D imaging with digital reconstruction,” Opt. Lett. 26, 157–159 (2001). 9. S.-H. Hong, J.-S. Jang, and B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express 12(3), 483–491 (2004). 10. Y. S. Hwang and E. S. Kim, “Perspective-variant reconstruction of a three-dimensional object along the depth direction with virtually generated elemental images by ray-based pixel mapping in integral-imaging,” Opt. Lasers Eng. 51(7), 797–807 (2013). 11. S.-H. Hong and B. Javidi, “Improved resolution 3D object reconstruction using computational integral imaging with time multiplexing,” Opt. Express 12(19), 4579–4588 (2004). 12. M. Cho and B. Javidi, “Computational reconstruction of three-dimensional integral imaging by rearrangement of elemental image pixels,” J. Disp. Technol. 5(2), 61–65 (2009). 13. D. H. Shin and H. Yoo, “Computational integral imaging reconstruction method of 3D images using pixel-topixel mapping and image interpolation,” Opt. Commun. 282(14), 2760–2767 (2009). 14. J.-Y. Jang, J. I. Ser, S. Cha, and S. H. Shin, “Depth extraction by using the correlation of the periodic function with an elemental image in integral imaging,” Appl. Opt. 51(16), 3279–3286 (2012). 15. J. Y. Jang, D. Shin, B. G. Lee, S. P. Hong, and E. S. Kim, “3D image correlator using computational integral imaging reconstruction based on modified convolution property of periodic functions,” J. Opt. Soc. Korea 18(4), 388–394 (2014). 16. A. Llavador, E. Sánchez-Ortiga, G. Saavedra, B. Javidi, and M. Martínez-Corral, “Free-depths reconstruction with synthetic impulse response in integral imaging,” Opt. Express 23(23), 30127–30135 (2015). Vol. 24, No. 18 | 5 Sep 2016 | OPTICS EXPRESS 20792 #269053 http://dx.doi.org/10.1364/OE.24.020792 Journal © 2016 Received 22 Jun 2016; revised 22 Jul 2016; accepted 10 Aug 2016; published 31 Aug 2016 17. J. S. Jang and B. Javidi, “Three-dimensional integral imaging of micro-objects,” Opt. Lett. 29(11), 1230–1232 (2004). 18. M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” ACM Trans. Graph. 25(3), 924–934 (2006). 19. Y. T. Lim, J. H. Park, K. Ch. Kwon, and N. Kim, “Resolution-enhanced integral imaging microscopy that uses lens array shifting,” Opt. Express 17(21), 19253–19263 (2009). 20. K.-Ch. Kwon, J.-S. Jeong, M.-U. Erdenebat, Y.-L. Piao, K.-H. Yoo, and N. Kim, “Resolution-enhancement for an orthographic-view image display in an integral imaging microscope system,” Biomed. Opt. Express 6(3), 736–746 (2015). 21. A. Llavador, E. Sánchez-Ortiga, J. C. Barreiro, G. Saavedra, and M. Martínez-Corral, “Resolution enhancement in integral microscopy by physical interpolation,” Biomed. Opt. Express 6(8), 2854–2863 (2015). 22. M. Martinez-Corral, G. Saavedra, E. Sánchez-Ortiga, A. Llavador, and J. Sola-Pikabea, “Microscopio integral, usos del mismo y sistema de microscopía integral,” Spanish patent P201531938. 23. R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a handheld plenoptic camera,” Tech. Rep. CSTR. 2 (2005). 24. H. Navarro, E. Sánchez-Ortiga, G. Saavedra, A. Llavador, A. Dorado, M. Martinez-Corral, and B. Javidi, “Nonhomogeneity of lateral resolution in integral imaging,” J. Disp. Technol. 9(1), 37–43 (2013). 25. M. Martinez-Corral, A. Dorado, A. Llavador, G. Saavedra, and B. Javidi, “Three-dimensional integral imaging and display” in Multi-Dimensional Imaging, B. Javidi, E. Tajahuerce, and P. Andres, eds. (John Willey and Sons, 2014), Chap. 11. 26. J. S. Jang and B. Javidi, “Three-dimensional synthetic aperture integral imaging,” Opt. Lett. 27(13), 1144–1146 (2002). 27. B. Javidi, I. Moon, and S. Yeom, “Three-dimensional identification of biological microorganism using integral imaging,” Opt. Express 14(25), 12096–12108 (2006).


Introduction
The acquisition and processing of three-dimensional (3D) information of microscopic samples have been extensively studied during the recent years due to their applications in many biological processes.There exist many techniques that can provide this 3D information.By axially scanning the sample it is possible to obtain a stack of images with optical sectioning.This is the case of the Light-sheet microscopy [1,2] and the structured illuminated microscopy [3].Moreover, confocal microscopy provides a stack of images with optical sectioning by 3D scanning the sample [4].An inconvenience of these techniques is that they need to capture many images in order to register the 3D information.An alternative is digital holographic microscopy, which has the capability to provide the 3D information of a coherently illuminated sample with a single shot [5].
A different approach is using the Integral Imaging (InI) technique.The original idea was proposed by Lipmann in 1908 [6], and consists of placing a microlens array (MLA) in front of an image sensor.Thus, it is possible to capture many elemental images (EIs) that provide information of different perspectives of the 3D scene.The integral image captured by the system contains the spatial and the angular information [7], so it is possible to reconstruct the scene at various depths [8][9][10].For performing this task, different reconstruction algorithms have been proposed [11][12][13][14][15][16].In [17,18], several works have been developed adapting InI technology to microscopy.Although results are very promising, the main drawback in Integral Microscopy (IMic), as in many InI systems, is the poor spatial resolution of the reconstructions.To solve this problem, different solutions have been proposed [19][20][21].Another problem which is not-tackled yet is the axial heterogeneity of lateral resolution.
In this work, we propose a different configuration of an integral microscope by inserting the lens array at the pupil plane of the microscope objective [22].This configuration constitute a new optical microscope, which we will name as Fourier Integral Microscope (FiMic), which has the ability of improving the spatial resolution and also the depth of field in the reconstructed images.The main fields of potential application of this new microscope are (a) bio-medicine, particularly for obtaining images in which the profile of the sample is of interest, or in which a depth section view is necessary; (b) Profilometry, for quality control of microelectronics and semiconductors, intra-ocular lenses and microlens testing, or forensic science.

The integral microscope
Let us start by considering the basic structure of an IMic.As shown in Fig. 1, a microlens array is placed at the image plane of a telecentric conventional microscope that is composed by an infinity-corrected microscope objective (MO) and a tube lens (TL) of focal length TL ' f .The camera sensor is then shifted a distance ML ' f from the MLA, where ML ' f is the focal length of the microlenses.After a single exposure, the CCD registers an integral image that is composed by a set of microimages that contain both the spatial and angular information of the 3D sample.To avoid overlapping of microimages, the numerical aperture (NA) of the objective, evaluated in the image space, has to match with the NA of the microlenses [23].In the case of an IMic, we can express this condition as follows: where M is the absolute value of the lateral magnification provided by the microscope, and p is the pitch or diameter of a single microlens (see Fig. 1).To illustrate the main characteristics of the IMic, we performed a first experiment.We used a 20x/0.4MO and a TL of focal length f' TL = 200 mm (see Fig. 1).The MLA had a pitch of p = 110 µm and a numerical aperture of NA ML = 0.01 (MLA #12-1192-106-000 from SUSS MicroOptics).The axial position of the MLA was fixed with precision of one micron.In our experiment we used as the sample a cleaning-lens tissue paper.These sheets are made for cleaning the lenses of optical instruments, but are also commonly used in testing microscopy experiments.They have low-density of threads, and about 100 microns of thickness.The tissue was stained with fluorescent ink proceeding from a fluorescent marking pen.The sample was illuminated with laser of wavelength 532 nm.Because we used a commercial MLA, its NA did not match with the NA MO .In order to satisfy Eq. ( 1), we inserted an additional aperture to reduce the MO effective numerical aperture to MO 0.2 eff NA = . With this configuration we recorded the integral image shown in Fig. 2, which is composed by 72x72 microimages with 27x27 pixels each.Due to the fact that the light emitted by any point of the sample is collected by many microlenses, this system is robust against low luminance conditions.Since the 3D information of the sample is present in the integral image, conventional postprocessing algorithm [12] permitted us to obtain a depth-reconstruction of the scene and also the different perspective views.In Fig. 3 Although IMic provides acceptable results in terms of 3D reconstruction, the poor lateral resolution of the reconstructions remains as its main drawback.To illustrate this drawback, in Fig. 4 we show the results of a second experiment performed with the IMic (see Fig. 1).In this case, we recorded the integral image of an incoherently-illuminated 1951 high-resolution USAF test chart, placed at different distances from the FFP of the MO.Note that when the object is placed at the FFP, the image provided by the microscope is just at the MLA plane, and therefore any microimage is a circle of constant irradiance (Fig. 4(a)).In this case the resolution of the reconstructed image is very poor, since it is limited by the number of microlenses in the array (Fig. 4(b)).Changing the axial position of the object allows to increase the lateral resolution of the reconstruction [24].As an example of this fact, in Figs.4(c) and 4(d) we see the captured integral image and the reconstructed image when the test chart is placed at a distance of 0.03 z = mm from the FFP.Although resolution is higher than the resolution of Fig. 4(b), it has been demonstrated that this resolution is still poor when it is compared with the resolution provided by a conventional microscope [21].

The Fourier integral microscope
To improve the spatial resolution in integral microscopy, we propose a different configuration in which we introduce the lens array at the pupil plane, i.e. the Fourier plane of the microscope (see Fig. 5(a)).We name this system as a Fourier Integral Microscope (FiMic).To understand this new design, it is convenient to calculate the relationship between the structure of the radiance map at the pupil plane coordinates 1 1 ( , ) x θ in Fig. 1, and at the image plane coordinates 0 0 ( , ) x θ .By applying ABCD matrices [25], we find 1 0 From Eq. ( 2), we see that there is a spatio-angular transposition between both planes.In other words, by inserting an adequate lens array at the pupil plane, we can capture directly the perspective views of the 3D sample and avoid the problems associated with working with a MLA of small pitch.To implement this device, there are three alternative architectures.One is to insert the lens array directly at the pupil aperture and place a single sensor at the back focal plane (BFP) of the lenslets.This way of proceeding has the advantage of compactness, but the drawback that the number of pixels of the integral image does not increase, as compared with the IMic architecture.The second possibility is to build an array of micro cameras, each with its own CCD, and place the camera array at the pupil aperture.This is the optimum arrangement in terms of compactness, resolution and depth of field, but has the constraint that it can work only with MOs with a large aperture stop.The third possibility is to combine the use of the micro-camera array and a relay system.This architecture is worse in terms of compactness, but optimum in the sense that it can works with any MO lenses.All these possibilities are fully described in a Spanish patent submitted recently [22].
To demonstrate the feasibility of our proposal, we implemented in our laboratory a FiMic following the third architecture, as shown in the scheme of Fig. 5(c).In our system the relay was composed by two lenses of focal length In the proof-of-concept experiment, we did not build the micro-camera array.Instead we implemented a syntheticaperture procedure [26].Specifically, the view images were captured by a digital camera that was mechanically displaced in both directions x and y .The camera was moved in accordance with the arrangement shown in Fig. 5(b) to achieve both acceptable parallax and resolution of the different views.In Fig. 6 we can see the integral image captured by the FiMic.We registered 3x3 EIs of 720x720 pixels each.With the information provided by the integral image we can reconstruct the irradiance of the 3D scene at different depths.Figures 7(a 7) and the ones obtained with the IMic (Fig. 3), it is easy to realize that FiMic provides images with higher resolution, much better depth of field and with axial homogeneity (note that now there is no central plane with poor resolution).
In order to quantify the improvement in resolution, we performed, as in the previous section, a second experiment using a high resolution USAF test chart as the object.The chart was incoherently illuminated and displaced at different axial positions in front of the MO.For any position we captured 3x3 EIs, with 720x720 pixels each.From any integral image we calculated the reconstruction at the corresponding depth.The result is shown in Visualization 6.From this movie we have extracted two frames, which correspond to a poor and a good IMic reconstruction planes.These frames are shown in Fig. 8. Comparing these images with the results shown in Fig. 4 we find that, for the parameters of our experiment, the IMic provides, in the best plane, a frequency cutoff of

Conclusions
We have presented a new configuration of an integral microscope (IMic) by placing a lenslet array at the Fourier plane of the microscope objective lens.This configuration provides images with lateral resolution improved by factor of 1.4, and with enlarged and depth of field, compared with the ones obtained from a conventional IMic.Our experimental results show the feasibility of the method.This approach may have broad applications to automated cell identification using integral microscopy [27].

Fig. 1 .
Fig. 1.Scheme of an Integral Microscope.The MLA is placed at the image plane of a conventional microscope and the CCD is shifted a distance ML ' f from the microlens array

Fig. 2 .
Fig. 2. Integral image acquired with our IMic of Fig. 1.The image is composed by 72x72 microimages with 27x27 pixels each.The structure of the microimages can be seen at the inset.
(a) we can see a frame of the depth reconstruction of the 3D sample.Note in the video (Visualization 1) the axial heterogeneity of the resolution.The worst resolution is obtained just in the MO front focal plane (FFP).On the hand, Figs.3(b) and 3(c) show two different views of the 3D scene.Note the existing parallax between these two images.

Fig. 3 .
Fig. 3. a) Single frame extracted from the depth reconstruction of the 3D sample obtained after applying conventional algorithm of refocusing (Visualization 1).In (b) and (c) we can see two perspective views of the microscopic object.A movie with continuous variation of horizontal and vertical perspective is shown in Visualization 2.

Fig. 4 .
Fig. 4. Capture and reconstruction of a 1951 high resolution USAF test.(a) and (c) show the integral image captured with the object placed at a distance of 0 0 z = mm and 0 0.03 z = mm from the FFP of the microscope objective.In (b) and (d) we show the depth reconstruction from (a) and (c), respectively.Additionally, in Visualization 3 we show a movie with the reconstructed images corresponding to object positions ranging from 0 0.02 z = − mm to

Fig. 5 .
Fig. 5. (a) Scheme of a FiMic.(b) Microlens arrangement at the pupil plane used in the experiment.(c) Experimental configuration of the FiMic with a relay system.

Fig. 6 .
Fig. 6.Integral image captured with the FiMic.The integral image is composed by 3x3 EIs, with 720x720 pixels each.
) and 7(b) show reconstructions at two different depths, whereas Figs.7(c) and 7(d) show two different perspectives and are extracted directly from the integral image.If we compare the results obtained with the FiMic (Fig.
contrary, in the FiMic the cutoff frequency, for a long range of depth planes is the homogenization in depth of lateral resolution.

Fig. 7 .
Fig. 7. Reconstructions of the 3D sample.In (a) and (b) we show two frames extracted from the depth reconstruction of the 3D sample obtained after applying conventional algorithm of refocusing (Visualization 4).In (c) and (d) we show two perspective views extracted from the capture shown in Fig. 6.A movie with variation of horizontal and vertical perspective is shown in Visualization 5.

Fig. 8 .
Fig. 8. Depth reconstruction of the USAF test chart captured with the FiMic when it is placed at a distance of (a) 0 0 z = mm and (b) 0 0.03 z = mm from the FFP of the microscope ob-