Video-rate, mid-infrared hyperspectral upconversion imaging

In this work we demonstrate, to the best of our knowledge, a novel wide field-of-view upconversion system, supporting upconversion of monochromatic mid-infrared (mid-IR) images, e.g., for hyperspectral imaging (HSI). An optical parametric oscillator delivering 20 ps pulses tunable in the 2.3–4 μm range acts as a monochromatic mid-IR illumination source. A standard CCD camera, in synchronism with the crystal rotation of the upconversion system, acquires in only 2.5 ms the upconverted mid-IR images containing 64 kpixels, thereby eliminating the need for postprocessing. This approach is generic in nature and constitutes a major simplification in realizing video-frame-rate mid-IR monochromatic imaging. A second part of this paper includes a proof-of-principle study on esophageal tissues samples, from a tissue microarray, in the 3–4 μm wavelength range. The use of mid-IR HSI for investigation of esophageal cancers is particularly promising as it allows for a much faster and possibly more observer-independent workflow than state-of-the-art histology. Comparing histologically stained sections evaluated by a pathologist to images obtained by either Fourier transform IR or upconversion HSI based on machine learning shows great promise for further work pointing towards clinical translation using the presented mid-IR HSI upconversion system.


INTRODUCTION
Mid-infrared (mid-IR) hyperspectral imaging (HSI) is of major interest in diverse fields ranging from medical diagnostics [1][2][3][4][5], to environmental monitoring [6][7][8][9], to geology for mineral identification [10], to within the food industry [11]. The widespread applicability of the mid-IR wavelength range (e.g., 2-15 μm) is intimately linked to nature-given properties of molecular absorption and includes the following main categories: (1) vibrational spectroscopy used for unique identification of complex molecules such as food or tissue; (2) gas spectroscopy characterized by fundamental band interaction between mid-IR light and most gaseous molecules including environmental gasses such as CH 4 , CO 2 , NO X , and SO X ; and (3) sensing of heat radiation from room temperature objects. Despite all the virtues of mid-IR spectroscopy and imaging, applications are in need of sensitive, fast detectors, and cameras operating preferably at room temperature [9]. Fourier transform IR (FTIR) spectroscopy is today the preferred technique for HSI in the mid-IR spectral range. A major limitation to FTIR imaging is that the cameras used for mid-IR detection are predominantly based on mercury cadmium telluride (HgCdTe), InSb, or microbolometers. Common for all these direct detectors are that they require cryogenic cooling for optimal performance and are of high cost [12,13]. Moreover, for HSI, large data storage and complex postprocessing software is currently needed to transform raw data into useful information [14]. Recent developments in mid-IR hyperspectral microscopy include tunable quantum cascade laser (QCL) illumination combined with either raster scanning [15] or microbolometer array detectors [16]. These systems have shown their potential to outperform FTIR systems for special applications; however, they still rely on direct detection of the mid-IR signal.
Nonlinear frequency upconversion provides an alternative route to fast, room-temperature mid-IR spectroscopy and imaging, due to an orders-of-magnitude higher sensitivity and speed compared to direct mid-IR detectors [17,18]. A significant drawback in previous realizations of monochromatic upconversion imaging has been the need for extensive postprocessing to obtain a large field of view (FoV), which has prevented its use for fast 2D data acquisition or real-time video-frame-rate imaging [19][20][21].
In this work, high-speed monochromatic mid-IR upconversion imaging is presented using a synchronously pumped optical parametric oscillator (OPO) delivering ∼20 ps pulses tunable in the 2.3-4 μm spectral range [22] for mid-IR illumination of an object. A lithium niobate (LN) crystal acts as the birefringent phase-matched nonlinear medium for the upconversion process, transferring the mid-IR signals to the NIR wavelength range, where a standard CCD camera is used for image acquisition.
To increase the FoV beyond the angular phase-match acceptance of the nonlinear medium, the idea of using angular rotation of the nonlinear crystal dating back to the late 1960s [23] is revitalized and integrated with modern camera technology. A rotation of the nonlinear crystal results in concentric rings in the object plane being effectively phase-matched, thus upconverting separate ring-shaped regions of the object as a function of time. Rotation of the LN crystal by a mere 1 deg is implemented here by mounting the crystal on a Galvano scanner (GVS), using "tangential phase-matching"; see Fig. 3 and Supplement 1, Sections S1-S3. This tiny rotation increases the FoV by a factor of approx. 5 compared to a fixed crystal angle configuration, thus increasing the number of pixels in the upconverted image by a factor of approx. 25. The FoV, using this approach, is limited primarily by the lens optics, well-known from standard imaging configurations. The concept is not specific to this setup and can be applied universally for any upconversion imaging system. Another important feature of the scanning technique is to eliminate the need for image postprocessing for large FoV monochromatic upconversion imaging. Adjusting the crystal rotation cycle time to match the camera integration time, an image with an increased FoV is acquired directly without postprocessing. Monochromatic mid-IR upconversion imaging with a frame acquisition time of 2.5 ms is demonstrated. The obtained spatial resolution is 35 μm within a 10-mm-diameter FoV in the object plane, providing approx. 64 kpixels (spatially resolvable elements). The developed system is demonstrated for histopathological mid-IR HSI.
Histopathological mid-IR HSI is an application of IR imaging that shows great promise, providing a molecular specific contrast from unstained tissue, utilizing the mid-IR absorbance from chemical bonds (vibrational modes) found within tissues. The molecular composition and hence spectral signatures have been shown to change with progression of disease. The spectral data can be used for computer-assisted classification of diseases. Histopathology, to date, has relied heavily upon the evaluation of haematoxylin and eosin stained (H&E) tissue sections. This workflow involves extensive laboratory work by technicians, followed by manual evaluation by a pathologist using a visible light microscope. This procedure could be improved by adding mid-IR HSI of unstained tissue sections into the toolbox of the pathologists. By collecting mid-IR transmission profiles, biochemical information such as protein, DNA, and lipid composition can be revealed, adding information about sample pathologies. Such an approach has the advantage that it requires less sample preparation compared to the H&E workflow and could furthermore help to provide an objective rather than subjective decision [24,25] based on the obtained chemical signatures.
We report on the use of mid-IR upconversion imaging using a silicon-based camera, demonstrating unsupervised computerassisted classification of biopsy sections in the 3-4 μm wavelength range. It is generally recognized that the 5-12 μm range is preferred in terms of chemical specificity [15,16]; however, the 3-4 μm wavelength range has also shown relevant chemical features [26] and has the additional advantage of being compatible with existing glass substrates (microscope slides) used presently in clinics [27]. Using the k-means algorithm, an unsupervised spectral clustering method based on spectral similarity of different regions of the sample, each pixel acquired from the tissue sample were segmented into groups, comparable to the histotypes identified by an expert gastrointestinal histopathologist. For reference, we compare the upconversion images with those obtained using FTIR imaging, and neighboring tissue slides stained with haematoxylin and Eosin evaluated by a histopathologist to make their pathology grading.

EXPERIMENTAL SETUP AND RESULTS
This section describes the setup for upconversion of monochromatic mid-IR images using the idler beam from an OPO as the mid-IR illumination source. As shown in Fig. 1, the OPO is synchronously pumped at 1064 nm by a picosecond Yb-fiber laser operating at a repetition rate of ∼80 MHz (Fianium, FP1060-20). The duration of the Yb-fiber laser pulses is ∼20 ps and the bandwidth is approx. 1 nm. The maximum available average power is 15 W, out of which 10 W is used to pump the OPO, while the remaining 5 W (attenuated to 1.5 W to avoid saturation of the CCD camera) serves as the pump field for the upconversion process, i.e., sum-frequency-generation (SFG). The synchronous upconversion pumping scheme deployed here inherently locks the mid-IR OPO pulses and the pump laser pulses, minimizing timing jitter to the picosecond scale. This feature is intrinsic to synchronously pumped picosecond OPOs, in contrast to nanosecond-pumped OPOs where the individual output pulse builds up from parametric fluorescence, introducing timing jitter in the nanosecond range [28]. The high temporal stability allows the upconversion setup to be operated passively, using a fixed delay line adjusted to synchronize the mid-IR OPO pulse and the pump pulse to coincide inside the nonlinear crystal [28]. Synchronous pumping is highly efficient since the high peak power pulses interact in perfect temporal synchronism, resulting in efficient upconversion.
A MgO-doped periodically poled LiNbO 3 (MgO:PPLN) crystal is used as the nonlinear medium in the OPO, generating the signal and the idler beam. The crystal is operated at constant temperature. Depending on the desired wavelength, either the Research Article signal or the idler beam is accessible for illumination. The signal wavelength can be tuned in the range of 1.43-1.98 μm with a bandwidth of <2 nm. The idler beam correspondingly covers the range of 4.1-2.3 μm with a bandwidth of 4-6 nm depending on the specific mid-IR wavelength. The illumination bandwidth defines the spectral bandwidth of the monochromatic images shown later. Further details of the OPO can be found in [22].
In this work, only the idler beam is used for illumination of the object. The idler beam was scaled to a diameter of approx. 10 mm, which then defines the FoV in the object plane. An average power of 750-900 mW (depending on the wavelength) is available in a near-Gaussian intensity distribution. The pump beam (1064 nm) has a waist diameter of 4 mm inside the nonlinear material (collimation optics not shown in Fig. 1).
A bulk LN crystal (5 × 5 × 10 mm 3 ) cut for birefringent phase-matching at an angle of 48 deg with respect to the c-axis acts as the nonlinear medium for the upconversion process. The large transverse dimension of the LN crystal compared to its PPLN counterpart allows the use of a large pump beam diameter, enabling high spatial resolution in the upconverted images (the pump beam here acts as a soft Gaussian aperture) [29]. It is further noted that other birefringent phase-matched crystals, e.g., AgGaS 2 , constitute an important class of nonlinear materials available for mid-IR applications beyond 5 μm, thus affirming the generic nature of the proposed concept and enabling further extension of the upconversion imaging into the 5-12 μm range.
The upconversion setup is implemented in a 4f configuration, where the upconversion process takes place in the Fourier plane relative to the object. A first lens, f 1 50 mm, Fourier transforms the 2D mid-IR object field to the center of the LN crystal, where synchronous upconversion to the near IR (NIR) takes place. A beam combiner (custom made, 25-mm-diameter, 3-mm-thick ZnSe substrate coated for high transmission at 1064 nm and high reflection at 3-4.5 μm) combines this signal and pump before entering the nonlinear crystal. A second lens, f 2 50 mm (100 mm), Fourier transforms the upconverted NIR signal to form an image at the CCD camera [0-21, 28]; see Fig. 1. A Si-based CCD camera (Andor Luca S) is used for image acquisition. Filters (shortpass 950 nm, longpass 700 nm) are used to block the residual pump and stray light from reaching the camera.
Different imaging configurations exist for the frequency upconversion process. Using simple collinear interaction in a birefringent phase-matched crystal typically leads to a narrow and often elliptical FoV [30]. However, choosing an appropriate offset angle between the mid-IR signal and the pump beam, a larger and more circular FoV can be realized (see Supplement 1, Section S2). This geometry is commonly referred to as tangential phase-matching [30,31]. For a specific monochromatic mid-IR signal and corresponding crystal angle, a uniquely defined cone of monochromatic light is optimally phase-matched, corresponding to a ring pattern when Fourier transformed to the image plane. Figure 2 illustrates the steps to model the upconversion process; see Supplement 1, Section S1.
To increase the upconverted monochromatic FoV, the crystal is angle-tuned to upconvert different sections (rings) of the object plane (Visualization 1 and Visualization 2), as seen in Figs. 3(a)-3(c). Figures 3(d)-3(f ) further confirms the theoretical model of upconversion, as illustrated in Fig. 2. Angular tuning of the crystal [23] is a particularly efficient approach, since a tiny crystal rotation can lead to a large change in the mid-IR phase-matching wavelength. Several other approaches to increase the FoV have been proposed in recent years; however, these are complicated to implement, of limited generality, or slow in nature [17][18][19][20][21]32].
To facilitate well-defined angular rotation, the nonlinear crystal is mounted on a GVS. An analog/digital signal generator (USB-6211 National Instruments) generates a synchronized voltage signal for the GVS and the CCD camera. A LabVIEW program controls the analog-to-digital (A/D) board to generate the GVS voltage and synchronization with the CCD camera. While scanning the crystal angle in a predefined range, a large FoV is obtained without the need for postprocessing, as the CCD camera passively integrates all the upconverted sub-images. In practical terms, the GVS sweep averages out the sinc 2 term imposed by the phase-matching condition, point by point. In particular, a nonlinear scan pattern can be used to further compensate for the Gaussian nature of the illumination source, thus generating a flat response over the full FoV, i.e., producing an isoplanatic imaging system [33]; see Supplement 1, Section S4.
To determine the spatial resolution experimentally, a standard USAF resolution target (Edmund Optics clear path resolution target) is used in the object plane; see Fig. 1. Figure 4 shows the upconverted image of the resolution target measured at 3.1 μm using GVS automatized acquisition. The smallest feature of the resolution target that can be resolved is approx. 35 μm (14 lines/mm); see Figs. 4(b)-4(c). The camera has a pixel size Fig. 2. Model of the upconversion process comprising three steps. The original mid-IR object is first rescaled, then convolved with the point spread function, which, in our case, is a Gaussian pump beam. Finally, the phase-match condition is multiplied to form the upconverted image. Multiplication of the sinc 2 term rather than convolution (i.e., blurring) is vital for superimposing different ring patterns to form a single upconverted image with increase FoV. Research Article of 10 μm × 10 μm, whereas the upconverted spatial features on the camera are smaller (i.e., ∼8 μm) due to the demagnification imposed by the upconversion process. Therefore, the lens, f 2 , was changed from 50 mm in Fig. 4(a) to 100 mm in Fig. 4(b).
With a FoV of 10 mm in diameter and a measured resolution of 35 μm, the number of resolvable elements corresponds to approx. 64 k pixels. A theoretical estimate of the spatial resolution based on the point spread function (PSF) using a pump beam diameter of 4 mm gives a resolution of 34 μm, thus in excellent agreement with experiments; see Supplement 1, Section S1. Generally, an uncertainty in the pump beam diameter and beam quality will affect the resolution. Furthermore, there may be a degradation in the upconverted image quality, caused by lens aberrations in the system and refraction in the surface of the rotating LN crystal; see Supplement 1, Section S5.
To illustrate high-speed postprocessing free image acquisition, butane was sprayed from a gas lighter in the object plane. By tuning the illumination wavelength and the upconversion process to match the absorption line of butane at 3.37 μm, narrowband images were recorded at a repetition rate of 40 Hz (limited by the camera update rate); see Visualization 3. With a 2.5 ms camera integration time, the signal to noise ratio was 424 in the center of the image; see Supplement 1, Section S6.
The monochromatic mid-IR upconversion system was then applied for computer-assisted biopsy classification. The mid-IR illumination beam diameter was rescaled to match the size of a typical medical biopsy, i.e., 2 mm in diameter. The scaled illumination beam was transmitted through the biopsy sample, followed by 5 × magnification to match the upconversion system; see Fig. 5(a). The samples shown in Fig. 5(a) are 7-μm-thick sections in a tissue microarray purchased from Biomax.us (sample ID: ES8011a). It contains several esophageal biopsy sample cores mounted on a 1-mm-thick CaF 2 substrate. The samples were dewaxed prior to measuring by an overnight n-hexane wash; see Supplement 1, Section S8. The acquisition of two hyperspectral images (each with 2.5 ms camera integration time), with and without the biopsy sample inserted, allows for a pixel-by-pixel calculation of the transmittance of the sample; a measured image at 3.34 μm is shown in Fig. 5(b). The spatial resolution at the object plane is approx. 9 μm. The speckled pattern observed in Fig. 5(b) can arise from both temporal and spatial coherence of the illumination source. In the present case the bandwidth of the mid-IR OPO pulses is approximately 5 nm, exceeding the free spectral range (FSR) of the 1 mm CaF 2 substrate by 30%, thus significantly reducing fringing from multiple reflections in the substrate; see Supplement 1, Section S7. To remove the speckles arising from the spatial coherence in the image Fig. 5(b), a spinning, ground glass diffuser (Thorlabs DG10-1500) was inserted in the beam path. In this configuration, the illumination source was reflected by the spinning diffuser and scaled to match the size of the biopsy. Figure 5(c) shows an image acquired using the diffuser, delivering 4-8 mW of incoherent mid-IR light to the sample. As can be seen, the interference patterns were minimized; however, the CCD camera integration time was increased to 400 ms to compensate for the reduced illumination power after insertion of the diffuser. No damage to the biopsies was observed at these power levels (corresponding to 1-2 mW∕mm 2 ). Figure 6 shows data from two medical biopsies, one cancerous and one healthy esophageal tissue sample, imaged with the upconversion system, FTIR imaging, and using conventional H&E staining. A GI-registry consultant histopathologist (Shepherd) performed the evaluation of the H&E stained images, and each pixel was manually coded in the mask image. This method  constitutes the gold standard. To produce hyperspectral data, 62 upconverted monochromatic mid-IR images were acquired in the 2770-3345 cm −1 range. These biopsy images were then loaded into R (the programming software) via the "hyperSpec" package [34]. For comparison, FTIR images were also included, cropped in the spectral dimension, to match the spectral range of the upconversion system. The upconversion spectra were then smoothed using a Loes function [34] and analyzed using R's generic k-means functionality [35]; see Supplement 1, Section S9. Figure 6 (first row) compares biopsy images obtained with the FTIR and the upconversion system, respectively. Figure 6 (third row) shows the result of computer-assisted classification as described above. Both morphology as well as the spectral classification of the images correlate well with the histopathology (bottom row), however, with the FTIR images appearing to resemble the pathologist's classification more closely. We believe that the differences are due to slightly better spatial and spectral resolution in the FTIR system. Figure 6 (second row) shows the k-means parameters used in the computer-assisted classification of the biopsies.
These represent the calculated mean spectra resulting from the k-means algorithm (not directly representing the spectral resolution of the instruments). The poorer resolution of the upconversion systems k-mean spectra is still unknown, but differences related to spatial resolution, number of spectra used, and signal to noise will all affect the k-means output.
Making a direct comparison between the presented upconversion system and the FTIR system is difficult due to fundamental differences in the sensing principle. The Agilent FTIR system uses 90 s for a 128 × 128 pixel image tile (pixel size: 5.5 × 5.5 μm 2 ), i.e., 15 min for a 2-mm-diameter biopsy (after a 30 min cooling time of the focal plane array), however, providing a full spectrum of the sample. For the upconversion system, the acquisition time scales directly with the number of spectral images. For a single image, we estimate that the presented upconversion system is in the order of 500 times faster than the FTIR system when using 4-8 mW of incoherent mid-IR light incident on the sample. The number of images needed for computer-assisted classification of biopsies may be as low as 10-20 [36], suggesting that a Research Article continuous spectrum is not required. A small number of images will, therefore, favor the upconversion system for this application.

CONCLUSION AND OUTLOOK
We have presented, for the first time to our knowledge, a monochromatic mid-IR upconversion imaging system with an enlarged FoV, operating at video frame rate (40 Hz) without the need for postprocessing of the images. In the present setup, a 2.5 ms exposure time still required an attenuated pump beam to avoid camera saturation, showing that a frame rate of 1 kHz can be realized using a faster silicon CCD camera. The high efficiency was obtained using a passively synchronized pumping geometry. We obtained 64 kpixels in the upconverted images, applying a 1 deg crystal rotation using tangential phasematching. This result is in good agreement with the theoretical estimated resolution. The FoV was increased by a factor of 5 compared to a static design, corresponding to an increase of 25 times in the number of spatially resolvable elements. This number can be further increased using a larger numerical aperture at the input side, combined with a correspondingly larger crystal rotation angle.
Eventually, the angular scanning of the nonlinear crystal for upconversion will be limited by image distortion, originating from refraction at the crystal surfaces; see Supplement 1, Section S5. Using, e.g., a 6-deg rotation combined with tangential phase-matching would provide approx. 370 kpixels. The mid-IR upconversion system operates at room temperature, unlike other low-noise mid-IR imaging systems with single-photon detection capability, and with extremely short response time, here demonstrated with 20 ps pulses.
Longer wavelengths can be upconverted using different nonlinear crystals such as AgGaS 2 (e.g., 5-12 μm). We are confident that the simple and generic approach using crystal angle tuning with a GVS, mitigating the restriction of FoV imposed by the nonlinear phase-matching properties, can find widespread use for mid-IR HSI applications. Two different examples were presented, highlighting the potential for online mid-IR imaging of gases and for computer-assisted biopsy screening by measuring disease-specific compositional changes.
This first demonstration of a mid-IR OPO-upconversion instrument to provide hyperspectral data from human tissue biopsy sections have confirmed that images from only 62 wavelengths could provide enough spectral data to enable preliminary, unsupervised clustering of tissue types, with a performance similar to FTIR imaging. This pilot study of mid-IR HSI upconversion imaging shows great potential for fast computer-assisted cancer biopsy screening, as demonstrated with sub-second monochromatic image acquisition time.