High performance image mapping spectrometer (IMS) for snapshot hyperspectral imaging applications

A high performance, snapshot Image Mapping Spectrometer was developed that provides fast image acquisition (100 Hz) of 16 bit hyperspectral data cubes (210x210x46) over a spectral range of 515-842 nm. Essential details of the opto-mechanical design are presented. Spectral accuracy, precision, and image reconstruction metrics such as resolution are discussed. Fluorescently stained cell samples were used to directly compare the data obtained using newly developed and the reference image mapping spectrometer. Additional experimental results are provided to demonstrate the abilities of the new spectrometer to acquire highly-resolved, motion-artifact-free hyperspectral images at high temporal sampling rates. © 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement


Introduction
Imaging spectrometers are specialized instruments capable of simultaneously acquiring spectral and spatial signatures.Typically, these instruments provide information about the observed scene in the form of a three dimensional data cube (x,y,λ).Due to the rapid developmental process of electronic components such as solid state detectors, imaging spectrometers can operate in a broad spectral range from ultra violet (UV) [1,2] to visible (VIS) [3] to infra-red (IR) [4,5].Hyperspectral imaging instruments found wide-spread use in industrial [6], research [7,8] and quality control [9] settings.Typically, these instruments are used to analyze chemical composition of astronomical bodies [10,11], detect explosives and aid in friend or foe decision making [7], identify vegetation [12] and its coverage [13], and quantify environmental condition factors such as canopy water content and plant stress states [14].Furthermore, hyperspectral imagers are used to analyze forensic traces [15,16], check the condition and authenticity of artwork [17,18], study the chemical makeup of food ingredients and products [19], aid in sorting of plastics in recycling settings [20], and assist in diagnostics for a broad range of biomedical applications [21][22][23].
Hyperspectral imaging spectrometers can be divided based on their principle of operation into scanning and snapshot instruments.Snapshot instruments capture the entire hyperspectral data cube in a single integration event of the imaging detector.Thanks to this property, each voxel in the hyperspectral data cube can be exposed for a time that is inversely proportional to the system's frame rate.This allows for detection of the weak fluorescent signatures at relatively high frame rates at low excitation levels, resulting in reduced phototoxicity [24].Additionally, since data is acquired during a single exposure event, snapshot hyperspectral data cubes do not require hardware level synchronization of the detector with the scanning platform to reconstruct the observed scene.Thus, snapshot imaging spectrometers are well suited to monitor transient events such as the observation of flagellated cell movement and morphogen diffusion, which would require at least 100 and 72 fps temporal acquisition rates respectively to provide motion-artifact free data [25].The ability to acquire data during a single integration event of the imaging detector allows for a great degree of synchronization between multiple snapshot imaging spectrometers, a feature exploited in remote sensing to acquire estimates of bidirectional reflectance-distribution function by a fleet of spatially, directionally and temporary coordinated instruments [26].Further, direct imaging snapshot hyperspectral spectrometers, a sub category of snapshot imaging spectrometers, enjoy the advantage of simple data reconstruction algorithms [27], which reduces computational load and provides the ability to display and analyze multidimensional data cubes at high frame rates, as compared to e.g.: computationally intensive tomographic snapshot spectrometers [27].Additionally, direct imaging snapshot spectrometers, which are based on the image mapping principle, enjoy the advantage of high light throughput [28], a critical component for light sensitive applications such as fluorescent imaging and observation of fast changing phenomena.
Our group specializes in the design and manufacturing of snapshot image mapping spectrometers (IMS) based on the image slicing principle.We have successfully built several generations of IMS systems that were optimized for microscopic applications and these systems were used to monitor both cell signaling events [29] and fluorescently stained samples [30].We have integrated our IMS spectrometer with a fundus camera to enable direct monitoring of blood perfusion and tissue oxygenation in the eye [31].Moreover, we used the IMS to build a snapshot optical coherence tomographic system (OCT) [32].All systems we developed to date were built around large area detectors, comparable in terms of the image diagonal to a full frame 36x24 mm 2 photographic film.The spectral range covered by instruments developed in the past varied between systems; for example, the spectrometer for hyperspectral microscopy operated in the 450-650 nm [33] range, the system designed for fluorescence microscopy acquired data in the 520-660 nm range [30], and the OCT system was optimized for the 610-645 nm spectral range [32].The read noise of the utilized detectors was on the level of 16e -and 15 e -for the BOBCAT ICL-B4820M and the Apogee U16 camera respectively, and combined with low quantum efficiency resulted in low light sensitivity.The maximum achievable temporal resolution reached was 7.2 fps and was limited by the respective camera internal electronics and data interface bandwidth.
Previous generations of IMS systems operated in the VIS part of the electromagnetic spectrum.This allowed their use in a range of applications including the assessment of skin perfusion [34] and retinal tissue oxygenation levels [35] using the reflectance mode of operation, as well as the direct observation of a range of fluorophores such as acridine orange, proflavine, DAPI and Alexa Fluor [33].While the visual part of the electromagnetic spectrum is preferred for direct observation of biological samples, due to low spectral absorption of tissue, contrast of exogenous fluorophores is typically limited by the background signal emitted by endogenous fluorophores e.g.: melanin, (NADH), collagen and elastin.From the perspective of multi-spectral observation, a spectrometer working on the border of the VIS and near infrared (NIR) part of the electromagnetic spectrum has several interesting properties.First, it allows deeper imaging into a tissue due to reduced scattering, which is proportional to λ −4 as described by the Rayleigh scattering law.Additionally, because absorption of the main tissue pigment, melanin, decreases with wavelength, tissue transmission increases proportionally with wavelength.Moreover, contrast of exogenous fluorophores emitting in the red and NIR part of the spectrum is not affected by the main endogenous tissue fluorophores such as NADPH, collagen and elastin, which have an emission spectra in 325-550 nm range.Furthermore, a spectrometer operating in NIR allows for the observation of new fluorophores such as indocyanine green (ICG) (Ex.600-900nm/Em.750-950nm), which is commonly used to diagnose heart conditions as well as to visualize blood flow in the retina.Aside from biomedical applications, a spectrometer operating within the VIS and NIR spectral bands can find multiple applications in remote sensing, as it may be used to delineate between vegetation species [9][10][11], identify minerals [21], and characterize urban environment landscapes (rooftop, pavement condition, and type identification) [22][23][24].Additionally, due to the low absorption of atmosphere in the NIR region, all of the above measurements can be performed at extended distances.
Taking the above statements into account, a new snapshot image mapping spectrometer operating in the VIS and NIR bands was developed around a sCMOS camera to provide low noise, high dynamic range data in the spectral range of 515-842 nm.The presented system is capable of reconstructing hyperspectral data cubes (x,y,λ) of a minimal size of 210x210x46 at a temporal sampling rate of 100 Hz.The newly developed spectrometer, as compared to previous generation systems, has about 16 times lower read noise, about 20% better quantum efficiency, and can acquire data at about 14 times higher frame rate.Previously developed IMS systems utilized detectors which imaging area was close to 36x24 mm 2 , a size similar to "full-frame" photographic film.Presented high performance system was built around sCMOS camera with imaging area of 16.6x14 mm 2 .Because the photosensitive area of the detector was smaller, we re-designed and miniaturized the prism and lenslet assembly.A detailed description of the new opto-mechanical design together with a discussion of mechanical tolerances of critical components is presented in the following paragraph.Additionally, the sCMOS detector photo-electron to intensity conversion characteristic was experimentally measured and is presented in paragraph 4.1.Quantitative and qualitative performance metrics of the high performance IMS system are experimentally verified.Advantages of an extended dynamic range and an increased spectral sensitivity in VIS and NIR together with a high temporal sampling rate are demonstrated in two experiments.In the first experiment, free falling objects selectively stained with NIR dye were monitored at 50 frames per second, and images of objects accelerating in the Earth's gravitational field were reconstructed in the VIS and NIR part of the spectrum without motion-related artifacts.In the second experiment, the emission spectrum of a halogen filament was continuously observed during an ON/OFF cycle at 100 fps to show the maximum temporal sampling rate of the newly developed system together with its potential to acquire data in the full dynamic range provided by the imaging detector.Performance of the new spectrometer is directly compared to a representative instrument from the previous generations of IMS systems in an experiment, during which a fluorescent BPAE cell sample was simultaneously observed by both systems.The presented system is to our best knowledge the first snapshot image mapping spectrometer utilizing the slicing principle that is capable of acquisition of data in the visible and near infrared part of the electromagnetic spectrum.Additionally, this is the first IMS type spectrometer with a temporal sampling rate adequate to monitor transient events.

Opto-mechanical design
An opto-mechanical schematic of the IMS is presented in Fig. 1(a).A three dimensional visualization of the opto-mechanical model for the IMS spectrometer is presented in Fig. 1(b), with the enclosure walls drawn in a semi-transparent mode to present the internal organization of the system components, and finally, a photograph of the assembled prototype is given in Fig. 1(c).A detailed description of the operating principle for the IMS can be found in our previous publications [24,28,30,33,34].Below, an abbreviated description of critical elements and their function is provided.The optical system of the IMS consists of three functional groups: the image relay microscope, the image mapping mirror, and the reimaging optical system made out of a collecting lens followed by a set of prisms and an array of miniature lenslets.The image relay microscope is made out of commercially available Zeiss components: a tube lens with focal length of f = 165 mm and a 2.5x Plan-Apo NA = 0.017 microscope objective.The primary role of this system is to provide a magnified, telecentrically projected image of a target on a segmented image mapping mirror.The segmented image mapping mirror is divided into regularly spaced rows of narrow, strip like facets.Each mirror is tilted around the x and y axes in order to break the image into spatially separated lines.Light reflected by the image mapping mirror is collimated by a 1x stereomicrosc subsequently behind the pr the image det which are gr spectrally scr using a transf The prese have selected scan modes r 600nm and ab favorably to t systems.

System c
A series of e accuracy and and a referenc target; a, reco Fig. 4  and controlled by AODS 20160-8 digital generator (Crystal Technology, Inc, USA) served as the light source.A beam filtered by a tunable AOTF filter was coupled to a liquid light guide, which in turn was plugged into the back illumination port of the Z1 Axio Observer microscope (Zeiss, Germany).Light reflected from an aluminum coated external mirror was measured sequentially by either the IMS or the reference spectrometer.An Ocean Optics USB2G38810 spectrometer (Ocean Optics, USA) was used as the reference spectrometer to measure the central wavelength and the Full Width at Half Maximum (FWHM) of each tested laser line.In total, nine measurements covering the 540 to 730 nm range were made.Tested spectral range was limited by technical capabilities of the acousto-optical tunable filter; please note that the Fianium supercontinuum laser can operate in wider range of wavelengths.A quantitative summary of the experimental data is given in Table 1.Please note that experimental results given for reference Ocean Optics spectrometer are for single acquisition event and data for the IMS spectrometer are averaged over the entire FOV.Central wavelengths and full width at half maxima (FWHM) of tested laser lines as measured by the reference spectrometer are given in the "Ocean Optics λ OO [nm]" and "Ocean Optics FWHM [nm]" columns respectively.Central wavelengths of laser lines as measured by the IMS system are given in the "IMS λ IMS [nm]" column.Standard deviations of central wavelengths measured over the entire field of view of the IMS system are given in the "IMS ±1σ [nm]" column.Full width at half maxima of laser lines measured by the IMS system is provided in the column labeled "IMS FWHM [nm]".Differences between central wavelength values measured by the IMS and the reference spectrometer are provided in the column labeled "Δλ λ OO -λ IMS [nm]".Central wavelengths were approximated by the abscissa of a vertex of a second degree polynomial fitted in the least square sense to three data points with the highest signal.Spectral plots obtained at 540, 650 and 730 nm laser lines for both the IMS and reference spectrometer are presented in Fig. 5 (a), (b) and (c), respectively.Spectral intensity values averaged over the field of view of the IMS system are drawn using red dots.The blue continuous line depicts a cubic spline curve fitted to the IMS data points.Raw spectra obtained from the reference Ocean Optics spectrometer are drawn with a continuous green line.All spectral profiles were normalized, and the inset in each figure shows the magnified spectra in the selected region of interest.Please note that lines filtered by the AOTF filter have secondary peaks across the visible spectrum, which amplitude decreases towards the red part of the spectrum.A set of office and flood lights was used to illuminate scene during experiment and because of directionality of the illumination a shadow of free falling balls was clearly visible in both "VIS" and "NIR" data sets, see for example Fig. 8(b).   of the Ronchi test were reconstructed without visible warping for all 46 wavelengths in 515-842 nm spectral range.The transition between dark and bright bars was uniform in length, and consistent in appearance throughout the field of view, which indicates good chromatic correction and suitable performance of image reconstruction routines.The inconsistency with the theoretically predicted lateral resolution can be ascribed to residual misalignment of the matrix of miniature lenses, which is inherently difficult to adjust using static optical components.

Experiments
A series of two experiments were performed to directly compare signal strength and throughput of the newly developed system to the previous generation spectrometer represented by an IMS built around an Imperx B4820M monochromatic camera.The reference system was capable of providing a (356x336x75) data cube in the spectral range of 470-670 nm at a maximum temporal sampling rate of 7.2 Hz [33].The two experiments performed were quantitative bead imaging and qualitative BPAE cells imaging.
Additionally, two experiments were performed to showcase the capability of the newly developed system to record data at 50 and 100 fps in experimental conditions requiring acquisition of intensity samples spanning the entire detector dynamic range.In the first experiment, free falling ping-pong balls (selectively stained with a near infrared absorbing dye) were observed by the IMS system and spectral data cubes were recorded at the 50 fps rate, the maximum frame rate at which the PCO camera can operate in global shutter mode.In the second experiment, a halogen bulb filament was observed during the ON/OFF cycle at a rate of 100 Hz, the maximum temporal sampling of the PCO camera operating in full frame mode.In the first experiment, the ability of the HS-IMS to acquire motion-artifact-free hyperspectral data cubes of fast moving objects is presented, and in the second experiment, dynamic range and high rate of temporal sampling required to monitor transient phenomena are shown.

Quantitative comparative evaluation of new IMS system against previous generation spectrometer
For the purpose of this experiment, the newly developed system will be hereafter referred to as HS-IMS and the formerly developed one will be simply denoted as P-IMS [34].The goal of this test was to quantitatively compare the reference P-IMS with the newly developed HS-IMS system.A Zeiss Z1 microscope served as the imaging platform and a mercury vapor X-Cite 120 (Excelitas, USA) provided excitation light, which was delivered to the epi-port of the microscope by a liquid guide cable.A double port adapter (T2-2x60N) with a 50/50 broadband beam splitter (operating range 400-750 nm, #21000, Chroma, USA) provided an image to both image mapping spectrometers, simultaneously.Both IMS systems used identical 2.5x (Plan-Apo 0.017 Zeiss, USA) microscope objectives in the fore-optical subsystem in order to guarantee identical light collection conditions.
The KAI-16000 detector installed in the P-IMS system has a 16 e -read noise, 14-bit maximum nominal bit depth, and 30,000 e -full well capacity.The CIS2521 sCMOS chip used in the HS-IMS system has a maximum root mean square read noise of 2.5 e -in the fast scan mode (the root mean square read noise drops to 1.1e -in the slow scan mode), 30,000 electron full well capacity and 16-bit analog to digital converter.In order to make a comparison that will be not biased by the camera A/D converter and signal conditioning circuitry, the intensity signal returned by each detector was converted to photo-electrons.The catalog specified conversion rate for the PCO5.5 Edge camera (stated as 0.46 e -/intensity count) was verified to be 0.504 e -/intensity count.Experimentally measured read noise was 2.6 e -in the fast scanning mode.To measure the conversion rate and read noise, the PCO camera was directly attached to the side port of the Z1 microscope and a series of defocused, uniformly illuminated background images were measured at multiple exposure times.Figure 6

BPAE cells imaging using HS-IMS and P-IMS systems
To qualitatively compare performance of the HS-IMS and P-IMS, a FluoCells slide #1 (F36924, Thermo-Fisher, USA) was observed in the setup described in the paragraph above.FluoCells slide #1 contains bovine pulmonary artery endothelial cells (BPAEC) stained with three fluorescent dyes: DAPI (λ Ex = 358/λ Em = 461), Alexa Fluor 488 (λ Ex = 505/λ Em = 512) and MitoTracker Red (λ Ex = 579/λ Em = 599).Both systems are capable of operation between 515 and 670 nm wavelength, and in that range signal emitted by AlexaFluor and MitoTracker was analyzed.It should be noted that due to the spectral properties of the 515 nm long pass filter, the newly developed HS-IMS system was able to record about half of the emission spectra of the AlexaFluor.Images of fluorescently stained samples were recorded by both systems at an identical exposure time of 500 ms.Intensity signals recorded by both systems were converted to photo electron count using the procedure described in the paragraph above to facilitate direct signal comparison.Emission spectra recorded at 537 nm wavelength by the HS-IMS and the P-IMS are presented in Fig. 7(a) and Fig. 7(b), respectively.The color scale on both images was set to cover an identical range to visualize differences in sensitivity between systems.Cross-sections through the 537 nm spectral pages (in the direction indicated by the white dotted line) and acquired by the newly developed and reference spectrometers are given in Fig. 7(c) and Fig. 7(d).Continuous blue and orange lines represent signals acquired by the MS-IMS and P-IMS, respectively.Cross-sections through the BPAEC sample are scaled individually on Fig. 7(c), with the left hand y-axis for HS-IMS and right hand side y-axis for P-IMS.Both plots with identical y-scales are drawn in Fig. 7(d).Spectral signatures of the BPAEC sample recorded at λ = 617 nm for both compared systems are shown in Fig. 7(d) and 7(e) for HS-IMS and P-IMS respectively.Both 2D images share the same color scale range to facilitate direct comparison.A cross-section through the HS-IMS and P-IMS data in the direction marked by the white lines in Fig. 7(e) and Fig. 7(f) is given in Fig. 7(g) and Fig. 7(h).Presented plots are individually scaled on Fig. 7(g), with the left and right y-axis scale range set to cover the HS-IMS and P-IMS data range respectively.The range of y-axis on Fig. 7(h) was selected to simultaneously visualize both plots.Please note that all spectral crosssections are provided for the common area of the two data sets, which is marked on all 2D images with a red rectangle.Source hyperspectral data cubes for both systems were numerically stitched from multiple laterally shifted fields of view to extend the visualized area and to eliminate a flat-field correction post-processing step [34].Additionally, all twodimensional images were low pass filtered to remove high frequency noise and their contrast,  Please not range of the cycle.

Conclusio
A new image capable of ac range covers verified spect average 1.1 measuring dis using a qualit target.Becau acquisition of range.The HS ng halogen bu he ability of olution in the sion of a halog o A1 Observ ype halogen b he newly deve ating in rollin recorded durin bulb was pow e halogen bulb Fig. 9(a) and ( the dynamic r tral plots, taken with red arrow re scaled to the malized to visua of intensity as ded for time t by two orders ent detector ex ned on, the fea atory ceiling m mitter in Fig. 9  It uses a small form factor imaging detector, has re-designed opto-mechanics, and is capable of data acquisition at a rate which is ~14 times faster compared to previous generations of the direct image mapping spectrometers.
Performance of the newly developed system was directly compared to a representative image mapping spectrometer from previous generations.Two samples were used to illustrate advantages of the newly developed system.Reference fluorescent beads and a BPAEC slide were simultaneously observed by both systems through a 50/50 beam splitter, and evidence to support superior performance of the newly developed systems was shown by directly comparing the spectral signature acquired by both compared systems.Dynamic range of the new spectrometer together with its ability to acquire data at 100 fps were presented in an experiment in which emission spectrum of a halogen bulb was continuously monitored during a power on/off cycle.The capability to monitor dynamic events in the full field of view without introduction of motion related artifacts was demonstrated in an experiment where free falling ping pong balls selectively stained with infrared dye were observed.
In the future we plan to employ the newly developed HS-IMS system to the observation of transient events in fluorescently stained living cells to study dynamic cellular processes.Additionally, we plan to take advantage of its high dynamic range by performing direct measurements of the absorbance/transmittance spectral signatures of thin pathological slides and integrate the newly developed spectrometer with a fundus camera to study retinal tissue perfusion.A small, compact version of this instrument will be developed in the future for use in remote sensing applications.
Fig. 1 compl assem Fig. 2 detect field p While the achromatic le accelerate dev A cross-se Fig. 3(b).Sep holes drilled i fixed by two aluminum, res C, D and F w dimensions su µm.The only (E, Fig. 3(a)), iterative proc procedure wa iteration until paper E, Fig. alignment of only 80 µm.machined fro technology.D slots housing limitations.T were fixed to Spacing betw thickness was critical from placed in the of lenslets an Focus of the between the b focus was set assembly was Entire assemb Ronchi ruling (c) ed in lateral shift ormation matri Data averaged e dispersion. in OpticStudio re given in Fig pectral range of elow 0.01 pixe differences bet d may be attr ic imperfection g the spectral ght source.A ousto-optic tuna an image ed during xel in the e look-up detail by patial and band light of the slit tial (x, y) mages of Fig. 5 540, 6 repres Optics spline IMS imag resolution bar IMS system inverted micro HD, Zeiss, configuration system shoul proving corre used to eva spectrometer.Germany) mi the IMS syste visualize geo Fig. 9 cycle.acquir at tim (to the laps re (

9 .
Images and spec A pseudo-colore red at time t = 0 (a me t = 2.75 s (b).S e detector's dynam ecording of the ha te that neutral d detector to ac , t = 1.7s an are shown in F ynamic range o of the spectral the Wien's disp = 1.7 s and t .Additionally, was held consta mitter are visib scent lamps.T errations introd halogen bulb emi image of the ha ic image of a tung plots taken at t = 0 ormalized are give an on/off cycle is ded at time t = that color map ata sets to ma nd t = 3.07s an Fig. 9(c) and Fi of the data set peak to shorte placement law = 3.07 s (Fig. , it should be ant.At time t ble only due to The blurred an duced by the g itter acquired durin alogen bulb emitte gsten emitter in an 0.9 s, t = 1.7 s and en in (c) and (dig.9(d).Plots p and plots give er wavelengths .It should be n .9(c) and (d)) pointed out th = 0 s until the ambient light nd distorted ba glass bulb.ng a power on/off er in an off state on state, acquired d t = 3.07 s scaled respectively.