A review of snapshot multidimensional optical imaging: Measuring photon tags in parallel

doi:10.1016/j.physrep.2015.12.004

Physics Reports

Volume 616, 29 February 2016, Pages 1-37

https://doi.org/10.1016/j.physrep.2015.12.004 Get rights and content

Abstract

Multidimensional optical imaging has seen remarkable growth in the past decade. Rather than measuring only the two-dimensional spatial distribution of light, as in conventional photography, multidimensional optical imaging captures light in up to nine dimensions, providing unprecedented information about incident photons’ spatial coordinates, emittance angles, wavelength, time, and polarization. Multidimensional optical imaging can be accomplished either by scanning or parallel acquisition. Compared with scanning-based imagers, parallel acquisition–also dubbed snapshot imaging–has a prominent advantage in maximizing optical throughput, particularly when measuring a datacube of high dimensions. Here, we first categorize snapshot multidimensional imagers based on their acquisition and image reconstruction strategies, then highlight the snapshot advantage in the context of optical throughput, and finally we discuss their state-of-the-art implementations and applications.

Section snippets

Introduction to multidimensional imaging

When performing optical measurement with a limited photon budget, it is important to assure that each detected photon provides as much information as possible. Conventional optical imaging systems generally capture light with just two characteristics ( $x, y$ ), measuring its intensity in a 2D ( $x, y$ ) lattice. However, this throws away much of the information content actually carried by a photon. This information can be written in nine dimensions as ( $x, y, z, θ, φ, λ, t, ψ, χ$ ): the spatial coordinates ( $x, y, z$

General acquisition schemes and advantages of parallel measurement in multidimensional imaging

To acquire a multidimensional datacube, a system must be able to differentiate photons with different characteristics. The most intuitive approach is to successively apply a variety of filters to the incident light and let photons with only desired characteristics pass through at each stage (Fig. 1a). Unfortunately, this results in a severe loss in optical throughput. By contrast, if an approach directs, rather than filters, photons with different tags towards distinct pixels on an FPA, the

Snapshot spectral imaging ( $x, y, λ$ )

Rather than simply capturing two-dimensional intensity images like a monochromatic camera or measuring spectra like a spectrometer, a spectral imager acquires entire 3D datacubes ( $x$ , $y$ , $λ$ ) for multivariate analysis, providing structural, molecular, and functional information about the sample with unprecedented detail [28], [29]. Using the conceptual framework in Fig. 2, snapshot spectral imagers can be divided into two categories. In the direct-measurement category, representative techniques

Discussions and outlook

In this review, we categorized snapshot multidimensional imagers based on their acquisition strategies and reconstruction strategies, and we discussed their state-of-the-art implementations in spectral imaging, plenoptic imaging, volumetric imaging, temporal imaging, and polarization imaging. Compared with their scanning-based counterparts, snapshot imagers have a remarkable advantage in optical throughput. The more datacube dimensions a snapshot imager measures, the greater the advantage in

Acknowledgments

The authors thank Professor James Ballard for close reading of the manuscript. This work was supported in part by National Institutes of Health grants DP1 EB016986 (NIH Director’s Pioneer Award) and R01 CA186567 (NIH Director’s Transformative Research Award). L.V.W. has a financial interest in Microphotoacoustics, Inc. and Endra, Inc., which, however, did not support this work.

References (251)

J. Allington-Smith
Basic principles of integral field spectroscopy
New Astron. Rev.
(2006)
H. Matsuoka et al.
Single-cell viability assessment with a novel spectro-imaging system
J. Biotechnol.
(2002)
B.K. Ford et al.
Computed tomography-based spectral imaging for fluorescence microscopy
Biophys. J.
(2001)
D.A. Basiji et al.
Cellular image analysis and imaging by flow cytometry
Clin. Lab. Med.
(2007)
M.B. Sinclair et al.
Hyperspectral confocal microscope
Appl. Opt.
(2006)
M.B. Sinclair et al.
Design, construction, characterization, and application of a hyperspectral microarray scanner
Appl. Opt.
(2004)
P.M. Kasili et al.
Hyperspectral imaging system using acousto-optic tunable filter for flow cytometry applications
Cytometry A
(2006)
Kodak, KAF-50100 IMAGE SENSOR, in: Kodak Products,...
C. Zhou et al.
Computational cameras: Convergence of optics and processing
IEEE Trans. Image Process.
(2011)
G. Wetzstein et al.
Computational plenoptic imaging
Comput. Graph. Forum
(2011)

R.M. Willett et al.

Compressed sensing for practical optical imaging systems: a tutorial

Opt. Eng.

(2011)

F. Bucholtz et al.

Compressive sensing demystified

Opt. Photonics News

(2014)

G. Chapman

Ultra-precision machining systems; an enabling technology for perfect surfaces

R. Williamson

Field Guide to Optical Fabrication

(2011)

A. Wagadarikar et al.

Single disperser design for coded aperture snapshot spectral imaging

Appl. Opt.

(2008)

A.A. Wagadarikar et al.

Video rate spectral imaging using a coded aperture snapshot spectral imager

Opt. Express

(2009)

D. Reddy, A. Veeraraghavan, R. Chellappa, P2C2: Programmable pixel compressive camera for high speed imaging, in:...

Nanotech, 250 UPL v2 Compact Diamond Turning Lathe, in: Nanotech...

N. Bedard et al.

Image mapping spectrometry: calibration and characterization

Opt. Eng.

(2012)

L.A. Gao et al.

Snapshot Image Mapping Spectrometer (IMS) with high sampling density for hyperspectral microscopy

Opt. Express

(2010)

L. Gao et al.

Compact Image Slicing Spectrometer (ISS) for hyperspectral fluorescence microscopy

Opt. Express

(2009)

B. Javidi et al.

Multi-Dimensional Imaging

(2014)

P. Jacquinot

The luminosity of spectrometers with prisms, gratings, or Fabry–Perot etalons

J. Opt. Soc. Am.

(1954)

N. Hagen et al.

Snapshot advantage: a review of the light collection improvement for parallel high-dimensional measurement systems

Opt. Eng.

(2012)

J.S. Tyo et al.

Review of passive imaging polarimetry for remote sensing applications

Appl. Opt.

(2006)

L.I.o. America

American National Standard for Safe Use of Lasers

(2000)

L. Gao et al.

Depth-resolved image mapping spectrometer (IMS) with structured illumination

Opt. Express

(2011)

M. Levoy et al.

Light field microscopy

K. Nakagawa et al.

Sequentially timed all-optical mapping photography (STAMP)

Nature Photon.

(2014)

L. Gao et al.

Single-shot compressed ultrafast photography at one hundred billion frames per second

Nature

(2014)

L. Gao et al.

Optical hyperspectral imaging in microscopy and spectroscopy–a review of data acquisition

J. Biophotonics

(2014)

G. Lu et al.

Medical hyperspectral imaging: a review

J. Biomed. Opt.

(2014)

A. Bodkin, A. Sheinis, A. Norton, J. Daly, C. Roberts, S. Beaven, J. Weinheimer, Video-rate chemical identification and...

A. Bodkin et al.

Snapshot hyperspectral imaging: the hyperpixel array camera

A. Bodkin et al.

Hyperspectral imaging systems

T.C. George et al.

Distinguishing modes of cell death using imagestream (TM) multispectral imaging cytometry

Cytometry A

(2004)

D.W.F.-H. Alistair Gorman et al.

Generalization of the Lyot filter and its application to snapshot spectral imaging

Opt. Express

(2010)

M.W. Kudenov et al.

Compact real-time birefringent imaging spectrometer

Opt. Express

(2012)

M.W. Kudenov et al.

Compact snapshot birefringent imaging Fourier transform spectrometer

M.W. Kudenov et al.

White-light Sagnac interferometer for snapshot multispectral imaging

Appl. Opt.

(2010)

B.K. Ford et al.

Large-image-format computed tomography imaging spectrometer for fluorescence microscopy

Opt. Express

(2001)

F. Henault, R. Bacon, R. Content, B. Lantz, F. Laurent, J. Lemonnier, S. Morris, Slicing the universe at affordable...

S. Vives et al.

Original image slicer designed for integral field spectroscopy with the near-infrared spectrograph for the James webb space telescope

Opt. Eng.

(2006)

R.T. Kester et al.

Development of image mappers for hyperspectral biomedical imaging applications

Appl. Opt.

(2010)

L. Gao et al.

Correction of vignetting and distortion errors induced by two-axis light beam steering

Opt. Eng.

(2012)

R.T. Kester et al.

Real-time snapshot hyperspectral imaging endoscope

J. Biomed. Opt.

(2011)

A.D. Elliott et al.

Real-time hyperspectral fluorescence imaging of pancreatic $β$ -cell dynamics with the image mapping spectrometer

J. Cell Sci.

(2012)

L. Gao et al.

Snapshot image-mapping spectrometer for hyperspectral fluorescence microscopy

Opt. Photonics News

(2010)

L. Gao et al.

Quantitative comparison between full-spectrum and filter-based imaging in hyperspectral fluorescence microscopy

J. Microsc.

(2012)

L. Gao et al.

Snapshot hyperspectral retinal camera with the Image Mapping Spectrometer (IMS)

Biomed. Opt. Express

(2012)

Cited by (131)

Lenslet-array-based snapshot hyperspectral imaging polarimeter using opposite spectral modulation
2024, Optics and Lasers in Engineering
We present a lenslet-array-based snapshot hyperspectral imaging polarimeter (LSHIP) that integrates spectral polarization modulation (SPM) into integral field imaging spectrometry. The LSHIP can simultaneously acquire three-dimensional spatial and spectral data-cubes for linear Stokes parameters in a single snapshot without any moving parts. The dual-scan technique is achieved through a custom-fabricated Wollaston prism that can simultaneously generate pairs of micro-spectra with opposite spectral modulations, maintaining the system's intrinsic spectral resolution during intensity spectrum reconstruction. We describe the principle and experimental setup of the LSHIP, as well as the system calibration and validation using measurements of polarized color blocks and a sphere made of polytetrafluoroethylene material. The prototype can cover the spectral range of 450–650 nm with 157 sample points, capturing approximately 3700 SPM micro-spectra spatially distributed on the detector in a single integration time. These results demonstrate the potential of the LSHIP for various applications, including remote sensing and biomedical diagnosis.
Trapezoidal dual-function splitter under second Bragg incidence
2024, Optics Communications
In this paper, a reflective trapezoidal dual-functional beam splitter grating is proposed, which performs two independent functions at 1550 nm. For TE polarization, the grating can achieve a uniform three-port output with efficiencies of 32.85%, 32.89%, and 32.75% for 0th, -1st, and -2nd orders respectively. For TM polarization, the grating can achieve a highly-efficient single-channel output of order -2nd with an efficiency of 93.27%. HfO₂ is chosen as the material of grating ridge layer, the structural parameters of the grating are studied by using rigorous coupled-wave analysis (RCWA) and simulated annealing algorithm (SAA), and the results are verified by using finite element method (FEM). In addition, the influence of structure and incident error on grating efficiency was also analyzed, the results indicate that the grating has good incident bandwidth and manufacturing tolerance with excellent performance.
Design and development of Fabry–Perot etalon based color filter using ebeam GLAD technique: Simulation and experiment
2023, Optics Communications
The design of Spatial Variable Color Filter (SVCF), based on the principle of Fabry–Perot (FP) etalon, has been analyzed in detail considering Ag as reflecting metallic layer and TiO₂ as cavity layer. The effect of different thickness ranges of cavity layer for different order of interference peaks and the effect of varying thickness of reflecting layer has been investigated through simulation. A SVCF has been designed in the broad visible wavelength range of 464 to 670 nm based on the 2nd order interference peak of FP etalon for optimized spectral performance and it has been experimentally demonstrated using ebeam evaporation technique. The reflecting Ag layer has been deposited at normal angle and the tapered cavity layer of TiO₂ has been deposited at 86° glancing angle. The distribution of thickness and dispersion relationship at various locations of the glancing angle deposited (GLAD) TiO₂ thin films has been analyzed by spectroscopic ellipsometry measurements. The effect of lateral variation of dispersion properties of cavity layer caused by GLAD, on the spectral performance of the developed SVCF, has been investigated by both simulation and through spectral transmittance measurement. Overall results demonstrate the potential of the axial glancing angle deposition technique for the first time for development of SVCF in visible wavelength with high peak transmittance ( $\sim$ 67%) and a peak wavelength tunability of $\sim$ 4 nm/mm.
Current advances and future perspectives of image fusion: A comprehensive review
2023, Information Fusion
Citation Excerpt :
This model comprises three parts: feature extraction, fusion, and reconstruction, which resolve the LD and feature representation issues. Photons carry a prosperity of information, which can be represented by nine-dimensional data (x, y, z, θ, φ, λ, t, ψ, χ), where (x, y, z) represents the spatial dimensions, (θ, φ,) represents the propagation angle of light, λ represents the wavelength, t represents the emission time of photons, (ψ, χ) represents the polarization state of light and ellipticity [218]. Imaging technology is a technology for recording target photon information.
Multiple imaging modalities can be combined to provide more information about the real world than a single modality alone. Infrared images discriminate targets with respect to their thermal radiation differences, and visible images are promising for texture details. On the other hand, polarized images deliver intensity and polarization information, and multispectral images dispense the spatial, spectral, and temporal information depending upon the environment. Different sensors provide images with different characteristics, such as type of degradation, important features, textural attributes, etc. Several stimulating tasks have been explored in the last decades based on algorithms, performance assessments, processing techniques, and prospective applications. However, most of the reviews and surveys have not properly addressed the issues of additional possibilities of imaging fusion. The primary goal of this paper is to give a thorough overview of image fusion approaches, including associated background and current breakthroughs. We introduce image fusion and categorize the methods based on conventional image processing, deep learning (DL) architectures, and fusion scenarios. Further, we emphasize the recent DL developments in various image fusion scenarios. However, there are still several difficulties to overcome, including developing more advanced algorithms to support more dependable and real-time practical applications, discussed in future perspectives. This study can assist researchers in coping with multiple imaging modalities, recent fusion developments, and future perspectives.
Pixelated polarizers array based coded aperture snapshot spectropolarimetric imaging with side information
2022, Optics Communications
Imaging spectropolarimetry is a promising technology in the fields such as biomedicine, planetary exploration, remote sensing, and so on. In this paper, a coded aperture snapshot spectropolarimetric imagery is developed by incorporating a pixelated polarizers array (PPA) into the coded aperture snapshot spectral imagery (CASSI). Aside from spatial coding in CASSI, polarization encoding is introduced in the proposed system by the PPA. As side information, an RGB image captured by a color camera is used to improve the reconstruction performance. The imaging model and reconstruction algorithm have been presented. An experiment has been carried out to validate the system’s feasibility. The significant advantages of the system are snapshot imaging, no scanning, and suitable for dynamic and static target detection. It gives us an alternative method for acquiring the spatial, spectral, and complete linear polarization information of the targets with a snapshot measurement.
Multidimensional photodetection of light fields based on metasurfaces or two-dimensional materials
2024, Applied Physics Letters

View all citing articles on Scopus

View full text

A review of snapshot multidimensional optical imaging: Measuring photon tags in parallel

Abstract

Section snippets

Introduction to multidimensional imaging

General acquisition schemes and advantages of parallel measurement in multidimensional imaging

Snapshot spectral imaging (x,y,λ)

Discussions and outlook

Acknowledgments

New Astron. Rev.

J. Biotechnol.

Biophys. J.

Clin. Lab. Med.

Hyperspectral confocal microscope

Appl. Opt.

Design, construction, characterization, and application of a hyperspectral microarray scanner

Appl. Opt.

Hyperspectral imaging system using acousto-optic tunable filter for flow cytometry applications

Cytometry A

Computational cameras: Convergence of optics and processing

IEEE Trans. Image Process.

Computational plenoptic imaging

Comput. Graph. Forum

Compressed sensing for practical optical imaging systems: a tutorial

Opt. Eng.

Compressive sensing demystified

Opt. Photonics News

Ultra-precision machining systems; an enabling technology for perfect surfaces

Field Guide to Optical Fabrication

Single disperser design for coded aperture snapshot spectral imaging

Appl. Opt.

Video rate spectral imaging using a coded aperture snapshot spectral imager

Opt. Express

Image mapping spectrometry: calibration and characterization

Opt. Eng.

Snapshot Image Mapping Spectrometer (IMS) with high sampling density for hyperspectral microscopy

Opt. Express

Compact Image Slicing Spectrometer (ISS) for hyperspectral fluorescence microscopy

Opt. Express

Multi-Dimensional Imaging

The luminosity of spectrometers with prisms, gratings, or Fabry–Perot etalons

J. Opt. Soc. Am.

Snapshot advantage: a review of the light collection improvement for parallel high-dimensional measurement systems

Opt. Eng.

Review of passive imaging polarimetry for remote sensing applications

Appl. Opt.

American National Standard for Safe Use of Lasers

Depth-resolved image mapping spectrometer (IMS) with structured illumination

Opt. Express

Light field microscopy

Sequentially timed all-optical mapping photography (STAMP)

Nature Photon.

Single-shot compressed ultrafast photography at one hundred billion frames per second

Nature

Optical hyperspectral imaging in microscopy and spectroscopy–a review of data acquisition

J. Biophotonics

Medical hyperspectral imaging: a review

J. Biomed. Opt.

Snapshot hyperspectral imaging: the hyperpixel array camera

Hyperspectral imaging systems

Distinguishing modes of cell death using imagestream (TM) multispectral imaging cytometry

Cytometry A

Generalization of the Lyot filter and its application to snapshot spectral imaging

Opt. Express

Compact real-time birefringent imaging spectrometer

Opt. Express

Compact snapshot birefringent imaging Fourier transform spectrometer

White-light Sagnac interferometer for snapshot multispectral imaging

Appl. Opt.

Large-image-format computed tomography imaging spectrometer for fluorescence microscopy

Opt. Express

Original image slicer designed for integral field spectroscopy with the near-infrared spectrograph for the James webb space telescope

Opt. Eng.

Development of image mappers for hyperspectral biomedical imaging applications

Appl. Opt.

Correction of vignetting and distortion errors induced by two-axis light beam steering

Opt. Eng.

Real-time snapshot hyperspectral imaging endoscope

J. Biomed. Opt.

Real-time hyperspectral fluorescence imaging of pancreatic β-cell dynamics with the image mapping spectrometer

J. Cell Sci.

Snapshot spectral imaging ( $x, y, λ$ )

Real-time hyperspectral fluorescence imaging of pancreatic $β$ -cell dynamics with the image mapping spectrometer