Carlos Osorio Quero, Daniel Durini, Jose Rangel-Magdaleno, Jose Martinez-Carranza, and Ruben Ramos-Garcia, "Deep-learning blurring correction of images obtained from NIR single-pixel imaging," J. Opt. Soc. Am. A 40, 1491-1499 (2023)
In challenging scenarios characterized by low-photon conditions or the presence of scattering effects caused by rain, fog, or smoke, conventional silicon-based cameras face limitations in capturing visible images. This often leads to reduced visibility and image contrast. However, using near-infrared (NIR) light within the range of 850–1550 nm offers the advantage of reduced scattering by microparticles, making it an attractive option for imaging in such conditions. Despite NIR’s advantages, NIR cameras can be prohibitively expensive. To address this issue, we propose a vision system that leverages NIR active illumination single-pixel imaging (SPI) operating at 1550 nm combined with time of flight operating at 850 nm for 2D image reconstruction, specifically targeting rainy conditions. We incorporate diffusion models into the proposed system to enhance the quality of NIR-SPI images. By simulating various conditions of background illumination and droplet size in an outdoor laboratory scenario, we assess the feasibility of utilizing NIR-SPI as a vision sensor in challenging outdoor environments.
Video showcasing the implementation of the diffusion model to improve the quality of single-pixel imaging.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
Cited By
You do not have subscription access to this journal. Cited by links are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.
You do not have subscription access to this journal. Figure files are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.
You do not have subscription access to this journal. Article tables are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.
You do not have subscription access to this journal. Equations are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.
Total Processing Time, Denoted as , Varies Depending on the Image Capture and Reconstruction Stagesa
(ms)
(ms)
(ms)
(ms)
(ms)
18–24
20–25
9
–
45–58
18–24
20–25
9
15
62–76
This includes considering the acquisition time ADC (${T_{\rm{aq}}}$), the Hadamard time projection array NIR-LEDs (${T_{\rm{Had}}}$), the exposure time of the bucket detector (${T_{\rm{ext}}}$), which is defined as the sum of pre-acquisition stage times (${T_{\rm{pre}}} = {T_{\rm{aq}}} + {T_{\rm{Had}}} + {T_{\rm{ext}}}$), reconstruction time (${T_{\rm{OMP}}}$), TOF fusion image-processing time (${T_{\rm{DL}}}$), and the application of a diffusion model (${T_{\rm{dm}}}$).
Algorithm 1.
OMP-GPU algorithm [26], Input: OMP-GPU algorithm input data: Patterns , input signal , target sparsity , Output: OMP-GPU algorithm output data: sparse representation that fulfills the relation
1: procedure OMP-GPU , , K:
2: set: , ,
3: set: , ,
4: whiledo
5: Finding the new atom
6: ifthen
7: Solver
8: Update of Cholesky
9:
10: Solver
11: Matrix-sparse-vector product for each path
12:
13: Calculate error
14: Calculate norm
15:
16: increasing iteration
17: return
Algorithm 2.
Pseudocode to estimate the maximum capture distance of SPI camera under rainy conditions, Input: background radiance (noise), photodetector’s quantum efficiency, material reflection index, wavelength, exposure time, time it takes to project the active illumination patterns, field-far measurement, horizontal polarized reflectivity, vertical polarized reflectivity, diameter of the water droplet, Output: Maximum measurement distance
Number of photons impinging on the photodetector [32]
Calculation of the general electrical noise floor [33].
ifthen
Break
ii=ii+1
Increase the step measurement
return
Table 2.
Relation Exposure Time (), Rain Rate (), and Capture Distance under Conditions of Light and Heavy Artificial Rain Based on Algorithm 2
Rainfall
(µs)
(mm/h)
Distance (m)
Light
8–25
1.93
0.1–2
Heavy
8–25
5.6
0.2–1.5
Table 3.
Evaluation of Image Reconstruction Quality Using NIR-SPI Compared to VIS under Rainy Conditions, with 2 mm Droplet Diameters and a Background Illumination of 15 KLux (Half-Cloudy)a
Image
PSNR (dB)
SSIM
FID
VIS
10
0.3
-
NIR-SPI
24
0.7
14
23
0.75
38
26
0.84
11
We measured PSNR, SSIM, and FID to assess the performance. The network model GAN [45] was applied, along with our diffusion model, to test the defined objects as shown in Fig. 6.
Table 4.
Evaluation of Spatial Resolution (mm) under Rainy Conditions with 2 mm Droplet Diameters, Background Illumination of 15 KLux (Half-Cloudy) for the Distance Measurement , , and Applying the Network Model GAN [45] and Our Diffusion Model
Image
VIS
45
48
60
NIR-SPI
5
25
45
5
18
40
5
15
25
Tables (6)
Table 1.
Total Processing Time, Denoted as , Varies Depending on the Image Capture and Reconstruction Stagesa
(ms)
(ms)
(ms)
(ms)
(ms)
18–24
20–25
9
–
45–58
18–24
20–25
9
15
62–76
This includes considering the acquisition time ADC (${T_{\rm{aq}}}$), the Hadamard time projection array NIR-LEDs (${T_{\rm{Had}}}$), the exposure time of the bucket detector (${T_{\rm{ext}}}$), which is defined as the sum of pre-acquisition stage times (${T_{\rm{pre}}} = {T_{\rm{aq}}} + {T_{\rm{Had}}} + {T_{\rm{ext}}}$), reconstruction time (${T_{\rm{OMP}}}$), TOF fusion image-processing time (${T_{\rm{DL}}}$), and the application of a diffusion model (${T_{\rm{dm}}}$).
Algorithm 1.
OMP-GPU algorithm [26], Input: OMP-GPU algorithm input data: Patterns , input signal , target sparsity , Output: OMP-GPU algorithm output data: sparse representation that fulfills the relation
1: procedure OMP-GPU , , K:
2: set: , ,
3: set: , ,
4: whiledo
5: Finding the new atom
6: ifthen
7: Solver
8: Update of Cholesky
9:
10: Solver
11: Matrix-sparse-vector product for each path
12:
13: Calculate error
14: Calculate norm
15:
16: increasing iteration
17: return
Algorithm 2.
Pseudocode to estimate the maximum capture distance of SPI camera under rainy conditions, Input: background radiance (noise), photodetector’s quantum efficiency, material reflection index, wavelength, exposure time, time it takes to project the active illumination patterns, field-far measurement, horizontal polarized reflectivity, vertical polarized reflectivity, diameter of the water droplet, Output: Maximum measurement distance
Number of photons impinging on the photodetector [32]
Calculation of the general electrical noise floor [33].
ifthen
Break
ii=ii+1
Increase the step measurement
return
Table 2.
Relation Exposure Time (), Rain Rate (), and Capture Distance under Conditions of Light and Heavy Artificial Rain Based on Algorithm 2
Rainfall
(µs)
(mm/h)
Distance (m)
Light
8–25
1.93
0.1–2
Heavy
8–25
5.6
0.2–1.5
Table 3.
Evaluation of Image Reconstruction Quality Using NIR-SPI Compared to VIS under Rainy Conditions, with 2 mm Droplet Diameters and a Background Illumination of 15 KLux (Half-Cloudy)a
Image
PSNR (dB)
SSIM
FID
VIS
10
0.3
-
NIR-SPI
24
0.7
14
23
0.75
38
26
0.84
11
We measured PSNR, SSIM, and FID to assess the performance. The network model GAN [45] was applied, along with our diffusion model, to test the defined objects as shown in Fig. 6.
Table 4.
Evaluation of Spatial Resolution (mm) under Rainy Conditions with 2 mm Droplet Diameters, Background Illumination of 15 KLux (Half-Cloudy) for the Distance Measurement , , and Applying the Network Model GAN [45] and Our Diffusion Model
Add to BibSonomy