Motion-resistant three-wavelength spatial frequency domain imaging system with ambient light suppression using an 8-tap CMOS image sensor

Yu Feng; Chen Cao; Yuto Shimada; Keita Yasutomi; Shoji Kawahito; Gordon T. Kennedy; Anthony J. Durkin; Keiichiro Kagawa

doi:10.1117/1.JBO.29.1.016006

18 January 2024 Motion-resistant three-wavelength spatial frequency domain imaging system with ambient light suppression using an 8-tap CMOS image sensor

Yu Feng, Chen Cao, Yuto Shimada, Keita Yasutomi, Shoji Kawahito, Gordon T. Kennedy, Anthony J. Durkin, Keiichiro Kagawa

Author Affiliations +

Journal of Biomedical Optics, Vol. 29, Issue 1, 016006 (January 2024). https://doi.org/10.1117/1.JBO.29.1.016006

Abstract

Significance

We present a motion-resistant three-wavelength spatial frequency domain imaging (SFDI) system with ambient light suppression using an 8-tap complementary metal-oxide semiconductor (CMOS) image sensor (CIS) developed at Shizuoka University. The system addresses limitations in conventional SFDI systems, enabling reliable measurements in challenging imaging scenarios that are closer to real-world conditions.

Aim

Our study demonstrates a three-wavelength SFDI system based on an 8-tap CIS. We demonstrate and evaluate the system’s capability of mitigating motion artifacts and ambient light bias through tissue phantom reflectance experiments and in vivo volar forearm experiments.

Approach

We incorporated the Hilbert transform to reduce the required number of projected patterns per wavelength from three to two per spatial frequency. The 8-tap image sensor has eight charge storage diodes per pixel; therefore, simultaneous image acquisition of eight images based on multi-exposure is possible. Taking advantage of this feature, the sensor simultaneously acquires images for planar illumination, sinusoidal pattern projection at three wavelengths, and ambient light. The ambient light bias is eliminated by subtracting the ambient light image from the others. Motion artifacts are suppressed by reducing the exposure and projection time for each pattern while maintaining sufficient signal levels by repeating the exposure. The system is compared to a conventional SFDI system in tissue phantom experiments and then in vivo measurements of human volar forearms.

Results

The 8-tap image sensor-based SFDI system achieved an acquisition rate of 9.4 frame sets per second, with three repeated exposures during each accumulation period. The diffuse reflectance maps of three different tissue phantoms using the conventional SFDI system and the 8-tap image sensor-based SFDI system showed good agreement except for high scattering phantoms. For the in vivo volar forearm measurements, our system successfully measured total hemoglobin concentration, tissue oxygen saturation, and reduced scattering coefficient maps of the subject during motion (16.5 cm/s) and under ambient light (28.9 lx), exhibiting fewer motion artifacts compared with the conventional SFDI.

Conclusions

We demonstrated the potential for motion-resistant three-wavelength SFDI system with ambient light suppression using an 8-tap CIS.

1. Introduction

Monitoring tissue metabolic and perfusion information such as oxygenation and hemoglobin concentration plays a crucial role in healthcare and therapy. Traditional methods commonly used to obtain such information include using a pulse oximeter¹ attached to the patient’s finger or employing near-infrared spectroscopy (NIRS)² using probes. However, neither of them can provide noncontact measurements nor cover a wide field of view. Spatial frequency domain imaging (SFDI) is an imaging technique that projects structured light with different spatial frequencies and phases onto diffusely reflecting objects to quantify absorption and reduced scattering coefficients.³^,⁴ SFDI is a noninvasive, noncontact, real-time, wide-field, quantitative imaging technique. Compact SFDI systems designed for bedside use have been employed in various experiments, including predicting diabetic foot ulcer,⁵^–⁷ oxygenation measurement during reconstructive breast surgery,⁸ monitoring skin cancer,⁹ imaging and staging pressure ulcers,¹⁰ among others. SFDI is a particularly promising method for diagnosing burn severity since changes in reduced scattering measured using SFDI have been shown to correlate with histologically verified changes in collagen as a function of burn severity. These changes can be detected as early as 24 h after the burn.¹¹^,¹² This change in scattering, along with tissue oxygen saturation ( ${StO}_{2}$ ) and total hemoglobin concentration ( $c_{tHB}$ ) derived from absorption coefficients, enable clinicians to make improved treatment decisions and can sometimes lead to early recovery for the patient.¹³ To accurately quantify tissue metabolic information, particularly the concentrations of oxy/deoxy hemoglobin, we require SFDI measurements at two or more wavelengths. However, since the presence of melanin in the epidermis can complicate the measurement results of other tissue chromophores, at least three wavelengths (as many as the number of chromophores) are generally required to achieve accurate determination of oxy and deoxyhemoglobin concentrations.

One of the limitations of SFDI is its susceptibility to ambient light.¹⁴ Most state-of-the-art SFDI systems require operating in low-light or dark conditions, as ambient light becomes a bias for the pattern projection and consumes detector dynamic range. However, there is an increasing demand for SFDI systems to be used in a variety of environments, including bright environments such as patient rooms, operating rooms, and outdoors. In these scenarios, the ambient light bias poses a challenge for accurate SFDI measurements.

Another challenge in SFDI is motion artifacts, which can be caused by the movement of the sample region of interest (ROI) during sequential pattern projection (i.e., motion due to respiratory cycles and shaking due to pain). As we assume that the captured raw images are stationary in the subsequent image processing, motion artifacts appear in the measurement results.¹⁴ They tend to be most pronounced at the subject’s edges and in regions with substantial variations in absorption and reduced scattering coefficients, such as in proximity to blood vessels. These artifacts, in turn, introduce errors into the measured skin optical properties.

In this study, to address these limitations, we present the development of a three-wavelength SFDI system based on an 8-tap complementary metal-oxide semiconductor (CMOS) image sensor (CIS) prototype. Our system offers a unique ability to simultaneously measure three wavelength bands, effectively addressing two prominent challenges faced by conventional SFDI systems: ambient light bias and motion artifacts. Specifically, we demonstrate the system’s efficacy in measuring the volar forearm of two human subjects under ambient light conditions and during controlled motion of the subjects’ arm. These two subjects were selected because their blood vessels are more visible, enabling easier assessment of any motion artifacts that may occur.

2. Materials and Methods

2.1.

Multitap CIS

The multitap CIS pixel is composed of a single photodiode (PD), and multiple pairs of charge transfer gates (TGs) and storage diodes (SDs), referred to as “taps.” This unique design enables the multitap image sensor pixel to output multiple pixel values upon readout, as opposed to the single pixel value output from a typical image sensor pixel. Schwarte et al.¹⁵ first introduced the original 2-tap pixel based on photogates. 4-tap pixels have been developed by Seo et al.¹⁶ from our research group utilizing the lateral electric field charge modulator (LEFM)¹⁷ and by Keel et al.,¹⁸ who employed the photogate. Hatakeyama et al.¹⁹ and Kuo and Kuroda²⁰ developed 4-tap pixels using TGs. Our research group successfully developed an 8-tap pixel utilizing the LEFM technology, specifically for the purpose of short-pulse-based time-of-flight range measurements.²¹

Figures 1(a) and 1(b) depict the structures of a typical image sensor pixel and an 8-tap sensor pixel, respectively. The charge TGs (G1 to G8) modulate the flow of the photogenerated charges, whereas the SD temporarily stores the transferred charges. In typical image sensor pixels, the TG turns on and transfers the photogenerated charges into the floating diffusion (FD) after the accumulation time. In contrast, in an 8-tap sensor, a designated TG among G1 to G8 turns on, and transfers the photogenerated charges to the corresponding SD. The gate for draining charges (GD) is used to make an insensitive period to the incident light, e.g., during the image readout. Subsequently, during the readout time, a typical image sensor pixel outputs a single pixel value, whereas an 8-tap image sensor pixel simultaneously outputs eight pixel values. Moreover, the 8-tap image sensor pixel allows multiple exposures, as the TG can be turned on and off multiple times during the accumulation time.

Fig. 1

(a) Simplified schematic diagram of a typical image sensor pixel (reset switch is omitted), (b) an 8-tap image sensor pixel, (c) and an 8-tap macropixel with a $2 \times 2$ -subpixel configuration.

In this study, we utilize a laboratory-designed 8-tap CIS pixel based on LEFM technology. Compared with the original 8-tap image sensor,²¹ the sensor features a smaller pixel size and a higher fill factor. Note that the pixel array of this image sensor is composed of macropixels with a $2 \times 2$ -subpixel configuration, where each subpixel consists of two taps and a drain, as illustrated in Fig. 1(c). The layout of the subpixel cell is similar to the reported 2-tap pixel.²² Thus, it can be defined as a “virtual” 8-tap pixel. The pixel design complexity is significantly reduced compared with the original 8-tap sensor. The specifications of this virtual 8-tap CIS are briefly shown in Table 1.

Table 1

Sensor specifications.

Parameters	Value
Macropixel size ( ${μ m}^{2}$ )	12.6 × 12.6
Subpixel number/macropixel	2 × 2
Macropixel number	700 (H) × 540 (V)
Tap number/subpixel	2 taps + drain
Max. frame rate (fps)	33

Despite the native resolution of $700 \times 540$ , the 8-tap sensor’s macropixel configuration, followed by interpolation similar to that used in RGB cameras, results in an effective resolution of 1400 × 1080 pixels. This effective resolution is comparable with typical cameras used in SFDI. The prototyped image sensor is based on $0.11 μ m$ CIS process, resulting in relatively large pixel size, about $12.6 \times 12.6 {μ m}^{2}$ . However, when a more advanced CIS process designed for mass production is available, the pixel size can be reduced, allowing more pixels within the same pixel area. While the taps and drains of the multitap sensor will still require some overhead for pixel shrinkage compared to a normal sensor, the overall functionality it provides remains beneficial.

2.2.

System Operation

Figure 2 shows the schematic diagram and a photograph of the SFDI system utilized in this study. The working distance of the 8-tap system is 24 cm. The system employs light-emitting diodes (LEDs) with wavelengths of 660, 730, and 850 nm to quantify oxyhemoglobin and deoxyhemoglobin, and the reduced scattering coefficients on a pixel-by-pixel basis. A two-dimensional sinusoidal patterned light is generated by a digital micromirror device (DMD; Keynote Photonics, LC4500NIR) and projected onto the sample. The reflected light is then captured by the 8-tap image sensor. During the imaging process, after the exposure for one pattern is finished, the field-programmable gate array (FPGA) board that controls the sensor sends a trigger signal to the DMD, initiating the projection of the next pattern. Most conventional SFDI systems utilize a sinusoidal pattern at a particular spatial frequency, projected three times, each time shifted by $2 π / 3$ radians for each wavelength. However, in this study, we employ an improved acquisition method based on the Hilbert transform,²³ which reduces the number of required images per wavelength (for a particular spatial frequency) from three to two, i.e., an image for planar illumination (direct current (DC) or sine wave at $0 {mm}^{- 1}$ ) and that for a single sinusoidal pattern (alternate current (AC) or sine wave at $0.1 {mm}^{- 1}$ ). In this study, we capture DC and AC images at the three wavelengths simultaneously using the 8-tap image sensor. Through the application of the Hilbert transform, we calculated the corresponding cosine image from the sine image. Additionally, we capture an image only for the ambient light. Since there is one spare tap available, a DC image for a wavelength (554 nm, LED) is captured. This wavelength is suitable for separating the melanin present in the epidermis. However, at the present stage, this additional DC image is not utilized in the subsequent processes but will be used in future work.

Fig. 2

(a) Schematic diagram of the SFDI system. (b) A photograph of the system (top view).

Figure 3(a) illustrates the timing chart of the 8-tap sensor operation. In this chart, $T_{0}$ represents the exposure time for each tap within an exposure cycle. In this work, to mitigate motion artifacts, the time for each pattern projection and sensor exposure is shortened to reduce time lag and make any motion among the frames imperceptible. During the accumulation period, the exposure cycle repeats N times to ensure that the brightness of the captured images from each tap is maintained. The total exposure time for each tap is denoted as $N$ $T_{0}$ . Unlike conventional high-speed image sensors that read out each image after each exposure, the 8-tap sensor reads out only after the accumulation period finishes, outputting all eight images simultaneously in the readout period.

Fig. 3

(a) Timing chart of the 8-tap image sensor operation. (b) An example of captured raw images of the 8-tap image sensor-based SFDI system under ambient light.

This unique multi-exposure capability results in fewer readouts. Although high-frame-rate image sensors can emulate the operation of multitap image sensors, they have drawbacks such as higher total read noise, heat dissipation, and processing power. When the number of exposure cycles is three and the image readout frame rate is 9.4 fps, which is used in the experiments below, the frame rate for equivalent high-frame-rate image sensors becomes 225.6 fps. Higher frame rates lead to higher heat dissipation leading to shorter battery-operating time and difficulty in miniaturization of the camera. Furthermore, higher computation power is required to process more images. Noise also is of significant concern. Because the readout noise is added to every frame, the total read noise becomes roughly three times higher than that of the multitap image sensor. The 8-tap sensor’s capability is particularly valuable in scenarios that do not require high frame rates, such as measuring the optical properties of tissue, where they are affected by respiratory cycles or shaking due to pain. In such scenarios, the 8-tap sensor can provide sufficient frame rates while enabling motion artifacts mitigation.

Figure 3(b) shows an example of the captured raw images. The numbers displayed on each image indicate the corresponding tap index. By subtracting the ambient light image from the other images, the ambient light bias is eliminated. The exposure time $T_{0}$ is set to 3 ms for each pattern due to the minimum projection time limitation of the DMD for 6-bit grayscale patterns. The exposure cycle was repeated three times to maintain the brightness of the captured images. The total exposure time for each tap is 9 ms. The 8-tap SFDI system operates at a frame rate of 9.4 fps, with each frame set comprising eight images. The frame interval, as shown in Fig. 3(a), includes both the accumulation period and the subsequent readout period.

2.3.

SFDI

As described in Ref. 3, the projected AC pattern, denoted as $S$ , is represented as

Eq. (1)

S = \frac{S_{0}}{2} [1 + M_{0} \cos (2 π f_{x} x + ϕ)],

where

S_{0}

,

M_{0}

,

f_{x}

, and

ϕ

represent the intensity of the illumination source, modulation depth, spatial frequency, and spatial phase, respectively.

The reflected intensity, denoted as $I$ , consists of a DC component $I_{DC}$ and an AC component $I_{AC}$ , which is expressed as

Eq. (2)

I (x, f_{x}) = I_{DC} (x) + I_{AC} (x, f_{x}) = α_{DC} M_{DC} (x) + α_{AC} M_{AC} (x, f_{x}) \cdot \cos (2 π f_{x} x + ϕ),

where,

M_{DC} (x)

represents the DC reflectance at a position

x

, and

M_{AC} (x, f_{x})

represents the AC reflectance of spatial frequency

f_{x}

at the position

x

.

α_{DC}

and

α_{AC}

are coefficients to convert the reflectance to the pixel value, which are removed in the calibration stage. For the acquisition of reflectance, we utilize the advanced SFDI acquisition method described in Ref. 23. This method sequentially projects one DC pattern and one AC pattern with a phase offset

ϕ

of 0 onto the sample. The AC pattern with the phase offset

ϕ

of π/2 radians is calculated by using the Hilbert transform. The DC and AC reflectance,

M_{DC} (x)

and

M_{AC} (x, f_{x})

, are then calculated at the position

x

as follows:

Eq. (3)

M_{DC} (x) = (1 / α_{DC}) I_{DC} (x),

Eq. (4)

M_{AC} (x, f_{x}) = (1 / α_{AC}) \sqrt{(I_{0} (x) - I_{DC} (x))^{2} + (I_{\frac{π}{2}} (x) - I_{DC} (x))^{2}},

where

I_{DC} (x)

,

I_{0} (x)

, and

I_{\frac{π}{2}} (x)

are the pixel values of the DC, the AC for the phase of 0, and the AC for the phase of

π / 2

radians, respectively. Due to the

2 \times 2

-subpixel configuration of the pixel, raw images are interpolated for each tap using the MATLAB function “interp2.”

Since the measured result considers factors such as the intensity of the light source and the f-number of the lens, among others, to obtain the sample’s reflectance, the measured results are calibrated using a tissue-mimicking phantom with known optical properties as a reference measurement.

Theoretically, the DC components of the AC image should be half of that of the corresponding DC image. However, it should be noted that the measured AC image’s DC component may not necessarily equal half of the DC component of the DC image, due to the differences in sensitivity among different taps of the multitap sensor pixel. The variation in the DC components introduces errors during the process of subtracting the DC images from the AC images, as Eq. (4) suggests. These errors can lead to residual sinusoidal patterns in the resulting $M_{AC}$ image.

To address this issue, a Fourier transform is performed on each set of input images for the reference measurement. Figure 4(a) shows the intensity of frequency components in the DC and sine images. Within each set of the DC image and sine image, the intensity of the DC components (i.e., the intensity at $f = 0 {mm}^{- 1}$ ) are different. The coefficient defined by the DC component in the DC image divided by the DC component in the sine image is used to equalize the DC components. The resulting intensity of frequency components after DC compensation is shown in Fig. 4(b). Note that the AC component in the sine image remains unchanged during this processing. The calibration to convert the pixel value to the reflectance $R_{DC}$ and $R_{AC}$ is applied after this processing.

Fig. 4

The intensity of frequency components in the DC and sine images: (a) before and (b) after DC compensation.

Using the true (calibrated) reflectance $R_{DC}$ and $R_{AC}$ at each wavelength, oxy/deoxyhemoglobin concentrations, and scattering parameters $a$ and $b$ are determined by referring to a precalculated look-up table (LUT). The scattering parameters a and b are denoted as

Eq. (5)

μ_{s}^{'} (λ) = a (\frac{λ}{500 nm})^{- b},

where

μ_{s}^{'}

is the wavelength dependent reduced scattering coefficient,

a

is the scattering amplitude normalized to 500 nm, and b is the scattering power.

In this work, we assume a skin model with a semi-infinite slab geometry with homogeneously distributed optical properties and assumes the absence of melanin for simplicity. The LUT is generated using the Monte Carlo simulation based on the model. Both the concentrations of chromophores and scattering parameters are considered in the simulation model and the Monte Carlo simulation then calculates the reflectance values, $R_{DC}$ and $R_{A C}$ , at each wavelength for the combinations of the input parameters for given spatial frequencies. Table 2 shows the upper and lower bounds for the input parameters. The lower and upper bounds for the parameters are determined based on previous work²⁴^–²⁶ while ensuring that they cover the range relevant to the subjects participating in the experiments.

Table 2

Parameters for LUT generation.

Parameter	Lower bound	Upper bound
$a$ ( ${mm}^{- 1}$ )	2.1	8.4
$b$	1	3
$c_{tHb} (μ M)$	4.6	300
${StO}_{2} (%)$	20	80

3. Results

To evaluate the performance of our 8-tap image-sensor-based system, we initially performed a side-by-side comparison with a conventional SFDI system based on three-phase demodulation and a commercial image sensor (Ximea, MQ013RG-E2, 1280 × 1024 pixels, 10 bit). The Ximea camera is placed under the lens, as shown in Fig. 2(b). The working distance of the Ximea system is 26 cm. The spatial frequency of the projected AC patterns was 0.10 mm⁻¹. We compared the diffuse reflectance maps obtained using both systems for three tissue-mimicking phantoms with known optical properties ( $μ_{a}$ and $μ_{s}^{'}$ ). We then performed in vivo volar forearm experiments under both the bright (ambient light intensity of 28.9 lx) and the dark (no ambient light) conditions. The ambient light is introduced by a fluorescent lamp, which contains little near-infrared light. In the in vivo volar forearm experiments where the subject’s arm remained stationary, we compared the chromophore concentrations and scattering parameter maps obtained by both systems. Finally, under ambient light while the subject’s arm was in motion, we compared videos of the chromophore concentrations and scattering parameter maps acquired by the 8-tap image sensor-based system. Specifically, we examined two configurations: one with a single exposure and another with shorter exposures repeated three times.

3.1.

Tissue Phantom Reflectance Experiment

First, the diffuse reflectance maps of three tissue-mimicking phantoms were measured using both the 8-tap SFDI system and the Ximea SFDI system. The three phantoms used in this study are referred to as the 2-tone, gum, and skin phantoms. Only for the reference phantom, a complete table of the absorption and reduced scattering coefficients for a wavelength range of 450 to 1000 nm was available, which was measured by an integrating sphere system.²⁷ The tissue-mimicking phantoms are made from silicone polymer, Black India Ink, and titanium-dioxide powder. The gum and skin phantoms have uniform optical properties. The 2-tone phantom has two distinct halves, each having different optical properties.

The measured optical properties of these phantoms, except for the reference phantom, were obtained using time-resolved NIRS equipment (TRS-20 or 80, Hamamatsu Photonics K.K., Japan) at either 797 or 801 nm. The specific values for these optical properties are shown in Table 3.

Table 3

Measured optical properties of phantoms using TRS (except “Reference”).

Phantom (wavelength)	μa (mm−1)	μs′ (mm−1)
Reference (800 nm)	0.0215	0.5576
2-Tone-dark (797 nm)	0.0487	0.8040
2-Tone-bright (797 nm)	0.0195	0.6038
Gum (801 nm)	0.0292	1.0875
Skin (801 nm)	0.0074	1.168

The exposure time for the Ximea system was 20 ms, whereas that for the 8-tap system was 9 ms (3 ms exposure repeated three times or 9 ms single exposure). To suppress the random readout noise and dark current shot noise, both systems averaged 10 sets of raw images during acquisition before performing image processing. The measured reflectance maps of the phantoms are compared in Figs. 5(a) and 5(b). The top two graphs in Fig. 5(b) show the mean reflectance for the DC component $R_{DC}$ , and the AC component $R_{AC}$ , within the ROI in Fig. 5(a). The size of the ROI is 200 × 200 pixels ( $\sim 23 \times 23 {mm}^{2}$ ) for the 2-tone phantoms and 300 × 300 pixels ( $\sim 34 \times 34 {mm}^{2}$ ) for other phantoms. Error bars represent the standard deviation. The bottom two graphs in Fig. 5(b) show the absolute relative error of the measurement results by the 8-tap system compared with those by the Ximea SFDI system. The measurements of the 8-tap system agreed with the Ximea system within ±1% for $R_{AC}$ and ±6% for $R_{DC}$ . The measurements for the 8-tap system showed a slightly larger standard deviation, which can be caused by the higher noise of the 8-tap image sensor compared with the Ximea system. This is a limitation of our current prototype image sensor.

Fig. 5

(a) Measured reflectance images of the phantoms, (b) the average reflectance’s DC and AC components within the ROI, and (c) the absolute relative error between the results of the two systems.

To validate the motion artifact suppression capability of our 8-tap system, we conducted experiments using a 2-tone phantom while it was under the influence of motion via a mechanical slider to characterize sensitivity to motion artifacts. The slider was set to move at speeds of $\sim 16.5$ and 8.25 cm/s. We are trying to mitigate motion associated with pain and tremor, which can be part of what the burn patient experiences, particularly while trying to remain motionless during imaging. While we have not been able to identify literature that quantifies that motion, we have been able to identify literature related to Parkinson’s disease tremor and have used this as a surrogate target. We surveyed articles on Parkinson’s disease tremor and found that most Parkinson’s disease tremors have a typical frequency of 4 to 6 Hz.²⁸^,²⁹ Assuming the tremor motion is sinusoidally oscillating with a peak-to-peak swing of 1 cm based on the supplementary online video in Ref. 29, the maximum speed of Parkinson’s disease tremor is calculated to be 12.6 to 18.8 cm/s. In addition, we require the technology to be useful across species, including preclinical experiments. Typically, these investigations involve animal subjects under anesthesia. However, motion related to respiration can confound data analysis. For rodents, respiration rates may be as high as 240 breaths/min.³⁰ Thus, the results that we have demonstrated here are relevant to investigations using preclinical animal models. Additional application areas may include studies of cerebral hemodynamics in preclinical models.³¹ The slider moved in parallel to the direction of the wave vector of the projected AC patterns, moving from right to left on the image. We aimed to evaluate the system’s ability to suppress the effects of pain-induced movement without physically restraining the subjects. The experiments aimed to compare the effects of long single exposure (9 ms) performed in the conventional SFDI with three times repeated short exposure (3 ms × 3) under two different lighting conditions: bright (ambient light intensity of 28.9 lx) and dark (no ambient light) conditions. As an example, Fig. 6 displays the measured $R_{AC}$ (660 nm) images. The results obtained using the 8-tap system, with the exposure repeated three times, demonstrated a noticeable reduction or absence of motion artifacts. In contrast, the single exposure results exhibited more pronounced motion artifacts, especially for the faster motion, resulting in residual sinusoidal patterns in the images. The $R_{AC}$ images at the other two wavelengths also showed similar behaviors. This experiment confirmed the superior performance of our 8-tap system in terms of its resistance to motion artifacts, thereby providing more reliable and accurate measurements in dynamic situations.

Fig. 6

Measured AC components (660 nm) of the reflectance of the 2-tone phantom (bright: with ambient light of 28.9 lx, dark: no ambient light).

3.2.

In Vivo Volar Forearm Experiment

Inin vivo volar forearm experiments, three healthy individuals (Asian males in their 20s) were recruited to participate. Figure 7 shows the $a, b, c_{tHb}$ , and ${StO}_{2}$ maps obtained from the volar forearm of the subjects in dark condition (no ambient light) using both the Ximea SFDI system and our 8-tap SFDI system. The lighting, image acquisition, and image averaging conditions were the same as those in Sec. 3.1. The $a, b, c_{tHb}$ , and ${StO}_{2}$ maps were generated using the LUT method described in Sec. 2.3. Figure 8 shows the average values of $a, b, c_{tHb}$ , and ${StO}_{2}$ within the ROI of the two systems in the dark condition as well as the 8-tap system in the bright conditions. The error bars represent the standard deviation. The size of the ROI is 80 × 80 pixels ( $\sim 9 \times 9 {mm}^{2}$ ). Table 4 displays the relative errors of the 8-tap system in the dark and bright conditions, compared with the Ximea system in dark conditions. In the dark conditions, it shows that the $a$ and $b$ values in the ROI were relatively close for the two systems. However, the $c_{tHb}$ and ${StO}_{2}$ values exhibited a larger difference between the two systems. When comparing the 8-tap system in both the dark and bright conditions, the results indicate that the values of $a, b$ , $c_{tHb}$ , and ${StO}_{2}$ values in the ROI showed good agreement. These results confirmed that the 8-tap SFDI system successfully eliminated the bias caused by the ambient light. The standard deviation of the parameter a and b are relatively large. This might be because a and b related to the exponential function are susceptive to noise.

Fig. 7

Obtained $a, b, c_{tHb}$ , and ${StO}_{2}$ maps of subject #1’s volar forearm: (a) Ximea dark, (b) 8-tap dark, and (c) 8-tap bright.

Fig. 8

The average $a, b, c_{tHb}$ , and ${StO}_{2}$ values in the ROI of the three subjects.

Table 4

Relative errors (%) of the 8-tap system in two conditions based on Ximea-dark.

System	Subject #	a	b	ctHb	StO2
8-Tap-dark	1	8.41	7.06	25.78	−15.57
	2	8.61	11.51	17.38	−12.69
	3	8.85	10.45	14.45	−13.02
8-Tap-bright	1	15.65	5.56	44.31	−8.08
	2	6.62	4.44	7.22	−15.68
	3	8.33	5.34	3.14	−16.11

Subsequently, in vivo volar forearm experiments during motion were performed with subject #1 and subject #2. In this work, only results of subject #1 are shown because the results of subject #2 exhibited the same trend as those of subject #1. The subject’s volar forearm was measured while the arm was placed on a mechanical slider, which was manually translated by the subject. The subjects moved the slider, and the total time was measured with a stopwatch. The measured speed of the slider represents the average speed. The subjects underwent several training sessions to ensure that the slider moved at a consistent speed during the measurements. The slider moved at a speed of $\sim 16.5 cm / s$ . The moving direction of the slider is the same as that in Sec. 3.1. The experiments were performed under two conditions: one with a single exposure and the other with exposures repeated three times, both in the bright condition (ambient light intensity of 28.9 lx). Videos 1 and 2 compare the estimated $c_{tHb}$ and ${StO}_{2}$ maps of the subject’s volar forearm during motion for the long single exposure and repeated exposure, respectively. The results shown in Fig. 9 are single frames extracted from the corresponding videos. The results indicate that the proposed method exhibits a reduction in motion artifacts compared with one with a single long exposure (conventional SFDI), especially in the region highlighted by the red circle. This demonstrates that our prototype 8-tap image sensor-based SFDI system is suitable for measuring subjects during motion under ambient light.

Fig. 9

Extracted frames from Videos 1 and 2. Obtained $c_{tHb}$ and ${StO}_{2}$ maps of subject #1’s volar forearm in bright condition: without repeated exposure (Video 1) and with exposures repeated three times (Video 2).

4. Discussion

In the tissue phantom reflectance experiment results shown in Fig. 5(b), it can be observed that the error of $R_{DC}$ between the two systems is relatively high for the skin phantom. Similarly, in the in vivo volar forearm experiment results shown in Fig. 8, the average $a, b, c_{tHb}$ , and ${StO}_{2}$ values for the three subjects also exhibit substantial discrepancies between the two systems. These disparities are potentially due to the skin and the skin phantom’s high scattering characteristics when compared with the reference phantom, as indicated in Table 3. Additionally, it should be noted that the capture angle for the 8-tap system is measured at $\sim 22.68 \deg$ due to the bulky size of the FPGA board in the 8-tap system, whereas the capture angle for the Ximea system is $\sim 0 \deg$ . The difference in capture angle may also contribute to the observed differences between the two systems.

It should be noted that the proposed method has not been verified with layered heterogeneous phantoms with known optical parameters. The experimental results in Fig. 6 only show that motion artifacts are suppressed for a single-layered heterogeneous (side-by-side) phantom. Although Fig. 9 implies that the proposed method could be also effective for layered heterogeneous tissues, it is important to verify the efficacy of the proposed method with layered heterogeneous phantoms mimicking skin with underlying blood vessels in future work.

Despite its advantages in motion resistance and ambient light bias suppression (ambient light intensity of 28.9 lx), our prototype 8-tap image sensor-based SFDI system exhibits a degradation in signal-to-noise ratio (SNR) compared with the conventional Ximea system. This degradation can be attributed to the design and operation of the sensor, as described in Ref. 22. The sensor in our system is specifically designed to detect short pulse light within consecutive time windows of 500 ns. This allows charges generated by ambient light and dark current to be effectively drained between time windows, leading to improved shot noise performance, and hence, the SNR in the system. However, in the case of our current 8-tap SFDI system, we used a 9 ms continuous-wave LED light source and noticeably short drain time, which does not adequately drain the charges generated by ambient light and dark current, resulting in a decrease in SNR. Acknowledging this problem, we limited our SFDI experiments to low ambient light conditions due to long LED emission time in this study. However, we suggest that our 8-tap SFDI system can overcome this limitation by employing a low duty ratio short pulse light source, which emits high-power light in a narrow pulse width (e.g., pulse with: a few $μ s$ ) periodically, whereas it turns off after each illumination and remains off for most of the time, as demonstrated in Ref. 22. In future work, we will focus on investigating the implementation of low duty ratio short pulse light sources and extending the drain time to meet with the sensor’s specification. By incorporating these improvements, we expect to increase the SNR of the system and achieve higher quality measurements under more realistic ambient conditions. By addressing the SNR limitation, our prototype 8-tap image sensor-based SFDI system will become even more robust and reliable, further expanding its potential applications in various practical and versatile scenarios.

5. Conclusion

In this work, we have demonstrated a prototype three-wavelength SFDI system that is capable of accurately measuring optical properties in a ROI that is subject to motion under ambient light. This is done using a unique 8-tap CIS developed in our laboratory. The Hilbert transform enabled faster and more efficient data acquisition in the SFDI system by reducing the required number of projected patterns per wavelength from three to two per spatial frequency. The system was validated by the reflectance results in phantom experiments and the tissue optical properties results in the in vivo volar forearm experiments. The phantom experiment during motion also confirms the system’s ability to suppress motion artifacts. The 8-tap system’s results in bright and dark conditions were compared to a reference system based on a commercial sensor and three-phase SFDI in the dark condition. The results showed good agreement in most phantom experiments, but differences were higher for skin phantoms and volar forearm measurements with high scattering coefficients. Scattering parameters a and b showed larger standard variation than other tissue parameters probably because they are susceptive to the image sensor noise. In vivo volar forearm experiments during motion were performed on subject #1 and #2. Our system effectively measured moving subjects (16.5 cm/s) under ambient light (28.9 lx) without significant motion artifacts. Overall, this work demonstrates the feasibility and potential of our motion-resistant and ambient light-resistant SFDI system, which can provide enhanced imaging capabilities for accurate measurements in real-world clinical settings.

6. Appendix: Supplementary Information

Video 1. Obtained $c_{tHb}$ and ${StO}_{2}$ maps video of subject #1’s volar forearm in bright condition without repeated exposure. The frame rate of the videos is about ×0.3 of the actual one (MP4, 380 KB [URL: https://doi.org/10.1117/1.JBO.29.1.016006.s1]).

Video 2. Obtained $c_{tHb}$ and ${StO}_{2}$ maps video of subject #1’s volar forearm in bright condition with exposures repeated three times. The frame rate of the videos is about ×0.3 of the actual one (MP4, 518 KB [URL: https://doi.org/10.1117/1.JBO.29.1.016006.s2]).

Ethics Approval

The study protocol and experimental procedures were approved by the ethics committee of Shizuoka University (registration number: 22-2). Written informed consent was obtained from all participants prior to their involvement in the study. The study was conducted in accordance with the ethical guidelines and principles outlined by Shizuoka University.

Disclosures

The authors declare that there are no conflicts of interest related to this article.

Code and Data Availability

Data and code developed in this study are available upon reasonable request to the corresponding author.

Acknowledgments

The work reported in this paper is an extension of SPIE proceedings.³² This work was supported in part by the JSPS KAKENHI (Grant Nos. 18H05240 and 21H04557). Drs. Durkin and Kennedy thankfully recognize support from the National Institutes of Health (NIH), including the National Institute of General Medical Sciences (NIGMS) (Grant No. R01GM108634). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIGMS or NIH.

References

1.

J. E. Sinex, “Pulse oximetry: principles and limitations,” Am. J. Emerg. Med., 17 (1), 59 –66 https://doi.org/10.1016/S0735-6757(99)90019-0 (1999). Google Scholar

2.

M. Ferrari, L. Mottola and V. Quaresima, “Principles, techniques, and limitations of near infrared spectroscopy,” Can. J. Appl. Physiol., 29 (4), 463 –487 https://doi.org/10.1139/h04-031 (2004). Google Scholar

3.

D. J. Cuccia et al., “Modulated imaging: quantitative analysis and tomography of turbid media in the spatial-frequency domain,” Opt. Lett., 30 (11), 1354 https://doi.org/10.1364/OL.30.001354 OPLEDP 0146-9592 (2005). Google Scholar

4.

D. J. Cuccia et al., “Quantitation and mapping of tissue optical properties using modulated imaging,” J. Biomed. Opt., 14 (2), 024012 https://doi.org/10.1117/1.3088140 JBOPFO 1083-3668 (2009). Google Scholar

5.

S. Lee et al., “SFDI biomarkers provide a quantitative ulcer risk metric and can be used to predict diabetic foot ulcer onset,” J. Diabetes Complications, 34 (9), 107624 https://doi.org/10.1016/j.jdiacomp.2020.107624 (2020). Google Scholar

6.

G. A. Murphy et al., “Quantifying dermal microcirculatory changes of neuropathic and neuroischemic diabetic foot ulcers using spatial frequency domain imaging: a shade of things to come?,” BMJ Open Diabetes Res. Care, 8 (2), e001815 https://doi.org/10.1136/bmjdrc-2020-001815 (2020). Google Scholar

7.

S. Jett et al., “Stratification of microvascular disease severity in the foot using spatial frequency domain imaging,” J. Diabetes Sci. Technol., 17 (1), 25 –34 https://doi.org/10.1177/19322968211024666 (2023). Google Scholar

8.

S. Gioux et al., “First-in-human pilot study of a spatial frequency domain oxygenation imaging system,” J. Biomed. Opt., 16 (8), 1 https://doi.org/10.1117/1.3614566 JBOPFO 1083-3668 (2011). Google Scholar

9.

D. J. Rohrbach et al., “Preoperative mapping of nonmelanoma skin cancer using spatial frequency domain and ultrasound imaging,” Acad. Radiol., 21 (2), 263 –270 https://doi.org/10.1016/j.acra.2013.11.013 (2014). Google Scholar

10.

A. Yafi et al., “Quantitative skin assessment using spatial frequency domain imaging (SFDI) in patients with or at high risk for pressure ulcers,” Lasers Surg. Med., 49 (9), 827 –834 https://doi.org/10.1002/lsm.22692 LSMEDI 0196-8092 (2017). Google Scholar

11.

A. Mazhar et al., “Noncontact imaging of burn depth and extent in a porcine model using spatial frequency domain imaging,” J. Biomed. Opt., 19 (8), 086019 https://doi.org/10.1117/1.JBO.19.8.086019 JBOPFO 1083-3668 (2014). Google Scholar

12.

A. Ponticorvo et al., “Spatial frequency domain imaging (SFDI) of clinical burns: a case report,” Burns Open, 4 (2), 67 –71 https://doi.org/10.1016/j.burnso.2020.02.004 (2020). Google Scholar

13.

A. M. Munster, M. Smith-Meek and P. Sharkey, “The effect of early surgical intervention on mortality and cost-effectiveness in burn care, 1978-91,” Burns, 20 (1), 61 –64 https://doi.org/10.1016/0305-4179(94)90109-0 BURND8 0305-4179 (1994). Google Scholar

14.

K. Kagawa, “[Invite Paper] Functional imaging with multi-tap CMOS pixels,” ITE Trans. Media Technol. Appl., 9 (2), 114 –121 https://doi.org/10.3169/mta.9.114 (2021). Google Scholar

15.

R. Schwarte et al., “New electro-optical mixing and correlating sensor: facilities and applications of the photonic mixer device (PMD),” Proc. SPIE, 3100 245 –253 https://doi.org/10.1117/12.287751 PSISDG 0277-786X (1997). Google Scholar

16.

M.-W. Seo et al., “A time-resolved four-tap lock-in pixel CMOS image sensor for real-time fluorescence lifetime imaging microscopy,” IEEE J. Solid-State Circuits, 53 (8), 2319 –2330 https://doi.org/10.1109/JSSC.2018.2827918 IJSCBC 0018-9200 (2018). Google Scholar

17.

S. Kawahito et al., “CMOS lock-in pixel image sensors with lateral electric field control for time-resolved imaging,” in Int. Image Sens. Workshop, (2013). Google Scholar

18.

M.-S. Keel et al., “7.1 A 4-tap 3.5 μm 1.2 Mpixel indirect time-of-flight CMOS image sensor with peak current mitigation and multi-user interference cancellation,” in IEEE Int. Solid-State Circuits Conf. (ISSCC), 106 –108 (2021). https://doi.org/10.1109/ISSCC42613.2021.9365854 Google Scholar

19.

K. Hatakeyama et al., “A hybrid indirect ToF image sensor for long-range 3D depth measurement under high ambient light conditions,” in IEEE Symp. VLSI Technol. and Circuits (VLSI Technol. and Circuits), 46 –47 (2022). https://doi.org/10.1109/VLSITechnologyandCir46769.2022.9830139 Google Scholar

20.

C.-C. Kuo and R. Kuroda, “A 4-tap CMOS time-of-flight image sensor with in-pixel analog memory array achieving 10kfps high-speed range imaging and depth precision enhancement,” in IEEE Symp. on VLSI Technol. and Circuits (VLSI Technol. and Circuits), 48 –49 (2022). https://doi.org/10.1109/VLSITechnologyandCir46769.2022.9830420 Google Scholar

21.

Y. Shirakawa et al., “An 8-tap CMOS lock-in pixel image sensor for short-pulse time-of-flight measurements,” Sensors, 20 (4), 1040 https://doi.org/10.3390/s20041040 SNSRES 0746-9462 (2020). Google Scholar

22.

C. Cao et al., “A dual NIR-band lock-in pixel CMOS image sensor with device optimizations for remote physiological monitoring,” IEEE Trans. Electron. Devices, 68 (4), 1688 –1693 https://doi.org/10.1109/TED.2021.3057035 IETDAI 0018-9383 (2021). Google Scholar

23.

K. P. Nadeau, A. J. Durkin and B. J. Tromberg, “Advanced demodulation technique for the extraction of tissue optical properties and structural orientation contrast in the spatial frequency domain,” J. Biomed. Opt., 19 (5), 056013 https://doi.org/10.1117/1.JBO.19.5.056013 JBOPFO 1083-3668 (2014). Google Scholar

24.

I. Nishidate et al., “Noninvasive imaging of human skin hemodynamics using a digital red-green-blue camera,” J. Biomed. Opt., 16 (8), 086012 https://doi.org/10.1117/1.3613929 JBOPFO 1083-3668 (2011). Google Scholar

25.

Y. Shimojo et al., “Measurement of absorption and reduced scattering coefficients in Asian human epidermis, dermis, and subcutaneous fat tissues in the 400- to 1100-nm wavelength range for optical penetration depth and energy deposition analysis,” J. Biomed. Opt., 25 (4), 045002 https://doi.org/10.1117/1.JBO.25.4.045002 JBOPFO 1083-3668 (2020). Google Scholar

26.

S. L. Jacques, “Optical properties of biological tissues: a review,” Phys. Med. Biol., 58 (11), R37 –R61 https://doi.org/10.1088/0031-9155/58/11/R37 PHMBA7 0031-9155 (2013). Google Scholar

27.

S. A. Prahl, M. J. C. van Gemert and A. J. Welch, “Determining the optical properties of turbid media by using the adding–doubling method,” Appl. Opt., 32 (4), 559 https://doi.org/10.1364/AO.32.000559 APOPAI 0003-6935 (1993). Google Scholar

28.

G. Deuschl, P. Bain and M. Brin, “Consensus statement of the movement disorder society on tremor,” Movement Disord., 13 (S3), 2 –23 https://doi.org/10.1002/mds.870131303 MOVDEA 0885-3185 (2008). Google Scholar

29.

H. Zach et al., “The clinical evaluation of Parkinson’s tremor,” J. Parkinsons Dis., 5 (3), 471 –474 https://doi.org/10.3233/JPD-150650 (2015). Google Scholar

30.

Y. Guan et al., “A noninvasive method for monitoring respiratory rate of rats based on a microcantilever resonant humidity sensor,” J. Micromech. Microeng., 29 (12), 125001 https://doi.org/10.1088/1361-6439/ab43e8 JMMIEZ 0960-1317 (2019). Google Scholar

31.

C. Crouzet et al., “Cerebral blood flow is decoupled from blood pressure and linked to EEG bursting after resuscitation from cardiac arrest,” Biomed. Opt. Express, 7 (11), 4660 https://doi.org/10.1364/BOE.7.004660 BOEICL 2156-7085 (2016). Google Scholar

32.

Y. Feng et al., “Demonstration of motion-resistant three-wavelength spatial frequency domain imaging system with ambient light suppression using an 8-tap CMOS image sensor,” Proc. SPIE, 12376 123760D https://doi.org/10.1117/12.2649418 PSISDG 0277-786X (2023). Google Scholar

Biographies of the authors are not available.

CC BY: © The Authors. Published by SPIE under a Creative Commons Attribution 4.0 International License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.

Citation Download Citation

Yu Feng, Chen Cao, Yuto Shimada, Keita Yasutomi, Shoji Kawahito, Gordon T. Kennedy, Anthony J. Durkin, and Keiichiro Kagawa "Motion-resistant three-wavelength spatial frequency domain imaging system with ambient light suppression using an 8-tap CMOS image sensor," Journal of Biomedical Optics 29(1), 016006 (18 January 2024). https://doi.org/10.1117/1.JBO.29.1.016006

Received: 1 August 2023; Accepted: 20 December 2023; Published: 18 January 2024

Access the abstract

JOURNAL ARTICLE
14 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

KEYWORDS

Image sensors

Imaging systems

Reflectivity

Tissues

Biomedical optics

Video

Image processing

Significance

Aim

Approach

Results

Conclusions

1.

Introduction

2.

Materials and Methods

2.1.

Multitap CIS

Fig. 1

Table 1

2.2.

System Operation

Fig. 2

Fig. 3

2.3.

SFDI

Eq. (1)

Eq. (2)

Eq. (3)

Eq. (4)

Fig. 4

Eq. (5)

Table 2

3.

Results

3.1.

Tissue Phantom Reflectance Experiment

Table 3

Fig. 5

Fig. 6

3.2.

In Vivo Volar Forearm Experiment

Fig. 7

Fig. 8

Table 4

Fig. 9

4.

Discussion

5.

Conclusion

6.

Appendix: Supplementary Information

Ethics Approval

Disclosures

Code and Data Availability

Acknowledgments

References

Show All Keywords

Keywords/Phrases

Search In:

Publication Years