Scattering-Assisted Computational Imaging

Abstract

Imaging objects hidden behind an opaque shelter provides a crucial advantage when physically going around the obstacle is impossible or dangerous. Previous methods have demonstrated that is possible to reconstruct the image of a target hidden from view. However, these methods enable the reconstruction by using the reflected light from a wall which may not be feasible in the wild. Compared with the wall, the “plug and play” scattering medium is more naturally and artificially accessible, such as smog and fogs. Here, we introduce a scattering-assisted technique that requires only a remarkably small block of single-shot speckle to perform transmission imaging around in-line-of-sight barriers. With the help of extra inserted scattering layers and a deep learning algorithm, the target hidden from view can be stably recovered while the directly uncovered view is reduced to 0.097% of the whole field of view, successfully removing the influence of large foreground occlusions. This scattering-assisted computational imaging has wide potential applications in real-life scenarios, such as covert imaging, resuming missions, and detecting hidden adversaries in real-time.

1. Introduction

How to detect objects hidden from direct view is of fundamental importance to many fields of research [1,2,3]. When an opaque barrier occurs, the region-of-interest tends to be kept out of in-line sight, which brings difficulties for direct measurement.

Over the years, numerous methods have been put forward to address imaging around barriers. For active optical methods, non-line-of-sight (NLOS) techniques [4,5] enter the mainstream. Most of them depend on transient imaging, requiring not only a short light pulse to create transient illumination but also the time of flight (ToF) [6] technique to enable raster-scanning. Herein, transient-imaging-based NLOS methods are not favorable for low-cost real-time acquisition [7]. Despite the work [8] that has clarified that NLOS imaging can be realized using passive illumination and entire single-shot speckle rather than transient-imaging techniques and the work [9] introduced the digital holography technique in NLOS imaging to avoid the scanning process, both of them still adopt the reflective detection structure that requires a wall to reflect light, which may not be feasible in the wild. In addition, the entire recorded speckle requires a large field of view (FoV). Moreover, various optical-based image inpainting methods [10,11] have been put forward to remove the occlusion effect in images. However, both of them adopt highly time-consuming algorithms and are not suitable for the large occlusion effect. Therefore, the existing methods are inapplicable to imaging using a small field of vision, limiting their applicability in certain situations, for instance, covert imaging. Hence, how to use the limited view and simple setup to acquire the object hidden behind a large occlusion is a problem needed to be addressed.

To address the high cost and the impracticality of existing methods outside laboratory conditions, we developed a scattering-assisted computational imaging technique to image objects hidden behind the in-line-of-sight opaque barriers using a remarkably small block of single-shot speckle. Although the scattering medium degrades the image quality and usually is regarded as a negative factor in imaging systems [12], there are many types of research that demonstrate that scattering medium can be beneficial such as acting as encoders for image encryption [13,14], breaking diffraction limit [15,16] and realizing lensless imaging [17]. In this work, we leverage the scattering medium to broaden the view. With the assistance of scattering medium, the proposed scattering-assisted method only requires a digital camera and a simple transmission detection setup to realize the obstacle-avoidance NLOS imaging. Compared with the wall that the reflective NLOS imaging method needs, the scattering medium such as heavy smoke and fogs that the transmission detection setup requires is more accessible in the wild.

In this paper, we demonstrate that a deep-learning-based scattering-assisted computational method performs imaging around large foreground occlusions while only a remarkably small speckle block rather than an entire one is required, which significantly broadens the valid view. Even when the illumination condition is photon-limited, the proposed still can realize imaging the hidden-of-view object using as little as 0.47 photons per pixel, further improving the practicality.

2. Methods

2.1. Proposed Method

The main ideas underlying the proposed method are given as follows. In the NLOS scenarios shown in Figure 1a, the direct view of the object decreases due to the in-line opaque barriers. Generally speaking, for a incident wave field $E\left({\mathit{r}}_i\right)$ at the incident plane ${\mathit{r}}_i$ in such an NLOS imaging system, the corresponding wave filed $E\left({\mathit{r}}_o\right)$ at the output location can be described as Equation (1):

$$E\left({\mathit{r}}_o\right)=\int G({\mathit{r}}_o,{\mathit{r}}_i)T\left({\mathit{r}}_o\right)E\left({\mathit{r}}_i\right)d{\mathit{r}}_i$$

(1)

where $G({\mathit{r}}_o,{\mathit{r}}_i)$ is Green’s function [18] describing the transmission scattering mapping relation between incident location ${\mathit{r}}_i$ and detection location ${\mathit{r}}_o$. $T\left({\mathit{r}}_o\right)$ represents the transmission matrix describing the occlusion effect caused by in-line-of-sight opaque barriers.

The recorded light intensity at output plane can de calculated using the output wave filed $E\left({\mathit{r}}_o\right)$ by:

$$\begin{array}{cc}\hfill I\left({\mathit{r}}_o\right)& =|E\left({\mathit{r}}_o\right){|}^2\hfill \\ \hfill & =|\int G({\mathit{r}}_o,{\mathit{r}}_i)T\left({\mathit{r}}_o\right)E\left({\mathit{r}}_i\right)d{\mathit{r}}_i{|}^2\hfill \end{array}$$

(2)

Obviously, as seen from the Equation (2), the light intensity $I\left({\mathit{r}}_o\right)$ at position $r_o$ on the image plane is integral of a certain transformation of the entire illuminated object.

Owing to the scattering effect, the propagation direction of incident light changes randomly. Therefore, part of the incident light that should travel in a straight line and be occluded by in-line-of-sight opaque barriers still can be measured. That is, with the help of a scattering medium, the information of hidden objects that should be occluded and lost will be encoded in each pixel of the recorded speckle. According to the information transferred in scattering media which has been studied in theory [19,20], it is possible to recover the target from detected speckles even though speckles tend to have random distribution in spatial because of the light scattering. Moreover, because each part of the speckle involves information about the whole object plane, there is a potential for using an extremely small speckle block to reconstruct the object image.

Because the above-mentioned reconstruction process that recovers object images from detected degraded speckles can be regarded as an inverse problem, the problem that retrieves the target from a partial-covered speckle can be described as a well-known Tikhonov–Wiener optimization problem as follows:

$$\widehat{I}=argmin\Vert \left(I\left({\mathit{r}}_o\right)\right)-H\left(I\right){\Vert }^2+\alpha \varphi \left(I\left({\mathit{r}}_o\right)\right)$$

(3)

where ${\Vert ·\Vert }^2$ denotes the $L^2$ norm. In general, the first term expresses fitness, such as matching in the least-squares sense the measurement to the forward model for the original object. H expresses the transform operator characterizing the mapping relation between the captured pattern $I\left({\mathit{r}}_o\right)$ and ground truth I, and $I\left({\mathit{r}}_o\right))=H\left(I\right)$. $\varphi$ is the regularizer function and tends to leverage prior knowledge to avoid the non-uniqueness of the ill-posed problem. $\alpha$ is the regularization parameter. The optimization problem described in Equation (3) can be solved by pseudo-inverse solution, or its Moore–Penrose improvement [21] when setting $\alpha$ as 0. The goal of this optimization problem is to minimize the difference between the estimation $\widehat{I}$ and the original object I.

When the forward operator in the above-mentioned reconstruction process is ill-posed or uncertain, the deep-learning-based approach has been proven to be especially attractive [22,23,24,25,26,27] while the common numerical method is out of work. In this work, inverse problems that recover object images from detected degraded speckles still can be solved using a deep learning network shown as in Figure 1b. As a data-driven algorithm, the deep learning model learns the forward operator H and regularizer function $\varphi$ simultaneously in a non-explicit way.

The deep-learning-based reconstruction model conforms an encoder–decoder frame and the detailed network structure is given in Figure 2a. The left part acts as an encoder, which consists of four dense blocks connected by max-pooling layers for downsampling. As seen from Figure 2b, each dense block contains multiple layers, in which each layer consists of batch normalization (BN), the rectified linear unit (ReLU) non-linear activation function, and the convolution (conv) operator. The high-level feature is extracted layer by layer from the input speckle with the encoder process, which takes advantage of the down-sampling process to enhance robustness, avoid overfitting, and reduce computation load. The right part plays a role in the decoder by upsampling operation. During the upsampling process, the feature map of the encoder path is copied and cropped to the corresponding symmetrical decoder path to preserve high-frequency information. After the decoder path, an additional convolutional layer and a fully connected layer followed by the last layer produce the output. It should be noted that the number of downsampling layers and upsampling layers changes according to the input size. Specifically, the number of downsampling layers and upsampling layers should be decreased to guarantee that the $\frac{N}{16}$ is an integer when $N<16$.

Our reconstruction model is designed to recover sparse objects. In order to train the model, the loss function is designed to measure differences between ground truths and predicted outputs. Here, the Negative Pearson correlation coefficient ($NPCC$) [28] and the mean square error ($MSE$) are combined as the loss function.

The detailed loss functions are defined as follows:

$$NPCC=-\frac{cov(I,\widehat{I})}{{\sigma }_I{\sigma }_{\widehat{I}}}=-\frac{{\sum }_{i=1}^{N_x}{\sum }_{j=1}^{N_y}(I(i,j)-\overline{I})(\widehat{I}(i,j)-\widehat{\overline{I}})}{\sqrt{{\sum }_{i=1}^{N_x}{\sum }_{j=1}^{N_y}{(I(i,j)-\overline{I})}^2}\sqrt{{\sum }_{i=1}^{N_x}{\sum }_{j=1}^{N_y}{(\widehat{I}(i,j)-\widehat{\overline{I}})}^2}}$$

(4)

$$MSE=\frac{1}{N_x\times N_y}\sum _{i=1}^{N_x}\sum _{j=1}^{N_y}{(I(i,j)-\widehat{I}(i,j))}^2$$

(5)

$$Loss=NPCC+\beta MSE$$

(6)

where $\overline{I}$, $\overline{\widehat{I}}$, $N_x$, $N_y$, and $\beta$ represent the mean value of the original input I, the mean value of the predicted output $\widehat{I}$, the size of inputs, and weight coefficient, respectively. In this work, $\beta$ is set as 0.001.

To train the network, 4500 speckle blocks of hidden objects are collected as training data. In addition, 300 speckle blocks are set as validation data and the remaining 200 speckle blocks are used as test data to test the trained model. Both test data and validation data are not included in the training data. During the training process, the initial learning rate is ${10}^{-3}$ and the learning rate multiplies by 0.9 for every 10 epochs. The number of training epochs is 800 and the batch size is set as 50. The adopts adaptive moment estimation (Adam) is used to minimize the loss function to realize model convergence.

The training process of the scattering-assisted network is performed on a 7th generation i7 server with two graphics processing units (NVIDIA TITAN Xp) using TensorFlow with Python 3.6. In this work, the average reconstruction time of a hidden object is about 21 ms.

2.2. Experimental Setup

To demonstrate the ability of imaging around opaque barriers, a typical NLOS imaging system is designed as Figure 3. The direct line of sight between the DMD and camera is blocked by an opaque barrier. In addition, the area of scattering layers is larger than that of occlusion to ensure partial scattering light can be captured. In the experiment, a helium-neon laser (wavelength = 632.8 nm) is used to illuminate the DMD (DLC9500P24) where the surface displays objects from two different databases: MNIST (a public handwritten digits dataset) and EnglishHnd (a public handwritten English letters dataset). The VND and expander are used to control the illumination power and laser beam size, respectively. The light around the obstacle passes through a 4f lens system and is captured by a CCD where the center block with 512 × 512 pixels is used. During the experiment, various laminated black cardboards with different shapes and sizes are set as in-line opaque barriers. In addition, a ground glass diffuser with 220 grit is chosen to be the insertable scattering layer. Both distances from the DMD to the scattering medium and the scattering medium to the barrier are ∼10 cm.

Using the above experimental setup, the speckle where partial is covered by an occlusion can be captured. In general, it is extremely difficult for human vision to recognize the target directly from a random speckle, not to mention a partial-covered of that. However, as an effective method for feature extracting, it is feasible for the deep learning method to retrieve images utilizing the invariant information coded in speckles. Herein, according to the thought in Figure 1, partial-covered speckles can be put into a deep neural reconstruction network shown as Figure 2 to retrieve hidden-from-view objects.

3. Results and Discussion

3.1. Obstacle-Avoidance Imaging

First of all, to evaluate the obstacle-avoidance imaging ability of the proposed scattering-assisted method, the founding year, ‘1896’, and name abbreviation, ‘SJTU’, of Shanghai Jiao Tong University are used as targets shown in Figure 4a. In addition, these objects displayed on DMD are occluded by the barrier with a fixed shape as the area above the red dotted line in the first line of Figure 4a. For the sake of comparison, before inserting the scattering layer, the direct measurement results are also recorded. As seen from the second line of Figure 4a, the majority view is covered so that the captured information is too little for a human to identify directly. However, with the help of an extra scattering medium, the proposed method retrieves corresponding object images successfully, which can be seen in the 4th and 5th lines in Figure 4a. In contrast, the intensity map along the solid line of the ‘8’ and ‘S’ images is plotted in Figure 4b. Although the intensity result of the larger cropped area is closer to the ground truth (GT), the small block, 32 × 32 pixels, merely accounting for 0.39% ($\frac{32\times 32}{512\times 512}$) FoV of the camera, still works compared with direct measurement. It is worth mentioning that there is a data pre-processing for captured speckles before the image reconstruction process. From blocks in the 3rd line of Figure 4a, 32 × 32 and 128 × 128 blocks from the lower right corner of speckles are cropped respectively as network inputs instead of using the entire speckle, which is 512 × 512 pixels. On one hand, the cropped operation can reduce calculation costs since most areas of speckle are covered and the corresponding information of those is lost. On the other hand, success-reconstructed results using small cropped speckle blocks verify that each part of the speckle involves complete information about the hidden target and in turn clarify the feasibility of the scattering-assisted computational imaging method. In addition, when the collusion area changes, we can crop a small speckle block from the public remained view and then use the well-trained network to restore the object image from it, which validly avoids retraining the neural network due to the change of occlusion set. It is worth mentioning that, even if the recorded speckle is completely out of the target area, the proposed scattering-assisted method still can recover the image hidden from view using a remarkably small speckle.

In addition, more complicated images like stick figures and faces are set as objects to perform scattering-assisted computational imaging. The corresponding results are shown in Figure 5, and the red box represents the remaining view due to the opaque barrier. The proposed method has great performance for eliminating the in-line occlusion effect in spatial sparse targets while failing to retrieve the detail feature in humans’ faces. In the future work, we will explore a more effective neural network architecture or a more powerful technique to improve the reconstruction performance for complicated non-sparse images.

Moreover, we further explore the feasibility when using dynamic scattering medium to assist the obstacle-avoidance imaging. We use the turbid Intralipid solution and keep stirring the solution to create the dynamic scattering environment. In the imaging system with the dynamic scattering solution and the in-line opaque barrier, the partial-occluded speckles are collected through and then cropped as 32 × 32 blocks to train the network. The corresponding reconstruction results are given in Figure 6a. As seen from the result, though the reconstruction quality needs to be improved, and the proposed scattering-assisted computational imaging method still works when using a dynamic scattering medium, which demonstrates its practicality in the practical scenario. Different shots of the same object “1” are recorded through the dynamic solution and the correlation map between speckle blocks of two different 32 × 32 speckle shots is given in Figure 6b. According to our previous works [22,23], the correlation relationship between two speckle blocks from different shots of the same object guarantees the feasibility of using a single well-trained network model to reconstruct the hidden object from speckles collected under a dynamic scattering environment. In addition, we will make our best effort to improve the reconstruction quality under a dynamic scattering condition in future work.

3.2. Comparison

For further accessing the obstacle-avoidance imaging performance using the limited view, five settings with different occlusion proportions are explored in Figure 7. When different occlusions are set, only the upper left view of a camera with 8 × 8, 16 × 16, 64 × 64, 128 × 128 and 256 × 256 pixels remain. As a contrast, results of the direct measurement and image inpainting are given. The image inpainting [29,30] algorithm is a common computer vision technique utilizing the redundancy information of an incomplete direct measurement to remove covered parts in a visually plausible way. However, these methods usually fail to guarantee the unique optimal output when there is a larger-scale occlusion. Without the scattering medium, it is hard to identify the target from direct measurement results in the first row of Figure 7a,b because of insufficient information caused by the occlusion effect. First of all, the direct measurement is used by an image inpainting method to recover the object image. In addition, only when the view block is up to 256 × 256 pixels do image inpainting results seem close to the ground truth. However, the proposed method works until the direct in-line-of-sight scattering area is as little as 8 × 8 pixels, which means ∼0.024% of the entire view and approximately 99.976% occlusion proportion of direct sight. To quantitatively evaluate the performance, the NPCC value of each iteration for the proposed method is given in Figure 7d and that of the image inpainting method is in Figure 7c for comparison. For NPCC in the training process, the closer that value is to −1, the more well-trained the network is. When captured speckle block comes to 16 × 16 pixels which respond to approximately 0.097%($\frac{16\times 16}{512\times 512}$) of the whole view, the scattering-assisted computational approach shows stable imaging performance. Meanwhile, until the speckle block comes to 256 × 256 pixels, the image inpainting network obtains great convergence, the speed of which is still slower than our method. There is no doubt that, for the transmission NLOS imaging scenarios, the proposed method is superior to a single image post-processing algorithm such as image inpainting because the scattering layer encodes the target information in the random speckle to guarantee the authenticity of reconstruction results and what image inpainting learns is just from the direct measurement where most target information is lost. As expected, the loss of the proposed method decreases concerning the visible block size until the value of NPCC nears −1. By the way, according to the set occlusion, the target is totally out of view when only the left upper blocks with 8 × 8 and 16 × 16 pixels are visible. In this case, the object is totally outside the view area where is a remarkably small speckle ratio($\frac{8\times 8}{512\times 512}$). That is, the proposed approach still works even when the target is out of sight as long as the scattered light can be captured. Therefore, with the assistance of scattering layers, the proposed method has great obstacle-avoidance imaging potential with a limited view even when the target is totally out of view.

3.3. Robustness Analysis

Figure 8a,b demonstrates the influence of occlusion positions, where three visible blocks of the same size are set as different-colored boxes show. In addition, corresponding results of digit ‘0’ are given as representative examples. The quantitative analysis is given in Figure 8c,d. For the inpainting method, when the selected visible region nears the central part, the loss of the inpainting method decreases obviously as Figure 8c shows, which is because the center tends to have richer target information than the corner. This clarifies that the position has a significant influence on the result of image inpainting. For our proposed method using speckle patterns, different occlusion positions have a little effect on reconstruction results as Figure 8d shows, implying that the method is viewing position-invariant. That is, our method not only performs well with the limited view but also is robust to the occlusion positions.

3.4. Generalization Analysis

Moreover, we further discuss the generalization performance of a single scattering-assisted network model when the position of the opaque barrier changes. We change the remaining view described as the solid box with 128 × 128 pixels and use the corresponding speckle block to train the reconstruction model, and then move the window along the direction of the arrow to crop the speckle as the test data. The movement distances are separately set as 0, 5, 10, 15, 20, 25, and 30 pixels. As seen from the result in Figure 9, the longer the movement distance is, the lower the image quality is. However, when the movement distance of the remaining view is under 10 pixels, the single well-trained model still can retrieve the high-quality image of the hidden object from the speckle block. Therefore, the proposed scattering-assisted computational imaging method is reasonably robust against the small change in the occlusion position caused by the camera jitter and other disturbances, performing great generalization performance and feasibility in practical obstacle-avoidance applications.

3.5. Photon-Limited Demonstration

Moreover, in view of the photon-limited illumination condition in the practical imaging scenarios, extending the scope of our current work to photon-limited imaging for hidden objects is also of considerable interest. In the experiment, the number of detected photons remains at a single-photon level by decreasing the incident light power. For convenience, to record the number of photons and clarify the photon-limited application, the single-photon counting camera (SPC) is used as the camera in Figure 3. In addition, the corresponding result of potential in photon-limited imaging is demonstrated in Figure 10. Meanwhile, the avoidance-obstacle imaging performance of different occlusion shapes has been explored simultaneously. The reconstruction results in Figure 10 show that the proposed scattering-assisted method can recover the image from speckles with different occlusion shapes, verifying the feasibility of imaging around the different opaque. Even more to the point, as little as 0.47 photons per pixel are required to retrieve the target, ’3’, demonstrating the powerful ability of imaging around opaque barriers in photon-limited illumination conditions. Combining with results in Figure 7, it is possible to use a part of the speckle collected at the photon-limited illumination condition to recover the image of hidden objects. Thus, the total number of photons required to recover the hidden object can be further reduced by cutting the recorded photon-limited speckle as a smaller block.

4. Conclusions

In conclusion, we present a scattering assisted computational imaging method for objects hidden behind opaque obstacles. With the assistance of a scattering medium, the deep-learning-based computational method performs imaging using remarkably small speckle blocks, significantly broadening the view and overcoming the influence of in-line-of-sight occlusions. It is demonstrated that the hidden object can be recovered from speckle blocks even when the object itself is totally out of view. Compared to the common NLOS imaging requiring the reflected wall and ToF technique, the proposed method only needs a remarkably small block of speckle captured by a ordinary digital camera and an in-line “plug and play” scattering medium such as fogs, which is more cost-effective and accessible in the wild. Furthermore, the method not only is robust and applicable to various in-line occlusions with different shapes and positions without any calibration operations, but also can be extended to the photon-limited imaging application while only a few photons can be detected, making it desirable for avoidance-obstacle imaging and covert imaging in a number of real-life scenarios, such as surveillance, rescue missions and tracking incoming hidden adversaries in real-time.