Optimization of retina-like illumination patterns in ghost imaging

Ghost imaging (GI) reconstructs images using a single-pixel or bucket detector, which has the advantages of scattering robustness, wide spectrum and beyond-visual-field imaging. However, this technique needs large amount of measurements to obtain a sharp image. There have been a lot of methods proposed to overcome this disadvantage. Retina-like patterns, as one of the compressive sensing approaches, enhance the imaging quality of region of interest (ROI) while not increase measurements. The design of the retina-like patterns determines the performance of the ROI in the reconstructed image. Unlike the conventional method to fill in ROI with random patterns, we propose to optimize retina-like patterns by filling in the ROI with the patterns containing the sparsity prior of objects. This proposed method is verified by simulations and experiments compared with conventional GI, retina-like GI and GI using patterns optimized by principal component analysis. The method using optimized retina-like patterns obtain the best imaging quality in ROI than other methods. Meanwhile, the good generalization ability of the optimized retina-like pattern is also verified. While designing the size and position of the ROI of retina-like pattern, the feature information of the target can be obtained to optimize the pattern of ROI. This proposed method paves the way for realizing high-quality GI.


Introduction
Multi-pixel detectors are usually used as light detection device in conventional optical imaging systems. However, multi-pixel detectors may become expensive or impractical at some specific wavebands, such as the infrared or deep ultraviolet [1]. Pittman et al. first use entangled photon pairs to demonstrate the feasibility of GI in 1995 [2]. GI, also known as correlated imaging or single-pixel imaging [3][4][5][6][7][8], provides object information by correlating illumination patterns of known light field distribution and a sequence of light intensity collected by a single-pixel or bucket detector. This unique imaging system makes GI have the advantages of scattering robustness, wide spectrum and beyond-visual-field imaging and have been adopted in many related fields including three-dimensional imaging [9][10][11][12], terahertz imaging [13][14][15], multispectral imaging [16][17][18] and scattering medium imaging [19][20][21].
In conventional GI, a large number of patterns are required to be illuminated on the target for high quality image while costing much time [22]. For obtaining good performance of GI, how to balance imaging quality and efficiency is still a major challenge to be overcome. Most of studies are focused on two aspects, one is the modulation patterns of illumination [23][24][25], and the other is reconstruction algorithm [26][27][28]. Nevertheless, the trade-off between efficiency and quality is still a problem in ROI imaging [29]. Inspired by the foveated vision found in the human eye, modulation patterns with retina-like structure are proposed to obtain higher imaging quality of ROI while not increasing sampling times [29]. In recent years, retina-like patterns have attracted the attention of many scholars. In 2019, Zhang et.al proposed a model of 3D GI combining with retina-like structure to improve imaging efficiency, and some retina-like properties such as invariant scaling and rotation have been realized in proposed imaging system [30]. In 2020, a foveal ghost imaging based on deep learning to realize selecting the ROI for foveal imaging intelligently was proposed by Zhai et al [31]. Selecting the ROI intelligently has been realized in foveal GI by applying generative adversarial networks based on single shot multiBox detector architecture [32]. The high PSNR of the ROI can be achieved compared with uniformresolution GI. Inspired by human consciousness controlling the eyes to acquire the ROI, Gao et al [33] proposed a new compressive GI called ROI-guided compressive sensing GI which used prior imaging information from fast Fourier single-pixel imaging to achieve a better visual effect and higher imaging quality. Our group presented parallel retina-like computational GI, which is based on a multi-pixel detector combined with retina-like patterns and has better performance than conventional parallel GI and retina-like GI (RGI) [34]. Although the advantages of RGI, to the best of our knowledge, the optimization of illumination retina-like patterns is lack of study. Actually, much attention is paid to the location and size of the ROI or the application of retina-like structure [30][31][32][33]. However, there is no study on patterns filled in the ROI of retina-like patterns. The illumination patterns applied to retina-like structure are most random binary patterns. Random patterns are one of the poor performance methods to obtain object information. In this paper, we proposed to improve the imaging quality of RGI by optimizing retina-like patterns, which is realized via filling the patterns containing the sparsity prior of objects in the ROI. The methods to obtain the sparsity prior of objects, such as imaging dictionary and principal component analysis (PCA), have been proposed in previous single-pixel imaging works [35]. We select PCA to generate patterns containing with sparsity prior of objects and then optimize the retina-like patterns to improve the imaging quality of ROI. We demonstrate the optimized retina-like patterns in GI using simulations and experiments. The imaging quality of the proposed scheme is evaluated by comparing the results of conventional RGI.

Principles
The principle of RGI using optimized patterns by PCA (PCA-RGI) is shown as Fig.  1. A sequence of optimized retina-like patterns is set up on the digital micromirror device (DMD) by computer. After a beam of light from the light source illuminated on the surface of DMD, the reflected light with modulated pattern reaches the object to be imaged. Reflected or projected light illuminated on the object is converged by a focus lens onto the single-pixel detector. The light intensity is collected by a data acquisition and then transmitted to the computer for reconstruction calculation with the corresponding patterns. The measurements principle of RGI can be described as follows: where It represents the light intensity collected by the signal-pixel detector and t is the time index; O (x, y) represents the object and (x, y) represents the 2D Cartesian coordinates in the scene; St (x, y) represents the optimized retina-like patterns. The conventional reconstruction algorithm is second-order correlation algorithm. However, compressive sensing (CS) algorithms have been proved to obtain a better performance in reconstructing images than conventional algorithm [36]. In this paper, we select the total variation (TV) regularization prior algorithm [37] for reconstructing images. Unlike the second-order correlation algorithm, the TV-based CS reconstruction algorithm transforms the imaging reconstruction into a constrained optimization problem. The optimization model of TVbased CS is shown as: where G represents the gradient calculation matrix and c is the corresponding coefficient vector; l1 represents the l1 norm; As for the PCA-based illumination patterns which is used to fill in the ROI of retina-like patterns, we generate them that can be used for DMD in three steps. The first step is to generate grayscale PCA-based patterns from training dataset. Large number of images with features are required to constitute a training dataset. The optimal patterns are then designed based on the common features extracted from the training dataset. It is assumed that dataset contains M objects and N variables. The original training dataset is represented by a matrix X of order M× N, which is shown as: 11 12 1 where each row represents one training image which is converted to a row and xmn represents the value of n th (n≤N) pixel in m th (m≤M) training image. The raw dataset is composed of many training images and the variables are generally not calculated in the same units. Therefore, the raw dataset needs to be standardized. The standardized process can be expressed as: where <xn> represents the average value of each column and Sn represents the variance of each column. Then, the covariance matrix  with a size of N × N is calculated, which is shown as: where cov represents covariance calculation; x * represents column vectors in X. The covariance matrix  is then decomposed into eigenvalues and eigenvectors, which is shown as: where Q represents the eigenvector matrix of  with size N× N and  represents the eigenvalue of  . Then the eigenvectors in Q are arranged in the descending order of corresponding eigenvalues. Each row vector of Q represents one principal component is also the illumination pattern after conversion to two-dimensional array. The second step is extracting the pixels with positive value of the PCA-based illumination patterns. The pixel intensity values of the principal components can be both positive and negative. However, in a practical optical system, only positive pixel intensity values can be applied to the projection device. In the previous work [36], both the pixels with positive and negative values are projected. Then the light intensity value corresponding to the original illumination pattern is obtained by subtraction. In our work, we extract the pixels with positive values from the PCA-based illumination patterns and the pixels with negative values are filled with zeros.
The third step is binarizing the grayscale PCA-based illumination patterns. Grayscale illumination patterns cannot be directly loaded into the DMD. There are two typical solutions to binarize the grayscale patterns. One is spatial dithering strategy and the other is temporal dithering strategy [4]. We select the temporal dithering strategy to binarize the grayscale patterns. The grayscale patterns processed by temporal dithering strategy are split into eight binary patterns. Considering the imaging efficiency, we use the lowest bit binary image to replace the original grayscale image.
In the previous work, the ROI of retina-like patterns was filled with random patterns which did not contain any features [30][31][32][33]. Unlike conventional methods, we fill in the ROI with PCA-based patterns containing with sparsity prior of objects to optimize the retina-like illumination patterns. We compare the proposed method with the GI using random patterns (Random-GI), GI using conventional retina-like patterns (Random-RGI) and GI using PCAbased patterns (PCA-GI) by simulations and experiments and demonstrate its advantages.

simulations
Simulations with two different objects were performed separately to evaluate the performance of PCA-RGI compared with Random-GI, Random-RGI and PCA-GI.
The performance of the final reconstructed images was compared quantitatively using the peak signal-to-noise ratio (PSNR) [38] and structural similarity index measure (SSIM) [39] as the evaluation indexes. The PSNR is defined as: where MSE represents the mean square error; O' (x, y) represents the reconstructed image; k is the number of bits set as 8. The higher the PSNR, the better the imaging quality. The SSIM is defined as: We first clarify several simulation settings. We select 60000 unlabeled grayscale images from STL-10 [40] dataset as the training dataset of PCA-based illumination patterns. The dataset belongs to 10 categories including bird, cat, car and others. We resized all images of dataset to 64×64 pixels which are same as the objects to be imaged. Sampling was simulated by two test images 'coco' and 'lena' under the different measurement conditions. These two images are chosen because coco belongs to the categories of the training dataset, while lena does not. The numbers of the illumination patterns required in the sampling of 10%, 20%, 30%, 40%, 50% and 100% are 410, 819, 1229, 1638, 2048 and 4096. For quantitative comparison, PSNR and SSIM are used to evaluate imaging quality of the overall area and ROI. The reconstruction results of 'coco' are shown in Fig. 2. It can be seen that the imaging quality of ROI for PCA-RGI is sharper than the other method, especially when the number of measurements is low. Besides, the methods with PCA-based illumination patterns correspondingly obtain better performance than the two methods with random-based patterns. The quantitative comparison results for 'coco' are shown in Table. 1. In terms of the imaging quality of overall image, the PSNR and SSIM of PCA-RGI are better than the other three methods in most conditions. However, we can observe that the PSNR and SSIM of methods with non-retina-like patterns are better than methods with retina-like patterns when the number of measurements is not very low. In terms of the imaging quality of ROI, the PSNR and SSIM of PCA-RGI are always better than the other three methods, followed by PCA-GI, Random-RGI and Random-GI. It is obvious that illumination patterns optimized by PCA have a better performance in obtaining object information than random patterns. In order to verify whether the optimized PCA-based retina-like illumination patterns own good generalization ability, the simulation was carried out on the test image 'lena', which not belongs to the training dataset. The settings of the second simulation are same as the first one. The comparison results for 'lena' test image are shown in Fig. 3. It can be observed that the image quality of ROI for PCA-RGI is sharper than the other three methods, which is same as the results of 'coco'. The optimized retina-like illumination patterns also perform better in terms of quickly acquiring information about the object. The quantitative comparison results of'lena' are shown in Table. 2. In terms of the imaging quality of overall image, the PSNR and SSIM of PCA-GI are better than the other three methods which is different with the results of 'coco' for more background information in the region of uninterest. However, in terms of the imaging quality of ROI, the PSNR and SSIM of PCA-RGI are better than the other three methods, followed by PCA-GI, Random-RGI and Random-GI. Obviously, the method with optimized retina-like illumination patterns has better performance because it contains sparsity prior information of the object, even the object does not belong to the category of training dataset. We compare the quantitative results of 'coco' and 'lena' in ROI. We used the PSNR and SSIM of PCA-RGI minus the PSNR and SSIM of random-RGI to express the increment of imaging quality by applying optimized retina-like patterns. The comparison results are shown in Fig. 4, it can be observed that the increment of PSNR and SSIM of 'coco' is more than that of 'lena'. This verifies that the method with optimized PCA-based retina-like patterns obtain better performance for object reconstruction of the same category of the training dataset. Meanwhile, the method also has generalization ability, although its increment is not as good as the objects of training category. In practical optical systems, measurements are always corrupted with noise from ambient light and circuit current. In the above simulations, noise was not considered in the reconstruction. We performed simulations on the influence of measurement noise and compare the robustness of methods with different illumination patterns in ROI. To simulate conditions of different noise levels, we add white Gaussian noise to the measurements of light intensity. The built-in function wgn() of MATLAB was used to add the white Gaussian noise. The reconstructed results of the "coco" with measurements of 2048 and 4096 are considered as original images. The comparison results are shown in Fig. 5, we can observe that the imaging quality of different methods decreases to different degrees with the increase of power of noise. In terms of the results of measurements with 2048, methods with nonretina-like patterns have a better performance than methods with retina-like patterns when power of noise is high. In terms of the results of measurements with 4096, methods with retina-like patterns have a better performance than methods with non-retina-like patterns. Quantitative comparison results are shown in Fig. 6, PCA-RGI has the best performance with the measurements of 2048 and 4096, but not for the condition that measurements are 2048 with the power of noise is more than -10dBw. However, the PCA-RGI always has a better performance than Random-RGI. The robustness of method with retina-like patterns is worse than methods with non-retina-like patterns under the condition measurement is low and power of noise is high enough. The reason is retina-like structure enhances the performance of ROI at the expense of the non-ROI information and also enhances the noise at the same time.

Experiments
The experimental setup is shown in Fig. 7. The setup consists of illumination part, detection part and objects. The illumination part consists of a light-emitting diode, DMD (Texas Instruments DLP Discovery 4100 development kit) and lens. The light-emitting diode operates at 400-760nm (@20W). The maximum binary modulation rate of DMD is up to 22kHz. The focal length of the projection lens is 150mm. The detection part consists of one photodetector (Thorlabs PDA36A, active area of 13 mm 2 ), data acquisition board (PICO6404E, sampling at 1 MS/s) and a computer. We selected three objects for the experiments, a modified United States Air Force resolution (USAF) test chart, an image of a cat, and an image of 'BIT'. The PCA-based patterns are trained by STL-10 dataset. The image of a cat belongs to the categories of training dataset and the others do not. The experiments were conducted in an environment with ambient noise. We first used the image of a cat for imaging. Measurements are set as 410, 819, 1229, 1638, 2048 and 4096. The experimental results are shown in Figs. 8 and 9, in terms of imaging of ROI, we can observe that the imaging quality of PCA-RGI and Random-RGI improved as the measurements increased. PCA-RGI has the best performance, followed by PCA-GI, Random-GI and Random-RGI. Affected by the ambient noise, the performance of Random-RGI was greatly affected when the measurements are low. Also affected by noise are Random-GI and PCA-GI, when close to full sampling, the image quality will begin to decrease. PCA-based patterns obviously improve the ability to better obtain information in ROI than other methods. This validates the superiority of optimized retina-like illumination pattern.  can be observed that PCA-RGI has the best imaging quality in ROI. The imaging quality Random-RGI influenced by noise and is even worse than Random-GI. The imaging quality of PCA-RGI is also affected by noise, but its robustness is better than Random-RGI.  We compared the quantitative results of 'CAT', 'USAF' and 'BIT' in ROI. The quantitative comparison results are shown in Fig. 12, the increment of imaging quality of 'CAT'is better than that of the other two objects. Because PCA-based illumination retinalike patterns improve more for objects belonging to the training dataset. Therefore, although our proposed method with optimized retina-like illumination patterns has better imaging quality in the ROI for the objects of the training dataset category. This method also has good generalization ability.

Discussions and Conclusions
The conventional retina-like patterns fill in the ROI with random patterns which has bad performance in obtaining object information. Illumination patterns with sparsity prior of objects can be obtained by performing PCA. In this paper, we present to optimize the retina-like illumination pattern in ROI by PCA. The optimized retina-like patterns are generated by filling the ROI with patterns trained by PCA and keeping the rest of conventional random retina-like patterns. The results of simulations and experiments suggest that PCA-RGI with optimized retina-like illumination patterns has better imaging quality in ROI than Random-GI, Random-RGI and PCA-GI. Meanwhile, PCA-RGI has good generalization ability. For objects that do not belong to the training dataset, PCA-RGI also has the ability to improve the imaging quality of ROI, and the improvement is more significant for objects that belong to the training dataset. However, PCA only can obtain patterns with features of dataset, which is similar to the function of one convolution layer in the deep learning network. Therefore, the patterns of ROI can be more improved by deep learning or other methods to add more priori knowledge. In the current applications of RGI, usually a small amount of sampling is performed to obtain the location and size of the ROI, and then the design and projection of the retina-like patterns is performed. While obtaining the size and location of the target, it can be classified to obtain features, so that adding prior knowledge to the design of the retina-like patterns can significantly improve the imaging quality of ROI. Our proposed method to optimize retina-like patterns paves the way for obtaing high-performance GI.