Implicit image processing with ghost imaging

: In computational ghost imaging, the object is illuminated with a sequence of known patterns and the scattered light is collected using a detector that has no spatial resolution. Using those patterns and the total intensity measurement from the detector, one can reconstruct the desired image. Here we study how the reconstructed image is modified if the patterns used for the illumination are not the same as the reconstruction patterns and show that one can choose how to illuminate the object, such that the reconstruction process behaves like a spatial filtering operation on the image. The ability to directly measure a processed image allows one to bypass the post-processing steps and thus avoid any noise amplification they imply. As a simple example we show the case of an edge-detection filter.

In computational ghost imaging one has a great deal of control over the choice of the projected patterns, allowing one to tailor them based on a knowledge of the nature of the object.In Principle Component Analysis the illuminating wavefront is designed to match the principle components of the object [12,13], and in other adaptive imaging works the spatial resolution of the wavefront is enhanced locally in response to the detection of high-frequency regions of the object [14,15].It is however less common to see the illumination basis modified in response to the way in which the image is to be processed [16].
In this paper we show that any post-processing step which can be described by a matrix multiplication with the image, such as convolution with an image filter, can be incorporated into the illumination basis.Doing so enables one to avoid image noise amplification by the filtering process, at the cost of increasing the complexity of the projected patterns.We demonstrate this technique experimentally for a basic edge-detection filter in a modified raster basis (edge-detection is one of the fundamental feature identification steps, and has been incorporated into recent computational imaging strategies [17][18][19][20][21]).We compare the resulting signal-to-noise ratios (SNRs) of our technique with those obtained via post-processing with the same filter.We also discuss a theoretical method to predict the performance of an arbitrary filter.

Changing the illumination basis
The ghost imaging measurement process can be described as follows in Bra-Ket notation, with N × N pixel projection patterns or images represented as N 2 × 1 element column vectors for convenience.The reconstructed image |I⟩ of the object |O⟩ can be written where |ψ j ⟩ is the j th pattern in the basis Ψ illuminating the object.The inner product ⟨ψ j |O⟩, which becomes the weighting coefficient of |ψ j ⟩ in the reconstruction, measures the spatial overlap between the projected pattern and object and is recorded with a bucket detector.Typically the illumination basis Ψ is the same basis in which the image is reconstructed, but this need not be the case.A change from an illumination basis Ψ to a new basis Φ can be written as Φ = B Ψ, where B is the matrix that performs the basis change.If one makes this substitution for the illuminating basis in Eq. ( 1) one can see by comparison with Eq. ( 1) that the reconstructed object effectively becomes (B T |O⟩).
The matrix B can then be chosen such that it performs any desired operation, as long as it can be expressed as a matrix multiplication (which includes convolution with any filter kernel K), directly during the measurement process.There is a lot of freedom in choosing Ψ and K (and thus B) but, as we discuss below, this choice has an impact on both the amount of signal available, and the complexity of the patterns to be projected.
To demonstrate this equivalence, we chose as our operation convolution with the edge-detection filter kernel K: A 1D convolution between two discrete signals can be written in the form of a matrix multiplication by converting one signal to the appropriate circulant (a subclass of Toeplitz) matrix [22].The matrix B that performs the convolution with a given kernel K can be constructed by extending this process to 2D signals with additional zero-padding and flattening steps.Examples of the resulting illumination patterns (⟨ψ j |B T ), are shown in Fig. 1 for the cases where the initial basis Ψ is either the canonical or Hadamard basis.
The modified illumination patterns are the result of convolving the original patterns with the filter kernel K, providing a more intuitive method of generating the matrix B. As with any image convolution a choice of boundary conditions is required, for which we have chosen cyclic conditions which wrap the values at opposite edges [23].
In the following sections we demonstrate this method experimentally for the edge-detection kernel (Eq.( 3)) in the canonical basis and compare with the results obtained when the filtering operation is instead applied after reconstruction.

Experimental setup
The experimental setup used to compare post-processed ghost images with those generated with a modified illumination basis is shown in Fig. 2. A 455 nm fibre-coupled LED is collimated by plano-convex lens L1 (f = 35 mm) and illuminates the 1080p resolution digital micromirror device (DMD).A beamsplitter allows the LED output to be monitored by a photodiode (PD1).The patterned light from the DMD is then imaged onto the object at reduced magnification by plano-convex lens L2 (f = 200 mm) and biconvex lens L3 (f = 35 mm).Finally, the light transmitted by the object is collected by biconvex lens L4 (f = 25.4 mm) and focused onto the photodiode PD2.The signal from PD2 can then be divided by the signal from PD1, compensating for fluctuations in the LED output.

Performing a fair comparison
One important consequence of modifying the illumination basis is that it might increase the number of unique intensity values required to generate the new projection patterns, unless spatial dithering techniques are employed [24].This can be seen in Fig. 1.To experimentally generate such patterns with a digital micromirror device, which is only capable of binary amplitude modulation, one needs to project and measure multiple times per desired pattern.The number of projectable sub-patterns needed is such that the desired pattern can be formed from a linear combination of sub-patterns.
For the edge-detection filter K in Eq. ( 3), this results in a factor of 2 increase in the total number of projected patterns, when comparing the canonical basis with its edge-detection counterpart (e.g.Fig. 1(a) and (b) respectively), as the negative patterns need to be projected separately, and the difference between the two patterns taken.This is similar to the standard difference measurements employed to achieve negative mask values when using Hadamard patterns [25].We compensate for this in our comparison by repeating each raster pattern twice and using their mean, keeping the total number of projections and measurement time for each method consistent.

Measurement and reconstruction
For the 64 × 64 resolution images used in this comparison, the number of patterns required in the raster basis is 4096.As discussed in Section 3.2, in the modified edge-detection basis a complimentary pair of measurements is required for each raster pattern, so the total number of measurements taken for each method is 8192, using repeat readings for the raster projections.
In the raster method an illumination similar to that in Fig. 1(a) is projected and photodiode 2 signal averaged for a given integration time, followed by a repeat reading for the same projection.The final weighting coefficient ⟨ψ j |O⟩ as in Eq. ( 1) is the mean of these two detector readings, divided by the power normalisation reading on photodiode 1, which is recorded once per pair of patterns.In this conventional raster imaging the reconstruction pattern |ψ j ⟩ for each pair is the same as the illumination pattern.This process is repeated for each of the 4096 patterns in the raster basis.The reconstructed image is then post processed by convolution with the kernel K of Eq. (3) to form the 'post-processed' result used in the comparisons.
For the 'basis-processed' results using a modified form of the raster illumination basis, the process is similar.The pattern pairs are no longer repeats, but look akin to the positive and then negative components of Fig. 1(b).The weighting coefficient ⟨ϕ j |O⟩ as in Eq. ( 2) is the photodiode 2 reading for the second pattern subtracted from that of the first, then divided by that of photodiode 1.The reconstruction pattern |ψ j ⟩ now differs from the illumination patterns and is the corresponding unmodifed raster pattern.This process is repeated for each of the 4096 raster patterns.No post-processing of the reconstructed image is necessary.
The detector integration times tested vary from 20 ms to 220 ms in 20 ms increments, and are as indicated in the figures for a given result.

Quantifying the signal-to-noise ratio
We characterize the quality of the reconstructed images from each method and over a range of detector integration times by means of the signal-to-noise ratio (SNR).We calculate the image signal-to-noise ratios by comparing the intensity values in defined peak signal (⟨I P ⟩) and background (⟨I B ⟩) regions as where ⟨.⟩ denotes the spatial average over a region and σ B the standard deviation of the intensity in the background region.The background region is selected manually and is shown in green in Fig. 3(b).We define the peak signal locations as those with the top 10% of intensity values, taken from a high SNR experimental image whilst excluding values at the borders.

Experimental results
The object, a USAF target as shown in Fig. 3, was imaged in the experimental configuration shown in Fig. 2 for a range of integration times.In Fig. 4 we show the main result of this paper, a comparison between the images obtained when projecting the modified illumination basis ('basis-processed', left column) and when using a raster basis which is then post-processed by convolution with K (Eq.( 3)) ('post-processed', right column).The experimental comparison between post-processing raster images and using a modified illumination basis, shown in Fig. 4, clearly demonstrates that the modified patterns perform the desired spatial filtering operation.
When the integration times and thus SNRs of the images are high (Fig. 4(a) and (b)), the difference in image quality is subtle, whilst in the low SNR regime (Fig. 4(e) and (f)) the improvement offered by using a modified illumination basis becomes quite clear.The visual differences apparent in Fig. 4 have two contributing factors.First, the SNR is higher in the basis-processed case.This is quantified in Fig. 5 as a factor of 2 enhancement over a range of integration times.The second difference is that the spatial character of the noise has been detrimentally modified in the post-processing case.A post-processing filter creates correlated noise due to the convolution theorem, whilst in the basis-processing approach the noise remains uncorrelated (white) and is less likely to be misinterpreted as part of the object signal.

Noise amplification model
In order to explain theoretically the difference in the signal to noise ratios of the two approaches we consider a simple theoretical framework .We use additive white Gaussian noise as a model for the detector noise in our measurement system, which we assume dominates.If an image is corrupted by additive white noise, a given filter kernel K will increase the standard deviation of the noise by a factor of √ E k , where E K is the energy of the filter defined as [26,27]: For K equal to the edge-detection filter of Eq. ( 3) used in the presented experiments, E K = 4 and the filter increases the standard deviation of the noise by a factor of 2. The additive Gaussian detector noise introduced with the measurement of the signal from the ith pattern is denoted by σ i , and are assumed to all follow the same distribution.
We represent the power normalisation step with a factor A, which tracks gradual changes in the lamp intensity with time.The normalisation will generally be imperfect due to two problems: first, the measurement of A is noisy in itself, and introduces a new detector noise σ A .Second, that either of the photodiode measurements may contain a background, e.g.due to stray light.These are denoted B m and B n for the measurement and normalisation signals respectively.
If the basis we want to use is such that it needs N patterns to be projected for the response to every basis element to be measured (for the kernel in Eq. ( 3), N = 2), a fair comparison requires us to repeat and average each measurement in the post-processing method N times, which results in a signal where * denotes the 2D convolution.In the case where A is large compared to both the noise and the background (i.e.there is a decent amount of signal compared with the artefacts), we can expand around A −1 = 0 to obtain (to first order): For the 'basis-processed' signal S B , the kernel will be a linear combination of patterns that can be projected, and thus K = ∑︁ N i=1 c i K i , where c i are the coefficients (for the kernel in Eq. ( 3), K = K 1 − K 2 ) and we can proceed similarly where we neglected all quadratic terms in A −1 .In both cases K * O is the desired signal, and the other terms represent the unwanted propagated noise (which is always a positive quantity).If the amount of stray light is small, i.e. both B n and B m are negligible the signal to noise ratio in the two cases is and thus If the σ A K * O term dominates, the ratio between the two signal-to-noise ratios is approximately √ E K , i.e. for the kernel in Eq. ( 3) we have an improvement of approximately 2, consistent with the experimental results.Notice that kernels with larger E K will lead to a higher ratio between the signal to noises.In the opposite limit, i.e. when σ A K * O is very small, the ratio of the SNRs will depend on how exactly the kernel has to be broken down to obtain a set of projectable patterns (i.e. it will depend on the coefficients c i ), but will be of the order of √ E K /N, which for the kernel in Eq. ( 3) is 1.

Conclusions
At its very core ghost imaging is the choice of basis patterns (the illuminations), the measurement of the coefficients (the intensity measured by the detector), and the reconstruction of the image using the two.A big advantage of the ghost imaging method is the total freedom in the choice of the basis used, and we have shown that the freedom of measuring the coefficient of the expansion in a different basis from the one used for the reconstruction allows for even more freedom.In particular one can use this freedom to recover any linear map of the image, thus skipping the post-processing step.We have shown this experimentally for the simple example of an edge-detection filter.
We also studied how the signal-to-noise ratios compare between performing a post-processing filtering, and directly measuring the processed image using ghost imaging.Whether the ghost imaging approach results in a smaller amount of noise depends on the linear map/filter used, and on the experimental details (e.g. the noise in the intensity normalization), but in our experiment we found that the ghost imaging SNR was a factor 2 better than the equivalent post-processed image.

Fig. 1 .
Fig. 1.Examples of illumination patterns before and after multiplication with B, in the canonical basis (a) then (b) and Hadamard basis (c) then (d) respectively.The matrix B is such that it performs the edge-detection operation of Eq. (3), with cyclic boundary conditions.The patterns shown are the 85 th in 16×16 resolution bases.

Fig. 2 .
Fig. 2. Schematic of the experimental setup.A fibre coupled blue LED is collimated and illuminates a DMD.The DMD is imaged at reduced magnification by lenses L2 and L3 onto a planar transmissive object (Obj).The transmitted light is then collected by lens L4 and focused onto photodiode 2 (PD2).The photodiode 1 (PD1) and beam-splitter combination allow for compensation for fluctuations in the light source intensity.

Fig. 3 .
Fig. 3. (a) USAF 1951 negative resolution target with the region imaged marked by a red dashed rectangle.(b) The region highlighted in (a), with the green dashed shape indicating the region defined as the background for later SNR calculations.

Fig. 4 .
Fig. 4.Comparison between the ghost images obtained using a modified projection basis ('basis-processed', left column) and those measured with a raster basis and then convolved with the edge detection kernel K ('post-processed', right column).The three rows show varying detector integration times, increasing from bottom to top as 20, 100 and 220 ms.The experimental method is as described in section 3. The images are 64 × 64 resolution.

Fig. 5 .
Fig. 5. Comparison of calculated SNRs from experimental images acquired using the basis vs post-processing methods.The error bars are calculated from the variation in three repeat measurements.Images were 64 × 64 resolution.Lines are a guide to the eye.