1000 fps computational ghost imaging using LED-based structured illumination

Single-pixel imaging uses a single-pixel detector, rather than a focal plane detector array, to image a scene. It provides advantages for applications such as multi-wavelength, three-dimensional imaging. However, low frame rates have been a major obstacle inhibiting the use of computational ghost imaging technique in wider applications since its invention one decade ago. To address this problem, a computational ghost imaging scheme, which utilizes an LED-based, high-speed illumination module is presented in this work. At 32 × 32 pixel resolution, the proof-of-principle system achieved continuous imaging with 1000 fps frame rate, approximately two orders larger than those of other existing ghost imaging systems. The proposed scheme provides a cost-effective and high-speed imaging technique for dynamic imaging applications. © 2018 Optical Society of America under the terms of the OSA Open Access Publishing Agreement OCIS codes: (110.1758) Computational imaging; (110.3010) Image reconstruction techniques. References and links 1. J. H. Shapiro, “Computational ghost imaging,” Phys. Rev. A 78(6), 061802 (2008). 2. T. B. Pittman, Y. H. Shih, D. V. Strekalov, and A. V. Sergienko, “Optical imaging by means of two-photon quantum entanglement,” Phys. Rev. A 52(5), R3429–R3432 (1995). 3. R. S. Bennink, S. J. Bentley, and R. W. Boyd, “’Two-Photon’ coincidence imaging with a classical source,” Phys. Rev. Lett. 89(11), 113601 (2002). 4. M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Process. Mag. 25(2), 83–91 (2008). 5. Y. Bromberg, O. Katz, and S. Silberberg, “Ghost imaging with a single detector,” Phys. Rev. A 79(5), 053840 (2009). 6. Z. Zhang, X. Ma, and J. Zhong, “Single-pixel imaging by means of Fourier spectrum acquisition,” Nat. Commun. 6, 6225–6230 (2015). 7. N. Radwell, K. J. Mitchell, G. M. Gibson, M. P. Edgar, R. Bowman, and M. J. Padgett, “Single-pixel infrared and visible microscope,” Optica 1(5), 285–289 (2014). 8. M. P. Edgar, G. M. Gibson, R. W. Bowman, B. Sun, N. Radwell, K. J. Mitchell, S. S. Welsh, and M. J. Padgett, “Simultaneous real-time visible and infrared video with single-pixel detectors,” Sci. Rep. 5(1), 10669 (2015). 9. L. Bian, J. Suo, G. Situ, Z. Li, J. Fan, F. Chen, and Q. Dai, “Multispectral imaging using a single bucket detector,” Sci. Rep. 6(1), 24752 (2016). 10. B. Sun, M. P. Edgar, R. Bowman, L. E. Vittert, S. Welsh, A. Bowman, and M. J. Padgett, “3D computational imaging with single-pixel detectors,” Science 340(6134), 844–847 (2013). 11. M.-J. Sun, M. P. Edgar, G. M. Gibson, B. Sun, N. Radwell, R. Lamb, and M. J. Padgett, “Single-pixel threedimensional imaging with time-based depth resolution,” Nat. Commun. 7, 12010 (2016). 12. G. A. Howland, D. J. Lum, M. R. Ware, and J. C. Howell, “Photon counting compressive depth mapping,” Opt. Express 21(20), 23822–23837 (2013). 13. B. Lochocki, A. Gambín, S. Manzanera, E. Irles, E. Tajahuerce, J. Lancis, and P. Artal, “Single pixel camera ophthalmoscope,” Optica 3(10), 1056–1059 (2016). 14. Z. Zhang, X. Wang, G. Zheng, and J. Zhong, “Fast Fourier single-pixel imaging via binary illumination,” Sci. Rep. 7(1), 12029 (2017). 15. E. J. Candes, “Compressive sampling,” in Proceedings of the International Congress of Mathematicians (International Mathematical Union, 2006), pp. 1433–1452. Vol. 26, No. 3 | 5 Feb 2018 | OPTICS EXPRESS 2427 #313782 https://doi.org/10.1364/OE.26.002427 Journal © 2018 Received 5 Dec 2017; revised 20 Jan 2018; accepted 20 Jan 2018; published 23 Jan 2018 16. D. L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006). 17. W. Gong, C. Zhao, H. Yu, M. Chen, W. Xu, and S. Han, “Three-dimensional ghost imaging lidar via sparsity constraint,” Sci. Rep. 6(1), 26133 (2016). 18. M.-J. Sun, M. P. Edgar, D. B. Phillips, G. M. Gibson, and M. J. Padgett, “Improving the signal-to-noise ratio of single-pixel imaging using digital microscanning,” Opt. Express 24(10), 10476–10485 (2016). 19. D. B. Phillips, M.-J. Sun, J. M. Taylor, M. P. Edgar, S. M. Barnett, G. M. Gibson, and M. J. Padgett, “Adaptive foveated single-pixel imaging with dynamic supersampling,” Sci. Adv. 3(4), e1601782 (2017). 20. E. J. Candès and M. B. Wakin, “An introduction to compressive sampling,” IEEE Signal Process. Mag. 25(2), 21–30 (2008). 21. M.-J. Sun, W. Chen, T.-F. Liu, and L.-J. Li, “Image retrieval in spatial and temporal domains with a quadrant detector,” IEEE Photonics J. 9(5), 3901206 (2017). 22. M. Herman, J. Tidman, D. Hewitt, T. Weston, and L. McMackin, “A higher-speed compressive sensing camera through multi-diode design,” Proc. SPIE 8717, 871706 (2013). 23. F. Ferri, D. Magatti, L. A. Lugiato, and A. Gatti, “Differential ghost imaging,” Phys. Rev. Lett. 104(25), 253603 (2010). 24. M. J. Padgett and R. W. Boyd, “An introduction to ghost imaging: quantum and classical,” Philos Trans A Math Phys Eng Sci 375(2099), 20160233 (2017). 25. W. Le and S. Zhao, “Fast reconstructed and high-quality ghost imaging with fast Walsh–Hadamard transform,” Photon. Res. 4(6), 240–244 (2016). 26. M. Alemohammad, J. R. Stroud, B. T. Bosworth, and M. A. Foster, “High-speed all-optical Haar wavelet transform for real-time image compression,” Opt. Express 25(9), 9802–9811 (2017). 27. B. T. Bosworth, J. R. Stroud, D. N. Tran, T. D. Tran, S. Chin, and M. A. Foster, “High-speed flow microscopy using compressed sensing with ultrafast laser pulses,” Opt. Express 23(8), 10521–10532 (2015). 28. Q. Guo, H. Chen, Z. Weng, M. Chen, S. Yang, and S. Xie, “Compressive sensing based high-speed time-stretch optical microscopy for two-dimensional image acquisition,” Opt. Express 23(23), 29639–29646 (2015). 29. L. Tian and L. Waller, “3D intensity and phase imaging from light field measurements in an LED array microscope,” Optica 2(2), 104–111 (2015). 30. N. Grossman, V. Poher, M. S. Grubb, G. T. Kennedy, K. Nikolic, B. McGovern, R. Berlinguer Palmini, Z. Gong, E. M. Drakakis, M. A. Neil, M. D. Dawson, J. Burrone, and P. Degenaar, “Multi-site optical excitation using ChR2 and micro-LED array,” J. Neural Eng. 7(1), 16004 (2010). 31. S. Onose, M. Takahashi, Y. Mizutani, T. Yasui, and H. Yamamoto, “Single Pixel Imaging with a High-FrameRate LED Array,” in JSAP-OSA Joint Symposia 2016 Abstracts (Optical Society of America, 2016), paper 13a_C301_10. 32. W. K. Pratt, J. Kane, and H. C. Andrews, “Hadamard transform image coding,” Proc. IEEE 57(1), 58–68 (1969). 33. M.-J. Sun, L.-T. Meng, M. P. Edgar, M. J. Padgett, and N. Radwell, “A Russian Dolls ordering of the Hadamard basis for compressive single-pixel imaging,” Sci. Rep. 7(1), 3464 (2017). 34. M.-J. Sun, M.-F. Li, and L.-A. Wu, “Nonlocal imaging of a reflective object using positive and negative correlations,” Appl. Opt. 54(25), 7494–7499 (2015). 35. P. Kner, B. B. Chhun, E. R. Griffis, L. Winoto, and M. G. Gustafsson, “Super-resolution video microscopy of live cells by structured illumination,” Nat. Methods 6(5), 339–342 (2009). 36. A. G. Godin, B. Lounis, and L. Cognet, “Super-resolution Microscopy Approaches for Live Cell Imaging,” Biophys. J. 107(8), 1777–1784 (2014).


Introduction
As an alternative to using a focal-plane detector array, single-pixel computational ghost imaging [1], an improved scheme of ghost imaging [2, 3] and closely related to single-pixel imaging [4-6], retrieves a two-dimensional image by recording only the total light intensities in each component of a spatial sampling basis, rather than capturing it with a pixelated array as in a digital camera. Though not performing as well as detector array in conventional visible imaging, computational ghost imaging provides advantages for applications such as multiwavelength imaging [7-9] and three-dimensional imaging [10-12], both of which pose difficulties for conventional cameras.
Low frame rate has been a major obstacle for computational ghost imaging to develop into a more practical technique. Many works have sought to address this problem and improved the frame rate to over 10 frames per second (fps) in recently years [11][12][13][14]. The approaches include using devices such as spatial light modulator (SLM) [1], and utilizing compressive sensing [15][16][17] to reduce the number of acquisitions that computational ghost imaging requires to reconstruct an image. However, even the digital-micromirror-device (DMD), the fastest SLM for the time being, has an upper limitation of 22 KHz modulation rate, which leads to ~10 fps at 32 × 32 pixel resolution for fully sampled differential ghost imaging [18,19]. Compressive sensing does decrease the required number of acquisitions for each frame, however, the subsequent convex optimization procedure to reconstruct the image requires a computational overhead, which poses difficulties for real-time dynamic applications. Furthermore, the problems to be solved by the convex optimization are ill-posed, therefore the solutions could be non-deterministic [20]. Other works aiming to increase the frame rate, including using a quadrant detector [21] or multi-diode design [22], are still limited by the modulation rate of the SLM.
In this work, we significantly improved the frame rate using a computational ghost imaging scheme based on a high-speed LED illumination module. With the help of the evolutionary compressive sensing [7, 11], which is a low computational and deterministic reconstruction algorithm, the proof-of-principle system experimentally achieved continuous imaging at a 1000 fps frame rate, which is approximately two orders higher than those of existing computational ghost imaging systems. The presented scheme enables us to capture images of dynamic scenes, which would be otherwise motion-blurred by other existing computational ghost imaging schemes as well as many conventional cameras. In computational ghost imaging, Fig. 1(a), an SLM creates spatial modulated light field E i (0) (here i refers to the ith modulation), which propagates distance L through free space and illuminates the object with the light field E i (L). The total transmitted or reflected intensity S i is then collected and measured by a large area "bucket" detector. More practically, computational ghost imaging is performed with an imaging lens, Fig. 1(b), by which the modulation plane and the object plane are imaged to each other. The intensity distribution is

Theory
, where β is the magnification determined by the imaging conjugation.
Based on the knowledge of intensity distribution P i and corresponding measurement S i the reconstruction of the ghost image can be undertaken with various degrees of sophistication [13,23]. The simplest algorithm weights each P i by the value of the fluctuation of S i and then sums these weighted P i [24] 1 ( ) , where N is the number of measurements the ghost imaging system recorded sequentially in time.
According to Eq. (1), the time to obtain one ghost image depends on the required acquisition time of N measurements. Early computational ghost imaging schemes [1, 5, 10, 23] used partially correlated patterns to provide modulation. It usually requires a large number of measurements to reconstruct an image, therefore the frame rate of the system is jeopardized. Recent works using orthogonal patterns [8,11,16,25] to provide more efficient modulation, which leads to a smaller N while yielding better image qualities compared to those using partial correlated patterns. However, the inversely proportion relationship still stands, except that now an n × n pixelated scene can be fully sampled with N = n × n orthogonal measurements. Specifically, to obtain images with pixel number N, the system frame rate is Rmod/N, where R mod is the modulation rate of system as in patterns/s. To further improve the frame rate of the ghost image, one approach is to decrease N by utilizing compressive sensing. Provided the scene is sparse, compressive sensing [15][16][17] can reconstruct ghost image with N < n × n measurements by under sampling the scene and solving the optimization problem. However, the optimization procedure could lead to a computational overhead and a non-deterministic reconstruction.
In conclusion, if the scene is not sparse and a deterministic reconstruction is desired, then at least N = n × n measurements are required to reconstruct an n × n pixelated scene. Consequently, to increase the frame rate of computational ghost imaging, the rational choice is to increase the modulation rate of the SLM.

Illumination module
Besides using SLM, structured illumination can be performed by using electro-optic modulation of broadband laser sources to conduct basis scan and compressive imaging [26][27][28]. Fast switching LED micro-arrays have been used 3D phase and intensity microscopy, super-resolution microscopy, and neural stimulation [29,30], and the concept of using LED arrays in the context of computational ghost imaging was recently considered theoretically in [31]. Here, taking advantage of the fact that the switching time of a single LED is less than 1 μs, we developed an LED array module to provide high-speed structured illumination for computational ghost imaging.
In this work, we chose to use the Hadamard basis [32] as the displayed patterns. There are three reasons for this choice. First, the patterns are orthonormal, therefore provide more efficient scene sampling. Second, the patterns are binary, therefore is easier and faster to be displayed by LEDs than greyscale patterns. Third, all patterns from the Hadamard basis contain a certain symmetry, which enables us to achieve a high pattern rate. More information on utilizing Hadamard matrix in computational ghost imaging or single-pixel imaging can be found in our previous works [15,33]. The details of developing the LED illumination module are specified as follow.
Display strategy. There are two conventional strategies to display binary patterns on an LED array. The point control strategy, Fig. 2(a), controls each LED with one I/O port individually, meaning n 2 I/O ports are required for an n × n resolution LED array. With this strategy, the display rate of the LED array can be as fast as the switching rate of a single LED. However, if the resolution of the LED array is increased, it is difficult for one FPGA to provide enough I/O ports. The line control strategy, Fig. 2(b), controls each row or each column with one port, and uses line scanning technique to display a pattern line by line. It requires only 2n I/O ports for an n × n resolution LED array, therefore avoids the problem of demanding too many I/O ports as resolution goes higher. However, it poses the problem of a much lower display rate because it scans the pattern line by line, and display one pattern in the time of n × LED switching time. Fortunately, for the particular case of Hadamard illumination, we found that all Hadamard patterns contain a symmetry, which enables us to separate each pattern into two parts. Importantly, both parts can be displayed by the line control approach, meaning that by sequentially displaying the two half parts, we can get the structured illumination of one Hadamard pattern in twice the LED switching time, as shown in Fig. 2(c). Pattern rate. In our module, the controlling signals sent by FPGA to 64 I/O ports were set to 1 MHz because the LED switching time is smaller than 1 μs. Using the display strategy we specified above, the pattern rate of the LED array was 500K patterns/s. Figure 3 illustrates the LED based computational ghost imaging experiment set-up, which is similar to Fig. 1 except that an LED array has replaced the SLM. The LED array was of 32 × 32 pixel resolution with the array size of 64 mm × 64 mm. The array displayed a preloaded sequence of Hadamard patterns at rate of 500K patterns/s. To remove noise of light fluctuation, differential signals were obtained by displaying each Hadamard pattern, followed by its inverse pattern, and taking the difference in the measured two intensities [15,20,34]. A camera lens (Nikon AF Nikkor, f = 50 mm, F/1.4D) was located 250 mm from the array and imaged the Hadamard patterns to 12 mm × 12 mm structured illumination on a rotating object. The object was a black disk uniformly carved with ten numbers from '0' to '9' and it rotated at the specified speed. The transmitting light intensities corresponding to different Hadamard patterns were collected by a singlet lens (BOCIC, OCL3-050-050, f = 50 mm, F/1) and measured by a single-pixel detector (Thorlabs PDA100A-EC, 320 nm -1100 nm, active area φ10 mm, gain 20 dB). A high dynamic range digitizer (PicoScope 6404D, sampling at 2 GS/s), synchronized with the LED array, acquired and transferred the intensity data to a computer for image reconstruction. Fig. 3. Experimental set-up. An illuminating 32 × 32 LED array consequentially displayed preloaded Hadamard patterns, which were imaged by a camera lens to provide structured illumination on a rotating object. The object was a black disk uniformly carved with ten numbers from '0' to '9' and it rotated at the speed we chose. A singlet lens collected the transmitting light into a single-pixel detector, which measured the light intensities corresponding to different Hadamard pattern illuminations. A digitizer, synchronized with the LED array, acquired and transferred the light intensity data to a computer for image reconstruction. To fully sample the object and reconstruct a 32 × 32 pixel image, 2048 patterns, 1024 Hadamard patterns and their inverse, were displayed. By utilizing evolutionary compressive sensing [7], it is also possible to reconstruct images with less patterns at a higher frame rate. In short, the evolutionary compressive algorithm chooses a subset of the Hadamard patterns to sample the scene, by selecting the patterns with the most significant intensities measured in the previous frame. To keep its robustness to dynamics, a fraction of randomly selected patterns which were not used previously were also selected. Figure 4(a) shows three reconstructed images of the static object by using different numbers of patterns. The image size was 12 mm × 12 mm with 32 × 32 pixel resolution, the same as that of the structured illumination. The times to display 2048, 1024 and 512 patterns were 4.1μs, 2.0μs and 1.0μs respectively. The image quality at 1000 fps was only mildly degraded when the image was reconstructed with 75% fewer patterns because evolutionary compressive sensing only samples the sub set of patterns having large weights. Noting here that the 1000 fps was achieved by using a 25% compressive rate, if full sampling was performed, the actual frame rate would be 250 fps. On the contrary, when the disc rotated at 8 rounds/s, i.e. the object moved at a speed of 1.51 m/s, at the tangential direction of disc circumference, Fig. 4(b) shows that the image qualities at 250 fps and 500 fps were compromised correspondingly by motion blur while the image quality at 1000 fps remained similar to that in the static case. The signal-to-noise ratio (SNR) of the images were obtained using red solid rectangle feature as signal and red dash rectangle background as noise. The values of the image SNR were in a good agreement with the observation above. It is worth mentioning that motion blur in computational ghost imaging is different from that in conventional imaging, though both lead to image quality degrading. The motion blur is related to the frame rate, the object size and the object moving velocity, detailed discussion can be found in Supplementary Note 3 of [10]. Figure 5(a) shows three images of the object at different rotational speeds. Because of the high frame rate, the image quality was still maintained as the object rotated faster. As an illustrative comparison, Fig. 5(b) shows three images captured by a commercial camera (Nikon D5300, with Nikon AF Nikkor lens, f = 50 mm, F/1.4D) operating at its highest frame rate (50 fps), where the motion blur became more severe as the object rotated faster. In the instance of Nikon D5300 camera, it is possible for it to get rid of the motion blur by taking a single shot with a 0.001 s shutter time (equivalent to 1000 fps). However, it is much more difficult for the camera to operating continuously at this frame rate. Visualization 1 shows an illustrative comparison of the video recorded by the conventional camera at 50 fps and the one obtained by our system at 1000 fps. We realize that there are expensive scientific cameras on the market which could reach higher frame rates with a higher pixel resolution, the video here only provided a visualization of the motion blur caused by low frame rates. Because commercial cameras perform calibration and pre-processing on the raw image data, we also performed Gaussian smoothing and thresholding on the data obtained by our system in Fig. 5 and Visualization 1, therefore these images didn't contain the background noise exhibited in Fig. 4. Although it might be unnecessary for human operator to see these images at 1000 fps, it could be crucial for machine vision to have the ability of capturing images of fast objects. As a matter of fact, the switching time of a single LED can be reduced to several nanoseconds, therefore the illumination rate could be further increased and potentially reach 100K fps.

Results and discussion
Illumination energy is an issue in the proposed system. Large drive currents could be applied to provide enough illumination power for long distance imaging application. However, the dense LED array would cause a severe heating problem. In this experiment, the current was limited to 2A. Given its relatively weak illumination but high frame rate, the imaging system we proposed here might be more suitable for living cell microscopy techniques, such as structured illumination microscopy, whose frame rate is still under 100 Hz [35,36].

Conclusion
In this work, we presented a computational ghost imaging scheme based on a high-speed LED illumination module. Taking advantage of the fact that the switching time of a single LED is smaller than 1 μs, and utilizing the symmetry presented in Hadamard patterns, we were able to develop a 32 × 32 LED array module which could illuminate a scene with 500 KHz pattern rate. The proof-of-principle system experimentally achieved 32 × 32 resolution imaging at 1000 fps frame rate, which is approximately two orders higher than that of existing computational ghost imaging systems. The presented scheme enables us to capture images of dynamic scene, which would be otherwise motion-blurred by other ghost imaging schemes as well as many commercial cameras. With the potential of reaching even higher frame rates, the LED array based computational ghost imaging system could be a high-speed, cost-effective imaging solution for living microscopy and many other applications.

Disclosures
The authors declare that there are no conflicts of interest related to this article.