Three-dimensional integral imaging displays using a quick-response encoded elemental image array

Mobile devices are a ubiquitous technology, and many researchers are trying to implement three-dimensional (3D) displays on mobile devices for a variety of applications. We investigate a method to store compressed and encrypted elemental images (EIs) used for 3D integral imaging displays in multiple quick-response (QR) codes. This approach allows user friendly access, readout, and 3D display with mobile devices. We first compress the EIs and then use double-random-phase encryption to encrypt the compressed image and store this information in multiple QR codes. The QR codes are then scanned using a commercial Smartphone to reveal the encrypted information, which can be decrypted and decompressed. We also introduce an alternative scheme by applying photon counting to each color channel of the EIs prior to the aforementioned compression and encryption scheme to generate sparsity and nonlinearity for improved compression and security. Experimental results are presented to demonstrate both 3D computational reconstruction and optical 3D integral imaging display with a Smart-phone using EIs from the QR codes. This work utilizing compressed QR encoded EIs for secure integral imaging displays using mobile devices may enable secure 3D displays with mobile devices.

Integral imaging, a promising three-dimensional (3D) imaging technique, has been extensively investigated in disciplines as diverse as entertainment, medical sciences, robotics, manufacturing, and defense [1][2][3][4][5][6][7][8][9].Photon counting has been incorporated with integral imaging [6,7] to reconstruct a 3D image from a photon-starved environment.Recently, doublerandom-phase encryption (DRPE) with photon counting has been incorporated with integral imaging [7].It was shown that integral imaging can provide better security performance along with the ability to secure 3D information.The DRPE technique [10][11][12][13][14][15][16][17][18] is able to encrypt and decrypt an image.There have been numerous improvements to the encryption system to enhance security [12][13][14] in the system.One encryption scheme involves applying a photon-counting technique to the amplitude of the encrypted image generating a photon-limited image [6,7,15,17].This technique limits the number of photons arriving at a pixel.As a result, the decrypted image is sparse, noise-like, and difficult to visually authenticate.In [17], a binary image was encrypted and compressed using the full-phase DRPE with photon counting and an iterative Huffman coding technique, respectively, and stored in a quick-response (QR) code.The QR code was then scanned by a QR reader built in commercial Smartphones revealing the data.The scanned data were then decompressed and decrypted revealing a noise-like decrypted image that can be authenticated using nonlinear correlation filters [19][20][21].In this Letter, we propose a method to securely store a RGB elemental image (EI) in a QR code by combining run-length encoding (RLE) [22] with the Huffman coding compression [23] scheme along with DRPE [10].By storing data in a QR code [24][25][26], it is possible to reduce the amount of information needed to transfer the EI.We can store multiple EIs in QR codes that can be used in 3D integral imaging reconstruction [1][2][3][4][5][6][7][8][9].In addition, we present an alternative scheme by first applying photon counting on the EI followed by RLE, Huffman coding, and encryption to generate sparsity and nonlinearity due to Poisson transformation for improved compression and security.Both computational and optical reconstruction of the EIs is presented.
Integral imaging [1][2][3][4][5][6][7][8][9] can produce a 3D image of a scene by recording multiple two-dimensional (2D) images of different perspectives of the scene, known as EIs.This is achieved by using various pickup methods such as an array of image sensors or a camera with a lenslet array, as shown in Fig. 1(a).Reconstruction of the 3D scene from the recorded EIs can be performed optically, as shown in Fig. 1(b), or computationally.Optical reconstruction is performed by projecting the EIs through a lenslet array to the image plane forming a 3D scene.Computational reconstruction uses a virtual pinhole array to inversely project the EIs to the image plane to obtain the reconstructed scene as follows [4,6,9]: where Rx; y; z is the intensity of the reconstructed 3D image at depth z; x and y are the index pixels, E kl is the intensity of the kth column and the l th row of the EIs, K and L are the total numbers of EIs in the column and row, respectively, N x and N y are the total numbers of pixels for each EI, M is the magnification factor z∕g; g is the focal length, p is the pitch between image sensors, c x ; c y are the spatial sizes of the image sensor, and Ox; y is the overlapping number matrix.DRPE is used to encrypt the input image [10].Onedimensional (1D) notation will be used in explaining the encryption method.To implement the encryption scheme, let x and υ denote the spatial and frequency domains, respectively, f x be the primary input image, and nx and bυ represent random noises that are uniformly distributed over the interval [0,1].The encrypted image is ψx ff x × expi2πnxg hx; ( where * denotes convolution and × denotes multiplication, expi2πnx is a phase mask, and hx is a function whose Fourier transform is expi2πbv.
The decryption process is the reverse of the encryption process.The Fourier transform of the DRPE image, ψx, is multiplied by the complex conjugate image of the phase mask, exp−i2πbν.The Fourier transform is taken once more.The intensity of the image can then be recorded, which produces the decrypted image, f decrypt x, if the primary input image is real and positive.
RLE [22] is a lossless compression technique that represents a series of repeated data by stating the number of times the data are repeated followed by the repeated integer.For example, [8 8 8 8 8] is represented by [5 8].Huffman coding [23] is a dictionary-based compression technique that can be combined with RLE to allow for sufficiently small images to be stored in a QR code.By combining the two compression methods, less information is stored in the QR code, improving the ability of a Smartphone to scan the QR code [24].
We compress each channel of the RGB image using RLE followed by Huffman coding.The DRPE [Eq.( 2)] is used to encrypt each channel of the compressed image.Due to the ciphered images being complex, both the real and imaginary parts must be retained.As a result, both are stored in separate QR codes, which are generated using the ZXing project [25].In addition, the encrypted images are rounded to one decimal place to minimize the amount of necessary information stored in the image.Each ciphered channel of the RGB image can then be stored in separate QR codes.Moreover, we can use the same encryption keys for each color channel.For a single RGB image, six QR codes are used in our experiments.The EI must be sufficiently small so that the compressed and encrypted data can be stored in a single QR code.
Once the QR code has been scanned, the data can be successfully decrypted if the decryption keys are known.In addition, the image can be decompressed if the dictionary associated with the Huffman code compression is known; no dictionary is required to decompress the RLE algorithm.
Figure 2(a) depicts a 19 × 19 pixel input RGB image.For brevity, the red channel of the RGB image is explained.The process can be repeated for the other channels.Figure 2(b) shows a QR code that stores the real part of the ciphered input image, while Fig. 2(c) shows the imaginary part of the ciphered input image.Figures 2(d) and 2(e) display the compressed and encrypted data stored in the QR codes using the iPhone SCAN application.After decompressing and decrypting the necessary data, the three color channels of the decrypted image are combined to form the decrypted RGB image as shown in Fig. 2(f).
To determine the degree of degradation (if any) due to DRPE combined with RLE and Huffman coding compression schemes, the mean squared error (MSE) is calculated as  where n and m are pixels in the x and y directions, respectively, N and M represent the total number of pixels in the x and y directions, respectively, f decrypt x n ; y m is the decrypted and decompressed RGB image, and f x n ; y m is the original RGB image.Since both Huffman coding and RLE are lossless, and no information is lost during DRPE, the MSE calculated between the images shown in Figs.2(a) and 2(f) is 0.
It is possible to store multiple encrypted and compressed images in multiple QR codes as shown in Fig. 3.Each QR code can be scanned allowing for every EI to be recovered.Thus, this approach can have applications in integral imaging reconstruction by storing the EIs in QR codes.
A 3D integral imaging experiment was conducted for computational integral imaging reconstruction.Each EI used identical encryption and decryption keys.Moreover, identical encryption keys were used in the spatial and frequency domains.Four 19 × 19 pixel RGB EIs, shown in Fig. 4(a), were generated using 3Ds Max.The 3D scene consists of a green "3" and red letter "D" located 65 and 115 mm away from a lenslet array.These images were compressed using RLE followed by Huffman coding compression and then encrypted with DRPE.These EIs were decrypted decompressed revealing the original EIs.
Computational reconstruction results using Eq. ( 1) were obtained.Figure 4(b) depicts the computational reconstruction at 65 mm focused on "3," while Fig. 4(c) depicts computational results at 115 mm focused on "D." Storing a large quantity of EIs in multiple QR codes may prove burdensome and inconvenient.Regardless, with improving camera resolution and hardware of Smartphones, it will be possible in the future to store more information in QR codes.Thus, it may be feasible to store a prodigious number of encrypted and compressed EIs in QR codes.An optical experiment was carried out using the proposed compression and encryption scheme to secure EIs for 3D display.Using 3Ds Max, 54 × 96 RGB EIs, similar to the EIs shown in Fig. 4(a), were used, which were each 19 × 19 pixels.As before, each EI used identical encryption and decryption keys.These images were then compressed using RLE followed by Huffman coding compression and encrypted with DRPE.Each EI had identical encryption keys for both the spatial and frequency domains.These EIs were then decrypted and decompressed revealing the EIs.An HTC One Smartphone was used to display the optical reconstruction.Figure 5 depicts the experimental setup for optical reconstruction using an HTC One Smartphone and a lenslet array.Table 1 presents the experimental parameters.
Figure 6(a) depicts the 3D optical reconstruction using the primary EIs, while Fig. 6(b) depicts the 3D optical reconstruction using the decompressed and decrypted EIs.The reconstruction results using our proposed method are as good as the conventional method.There is no visible loss of information due to compression, encryption, and QR encoding.
Storing data in QR codes holds many practical advantages due to not requiring any specialized hardware to scan the QR code.Moreover, Smartphones are a ubiquitous technology.The proposed technique of storing QR codes can potentially be modified by creating a video barcode as described in [26] to allow a faster user experience when scanning multiple QR codes.
We now introduce an alternative compression and security approach.We first apply photon counting [6,7,15,17] to each channel of an RGB EI, f x; y.Photon counting is described by a statistical method for a limited number of photons that arrive at a pixel.Thus, the lower the total number of photons in the image, the fewer the number of photons that arrive at a pixel.By reducing the number of photons, it is possible to have more sparsity, improving the RLE compression scheme used prior to Huffman coding.Also, the nonlinear transformation introduced by photon counting improves the security.Photon counting can be modeled as a Poisson distribution:   Pl j ; λ j λ j l j e −λ j l j ! ; for λ j > 0; l j ∈ f0; 1;2; …g; (4) where l j is the number of photons detected at pixel j and λ j is the Poisson parameter defined as N p x j , where N p is the number of photons in the scene and x j is the normalized irradiance at pixel j such that P J j1 x j 1 with J being the total number of pixels.Moreover, the normalized irradiance is defined as jf x m ; y n j∕ P M m1 P N n1 jf x m ; y n j, where m and n denote pixels in the x and y directions, respectively, and M and N represent the total numbers of pixels in the x and y directions, respectively.
A 54 × 96 RGB EI array with each EI being 19 × 19 pixels, similar to the EI depicted in Fig. 7(a), was used for 3D photoncounting imaging compression and security.The object chosen was a color gradient teapot generated by 3Ds Max.
Photon counting was applied to each channel of the RGB EI as shown in Fig. 7(b) using 3000 photons or about eight photons per pixel on each channel of the 19 × 19 RGB image.RLE is then applied, followed by coding and DRPE.The image was then decrypted and decompressed.Optical reconstruction using an HTC phone (see Table 1) was used to reconstruct the image [see Eq. ( 1)].The MSE for the red color channel was calculated as 62.94, the green channel was 1.07, and the blue channel was 19.73.Thus, there was some degree of degradation; however, visually the 3D optical reconstruction of the photon-counted image can still be discerned.
A comparison of the compression with and without photon counting was conducted.It was found that for a sufficiently lower number of photons, the photon-counting method helped to improve the compression scheme.Using the EIs in Figs.7(c) and 7(d), the total length of the Huffman code following RLE was compared.It was found that the red channel was 21.7% shorter and the blue channel was 18.53% shorter; there were no green channel components in the test image.These results show that photon counting may allow for larger EIs to be stored in QR codes after compression and encryption.
We have presented a method and experiments for secure integral imaging as a potential approach to store EIs in QR codes for 3D displays.The compressed and encrypted EIs using DRPE can be stored in multiple QR codes to be scanned by a commercial Smartphone revealing the encrypted and compressed EIs.This data can then be decrypted and decompressed to obtain the original EIs for secure 3D integral imaging displays.We also present an alternative compression and security scheme by applying photon counting to the input image followed by compression and encryption.By using photon counting, the compression and security system improve at the cost of diminished image quality.Future work may include alternative algorithms for encrypting the EIs, compressing the EIs, and video scanning the QR codes.

Fig. 2 .
Fig. 2. (a) 19 × 19 pixel RGB image; (b) and (c) display the QR codes containing the real and imaginary information for the red color channel from the compressed and ciphered image shown in (a), respectively; (d) and (e) depict a scanned QR code using the iPhone SCAN application revealing the real and imaginary information from the compressed and ciphered image, respectively, for the red color channel of the image shown in (a); (f) shows the decrypted and decompressed image.

Fig. 5 .
Fig. 5. Optical 3D integral imaging display setup using a Smartphone and a lenslet array.
Figure 7(c) depicts optical reconstruction showing the original EI, while Fig. 7(d) depicts the optical reconstruction using the photonlimited EI.The MSE [see Eq. (3)] was calculated to determine the degree of degradation for each individual color channel.For comparison, both images were normalized to [0,255].

Fig. 6 .
Fig. 6. 3D display results using Smartphone.(a) 3D optical reconstruction with integral imaging using the primary EIs.(b) Optical 3D reconstruction using the decrypted and decompressed EIs obtained from the compressed DRPE EIs in QR codes.

Fig. 7 .
Fig. 7. 3D photon counting integral imaging compression and security experiments.A 54 × 96 EI array consisting of 19 × 19 pixel RGB EIs was used.(a) Shows an EI, while (b) depicts the corresponding photonlimited EI using about eight photons per pixel on each color channel of the EI.3D optical reconstruction after decryption and decompression is shown (c) using original EIs and (d) when photon counting on the EIs was used.

Table 1 .
Specifications of 3D Display Setup