Real-time relighting of digital holograms based on wavefront recording plane method

Relighting is an important technique in photography which enables the optical properties of a picture to be modified without retaking it again. However, different from an optical image, a digital hologram cannot be relit by simply varying the value of individual pixel, as each of them is representing holistic information of the entire object scene. In this paper, we propose a fast method for the relighting of a digital hologram. First, the latter is projected to a virtual wavefront recording plane (WRP) that is located close to the object scene. Next, the WRP is relit, and subsequently expanded into a full hologram. Experiment results have demonstrated that our proposed method is capable of relighting a 2048x2048 hologram at a rate of over 50 frames per second. To the best of our knowledge, this is the first time relighting is considered in the context of holography. ©2012 Optical Society of America OCIS codes: (090.0090) Holography; (090.1995) Digital holography; (090.1760) Computer holography. References and links 1. T. Shimobaba, H. Nakayama, N. Masuda, and T. Ito, “Rapid calculation algorithm of Fresnel computergenerated-hologram using look-up table and wavefront-recording plane methods for three-dimensional display,” Opt. Express 18(19), 19504–19509 (2010). 2. S. C. Kim and E. S. Kim, “Fast computation of hologram patterns of a 3D object using run-length encoding and novel look-up table methods,” Appl. Opt. 48(6), 1030–1041 (2009). 3. H. Sakata and Y. Sakamoto, “Fast computation method for a Fresnel hologram using three-dimensional affine transformations in real space,” Appl. Opt. 48(34), H212–H221 (2009). 4. P. W. M. Tsang, J.-P. Liu, W. K. Cheung, and T.-C. Poon, “Fast generation of Fresnel holograms based on multirate filtering,” Appl. Opt. 48(34), H23–H30 (2009). 5. T. Yamaguchi, G. Okabe, and H. Yoshikawa, “Real-time image plane full-color and full-parallax holographic video display system,” Opt. Eng. 46(12), 125801 (2007). 6. P. W. M. Tsang, K. W. K. Cheung, T. Kim, Y. S. Kim, and T.-C. Poon, “Fast reconstruction of sectional images in digital holography,” Opt. Lett. 36(14), 2650–2652 (2011). 7. X. Zhang and E. Y. Lam, “Edge-preserving sectional image reconstruction in optical scanning holography,” J. Opt. Soc. Am. A 27(7), 1630–1637 (2010). 8. L. Denis, D. Lorenz, E. Thiébaut, C. Fournier, and D. Trede, “Inline hologram reconstruction with sparsity constraints,” Opt. Lett. 34(22), 3475–3477 (2009). 9. P. Tsang, W.-K. Cheung, T.-C. Poon, and C. Zhou, “Holographic video at 40 frames per second for 4-million object points,” Opt. Express 19(16), 15205–15211 (2011). 10. T.-C. Poon, ed., Digital holography and three-dimensional display: Principles and Applications (Springer, 2006).


Introduction
In traditional photography, it is often necessary to modify a picture to enhance its visual quality, or to create special effects that are absent in the image acquisition process.Amongst different techniques, relighting is perhaps one of the most popularly practiced methods as it allows the optical properties (such as illumination), which may be difficult to control in the real world environment, to be synthesized.Nowadays, relighting can now be conducted by both professional and novice users through a comprehensive choice of commodity photo-editing software.In brief, a picture is relit by modifying the value of each pixel according to a given criteria, an operation commonly referred to as the 'point' process.For example, the effect of a spotlight can be simulated by modulating the luminance of each pixel with the spatial distribution of the illumination.Apparently, it will be desirable if the relighting mechanism can be applied to digital holograms to enhance their impact to the observers.The problem is, rendering a digital hologram with the point process is erroneous, as each pixel is representing the holistic information from the entire object scene.Until now, research on hologram relighting has not been investigated.A straightforward solution is to render the original object scene, if it is still available, whenever a relighting task is required, and then regenerate the hologram afterwards.Despite the effectiveness and simplicity of this method, the process is time-consuming as the numerical generation of a digital hologram involves enormous amount of arithmetic operations.Although there are quite a number of fast algorithms, such as [1][2][3][4][5] which attempt to alleviate the problem, they are not capable of generating holograms in real time (e.g., at the video frame rate) if there are large number of object points.Besides, if the digital hologram is captured with optical means, the original scene may not be available afterwards.If that is the case, theoretically we can apply some sort of inverse mapping [6][7][8] to reconstruct, and then relight the scene image.Subsequently, the rendered scene image can be converted into a hologram.However, as reported in some literatures, the inverse process itself is complicated.Besides, so far only the reconstruction of holograms representing sparse images (i.e., images that contain few number of object points) had been successfully demonstrated.In this paper, we propose a fast method for relighting digital hologram without the presence, or the reconstruction of the original object scene.Our scheme is based on the wavefront recording plane (WRP) which is originally employed in fast generation of digital holograms [1] [9].To our knowledge, it is the first time we have applied the WRP concept to achieve real-time relighting of holograms.In our proposed method, a digital hologram is projected onto a virtual wavefront recording plane (WRP) which is placed sufficiently near to the object points in the scene.At close proximity, each object will only cast its optical wave on a small region on the WRP.Hence, relighting the intensity of a pixel in the WRP is equivalent to modifying the intensities of a small cluster of object points that is contributing to the pixel of interest.On this basis, we apply relighting to the WRP, and subsequently expand it to a full hologram.The entire process mainly involves 4 Fast Fourier Transform (FFT) operations which can be realized with the graphical processing unit (GPU) in less than 20ms for a hologram comprising of 2048x2048 pixels.Experimental results demonstrated that the target relighting effects are correctly synthesized in the reconstructed images of holograms that are relit with our proposed method.

Hologram relighting
The concept of the proposed hologram relighting can be illustrated with Fig. 1.To start with, we insert a hypothetical diffraction plane, known as the wavefront recording plane (WRP), between the digital hologram and the scene.Given an arbitrary object point, its optical wave will propagate by diffraction to the entire hologram.Other object points in the scene are contributed to the hologram in a similar manner.Hence, modifying a hologram pixel will affect the diffracted waves contributed by the entire scene image, instead of localizing in the region around the pixel of interest.However, as shown in the diagram, an object point will only cover a small area on the WRP (the dotted region).The closer the distance between the object point and the WRP, the smaller will be the coverage (a.k.a. the support) of the diffraction pattern on the latter.As such, relighting a pixel in the WRP will only affect the diffraction pattern of a small cluster of object points that share the same support.Our proposed relighting method is realized in 3 stages.First, we derive the WRP based on the mathematical framework in [1] that described the relationship between the object points in a 3D scene, the field distribution on WRP   , w u x y , and the hologram   , u x y .These three entities are assumed to have the same horizontal, as well as vertical extents of X and Y units.-Thecomplex wavefront contributed by the object points on the WRP is given by (Eq.(1).
where 0 j xX  and 0 j yY  are the horizontal and vertical positions of the jth object point.j a and are the amplitude of the 'jth' object point and its distance from the WRP, respectively.j d is the perpendicular distance from the jth object point to the WRP and  is the wavelength of the reference light.As the object scene is very close to the WRP, the diffracted beam of each object point only covers a small square window of size WW  (the dotted window in Fig. 1).As such, Eq. ( 1) can be rewritten as (Eq. ( 2).
  where Next, the WRP is expanded to a hologram   , u x y given by (Eq.( 3). where denote the forward and inverse fast Fourier transform (FFT), respectively.
2 exp , exp is a fixed impulse function for a given separation w z between the WRP and the hologram.From Eq. (3), the inverse process projecting the hologram to the WRP can be determined as (Eq.(4).
In the second stage, the WRP obtained in Eq. ( 4) is modulated with the relighting image (RI)   , G x y that simulates a given relighting condition.For example, the RI in Fig. 2 emphasizes the intensity within a circular region around the center of the scene, creating the effect of a spotlight.The relit WRP is given by (Eq.(5). Subsequently,

 
, L w u x y is expanded to a hologram as (Eq.( 6) The complete relighting process of our proposed method involves 4 FFT operations (2 in Eq. ( 4) and 2 in Eq. ( 6)), which constitutes to the majority of arithmetic operations.The pair of terms,   F can be pre-calculated in advance and stored in a look up table (LUT), and hence not count towards the computation load.With a typical PC and a GPU, the 4 FFTs can be executed in less than 20ms.The computation time on the rest of the process, comprising of multiplication between pairs of 2D arrays (e.g.Equation ( 5)), is negligible.

Experimental results
Our proposed hologram relighting method is demonstrated with the double depth image shown in Fig. 3(a).The image is evenly partitioned into a left side and a right side, located at 0.55m and 0.6m from the hologram, respectively.A digital Fresnel hologram is generated with the 'point-light' method described in [10].The wavelength of the optical beam and the pixel size of the hologram are 650nm and 7um , respectively.A relighting image   , G x y shown in Fig. 3(b) is employed to simulate the directional illumination emerging from the upper right corner.  , G x y is divided into an illuminated (the white area) and a shadow (the grey area) regions that are separated by a sharp boundary.When relighting is not applied, the numerical reconstructed images at the two depth planes are shown in Figs.3(c) and 3(d).When either side of the reconstructed image is in focus, it is a good recovery of the original content.Next we relight the hologram directly with   , G x y by multiplying the two images on a pixel by pixel basis.A hologram   , u x y , after direct relighting with an image   , G x y , is given by (Eq.( 7) The reconstructed images of   , L D u x y at the two depth plane are shown in Fig. 4(a) and 4(b).We observe that the relighting effect is not totally in line with the relighting image.Notably, the boundary between the illuminated and the shadow regions are fuzzy, and the area around it is heavily contaminated with slanting bars.The defects exhibited in Figs.4(a 4), and multiplied with the relighting image   , G x y .Subsequently, the result is expanded into a relit hologram based on Eq. ( 6).The reconstructed images of the relit hologram at the two depth planes are shown in Figs.5(a) and 5(b).We observe that the relighting effect is in good agreement with the relighting image, with a clear boundary between the illuminated and the shadow regions.To further illustrate our proposed method, we generated the digital hologram   , u x y of a hemisphere which is rendered with the texture of the earth image as shown in Fig. 6(a).The hemisphere has a radius of 0.005m, with the tip located at 0.001m and 0.3m from the WRP and the hologram, respectively.A real, off-axis hologram   where   RE  denotes the real part of a complex variable.The real, off-axis hologram is displayed on a liquid crystal on silicon (LCOS) modified from the Sony VPL-HW15 Bravia projector having a horizontal and vertical resolution of 1920 and 1080, respectively, and a dot-pitch of 7μm .The optical reconstructed image displayed on the LCOS device is shown in Fig. 6(b).Subsequently, we applied our proposed method to relight the hologram   , u x y with the image in Fig. 2. The latter is translated horizontally in a back and forth manner to generate the effect of a panning spotlight.Each relit hologram (corresponding to a particular spotlight position) is then converted into a real hologram based on Eq. ( 8), and reconstructed on the LCOS display.A single frame excerpt of the optical reconstructed animation clip (Media 1) is shown in Fig. 6(c).The clip is showing 25 frames per second.It can be seen from the excerpt, as well as in the animation clip, that the effect of the panning spotlight is correctly generated in the sequence of reconstructed images.

Conclusion
This paper reports a fast method for relighting a digital hologram without the need of regenerating the latter from the original object scene.We note that traditional photographic relighting techniques based on the 'point' process is not applicable on a hologram.To overcome this problem, we project the digital hologram to a hypothetic WRP that is inserted near to the object scene.Due to the close proximity, the wavefront of each object point is occupying a small neighborhood on the WRP.As such, relighting can be conducted by modulating the WRP with a relighting function, and expanded the modified WRP to a full hologram.Our proposed method is realized with the GPU, and the time taken to relight a 2048x2048 hologram is less than 20ms, equivalent to a rate of over 50 frames per second.Experimental results reveal that the reconstructed images of holograms that are processed with our proposed method are correctly relit with the target optical effects.In our evaluation, we have only adopted some simple relighting effects for synthesizing different kinds of illumination, mainly to demonstrate the effectiveness of our approach.However, it is evident that the current framework can be easily applied to handle more sophisticated effects, such as the image-based and the geometry-based relighting.

Fig. 1 .
Fig. 1.Spatial relation between the object point, the WRP, and the hologram.

Fig. 3 .
Fig. 3. (a) Scene image evenly divided into a left and a right sections, positioned at 0.55m and 0.6m from the hologram plane, respectively.(b) Relighting image simulating the directional illumination emerging from the upper right corner.(c),(d) Numerical reconstructed image of the digital hologram representing the image in Fig. 3(a), at a distance of 0.55m and 0.6m, respectively.
) and 4(b) are expected, as direct modification of a small part of the hologram will change the diffraction waves contributed by the entire object scene instead of localizing in the neighborhood of the modified area.To overcome this problem, we applied our proposed method to relight the digital hologram.The latter is first converted to the WRP

Fig. 4 .
Fig. 4. Numerical reconstructed image of the digital hologram that has been directly relit with the image in Fig. 3(b), at a focal distance of 0.55m and 0.6m, respectively.

Fig. 5 .
Fig. 5. Numerical reconstructed image of the digital hologram (relit with our proposed method based on the relighting image in Fig. 3(b) representing the image in Fig. 3(a), at a focal distance of 0.55m and 0.6m, respectively.

,
H x y is generated by adding a planar reference wave   Ry (illuminating at an inclined angle 1.2 o on the hologram) to   , u x y , and taking the real part of the result to give (Eq.(8)

Fig. 6 .
Fig. 6.(a) Hemisphere rendered with the texture of the earth image.(b) Optical reconstructed image of the hologram representing the hemisphere shown in (a).(c) Single frame excerpt of the optical reconstructed image of the hologram representing the hemisphere in (a), which has been relit with the spotlight image shown in Fig. 2.