High-speed full analytical holographic computations for true-life scenes

: We develop a novel method to generate hologram of three-dimensional (3D) textured triangle-mesh-model that is reconstructed from ordinary digital photos. This method allows analytically encoding the 3D model consisting of triangles. In contrast to other polygon based holographic computations, our full analytical method will free oneself from the numerical error that is in the angular spectrum due to the Whittaker-Shannon sampling. In order to saving the computation time, we employ the GPU platform that is remarkably superior to the CPU’s. We have rendered a true-life scene with colored textures as the first demo by our homemade software. The holographic reconstructed scene possesses high performances in many aspects such as depth cues, surface textures, shadings, and occlusions, etc. The GPU’s algorithm performs hundreds of times faster than those of CPU.


Introduction
3D imaging attracts increasing attention nowadays [1].3D data for real existent or nonexistent objects can be obtained more conveniently since the information technology made great progress during the past decades [2,3].They have been widely used in games, movies, virtual reality, reverse engineering, etc.However, the existing two-dimensional display limits the performance.One of the promising technologies for true 3D display is computer generated holograms (CGHs) [4].It can use the prescribed 3D object data stored in computer to produce wavefronts with amplitude and phase distribution for creating the most accurate depth cues of 3D objects [5].
However, the holographic computation is time consuming.Researchers have developed fast algorithms to accelerate the calculation [6,7].These methods treat 3D models as a combination of individual self-luminous points.The object wave distribution on the hologram is calculated for each point and then superimposed.It requires a large number of points to achieve the solid effect, and enormous memory size for sampling [7].Alternatively, 3D objects may be constructed of planar segments whose elements number is much less than that of point sets.In this way, the time will decrease significantly.In these methods, the numerical algorithm-fast Fourier transform (FFT), with the extra interpolated should be employed for each plane [8][9][10].Recently, a triangle-mesh-based algorithm is proposed to further reduce the computational time [11,12].The analytical angular spectrum of tilted triangle is calculated in frequency domain and a single FFT is required for the whole wave field.However, the uses of FFT will introduce numerical errors when the object goes beyond a certain distance.This limitation comes from the Whittaker-Shannon sampling theorem [5,[13][14][15][16][17].As a result, the diffraction field of the triangular aperture becomes noisy when the object moves away from the hologram [17].Such noises will mainly affect the surface performances (e.g.introducing additional artifacts) in the holographic reconstruction.The additional artifacts will bring a blur defocus feeling and weaken the quality of the reconstruction scene.In addition, if the hologram becomes large, the huge number of sampling points for FFT will bring some calculation problems [18].Using analytical Fourier CGH may save the last FFT, but it will also suffer from the short depth-of-field because of the limitation of aperture and focus depth of lens [12].
On the other hand, people use high-class hardware to accelerate the computation speed.For example, a MIT group has developed a special purpose computational board for holographic rendering [19].They achieved a record that the implementation was 50 times faster than workstation of the time.Most recently, some researchers presented parallel algorithms based on commodity graphic processing units (GPUs) [20][21][22].These approaches, using GPGPU (General-Purpose computation on GPU) techniques, can accelerate the CGH calculation in an inexpensive way.
In this paper, we derive the analytical formulas to describe the diffraction field of 3D truelife scenes.Since there no need for FFT, we can get rid of the numerical errors from the discrete algorithm.In addition, we propose a simple and useful phase adjustment technique to remove the visible mesh edges that appear in previous analytical polygon-based methods.Therefore, the smooth reconstructed surface, as well as the high quality 3D display, can be accomplished.By using CUDA [23], another GPU technique, the algorithm performs hundreds of times faster than those of CPUs.

Analytical diffraction field of 3D triangle-mesh model
In the common computer graphics field, 3D surface models are represented by polygons, for instance, triangles, as shown in Fig. 1(a).If the diffraction field emitted from each triangle of the 3D model can be calculated explicitly on the hologram plane, the whole objects field distribution will be obtained through their linear superposition., and where ( c x , c y , c z ) is the center of mass of the triangle in global coordinate system, and l z on the triangle facet is 0. Let three vertices of the triangle be 1, 2, 3 ( , , ), coordinate system, and ( , , 0), x y 1, 2, 3 i = in the local coordinate system.Consider the diffraction of monochromatic plane wave by the finite triangular-shape aperture in an infinite opaque screen, as shown in Fig. 1

(b). The wave propagates along the axis
where 2 / k π λ = and λ is the wavelength; is the distance between the points within triangle and the hologram pixels.( , ) H H x y denotes the hologram coordinate system.Combined by Eq. ( 1), it can be found that the quantity r is a square root function with the local coordinate variables ( , ) l l x y .Therefore, if the assumption and is satisfied, then r can be expanded into: r x x r y y r z x r x x r y y r z y x y r r r r where ( ) ( ) is the distance between the triangular mass center and the hologram pixel.In fact, Eq. ( 3) can be regarded as the modified paraxial approximation.It indicates that the linear scale of each triangle should be much less than the distance between the mass center of triangle and the hologram plane.This restriction can be easily satisfied as the scene usually contains numbers of triangles and each triangle is small.In this way, Eq. ( 2) becomes, ( ) where ( ) . Note that Eq. ( 5) is similar in form to the well known Fresnel diffraction between two parallel planes [5].As we know the Fresnel diffraction formula can be simplified to Fraunhofer form when an infinity aperture lens set immediately in front of the object [5].Numerous experiment results of Fraunhofer holography reveal that the quadratic phase factor within Fresnel integral does not affect the reconstruction performance [5].So in the CGH, the quadratic phase factor ( , ) l l Q x y in Eq. ( 5) can also be discarded but would not affect the reconstruction result.Thus, Eq. ( 5) will be simplified as where [ ] where  6) and ( 7), we can compute the diffraction field of an arbitrary tilted triangle analytically.The reconstruction results of a slanted triangle are presented in Fig. 2 for verification purpose.The reconstructions at different distances reveal that the different part of a triangle is focused while other parts are defocused.For example, in Fig. 2(b), point A is focus but point B and C are blur, when we focus on plane I where point A locates; in Fig. 2(c), point B is focus but point A and C are blur, when we focus on plane II where point B locates.This is a very important result, as it allows us to generate the hologram directly pixel by pixel without FFT, and easily to perform

Texture mapping, occlusion, and phase adjustment
To achieve a better performance, some specifics should be taken into account.For example, we adopt a modified Lambert brightness to avoid the unexpected shading [10].Another important feature is texture of the whole scene.3D model with texture will bring us a more realistic sense than those without texture.In our method, the amplitude of each triangle is a constant that is proportional to the color of the triangular mass center.It requests the texture of each triangle is as color-similar as possible.Otherwise, one should subdivide the triangle into small ones before extracting the texture.Hidden-surface removal is a necessary and troublesome issue since an object may occlude or partially overlapping by another one.Before coding, we let all the triangles be projected onto the hologram plane, and then sort them according to their distance.The shielded rear triangles are discarded.For nowadays the bandwidth of SLM is narrow, this method is quite well for our situation.One problem of using analytical formulas is that there are some visible mesh-edges in the reconstruction.This is because of the interferences between the diffraction fields of the same edge that belongs to two conjoint triangles.In other words, we record two virtual triangular edges with an additional phase difference ( z δ ) instead of one real edge.Here, we develop a phase adjustment technique to smooth such visible edges.Our method only needs to adjust the z component of mass center of each triangle ( c z ) to an integral multiple of wavelength.There are two reasons.It is the fact that, when we reconstruct the scene, the view angle is narrow along the z axis due to the narrow SLM bandwidth nowadays.Thus, only the z component of the phase adjustment is considered.Another is that, the light intensity of the triangular edge is equal to ( )  , where α relates to the response of coherent system to sharp triangular edges, with an approximate value of 0.5 at the edge [5].When z δ is an integer multiple of the wavelength, it yields ~1 I .It means that the intensity of triangular edge is almost the same as that inside the triangle, and thus, the visible edges is smoothed.In practice, we let all the z components of mass center of triangles (zc) to be an integer multiple of the wavelength, so that z δ is also the integer multiple of the wavelength.
In addition, this simple phase adjustment can be easily integrated into our analytical algorithm, and make us employ using the GPU platform.Since the adjustment is wavelength order of magnitude, it is too slight to affect the reconstruction positions of the macroscopical triangle.The reconstructed scene shown below will demonstrate how the phase adjustment works well.

Parallel algorithm and implementation on CUDA
To accelerate the computation in an inexpensive way, we consider the parallel algorithm based on GPU.The CUDA compiler is employed as a programming environment.It can compile a C-like language source code for parallel application on the GPU [23].Data communication is always a troublesome problem in the parallel calculation.In our full analytical encoding method, however, the sampling points on the hologram can be calculated individually, leading that the data communication time could be extremely decreased.We use a CPU server (Dell PE2900, Quad-Core Intel Xeon Pro X5460 2X3.16GHz) and a graphics card with GPU chipset (GeForce GTX 285) for comparison.From the Table 1, it can be seen that while the triangles is 120, we can achieve a video rate.The performance in the case of huge number is more encouraging.When the number of triangles is more than three thousand, the accelerated effect by CUDA is amazing.The CUDA algorithm runs more than 500 times faster than that of serial algorithm in CPU.

Results
To calculate the hologram for true life, we should firstly bring the real 3D scene into computer.The image-based modeling and photogrammetry software-ImageModeler [25] is adopted to generate 3D models from ordinary 2D digital photos.This software provides an inexpensive, flexible and fast way to record the real scene and even for a living.Figures 3(a) and 3(c) are the digital photos we take from two arbitrary different aspects for modeling.Needless to input information about the camera, we use the ImageModeler for calibration and modeling.After that, photorealistic texture maps of the model are automatically extracted from the pixel information of the source images.Till then, we obtain the textured 3D scene model.Finally, we encode them in the full analytical CGH algorithm as proposed above.The 3D scene model here contains 22408 triangles, and the resolution of CGH is 4096 3072 × , while pixel size is 2 m µ .The scene is at a distance of 15cm from the hologram, and the pawns in back are 1.76cm apart from the king in front.To verify the texture technique, numerical colored reconstruction is employed.The textured scene model is separated into three primary color components; i.e., red, green, and blue.Each monochromatic target is encoded and reconstructed independently.The final colored result is merged from these three monochromatic reconstructions.Figure 3 shows five aspects of the reconstructed scene to demonstrate the enhancement on the realistic effect.The reconstructed views in Figs.Thanks to our phase adjustment technique, the artificial edges now disappear in our reconstruction, so that the reconstructed scenes perform a well solid effect.Meanwhile, the objects retain their positions and are not affected by the phase adjustment.The shading technique used here also adds to the impression of surface curvature.
In Fig. 4, simulation results of two cases focused on the king [Fig.4(a)] and pawn [Fig.4(b)] are presented.It is clear that the pawn becomes defocus blur when focusing on the king, and vice versa.This defocus effect further enhances the depth sensation, and further demonstrates that the 3D true-life scene is faithfully reconstructed.

Conclusion
We proposed a novel method for high-speed hologram generation of 3D real scenes.A full analytic theory is derived to directly describe the object field on the hologram plane.FFT is no need, and thus, we can get rid of some limitation from it.To remove the artificial visible edges in analytical Fourier transforms for polygon based model, we develop a simple but useful phase adjustment technique.Finally we employ the GPU to accelerate the calculation, and obtain a five hundred times faster than the serial CPU algorithm.One limitation of our theory is the modified paraxial approximation.But it seems that this limitation can be further overcome by using a curved array of spatial light modulators [26].
Since we just use a single GPU and achieve such an amazing speed, we believe that our full analytical algorithm based on Multi-GPU will make the real time display come true in the future.Our method described above is suitable for 3D data from not only image-based modelings, but also any other ways such as CT, NMR, laser scanner, 3D design software, and so on.Furthermore, if we have two different aspects synchronized videos, we can record and display a motional scene.

Fig. 1 .
Fig. 1.(a) Local coordinate system of triangle mesh object and the global coordinate system.(b)Diffraction by one of the trianglesFor the yellow triangle of the object [Fig.1(a)], the local coordinate system ( l x , l y , l z ) and the global coordinate system ( g x , g y , g z ) can be related by the rotation matrix: Rayleigh-Sommerfeld diffraction integral[5], the diffraction field of the yellow triangle on the hologram [see Fig.1(b)] can be expressed,

#
115096 -$15.00USD Received 31 Jul 2009; revised 20 Jan 2010; accepted 29 Jan 2010; published 2 Feb 2010 (C) 2010 OSAthe theory on the GPU's parallel platform.Finally, the whole objects field can be obtained through the linear superposition of the diffraction field per triangle.

Fig. 2 .
Fig. 2. (a) A schematic illustration of a slanted triangle reconstruction.(b) and (c) are reconstruction results at the plane I and II where point A and B locate.

Fig. 3 .
Fig. 3. Images for different aspects of the scene: (a), (c) original photos of the true-life scene taken by digital camera.(b), (d) and (e)-(g) numerical reconstructions of the scene from different aspects.
3(b) and 3(d) are consistent with Figs.3(a) and 3(c), respectively.Moreover, since we obtained the 3D scene model, we can reconstruct not only the same aspects as those of photos but also other aspects, as shown in Figs.3(e)-3(g).The occlusion effects between chesses cause a strong depth sensation.For instance, the king in front shields a part of the right bishop in Fig. 3(e); when the scene rotates a slight clockwise angle, the king does not shield any chesses [Fig.3(f)]; when it rotates more, the king shields the back pawn.It demonstrates our occlusion methods works quite well.

Fig. 4 .
Fig. 4. Images of numerical reconstruction: (a) focus on king, (b) focus on pawn the quadratic phase factor; and'