Abstract
Free-viewpoint video (FVV) is a promising approach that allows users to control their viewpoint and generate virtual views from any desired perspective. The individual user viewpoints are synthesized from two or more camera streams and correspondent depth sequences. In case of continuous viewpoint changes, the camera inputs of the view-synthesis process must be changed in a seamless way, to avoid the starvation of the viewpoint synthesizer algorithm. Starvation occurs when the desired user viewpoint cannot be synthesized with the currently streamed camera views, and thus, the FVV playout interrupts. In this paper, we propose three different camera handover schemes (TCC, MA, and SA) based on viewpoint prediction to minimize the probability of playout stalls and find the trade-off between the image quality and the camera handover frequency. Our simulation results show that the introduced camera switching methods can reduce the handover frequency with more than 40%, and hence, the viewpoint synthesis starvation and the playout interruption can be minimized. By providing seamless viewpoint changes, the quality of experience can be significantly improved, making the new FVV service more attractive in the future.
References
Chiariglione, L., Szabó, C.A.: Multimedia communications: technologies, services, perspectives, Part II: applications, services and future directions. Infocommun J VI(3), 51–59 (2014)
Levoy, M., Hanrahan, P.: Light field rendering. In: SIGGRAPH ‘96 Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, August, pp. 31–42 (1996)
Fehn, C.: Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV. In: Proc. of SPIE, vol. 5291, Stereoscopic Displays and Virtual Reality Systems, pp. 93–104 (2004)
Smolic, A.: 3D video and free viewpoint video-From capture to display. Pattern Recog. 44(9), 1958–1968 (2011)
Zilly, F., et. al.: Real-time generation of multi-view video plus depth content using mixed narrow and wide baseline. J. Vis. Commun. Image Rep. 25(4), 632–648 (2014)
Kuster, C., Popa, T., Zach, C., Gotsman, C., Gross, M., Eisert, P., Hornegger, J., Polthier, K.: Freecam: A hybrid camera system for interactive free-viewpoint video. In: Proceedings of vision, modeling, and visualization (VMV), pp 17–24 (2011)
Hilton et al.: 3D-TV production from conventional cameras for sports broadcast. IEEE Tr. Broadcast. 57(2), 462–476 (2011)
Mueller, K., Merkle, P., Smolic, A., Wiegand, T.: Multiview coding using AVC. MPEG2006/m12945, 75th MPEG meeting, Bangkok, Thailand (2006)
Merkle, P., Smolic, A., Mueller, K., Wiegand, T.: Efficient prediction structures for multiview video coding. IEEE Trans. Circ. Syst. Video Technol. Spec. Issue Multiview Video Coding 3DTV 17(11), 1461–1473 (2007)
Merkle, P., Morvan, Y., Smolic, A., Farin, D., Muller, K., Wiegand, T.: The effects of multiview depth video compression on multiview rendering. Signal Process. Image Commun. 24(1), 73–88 (2009)
MSR 3D Video Dataset. http://research.microsoft.com/en-us/downloads/5e4675af-03f4-4b16-b3bc-a85c5bafb21d/
Su, G.M., Lai, Y.C., Kwasinski, A., Wang, H.: 3D video communications: challenges and opportunities. Int. J. Commun. Syst. 24/10, 1261–1281 (2011)
Sullivan, G.J., Ohm, J.-R., Han, W.-J., Wiegand, T.: Overview of the high efficiency video coding (HEVC) Standard. IEEE Trans. Circ. Syst. Video Technol. 22(12), 1649–1668 (2012)
Tech, G., Chen, Y., Müller, K., Ohm, J.-R., Vetro, A., Wang, Y.-K.: Overview of the multiview and 3D extensions of high efficiency video coding. IEEE Trans. Circ. Syst. Video Technol. 26(1), 35–49 (2015)
Sullivan, G.J., Boyce, JM, Chen, Y., Ohm, J.-R., Segall, C.A., Vetro, A.: Standardized extensions of high efficiency video coding. IEEE J. Select. Topics Signal Process. 7(6), 1001–1016 (2013)
Domański, M., Gotfryd, M., Wegner, K.: View synthesis for multiview video transmission. In: The 2009 International Conference on Image Processing, Computer Vision, and Pattern Recognition, Las Vegas, USA, pp. 1–4 (2009)
Jo, S., Lee, D., Kim, Y., Yoo, C.: Development of a simple viewpoint video system. In: IEEE International Conference on Multimedia and Expo, Hannover, Germany, pp. 1577–1580 (2008)
Kimata, H., Shimizu, S., Kunita, Y., Isogai, M., Kamikura, K., Yashima, Y.: Real-time MVC viewer for free viewpoint navigation. In: 2008 IEEE International Conference on Multimedia and Expo, Hannover, Germany, pp. 1437–1440 (2008)
Starck, J., Kilner, J., Hilton, A.: A free-viewpoint video renderer. J. Gr. GPU Game Tools. 14(3), 57–72 (2009)
Zefeng, N., Tian, D., Bhagavathy, S., Llach, J., Manjunath, B.S.: Improving the quality of depth image based rendering for 3D Video systems. In: Conf. on Image Processing (ICIP), 7–10 Nov. 2009, pp. 513–516 (2009)
Ndjiki-Nya, P., et al.: Depth image-based rendering with advanced texture synthesis for 3-D video. IEEE Trans. Multimedia 13(3), 453–465 (2011)
Farid, M.S., et al.: Depth image based rendering with inverse mapping. In: Multimedia Signal Processing (MMSP), 2013 IEEE 15th International Workshop, Pula, Italy, pp. 135–140 (2013)
Schmeing, M., Jiang, X.: Depth image based rendering: a faithful approach for the disocclusion problem. In: 2010 3DTV-conference: the true vision—capture, transmission and display of 3D video, tampere, Finland, pp. 1–4 (2010)
Merkle, P., Morvan, Y., Smolic, A., Farin, D., Müller, K., de With, P.H.N., Wiegand, T.: The effects of multiview depth video compression on multiview rendering. Signal Process. Image Commun. 24(1–2), 73–88 (2009)
Bosc, E., et al.: Towards a new quality metric for 3-D synthesized view assessment. IEEE J. Select. Topics Signal Process. 5(7), 1332–1343 (2011)
Farid, M.S., Lucenteforte, M., Grangetto, M.: Edge enhancement of depth based rendered images. In: 2014 IEEE International Conference on Image Processing (ICIP), Paris, pp. 5452–5456 (2014)
Tanimoto, M., Fujii, T., Tehrani, et al.: Reference software for depth estimation and view synthesis. In: MPEG/M15377, Archamps, France (2008)
Huszák, A.: Predictive multicast group management for free viewpoint video streaming. In: International Conference on Telecommunications and Multimedia (TEMU 2014), ISBN 978-1-4799-3199-6, Heraklion, Greece (2014)
Han, Z., Dai, Q.: A new scalable free viewpoint video streaming system over IP network. In: Acoustics, Speech and Signal Processing, ICASSP 2007, pp. II-773-II-776 (2007)
Petrovic, G., de With, P.H.N.: Near-future streaming framework for 3D-TV applications. In: Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), pp. 1881–1884 (2006)
Huszák, A.: Optimization of distributed free-viewpoint video synthesis. In: 3DTV-conference: the true vision—capture, transmission and display of 3D Video (3DTV-CON), Budapest, Hungary (2014)
Orman, N., Kim, H., Sakamoto, R., Toriyama, T., Kogure, K., Lindeman, R.: GPU-based optimization of a free-viewpoint video system. In: ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, Redwood City, CA, USA (2008)
Kurutepe, E., Aksay, A., Bilen, C., Gurler, C. G., Sikora, T., Akar, G. B., Tekalp, A. M.: A standards-based, flexible, end-to-end multi-view video streaming architecture. In: Proc. Int. Packet Video Workshop, Lausanne, Switzerland, pp. 302–307 (2007)
Gürler, C.G., Görkemli, B., Saygili, G., Tekalp, A.M.: Flexible transport of 3-D video over networks. Proc. IEEE. 99(4), 694–707 (2011)
De Raffaele, C., Debono, C.J.: A comparison of the performance of prediction techniques in curtailing uplink transmission and energy requirements in mobile free-viewpoint video applications. Int. J. Adv. Telecommun. 4(1 & 2), 1–11 (2011)
Hamza, A., Hefeeda, M.: Adaptive streaming of interactive free viewpoint videos to heterogeneous clients. In: 7th International Conference on Multimedia Systems (MMSys ‘16), Article No. 10. ACM, New York, NY (2016)
Hoßfeld, T., Schatz, R., Krieger, U., Fischbach, K., Krieger, U.: QoE of YouTube video streaming for current Internet transport protocols. Measure. Model. Eval. Comput. Syst. Depend. Fault Toler. 8376, 136–150 (2014)
Van Kester, S., Xiao, T., Kooij, R.E., Brunnstrom, K., Ahmed, O.K.: Estimating the impact of single and multiple freezes on Video quality. In: Proc. SPIE 7865 Human Vision and Electronic Imaging XVI (2011)
J. Kalman, https://sourceforge.net/projects/jkalman/
Acknowledgements
The authors are grateful for the support of the Hungarian Academy of Sciences through the Bolyai János Research Fellowship.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by S. Kopf.
Rights and permissions
About this article
Cite this article
Huszák, Á., Háló, B. Optimized camera handover scheme in free-viewpoint video streaming. Multimedia Systems 24, 241–256 (2018). https://doi.org/10.1007/s00530-017-0537-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00530-017-0537-x