Skip to main content
Log in

Optimized camera handover scheme in free-viewpoint video streaming

Multimedia Systems Aims and scope Submit manuscript

Abstract

Free-viewpoint video (FVV) is a promising approach that allows users to control their viewpoint and generate virtual views from any desired perspective. The individual user viewpoints are synthesized from two or more camera streams and correspondent depth sequences. In case of continuous viewpoint changes, the camera inputs of the view-synthesis process must be changed in a seamless way, to avoid the starvation of the viewpoint synthesizer algorithm. Starvation occurs when the desired user viewpoint cannot be synthesized with the currently streamed camera views, and thus, the FVV playout interrupts. In this paper, we propose three different camera handover schemes (TCC, MA, and SA) based on viewpoint prediction to minimize the probability of playout stalls and find the trade-off between the image quality and the camera handover frequency. Our simulation results show that the introduced camera switching methods can reduce the handover frequency with more than 40%, and hence, the viewpoint synthesis starvation and the playout interruption can be minimized. By providing seamless viewpoint changes, the quality of experience can be significantly improved, making the new FVV service more attractive in the future.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23

References

  1. Chiariglione, L., Szabó, C.A.: Multimedia communications: technologies, services, perspectives, Part II: applications, services and future directions. Infocommun J VI(3), 51–59 (2014)

  2. Levoy, M., Hanrahan, P.: Light field rendering. In: SIGGRAPH ‘96 Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, August, pp. 31–42 (1996)

  3. Fehn, C.: Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV. In: Proc. of SPIE, vol. 5291, Stereoscopic Displays and Virtual Reality Systems, pp. 93–104 (2004)

  4. Smolic, A.: 3D video and free viewpoint video-From capture to display. Pattern Recog. 44(9), 1958–1968 (2011)

  5. Zilly, F., et. al.: Real-time generation of multi-view video plus depth content using mixed narrow and wide baseline. J. Vis. Commun. Image Rep. 25(4), 632–648 (2014)

  6. Kuster, C., Popa, T., Zach, C., Gotsman, C., Gross, M., Eisert, P., Hornegger, J., Polthier, K.: Freecam: A hybrid camera system for interactive free-viewpoint video. In: Proceedings of vision, modeling, and visualization (VMV), pp 17–24 (2011)

  7. Hilton et al.: 3D-TV production from conventional cameras for sports broadcast. IEEE Tr. Broadcast. 57(2), 462–476 (2011)

  8. Mueller, K., Merkle, P., Smolic, A., Wiegand, T.: Multiview coding using AVC. MPEG2006/m12945, 75th MPEG meeting, Bangkok, Thailand (2006)

  9. Merkle, P., Smolic, A., Mueller, K., Wiegand, T.: Efficient prediction structures for multiview video coding. IEEE Trans. Circ. Syst. Video Technol. Spec. Issue Multiview Video Coding 3DTV 17(11), 1461–1473 (2007)

  10. Merkle, P., Morvan, Y., Smolic, A., Farin, D., Muller, K., Wiegand, T.: The effects of multiview depth video compression on multiview rendering. Signal Process. Image Commun. 24(1), 73–88 (2009)

    Article  Google Scholar 

  11. MSR 3D Video Dataset. http://research.microsoft.com/en-us/downloads/5e4675af-03f4-4b16-b3bc-a85c5bafb21d/

  12. Su, G.M., Lai, Y.C., Kwasinski, A., Wang, H.: 3D video communications: challenges and opportunities. Int. J. Commun. Syst. 24/10, 1261–1281 (2011)

    Article  Google Scholar 

  13. Sullivan, G.J., Ohm, J.-R., Han, W.-J., Wiegand, T.: Overview of the high efficiency video coding (HEVC) Standard. IEEE Trans. Circ. Syst. Video Technol. 22(12), 1649–1668 (2012)

  14. Tech, G., Chen, Y., Müller, K., Ohm, J.-R., Vetro, A., Wang, Y.-K.: Overview of the multiview and 3D extensions of high efficiency video coding. IEEE Trans. Circ. Syst. Video Technol. 26(1), 35–49 (2015)

  15. Sullivan, G.J., Boyce, JM, Chen, Y., Ohm, J.-R., Segall, C.A., Vetro, A.: Standardized extensions of high efficiency video coding. IEEE J. Select. Topics Signal Process. 7(6), 1001–1016 (2013)

  16. Domański, M., Gotfryd, M., Wegner, K.: View synthesis for multiview video transmission. In: The 2009 International Conference on Image Processing, Computer Vision, and Pattern Recognition, Las Vegas, USA, pp. 1–4 (2009)

  17. Jo, S., Lee, D., Kim, Y., Yoo, C.: Development of a simple viewpoint video system. In: IEEE International Conference on Multimedia and Expo, Hannover, Germany, pp. 1577–1580 (2008)

  18. Kimata, H., Shimizu, S., Kunita, Y., Isogai, M., Kamikura, K., Yashima, Y.: Real-time MVC viewer for free viewpoint navigation. In: 2008 IEEE International Conference on Multimedia and Expo, Hannover, Germany, pp. 1437–1440 (2008)

  19. Starck, J., Kilner, J., Hilton, A.: A free-viewpoint video renderer. J. Gr. GPU Game Tools. 14(3), 57–72 (2009)

    Article  Google Scholar 

  20. Zefeng, N., Tian, D., Bhagavathy, S., Llach, J., Manjunath, B.S.: Improving the quality of depth image based rendering for 3D Video systems. In: Conf. on Image Processing (ICIP), 7–10 Nov. 2009, pp. 513–516 (2009)

  21. Ndjiki-Nya, P., et al.: Depth image-based rendering with advanced texture synthesis for 3-D video. IEEE Trans. Multimedia 13(3), 453–465 (2011)

  22. Farid, M.S., et al.: Depth image based rendering with inverse mapping. In: Multimedia Signal Processing (MMSP), 2013 IEEE 15th International Workshop, Pula, Italy, pp. 135–140 (2013)

  23. Schmeing, M., Jiang, X.: Depth image based rendering: a faithful approach for the disocclusion problem. In: 2010 3DTV-conference: the true vision—capture, transmission and display of 3D video, tampere, Finland, pp. 1–4 (2010)

  24. Merkle, P., Morvan, Y., Smolic, A., Farin, D., Müller, K., de With, P.H.N., Wiegand, T.: The effects of multiview depth video compression on multiview rendering. Signal Process. Image Commun. 24(1–2), 73–88 (2009)

  25. Bosc, E., et al.: Towards a new quality metric for 3-D synthesized view assessment. IEEE J. Select. Topics Signal Process. 5(7), 1332–1343 (2011)

  26. Farid, M.S., Lucenteforte, M., Grangetto, M.: Edge enhancement of depth based rendered images. In: 2014 IEEE International Conference on Image Processing (ICIP), Paris, pp. 5452–5456 (2014)

  27. Tanimoto, M., Fujii, T., Tehrani, et al.: Reference software for depth estimation and view synthesis. In: MPEG/M15377, Archamps, France (2008)

  28. Huszák, A.: Predictive multicast group management for free viewpoint video streaming. In: International Conference on Telecommunications and Multimedia (TEMU 2014), ISBN 978-1-4799-3199-6, Heraklion, Greece (2014)

  29. Han, Z., Dai, Q.: A new scalable free viewpoint video streaming system over IP network. In: Acoustics, Speech and Signal Processing, ICASSP 2007, pp. II-773-II-776 (2007)

  30. Petrovic, G., de With, P.H.N.: Near-future streaming framework for 3D-TV applications. In: Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), pp. 1881–1884 (2006)

  31. Huszák, A.: Optimization of distributed free-viewpoint video synthesis. In: 3DTV-conference: the true vision—capture, transmission and display of 3D Video (3DTV-CON), Budapest, Hungary (2014)

  32. Orman, N., Kim, H., Sakamoto, R., Toriyama, T., Kogure, K., Lindeman, R.: GPU-based optimization of a free-viewpoint video system. In: ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, Redwood City, CA, USA (2008)

  33. Kurutepe, E., Aksay, A., Bilen, C., Gurler, C. G., Sikora, T., Akar, G. B., Tekalp, A. M.: A standards-based, flexible, end-to-end multi-view video streaming architecture. In: Proc. Int. Packet Video Workshop, Lausanne, Switzerland, pp. 302–307 (2007)

  34. Gürler, C.G., Görkemli, B., Saygili, G., Tekalp, A.M.: Flexible transport of 3-D video over networks. Proc. IEEE. 99(4), 694–707 (2011)

    Article  Google Scholar 

  35. De Raffaele, C., Debono, C.J.: A comparison of the performance of prediction techniques in curtailing uplink transmission and energy requirements in mobile free-viewpoint video applications. Int. J. Adv. Telecommun. 4(1 & 2), 1–11 (2011)

  36. Hamza, A., Hefeeda, M.: Adaptive streaming of interactive free viewpoint videos to heterogeneous clients. In: 7th International Conference on Multimedia Systems (MMSys ‘16), Article No. 10. ACM, New York, NY (2016)

  37. Hoßfeld, T., Schatz, R., Krieger, U., Fischbach, K., Krieger, U.: QoE of YouTube video streaming for current Internet transport protocols. Measure. Model. Eval. Comput. Syst. Depend. Fault Toler. 8376, 136–150 (2014)

    Google Scholar 

  38. Van Kester, S., Xiao, T., Kooij, R.E., Brunnstrom, K., Ahmed, O.K.: Estimating the impact of single and multiple freezes on Video quality. In: Proc. SPIE 7865 Human Vision and Electronic Imaging XVI (2011)

  39. J. Kalman, https://sourceforge.net/projects/jkalman/

  40. http://www.hit.bme.hu/~huszak/publ/Optimized_Cam_src.rar

  41. http://www.fujii.nuee.nagoya-u.ac.jp/multiview-data/

Download references

Acknowledgements

The authors are grateful for the support of the Hungarian Academy of Sciences through the Bolyai János Research Fellowship.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Árpád Huszák.

Additional information

Communicated by S. Kopf.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huszák, Á., Háló, B. Optimized camera handover scheme in free-viewpoint video streaming. Multimedia Systems 24, 241–256 (2018). https://doi.org/10.1007/s00530-017-0537-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-017-0537-x

Keywords

Navigation