Skip to main content
SpringerLink
Log in

Proposed Framework for Evaluating Quality of Experience in a Mobile, Testbed-oriented Living Lab Setting

  • Published:
Mobile Networks and Applications Aims and scope Submit manuscript

Abstract

The framework presented in this paper enables the evaluation of Quality of Experience (QoE) in a mobile, testbed-oriented Living Lab setting. As a result, it fits within the shift towards more user-centric approaches in innovation research and aims to bridge the gap between technical parameters and human experience factors. In view of this, Quality of Experience is seen as a multi-dimensional concept, which should be considered from an interdisciplinary perspective. Although several approaches for evaluating perceived QoE have been proposed in the past, they tend to focus on a limited number of objective dimensions and fail to grasp the subjective counterparts of users’ experiences. We therefore propose a distributed architecture for monitoring network Quality of Service (QoS), context information and subjective user experience based on the functional requirements related to real-time experience measurements in real-life settings. This approach allows us to evaluate all relevant QoE-dimensions in a mobile context.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. http://amorg.aut.bme.hu/projects/symtorrent

  2. http://www.lcc.uma.es/∼pedro/mobile/Software/sympa.html

References

  1. Crisler K, Turner T, Aftelak A, Visciola M, Steinhage A, Anneroth M et al (2004) Considering the user in the wireless world. IEEE Commun Mag 42:56–62

    Article  Google Scholar 

  2. Haddon L, Mante E, Sapio B, Kommonen K-H, Fortunati L, Kant A (eds) (2005) Everyday innovators: researching the role of users in shaping ICT’s. Springer, Dordrecht

    Google Scholar 

  3. Cabral A (2007) Quality of experience: a conceptual essay. In: Wang W, Wang W (eds) IFIP international federation for information processing, volume 252, integration and innovation. Springer, Boston, pp 193–199

    Google Scholar 

  4. De Marez L, De Moor K (2007) The challenge of user- and QoE-centric research and product development in today’s ICT environment. Observatorio (OBS*) 1(3):1–22

    Google Scholar 

  5. Rickards T (2003) The future of innovation research. In: Shavinina LV (ed) The international handbook on innovation. Pergamon, Elsevier, Oxford, pp 1094–1100

    Chapter  Google Scholar 

  6. Trott P (2003) Innovation and market research. In: Shavinina LV (ed) The international handbook on innovation. Pergamon, Elsevier, Oxford, pp 835–844

    Chapter  Google Scholar 

  7. Deryckere T, Joseph W, Martens L, De Marez L, De Moor K (2008) A software tool to relate technical performance to user experience in a mobile context. Paper presented at WOMWOM 2008 International Symposium on a World of Wireless, Mobile and Multimedia Networks. http://doi.ieeecomputersociety.org/10.1109/WOWMOM.2008.4594902. Accessed 25 October 2008

  8. Kumar K (2005) A marriage made in QoE heaven. CED Magazine, July 2005, 37–39

  9. Nokia (2004) Quality of Experience (QoE) of mobile services: can it be measured and improved? White paper. http://www.nokia.com/NOKIA_COM_1/About_Nokia/Press/White_Papers/pdf_files/whitepaper_qoe_net.pdf. Accessed 3 October 2007

  10. Kilkki K (2008) Quality of experience in communications ecosystem. J Univers Comput Sci 14(5):615–624

    Google Scholar 

  11. De Marez L, Verleye G (2004) Innovation diffusion: the need for more accurate consumer insight. Illustration of the PSAP scale as a segmentation instrument. J Target Meas Anal Market 13(1):32–49

    Article  Google Scholar 

  12. Veryzer RW, Borja de Mozota B (2005) The impact of user-oriented design on new product development: an examination of fundamental relationships. J Prod Innovat Manag 22(2):128–143

    Article  Google Scholar 

  13. Väätäjä H (2008) Factors affecting user experience in mobile systems and services. Proceedings of the 10th international conference on Human-Computer Interaction with mobile devices and services, 551–551

  14. Knoche H, Papaleo M, Sasse MA, Vanelli-Coralli A (2007) The Kindest Cut: Enhancing the User Experience of Mobile TV through Adequate Zooming. Proceedings of the 15th ACM International conference on Multimedia, 87–96.

  15. De Koning TC, Veldhoven P, Knoche H, Rooij RE (2007) Of MOS and men: bridging the gap between objective and subjective quality measurements in mobile TV. In Creutzburg R, Takala JH, Cai J (Eds) Multimedia on Mobile Devices 2007: Proceedings of the SPIE, Volume 6507, 65070P

  16. Singh K, Schulzrinne H (2005) Peer-to-peer internet telephony using SIP. Proceedings of the international workshop on Network and operating systems support for digital audio and video, 63–68

  17. Verkasalo H (2008) Contextual patterns in mobile service usage. J Pers Ubiquit Comput 13(5):331–342

    Article  Google Scholar 

  18. Tsai CC, Lee G, Raab F, Norman GJ, Sohn T, Griswold WG, Patrick K (2007) Usability and feasability of PmEB: a mobile phone application for monitoring real time caloric balance. Mob Netw Appl 12(2–3):173–184

    Article  Google Scholar 

  19. Ballon P, Pierson J, Delaere S (2007) Fostering innovation in networked communications: test and experimentation platforms for broadband systems. In: Heilesen SB, Siggaard Jensen S (eds) Designing for networked communications—strategies and development. Hershey (PA), London, pp 137–166

    Google Scholar 

  20. Intille SS, Tapia EM, Rondoni J, Beaudin J, Kukla C, Agarwal S et al (2003) Tools for studying behavior and technology in natural settings. In: Dey AK, Schmidt A, McCarthy JF (eds) UbiComp 2003. Springer-Verlag, Berlin, pp 157–174

    Google Scholar 

  21. Følstad A (2008) Towards a living lab for the development of online community services. Electron J Virtual Organ Netw 10:47–58

    Google Scholar 

  22. Ponce de Leon M, Eriksson M, Balasubramaniam S, Donelly W (2006) Creating a distributed mobile networking testbed environment—through the Living Labs approach. Proceedings of the 2nd International IEEE/Create-Net Conference on Testbeds and Research Infrastructures for the Development of Networks and Communities, 135–139.

  23. Perkis A, Munkeby S, Hillestad OI (2006) A model for measuring quality of experience. Proceedings of the 7th Nordic Signal Processing Symposium (NORSIG 2006), 189–201

  24. Fehnert B, Kosagowsky A (2008) Measuring user experience—complementing qualitative and quantitative assessment. Proceedings of the 10th International conference on Human-Computer Interaction with mobile devices and services, 383–386

  25. Schmidt A, Strohbach M, van Laerhoven K, Friday A, Gellersen H-W (2002) Context acquisition based on load sensing. In: Boriello G, Holmquist LE (eds) Lecture notes in computer science 2498. Springer, Berlin, pp 333–350

    Google Scholar 

  26. Niitamo V-P, Kulkki S, Eriksson MH (2006) State-of-the-art and good practice in the field of living labs. Proceedings of the 12the International Conference on Concurrent Enterprising: Innovative Products and Services through Collaborative Networks, 349–357

  27. Henderson T, Anthony DA (2005) Measuring wireless network usage with the experience sampling method. Proceedings of the First Workshop on Wireless Network Measurements, Trentino, Italy

  28. Andrews M, Cao J, McGowan J (2006) Measuring human satisfaction in data networks. Proceedings of the 25th IEEE International Conference on Computer Communications, 1–12.

  29. Reichl P (2007) From ‘Quality-of-Service’ and ‘Quality-of-Design’ to ‘Quality-of-Experience’: A holistic view on future interactive telecommunication services. Proceedings of the 15th International conference on Software, Telecommunications and Computer Networks, 2007, 1–6.

  30. Lopez D, Gonzalez F, Bellido L, Alonso A (2006) Adaptive multimedia streaming over IP based on customer oriented metrics. Proceedings of the International Symposium on Computer Networks, 185–191.

  31. Van Ewijk A, De Vriendt J, Finizola L (2006) Quality of Service for IMS on fixed networks. White Paper. http://www1.alcatel-lucent.com/publications/abstract.jhtml?repositoryItem=tcm%3A172-850091635. Accessed 3 October 2007

  32. O'Neill TM (2002) Quality of experience and quality of service for IP video conferencing. White paper. http://hive2.hive.packetizer.com/users/h323forum/papers/polycom/QualityOfExperience+ServiceForIPVideo.pdf. Accessed 3 October 2007

  33. Siller M, Woods JC (2003) QoS arbitration for improving the QoE in multimedia transmission. Proceedings of the International Conference on Visual Information Engineering, 238–241

  34. Empirix (2001) Assuring QoE on next generation networks. Whitepaper. http://www.triple-play-news.com/voip/whitepapers/whitepaper_NGNT_AssuringQoE.pdf. Accessed 3 October 2007

  35. Soldani D (2006) Means and methods for collecting and analyzing QoE measurements in wireless networks. Proceedings of the International Symposium on a World of Wireless, Mobile and Multimedia Networks, 531–535

  36. Eberle W, Bougard B, Pollin S, Catthoor F (2005) From myth to methodology: cross-layer design for energy-efficient wireless communication. Proceedings of the 42nd Design Automation Conference, 303–308.

  37. Forlizzi J, Battarbee K (2004) Understanding Experience in interactive systems. Proceedings of the DIS04 Conference, 261–268

  38. Gaggioli A, Bassi M, Delle Fave A (2003) Quality of experience in virtual environments. In: Riva G, Davide F, Ijsselsteijn WA (eds) Being there: concepts, effects and measurement of user presence in synthetic environments. Ios, Amsterdam, pp 122–136

    Google Scholar 

  39. Alben L (1996) Defining the criteria for effective interaction design. Interactions 3(3):11–15

    Article  Google Scholar 

  40. Arhippainen L (2003) Capturing user experience for product design. In Laukkanen S, Sarpola S (Ed) Electronic Proceedings of the 26th Information Systems Research Seminar in Scandinavia (1–10)

  41. Vyas D, Van Der Veer GC (2005) Experience as ‘meaning’: creating communicating and maintaining in real-spaces. Proceedings of the Tenth IFIP TC13 International Conference on Human-Computer Interaction, 1–4

  42. Hassenzahl M, Tractinsky N (2006) User experience—a research agenda. Behav Inf Technol 25(2):91–97

    Article  Google Scholar 

  43. Blythe MA, Overbeeke K, Monk AF (2004) Funology: usability to enjoyment (Human-Computer Interaction). Kluwer Academic, Dordrecht

    Google Scholar 

  44. McNamara N, Kirakowski J (2005) Defining usability: quality of use or quality of experience? Proceedings of the Professional Communication Conference 2005, 200–204

  45. Abowd G, Mynatt E, Rodden T (2002) The human experience [of ubiquitous computing]. IEEE Pervasive Comput 1:48–57

    Article  Google Scholar 

  46. Roto V (2006) Web browsing on mobile phones—characteristics of user experience (Unpublished doctoral dissertation). University of Technology, Helsinki

    Google Scholar 

  47. Corrie B, Wong H-Y, Zimmerman T, Marsh S, Patrick AS, Singer J et al (2003) Towards quality of experience in advanced collaborative environments. Unpublished paper. http://www.andrewpatrick.ca/cv/WACE-2003-Corrie-et-al.pdf. Accessed 3 October 2007

  48. McNamara N, Kirakowski J (2006) Functionality, usability and user experience: three areas of concern. Interactions 13(6):26–28

    Article  Google Scholar 

  49. ITU-T (1994) ITU-T Recommendation E.800. Terms and definitions related to quality of service and network performance including dependability. International Telecommunication Union, CH-Geneva. ITU-T Recommendation E.800. Terms and definitions related to quality of service and network performance including dependability. International Telecommunication Union, CH-Geneva. Geneva: International Telecommunication Union

  50. Sullivan M, Pratt J, Kortum P (2008) Practical issues in subjective video quality evaluation: human factors vs. psychophysical image quality evaluation. Proceedings of UXTV 2008, 1–4

  51. da Silva AP, Varela M, de Souza e Silva E, Leão RM, Rubino G (2008) Quality assessment of interactive voice applications. Comput Netw 52(6):1179–1192

    Article  Google Scholar 

  52. Khirman S, Henriksen P (2002) Relationship between quality-of-service and quality-of-experience for public internet service. Unpublished paper. http://www.pamconf.net/2002/Relationship_Between_QoS_and_QoE.pdf. Accessed 3 October 2007

  53. Li-yuan L, Wen-an Z, Jun-de S (2006) The research of quality of experience evaluation method in pervasive computing environment. Proceedings of the 1st International Symposium on Pervasive Computing and Applications, 178–182

  54. Díaz A, Merino P, Rivas FJ (2008) Customer-centric measurements on mobile phones. Proceedings of the IEEE International Symposium on Consumer Electronics (ISCE 2008), 1–4

  55. Wheeler L, Reis H (1991) Self-recording of everyday life events: origins, types, and uses. J Personal 59(3):339–354

    Article  Google Scholar 

  56. Bolger N, Davis A, Rafaeli E (2003) Diary methods: capturing life as it is lived. Annu Rev Psychol 54(1):579–616

    Article  Google Scholar 

  57. Csikszentmihalyi M, Larson R (1987) Validity and reliability of the experience-sampling method. J Nerv Ment Dis 175(9):526–536

    Article  Google Scholar 

  58. Consolvo S, Harrison B, Smith I, Chen M, Everitt K, Froehlich J et al (2007) Conducting in situ evaluations for and with ubiquitous technologies. Int J Hum Comput Interact 22(1–2):103–118

    Article  Google Scholar 

  59. Mulder I, Velthausz D, Strating P, ter Hofte GH (2006) Bring the lab to the cities: experiences from two Dutch Living Labs. Proceedings of e-Social Science conference. http://www.ncess.ac.uk/events/conference/2006/papers/papers/MulderBringLabToCities.pdf. Accessed 7 October 2007

  60. De Moor K, Berte K, De Marez L, Joseph W, Deryckere T, Martens L (2009) User involvement in living lab research: experiences from an interdisciplinary study on future mobile applications. Proceedings of the Third International Seville Seminar on Future-Oriented Technology Analysis, 103–104

  61. Majumdar K, Das N (2005) Mobile user tracking using a hybrid neural network. Wirel Netw 11(3):275–284

    Article  MathSciNet  Google Scholar 

  62. Mäkelä J-P, Pahlavan K (2005) Performance of neural network handoff algorithm under varying mobile velocities. Proceedings of the 2005 Finnish Signal Processing Symposium (FINSIG’05), 42–45

  63. Schmitt J, Hollick M, Roos C, Steinmetz R (2008) Adapting the user context in realtime: tailoring online machine learning algorithms to ambient computing. Mobile Netw Appl 13(6):583–598

    Article  Google Scholar 

  64. Soldani D, Li M, Cuny R (2006) QoS and QoE Management in UMTS Cellular Systems. Wiley, Chichester

    Book  Google Scholar 

  65. Verkasalo H (2005) Handset-based monitoring of mobile customer behavior. Master’s Thesis Series. Networking Laboratory, Department of Electrical and Telecommunications. Helsinki: Helsinki University of Technology

  66. Corvaja R, Zanella A, Dossi M, Tontoli A, Zennaro P (2004) Experimental Performance of the Handover Procedure in a WiFi Network. Proceedings of the Seventh International Symposium on WIRELESS PERSONAL MULTIMEDIA COMMUNICATIONS (WPMC04) http://www.dei.unipd.it/∼zanella/PAPER/CR_2004/WPMC04_czdtz_WiFi_cr.pdf. Accessed 7 October 2007

  67. Becvar Z, Zelenka J, Bestak R (2006) Comparison of handovers In UMTS And WiMAX. Paper presented at the 6th International Conference ELEKTRO 2006. http://fireworks.intranet.gr/Publications/Fireworks_6CTUPB008a.pdf. Accessed 7 October 2007

  68. Hornsby A, Bangash S, Benchimol S, Defee I (2008) An approach to handover between DVB-H and Wi-Fi networks. Proceedings of the 2008 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting, 1–8

  69. Huang S-M, Wu Q, Lin Y-B, Yeh C-H (2006) SIP mobility and IPv4/IPv6 dual-stack supports in 3G IP multimedia subsystem: Research Articles. Wirel Commun Mob Comput 6(5):543–739

    Article  Google Scholar 

  70. Calvagna A, Di Modica G (2004) A user-centric analysis of vertical handovers. Proceedings of the 2nd ACM international workshop on Wireless mobile applications and services on WLAN hotspots, 137–146

  71. Kim HJ, Lee DH, Lee JM, Lee KH, Lyu W, Choi SG (2008) The QoE evaluation method through the QoS-QoE correlation model. Proceedings of the 2008 Fourth International Conference on Networked Computing and Advanced Information Management, 719–725

  72. Jain R, Chiu D, Hawe W (1984) A quantitative measure of fairness and discrimination for resource allocation in shared computer systems. Research Report. http://www.cs.wustl.edu/∼jain/papers/ftp/fairness.pdf. Accessed 20 November 2008

  73. Piamrat K, Ksentini A, Viho C, Bonnin J-M (2008) QoE-based network selection for multimedia users in IEEE 802.11 wireless networks. Proceedings of the 33rd IEEE Conference on Local Computer Networks, 388–394

  74. Bharatula NB, Tröster G (2006) On-body context recognition with miniaturized autonomous sensor button. Proceedings of 4 th GMM Workshop on Energieautarke Sensorik, 621–628

  75. Stägera M, Lukowiczb P, Tröster G (2007) Power and accuracy trade-offs in sound-based context recognition systems. Perv Mob Comput 3(3):300–327

    Article  Google Scholar 

  76. De Pessemier T (2008) Proposed architecture and algorithm for personalized advertising on iDTV and mobile devices. IEEE Trans Consum Electron 54(2):709–713

    Article  Google Scholar 

  77. Canalyis Ltd (2009) Smart phone market shows modest growth in Q3—But Apple and RIM hit record volumes. Press release, http://www.canalys.com/pr/2009/r2009112.htm. Accessed 3 November 2009

  78. Cooper JW (1998) The design patterns java companion, Online book, http://www.patterndepot.com/put/8/DesignJava.PDF. Accessed 20 November 2008

  79. http://www.tcpdump.org/. Accessed 20 October 2009

Download references

Acknowledgements

This work was supported by the interdisciplinary GR@SP-project (Bridging the gap between QoE and QoS), funded by IBBT (Interdisciplinary Institute for BroadBand Technology), a research institute founded by the Flemish Government in 2004 (www.ibbt.be). W. Joseph is a Post-Doctoral Fellow of the FWO-V (Research Foundation—Flanders).

Author information

Authors and Affiliations

  1. Research Group for Media and ICT, Department of Communication Sciences, Ghent University / IBBT, Korte Meer 7, 9000, Ghent, Belgium

    Katrien De Moor, Lieven De Marez & Gino Verleye

  2. Department of Information Technology, Ghent University / IBBT, G. Crommenlaan 8, 9050, Ghent, Belgium

    Istvan Ketyko, Wout Joseph, Tom Deryckere & Luc Martens

Authors
  1. Katrien De Moor
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Istvan Ketyko
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wout Joseph
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tom Deryckere
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lieven De Marez
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Luc Martens
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Gino Verleye
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Katrien De Moor.

Rights and permissions

Reprints and permissions

About this article

Cite this article

De Moor, K., Ketyko, I., Joseph, W. et al. Proposed Framework for Evaluating Quality of Experience in a Mobile, Testbed-oriented Living Lab Setting. Mobile Netw Appl 15, 378–391 (2010). https://doi.org/10.1007/s11036-010-0223-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11036-010-0223-0

Keywords

Search

Navigation