1 Introduction

Multimedia applications provide partial sensorial experience to the users when utilizing only their vision and audition. A potential direction for multimedia is to make it diffuse in our environment in an effort to build an ambient intelligent environment [1]. One predecessor for this diffusion is the ability to increase the natural interaction between the user and the multimedia applications. The tendency will be to bring the sense of “touch”—or haptics—into multimedia applications and systems thus making it more intrinsic and natural to the user. In addition to traditional multimedia such as image, audio, and video, haptic—as a new media—plays a prominent role in making real world objects physically palpable in collaborative/shared environments. The haptic modality is tightly coupled with the visual sense; the eyes and hands work collectively to explore and manipulate objects. Therefore, there is a trend in the design of interfaces towards multimodal human-computer interaction that incorporates the sense of touch [11].

Haptics, as a research terminology, refer to the science of manual interactions with the multimedia environment through touch, including exploration for information extraction (called tactile feedback) and/or manipulation for modifying the environment (named kinesthetic feedback). Applications of haptic technology have been spreading rapidly from devices applied to graphical user interfaces (GUI’s), games, scientific discovery and visualization, arts and creation, the vehicle industry, Tele-robotics and Tele-operations, education and training, as well as medical simulation and rehabilitation [11].

Given all the promising advantages of haptic technology (from sensorial enriching and natural interaction to promised excitement), very few research efforts have objectively analyzed the way haptics improves the user’s experience. Quality of Experience (QoE) is an approach that describes the evolving reality that what ultimately matters in a multimedia system is how users perceive its performance [22]. In contrast to the traditional methodology of Quality of Service (QoS) (which manages network configuration and performance and is ineffective to quantify the user experience), QoE examines all elements that influence user’s perception of the interaction. The QoE approach is based on optimizing and monitoring the actual user experience by considering subjective quality parameters such as user satisfaction, usefulness, fatigue, distraction, among others.

The incorporation of haptic modality in multimedia systems depends on whether it enhances the overall quality of user experience. Furthermore, questions can be raised as how do the advantages of heightened senses overweigh the costs of adapting to a new technology? Moreover, how will users experience such advantages and whether they will be overwhelmed or exhausted? In this paper, we attempt to answer such questions by exploring two haptic applications: the haptic learning tool [21] (to measure the QoE associated with kinesthetic haptic feedback) and the haptic YouTube application [2] (to measure the QoE associated with tactile haptic feedback). We base our analysis on the fact that most computer users are adjusted to using the mouse and keyboard and are acquainted with online video streaming websites such as YouTube. We administer two user studies in this paper. In one study, the participants evaluate a writing application that integrates force feedback through a stylus-based haptic device. The other study introduces the user to a new way of watching online streaming videos; videos with tactile haptic feedback. The studies also focus on important haptic properties, including positive and negative attributes. The studies along with their accompanied questionnaire are geared to explore the effect on the QoE when haptics are involved in edutainment applications.

Another quality measure that is user-centric is the Quality of Performance (QoPer) approach [28]. QoPer methods refer to observational procedures or testing, that are designed to evaluate the correctness of performing a particular task—or how well the user performs. Compared to QoPer approach that is task and procedure oriented, QoE approach measures the implications of the interaction at the cognitive level. Haptics enriches the interaction and eventually the user experience, and therefore the QoE approach is more indicative to measure the user experience.

The rest of the paper is organized as follows. In section 2, we review previous work concerning QoE and haptic-based applications as well as haptic effect on users. Sections 3 and 4 describe the experimental setup and introduce our test bed and the applications we used to assess the QoE associated with both kinesthetic and tactile haptic feedback. We also introduce our analysis methodology and present the evaluation results in those two sections. In section 5, we provide further analysis and comparisons between the two sets of evaluation results and the QoE associated with haptic modality in general. Finally, we conclude the paper in section 6 by summarizing the paper contents and providing perspectives for future work.

2 Related work

2.1 Quality of experience in virtual reality

The official definition of QoE given by the International Communication Union (ITU) is “The overall acceptability of an application or service, as perceived subjectively by the end-user” [26]. QoE is gradually becoming an important measure for the evaluation of multimedia applications. As [19] puts it, we require improved performance measures over the well-established QoS measures to deal with the subjectivity of the user.

In [33], the authors discuss some of the methods and challenges in determining those performance measures in the context of virtual reality applications. They indicate that there are three ways of assessing QoE performance measures: subjective ways through interviews and questionnaire, task performance measures through observation of the user, and physiological approach via biological indicators such as heart rate.

The relation between QoS and QoE has been addressed in [34]. Instead of extending the QoS metrics, the paper relates the performance measures of the QoS to QoE measures according to quantified correlation. The result is a theoretical framework for computing QoE using both QoS and QoE metrics. The work is general to all multimedia systems and not specific to virtual reality. The paper does list some performance measures related to virtual reality but does not count all of them.

The move from QoS metrics to QoE performance measures is being addressed by the research community. The author of [8] stresses the importance of this move as well as including new multimedia data types and human computer interaction. In our previous work [15–17], we have attempted to collect all possible parameters for QoE evaluation of multimedia applications including virtual reality and haptics applications.

More specifically in [15], we have constructed a taxonomy for QoE evaluation in haptic-audio-visual applications. Our taxonomy consisted of an extensive listing of the QoE parameters classified into two higher-level categories: QoS and User Experience. We did acknowledge that QoE assessment should include the QoS parameters but it should not solely be based on them. Indeed, the inclusion of the User Experience category is important in our QoE model. The user experience is divided into four parts: perception measures, rendering quality, physiological measures, and psychological measures.

Each of the User Experience subcategories is related to one aspect of the user’s experience. For example, rendering quality measures the quality of the three rendering modalities usually included in virtual reality applications, which are audio, video, and haptics. Interested readers should consult [15] for the complete list of the categories’ definitions and the parameters that fall underneath them.

As a proof of concept, we constructed a fuzzy logic system to evaluate a virtual reality application using five selected parameters from the QoE taxonomy (media synchronization, fatigue, haptic rendering, degree of immersion, and satisfaction). We conducted user experimentation and developed a fuzzy logic model based on the results of the experimentation (membership functions, fuzzy rules, etc.). The system was able to compute a value for the QoE based on the inputs, which are the five parameters we have selected.

The fuzz logic evaluation takes into account the three media modalities as whole. In this paper, however, we attempt to find the effect of just the haptics modality on the QoE of the application. We chose parameters from the taxonomy that we have constructed in [16]. However we did not use a fuzzy logic system to evaluate QoE, we relied on the Mean Opinion Score (MOS) [25] values to draw correlation and statistical analysis from the results as it will be described later.

2.2 Effect and benefits of haptic devices on the user

Haptic devices are the instruments that enable us to feel the objects in a virtual environment. Each application is usually designed to adapt to a certain type of haptic devices. Human haptic senses can be divided into two distinct categories: tactile sensation and force feedback (kinesthetic) sensation [12].

Tactile sensation deals with information about the physical surface such as contact force, geometry of the object and temperature. Kinesthetic sensation, on the other hand, deals with forces resulting from position and velocity of the hand motion. Net forces experienced by humans are usually a combination of both types of haptic sensation. However, haptic devices can be classified into tactile haptic devices or force feedback (kinesthetic) haptic devices. The two categories of haptic devices can be distinguished by the users’ interaction with the virtual environment. Tactile devices or tactile displays distributes the tactile forces over the region of contact, and simulate that contact to the skin. Currently tactile devices are composed of shape-memory alloys, pneumatic actuators, or vibrotactile elements. Kinesthetic devices will simulate the force and torque to the user through tool such as a rigid stick [29].

Since haptic devices primary target the human sense of touch, evaluating these devices and their advantages became an essential research topic. G. De-La-Torre [27] discusses the importance of haptics technology by discussing the biological sense of touch in humans. He stresses that it is hard to cope without the sense of touch in the real world, so in the virtual world vital information would be lost without the inclusion of haptic devices. Basdogan et al., discuss the effect of the touch sense in collaborative virtual environments [5]. In their experiment, they have asked users in separate locations to perform a shared virtual task while using haptic devices. They have concluded that haptic devices increase performance as well as the sense of co-presence.

Force feedback effect has been considered in the literature. Pawar and Steed [24] discuss the effect of haptic cues on three dimensional selection tasks. They argue that the conception of haptic cues always improving performance is not always true. On the positive side, the authors of [32] discuss the benefit of force feedback in laparoscopic surgery. Their analysis is based on the number of errors and precision of dissection. Reinforcing this idea, [14] discusses force feedback effect on surgical performance in minimally invasive surgery.

On the other hand, the effects of tactile feedback on movement accuracy and speed were studied by [31]. The researchers concluded that tactile feedback may enhance performance when feedback is event related. Tactile feedback has also been proved constructive for several other applications including military and robotics tasks in multitasking environments [6, 13], visual speech-reading [3], task performance [23], as well as grasping in laparoscopic surgery [20, 30].

It can be noticed that medical procedures and simulations have received special attention in the literature when investigating the effect of haptic feedback on medical applications. A survey of the effect of both types of haptic feedback on medical simulators has been addressed in [7]. At the same time, the literature seems to lack studies about the effects of haptic modality onto the quality of user experience.

In [18], we have discussed initial thoughts and observations regarding the effect of haptics on the QoE. More specifically, the paper dealt with force feedback effect on a haptic writing application as opposed to using a standard input such as the mouse. User studies were conducted and based on the questionnaire results, correlation values were mapped between several metrics and the overall quality ratings submitted by the user. Drawing on these correlation values, we compared our findings with other user-centric studies conducted using similar type of haptic devices.

3 Force feedback QoE experiment

3.1 Application description

The haptic handwriting learning system [10] is a haptic GUI application designed to facilitate learning of alphabetic handwriting of various languages by incorporating visual, auditory, and haptic feedback. The application is divided functionally into four blocks: the alphabets keyboard, the preview window, the workspace area, and the control panel (Fig. 1). The alphabets keyboard contains all the characters of the selected language. The system supports in its current state five languages: Arabic, English, Chinese, Japanese, and French. The alphabets are stored in a language repository that stores the characters using an XML-based description. Users can select their desired language through the File menu which prompts the system to load the corresponding alphabets from the repository. The preview window replays graphically the selected character in vivid fonts to act as a reference for the user when he/she is practicing writing the letter in the workspace. The workspace in turn enables users to experience the handwriting with haptic and graphic feedback. As users are grasping the haptic device and moving the virtual cursor along the whiteboard plane, the graphic feedback allows them to see their progress whereas the haptic feedback allows them to physically experience it. Finally, the control panel controls the playback mode that the user wants to permit in the workspace. Depending on the user’s selection, graphic or haptic feedback can be enabled as well as setting the playback guidance level such as no guidance for confident users, partial guidance for users in doubt, or full guidance for novice users. The guidance (if set to full) will move the haptic device in real time according to the selected character, and will display the graphic or/and haptic feedback according to the selected options. If no guidance is selected the haptic device will be transparent allowing the free movement of the user. The partial guidance is currently not implemented.

Fig. 1
figure 1

Haptic handwriting learning tool screenshot

Full size image

3.2 Experimental setup

The application ran on a Pentium 4 dual processor machine (3.40 GHz and 3.39 GHz) with one GB of RAM, and an NVIDIA Quadro FX 1000 video card. The screen was placed in front of the users while the haptic device was placed on their working hand side as shown in Fig. 2. The PHANTOM Omni was the haptic device of choice since it has six degrees of freedom positional sensing which allows the users to handle the haptic stylus like handling a pen. Moreover, the three degrees of freedom force feedback provide the desired force in the x, y, and z directions as if writing on paper.

Fig. 2
figure 2

Experimental setup where the user is manipulating the haptic device to write a particular letter on the writing pane

Full size image

Twenty four test subjects (19 males, 5 females, ages 18–34) were asked to participate in the evaluation study of the haptic handwriting learning tool. Some of the subjects did have working experience with haptics, while others did not. However, none of the participants had Japanese writing background before. The users were introduced to the haptic device if it was unfamiliar to them. They were also introduced to the application and shown the different features and aspects including the various languages that can be loaded and the guidance option which was not enabled beyond the introduction of the application. The subjects were asked to practice the handwriting of three Japanese characters (ho, ki, and yo), two times each with only graphic feedback enabled using the mouse first and then PHANTOM Omni haptic device. The graphic only feedback with the PHANTOM Omni mimics the mouse operation while keeping the hardware setup similar to the force feedback testing. Afterwards, using the PHANTOM Omni haptic device, the users practiced writing the characters three times but this time the haptic and graphic feedbacks were enabled. The playback guidance was set to none to allow the free movement of the users according to their own pace. Following that, the users were asked to write the three Japanese characters on a sheet of paper and a score was given for each. The subjects were reminded that we are evaluating the application and not the users themselves. Finally, the subjects were asked to complete a Likert Scale questionnaire. Throughout the procedure, from the experiment to the questionnaire, the steps and objectives were explained to the user. Any concern they might have had was answered.

3.3 The questionnaire

In the final stage of the experimentation, the users were given a questionnaire to reflect on their own experience with the haptic device when force feedback is enabled. The questionnaire provided adheres to the Likert scale format. Essentially, most questions are followed by a five point scale in which the users are required to circle the point that is closest to their subjective consideration. Each extreme poles of the scale is marked by opposing descriptive labels based on the content of the questions. The descriptive labels help the user reflect on the question as he/she is writing the questionnaire. For example, the question below asks the users about their rating of the usefulness of the haptic feedback:

figure d

The questionnaire consisted of the following questions:

  1. Q1.

    How would you rank yourself in Japanese handwriting?

  2. Q2.

    To what extent do you think the haptic feedback was realistic?

  3. Q3.

    To what extent do you think the haptic feedback was useful?

  4. Q4.

    To what extent, if any, did using the haptic device increase your ability to interact with the application?

  5. Q5.

    To what extent, if any, did using the haptic device cause fatigue?

  6. Q6.

    To what extent did using the haptic device make for a better experience than using the mouse?

  7. Q7.

    How would you rank your experience using the haptic device?

  8. Q9.

    Have you ever used a haptic device before?

  9. Q11.

    Give a grade, over 100, for the overall quality of the application?/100

Questions 2 to 7 all followed the above mentioned format (Likert). Questions 8 and 10 were omitted because they are related to features that were not implemented yet. Questions 1 and 9 ask the users about their background in haptic devices and Japanese characters. Question 11 asks the user to evaluate the quality of the application. This is used as the QoE rating of the user. This rating is in accordance with the ITU definition that states that the QoE is the end user’s overall acceptability of the application as described in the related work section.

3.4 Results and analysis

The correctness of the letters is irrelevant to the scope of this paper. What is important is the feedback of the users portrayed in their questionnaire data. The results of the questionnaire are summarized in Table 1. The table’s headings are related to haptic parameters. For instance, Realism refers to haptic realism.

Table 1 Results of the questionnaire conducted for the force feedback haptic application
Full size table

The most relevant question in the questionnaire that relates to the effect of haptics on the QoE is displayed in row six labeled Haptic vs. Mouse. Looking back at the questionnaire, Q6 investigates the users’ experience with the haptic device compared with their experience using the mouse, in which the users were instructed to consider only the force feedback factor (hence the part where they used the haptic device without force feedback). The descriptors on the opposite labels of the five point Likert scale are ‘Not at all’ and ‘Completely’. The mean value of the users’ selection is 4.00 ± 0.88 which suggests a preference towards the haptic device as the mean is in the high range. The results are visually shown in Fig. 3, which displays the number of users who rated their preference choice for haptic instruments opposed to the mouse.

Fig. 3
figure 3

Frequency diagram of haptic experience preference rating

Full size image

Seven users concurred that their haptic experience is completely better than their mouse experience, in this given context. Only two users preferred using the mouse over the haptic device. Three users found their haptic experience to be roughly the same as the mouse experience as their rating was in the middle of the Likert Scale. The rest of the users lay above average which further supports the favoritism towards the haptic device compared to the mouse. It can be noticed that the majority of the users (19 out of 24) have ranked their preference high (4 or 5).

Nonetheless, the results seem ambiguous when viewed unaided by supporting data. The term ‘better experience’ certainly suggests users’ tendency towards haptic devices, but does not necessary mean that the haptic device would be their desired hardware interface if they had to choose. To further investigate the claim of haptics providing a better experience, we have to observe the last column of Table 1, entitled Overall QoE. Here users rated the quality of the application with a score out of a 100. If indeed haptics increase the overall QoE then users who strongly agree that their haptic experience exceeds their mouse experience will rate the overall quality higher than others. The previous prediction—that haptics add to the QoE—is valid since the majority of users established their strong affinity towards haptics.

Figure 4 displays the correlation between the two variables. The diagram suggests a linear, albeit not perfect, correlation as indicated by the straight line that reveals the overall trend of the scattered dots. We calculated the exact correlation of the two data series according to

Fig. 4
figure 4

Scatter diagram of the results where the correlation mapping between the two parameters is shown by the straight line that represents the best linear fit

Full size image
$$ r = \frac{{\sum {\left[ {\left( {x - Mx} \right)\left( {y - My} \right)} \right]} }}{{\sqrt {{\left( {SSx} \right)\left( {SSy} \right)}} }} $$
(1)

where X is the overall QoE series and Y is the Haptic Experience vs. Mouse series (with means of Mx, My and sum of squared deviation of SSx, SSy respectively).

The correlation between the haptic/mouse experience and the overall QoE is found to be 0.666 (degrees of freedom (DF) = 24−2 = 22, two-tailed significance at p < 0.001). This is a high correlation value, which suggests that if users found their haptic experience better than the mouse experience (which is the majority of users) then they are more likely to have a better overall QoE when using the haptic device.

In accordance, we have divided our users into two populations. The first group rated the Haptic vs. Mouse category in the high range; 4 or 5. The other population is the rest of the users. We formulate our hypothesis as the following: The population with high Mouse vs. Haptics rating will rate QoE higher on average. The null hypothesis is that there is no real difference between the two populations, i.e. preferring haptics have no real effect on the QoE.

Performing a t-test on the results we can accept the hypothesis and reject the null hypothesis (t = 3.26, one-tailed test, p < 0.01). Hence, the two groups of subjects significantly differ, which indicates that preference for the haptic device does actually add to the QoE.

It would be possible to examine other metrics that were also included in the questionnaire. Table 2 summarizes the correlation results of each parameter. The realism of the haptic rendering correlates by a relatively high level with the overall QoE. Although it is not as significant as other factors, it is still a high correlation and could be due to the importance of haptic rendering quality in haptic-based applications. If users feel degradation in haptic rendering they would also feel a loss of reality in their touch feedback which would lead to lower perceived QoE [34].

Table 2 Haptic force feedback factors and their correlation with QoE
Full size table

It is also important for users to appreciate the purpose of the haptic interface. If they feel that the device is an unnecessary accessory or a burden then that would expectedly affect the overall quality. On the other hand, if they find that the interface is useful in delivering its purpose then the overall quality would be higher. Evidently, the correlation between haptic usefulness and overall quality is 0.532, significant at the p < 0.01 level.

Surprisingly, there is no significant correlation between intuitiveness and QoE. Some of the users did not have any prior experience with haptic devices while others might not be completely used to them, nevertheless, this does not seem to correlate significantly with the overall QoE. Eventually, people will get used to the interface they are using similar to when the mouse was introduced to the public and people had to adjust for the buttons. However, the results do indicate that the users found the haptic device to be intuitive. Indeed, looking at Table 1, the users have chosen high values for intuitiveness (μ = 3.92, sd = 1.06). The haptic device was intuitive for the majority of users but it did not correlate with the overall QoE. Hence, it is not a major factor in our analysis given that the user will have necessary time for adjustment.

The last factor is fatigue. It has a high level of correlation in magnitude but with an inverse direction (−0.541, p < 0.01). Given that fatigue correlates highly with perceived QoE it is an important factor in determining the desired result of haptic interfaces and whether they would actually increase the application’s QoE, or decrease it.

It is also possible to compute the statistical significance of the haptic parameters. The t-tests results are displayed in Table 3. Most haptic factors investigated here are significant except for the haptic intuitiveness. This reinforces the correlation results since all factors correlated significantly with QoE except for haptic intuitiveness.

Table 3 Statistical analysis results for force feedback parameters
Full size table

The idea that haptics is one aspect of causality for increasing overall QoE perceived by users is emphasized by the results. The majority of the application’s users who found the haptic interface to be realistic, useful, causing less fatigue, and on top of that has a better experience than the mouse also found the application to be more rewarding with a higher QoE.

4 Tactile QoE experiment

4.1 Application description

The application is composed of a client browser (implemented using Java-based SWT) and the tactile arm band device [2]. The haptic rendering logic is embedded in the client browser and necessary Bluetooth communication module is used to connect the arm band to the computer.

The arm band device embeds vibro-tactile motors that generate vibrations at controllable amplitude, frequency, and duration to simulate different tactile feedback (shown in Fig. 5). The application streams a YouTube video onto the local machine and presents the video to the user via the tactile player. At the server, the YouTube video is annotated with tactile feedback using XML notation. This tactile content is stored in XML file with timestamps that specify when the actuator is triggered [2].

Fig. 5
figure 5

The haptic arm band tactile device

Full size image

The video annotation can be done by the owners wishing to add the tactile feedback to their online video. At the desired time ranges, the authors can add the tactile content with a specific amplitude and frequency for the given duration. Those desired time ranges are decided by the owners of the video at interactive or intense peak moments to add dramatic or entertaining effects.

The user wearing the arm band will feel the tactile sensation as series of electrical vibration that is gently stroking through his/her arm. The vibrations intensity will be influenced by the annotation that the author of the video has created and stored on the server using a haptic authoring interface. The vibrations time range would occur at certain events that the author deems worthy of conveying certain tactile simulation to the user. Hence the YouTube annotation is dependent on the scenario of the video.

4.2 Experimental setup

A snapshot of the experimental setup is shown in Fig. 6. The application ran on a Pentium 4 dual processor laptop (3.40 GHz and 3.39 GHz) with one GB of RAM. The screen was placed in front of the users while the haptic device was placed on their working hand side as shown in Fig. 6. The haptic arm band tactile device, that was developed as a prototype at the MCRlab, University of Ottawa, was the tactile device of choice due to its availability and suitability for the experimental application.

Fig. 6
figure 6

Experimental setup where the user is watching a video and receiving synchronous tactile feedback via the arm bend device

Full size image

In this particular experiment, the video that was shown contained several parts. The video was created as a demo for the haptic jacket [9]. The main scenario was a distant husband being reprimanded by his spouse. The two are communicating through a webcam and the husband is wearing a haptic tactile jacket. As the wife is scolding him she humorously punches him. Meanwhile, as the husband amicably ‘feels’ the punch through his haptic jacket, the user watching the video feels the tactile sensation on the wrist arm band (the video was annotated during the punches and during the virtual hugs as well).

Twenty subjects (14 males, 6 females, ages 18–35) took part in the tactile YouTube experiment; they were undergraduate or graduate students from the School of Information Technology and Engineering at the University of Ottawa. No particular reward was given to them for their collaboration. Prior to the start of the experimental session, the application is introduced to the users along with the objectives of the experiment. The experiment video was shown and explained. The participants were provided with assistance for wearing the armband. All the users were presented with the same tactile video material (described above) and were asked to complete a Likert Scale questionnaire immediately after the experiment (to get as instantaneous feedback as possible). The users were debriefed briefly after the experiment. The subjects were reminded that we are evaluating the application and not the users themselves.

4.3 The questionnaire

The questionnaire was similar to the one given in the force feedback experimentation. However the two experiments were not designed in parallel and there are some changes. Most importantly the scale chosen for the tactile experiment was a seven point Likert-type scale in contrast to five points chosen for the kinesthetic experiment. Both questionnaires are considered a likert-type scale since there is a middle point and therefore an equal amount of positive and negative items in most questions.

Moreover, there are few choice differences between tactile and kinesthetic parameters to reflect the variation of the haptic devices used. Intuitiveness was replaced by excitement and fatigue by discomfort since the users are not manipulating the haptic device, instead they are wearing it.

The following question constituted the core of the questionnaire:

  1. Q1.

    Do you prefer using a haptic device while watching an online video (such as YouTube)?

  2. Q2.

    How realistic is the haptic feedback?

  3. Q3.

    To what extent, if any, did using the haptic device create discomfort?

  4. Q4.

    To what extent do you think the haptic feedback was useful?

  5. Q5.

    To what extent, if any, did using the haptic device increase excitement?

  6. Q6.

    Rate the overall experience you had during the demo

Questions 1 to 5 are a seven-point Likert-type scale. Question 6 asks users to rate their overall experience during the demo. This was the overall QoE rating. In this case also, the QoE rating is in agreement with the ITU definition.

4.4 Results and analysis

The result of the questionnaire conducted is displayed in Table 4. The correlation between the QoE and the different parameter are presented in Table 5. Most parameters correlated significantly with QoE except for excitement which did not have a significant correlation. The parameter of focus in the tactile experimentation is the haptic preference which has a correlation of 0.77, p < 0.001 with QoE. This is the user’s answer to the question, “Do you prefer using a haptic device while watching an online video (such as YouTube)?”. The frequency distribution of the results is shown in Fig. 7. Fifteen out of the 20 users rated their preference high (five or above), while five users were either neutral or preferred to watch online video without haptic feedback.

Table 4 Results of the questionnaire conducted for the tactile haptic application
Full size table
Table 5 Tactile factors and their correlation with QoE
Full size table
Fig. 7
figure 7

Frequency diagram of tactile haptic device preference rating

Full size image

In the tactile case as well, discomfort had a significant inverse correlation, while realism and usefulness had high correlation values with QoE. Discomfort parameter is an important factor in the user perceived QoE, since any slight discomfort will reduce the quality of the application. The realism and usefulness in this case is dependent on the scenario of the video the users were subjected to. If the video has some action elements that are suited to be represented by tactile feedback then users will find it useful. The video we demonstrated to the subjects contained such a scenario, which clarifies why usefulness and realism had high correlation values.

All factors including excitement were statistically significant (Table 6). Excitement did not correlate significantly with QoE, but the population of users who rated excitement higher also viewed QoE higher significantly than the rest of the users. Therefore we cannot exclude the excitement factor from the analysis, as we did with intuitiveness in the force feedback analysis. Intuitiveness effect will fade as users get used to the application, but excitement should be present, although in certain cases users get excited about new technologies but without having a decent QoE.

Table 6 Statistical analysis results for tactile haptic parameters
Full size table

5 Discussion

In this section, the results obtained from sections 3.4 and 4.4 are compared. The basis of comparison is to detect similarities between the two sets of results. Since the experiments were different in content and tasks, it cannot be deduced that one type of feedback is more positively or negatively perceived by the user. The separate effect of each type of feedback on the user is the desired basis for the analogy.

Comparing both the force feedback analysis and the tactile analysis, it is remarkable the similarity of the results between the two. The QoE correlation results followed a strikingly similar pattern with intuitiveness parameter in the force feedback application being replaced by the excitement parameter. Discomfort/fatigue had an inverse high correlation, while realism and usefulness had a positive high correlation.

Majority of users in each scenario (79.2% for force feedback and 75% for tactile feedback) had a preference for haptic feedback over visual feedback learning applications or over online videos without tactile feedback. Although the two groups which performed the kinesthetic experiment and the tactile experiment were independent of each other still they have managed to keep tight results. This is furthermore manifested in the mean QoE value for both applications along with the standard deviation. For force feedback experiment, where users were asked to enter an overall experiment rating out of a 100, it was 82.29 ± 14.03 while for the tactile experiment it was 81.0 ± 15.53.

Statistical analysis presented a different way than correlation to look at the data. Correlation shows the direction of the results and the relationship between a given parameter and the QoE. On the other hand, statistical analysis provides causality and an indication that the factors presented to the user indeed made a difference in their perceived QoE.

More importantly is that the majority of the users in each case were in favor of utilizing haptic devices. And since they had a higher QoE, it is an indication that for the majority of our sample population, haptic did cause a higher QoE and the results are statistically significant in both the tactile and the kinesthetic case.

Final thought on the comparison between tactile and kinesthetic results is about the statistical effect size that has become a preferred element in recent years to be included with hypothesis testing [4]. Effect size is calculated according to the following equation

$$ r = \frac{{Mx1 - Mx2}}{{Sx}} $$
(2)

which is the difference of means between the two populations of the hypothesis over the standard deviation of the population.

It seems that the tactile experimental group was more emphatic than the kinesthetic group with their results. Looking at Table 3 compared with Table 6, the t-values in the latter are much higher for most significant parameters. Since both groups have a similar degrees of freedom (24 users and 20 users for force and tactile feedback studies respectively) and similar standard of deviation for the overall QoE value, the higher t-values (which depends on the difference of means as well) indicate a higher effect size in terms that the null hypotheses can be rejected with a higher probability when considering the significant metrics. As an example, the effect size for haptic vs. mouse parameter in the force feedback experiment is 1.37. On the other hand the effect size for the haptic preference in the tactile feedback experiment is 2.09. Although both results are significant, in the case of the tactile feedback we can reject the null hypothesis and accept the research hypothesis more strongly.

6 Conclusion

In this paper we have presented two user studies steered towards determining the effect of haptics on the perceived QoE by the users of two haptic-enhanced applications. One user study was aimed to examine force feedback (kinesthetic) haptic metrics and their correlation with the overall QoE as well as the statistical significance of these metrics. The other study considered tactile haptic metrics and similarly examined correlation and statistical significance.

The conclusion for both studies is that haptic devices, whether tactile or kinesthetic, will add to the QoE if users of similar applications prefer to use these devices over traditional media setup. The majority of the users did state their preference towards haptic feedback and based on the result of the user studies, we can be statistically confident that haptics will improve the QoE of users of stylus-based education applications and tactile-enhanced entertainment videos.

For future work, we can look at different types of applications that utilize haptic devices. For instance, gaming applications and surgical simulations are becoming more integrated with both tactile and force feedback devices. We can sort the QoE influence and effect size according to the type of the application it is associated with.