Adult listening behaviour, music preferences and emotions in the mobile context. Does mobile context affect elicited emotions?

Abstract After the introduction of mobile computing devices, the way people listen to music has changed considerably. Although there is a broad scientific consensus on the fact that people show music preferences and make music choices based on their feelings and emotions, the sources of such preferences and choices are still debated. The main aim of this study is to understand whether listening in ecological (mobile) contexts differs from listening in non-mobile contexts in terms of the elicited emotive response. A total of 328 participants listen to 100 classical music tracks, available through an ad-hoc mobile application for mobile devices. The participants were asked to report their self-evaluation of each of the tracks, according to the Pleasure-Arousal-Dominance model and filled out a questionnaire about their listening behaviour. Our findings show that the same factors that affect music listening in non-mobile contexts also affect it in a mobile context.


Introduction
In recent years, the way people listen to music has changed considerably (Avdeeff, 2012;Crawford, 2017). The spread of mp3 players first, and then, even more significantly, the wide diffusion of ABOUT THE AUTHORS Rosa Angela Fabio is Full Professor of General Psychology at the University of Messina. She mainly works in the area of experimental psychology on automatic and controlled processes of attention.
Tindara Caprì is a researcher and psychologist. She mainly works in cognitive processes, general psychology and experimental psychology.
Giancarlo Iannizzotto is Associate Professor of Human-Computer Interaction at the University of Messina. He works in the areas of Artificial Intelligence and Computer Vision applied to Human-Computer Interaction and assistive technologies.
Andrea Nucita is Assistant professor of Web Programming at the University of Messina, Italy. His research interests include Data Management and Medical Informatics.

PUBLIC INTEREST STATEMENT
This study investigates whether listening to music in a mobile context affects the emotions of the listener in a different way than in non-mobile contexts. Such an understanding will provide a deeper insight into the mechanisms involved in emotion elicitation and help better to inform automatic content selection and suggestion engines. More than three hundred participants were involved in the experimentation, listening to one hundred classical music tracks and providing their feedback through a mobile app specifically developed for this investigation. Our findings show that there is no detectable difference in the factors affecting the user's emotions between the mobile and non-mobile contexts. A possible explanation is that the users tend to use the same devices and streaming services in both contexts, thus enjoying very similar experiences. streaming services and music apps for smartphones, shifted music listening towards more ecological contexts, induced by the use of mobile devices while immersed in the different activities of everyday life (Krause & North, 2016;Srivastava, 2010). The world of music is remarkably complex and varied and although there is a broad scientific consensus on the fact that people show music preferences and make music choices based on their feelings and emotions, the sources of such preferences and choices are still debated (Picking, 2010).
Previous research concerning music preference studied music attributes that describe psychological characteristics of musical pieces. One of the first analyses concerned the putative inverted-U relationship between the degrees of pleasure and arousal evoked by music (Kellaris, 1992;North & Hargreaves, 1996). Berlyne's theory (1971) argues that there is an inverted-U between liking for music and the degree of arousal it evokes so that moderately arousing music is liked the most. Laboratory research confirmed such an inverted-U relationship. Other studies mainly focused on the factor structure of music attribute preference and its relationship with personality. Rentfrow and Gosling (2007) rated music excerpts based on 14 sonic and psychological attributes and showed that much of the variance in music genre preferences can be attributed to the liking of musical features. Ju-Chiang Wang, Yi-Hsuan, Hsin-Min and Shyh-Kang (2015) statistically modelled the affective reaction of the users while listening to music excerpts, by means of a Gaussian Mixture Model (GMM) on the valencearousal (VA) plane. Xiao Hu and Yi-Hsuan Yang (2017) investigated the effects of different cultural background of the users and of different choices of datasets on the mood estimation of musical excerpts, again assuming a two-factor VA structure. Greenberg, Rentfrow and Baron-Cohen (2016) had subjects rate 102 song excerpts on 38 attributes. They found a three-factor structure: Arousal (e.g. tense, strong, warm), Depth (e.g. sophisticated, relaxing, party music) and Valence (e.g. joyful, lively, sad). Also Fricke and Herzberg (2017) aimed to confirm this three-factor structure and the relationships with the personality domains. The authors used self-reported music attribute preference and replicated the findings of Greenberg et al. (2016). In studying music attributes that describe psychological characteristics of musical pieces, Juslin (2000) and Bradley and Lang (1994) used the Self-Assessment Manikin (SAM) scale to evaluate pleasure, arousal and dominance. The SAM scale is based on an environmental psychology theory that states that human emotional responses result from variations in pleasure, arousal and dominance. According to the theory, these three dimensions are necessary and sufficient to characterize a subject's feelings, which are influenced both by the personal and environmental stimuli. In turn, a subject's emotional state regulates his behaviour in a situation. In the present study, we used the SAM scale for two reasons: 1) it is based on a consolidated theory and 2) SAM-based evaluations can be easily collected through an automatized procedure realized by a client-server mobile application.
The digitization of music has altered consumption practices (Avdeeff, 2012), particularly by allowing more interactivity (Kibby, 2009) so that listeners are no longer constrained by, for instance, albums and track orders (Molteni & Ordanini, 2003). Moreover, due to the introduction of mobile computing devices (e.g. mp3 players and smartphones and tablets), people are able to exert greater control than ever before on how, when and where they experience music (Bryan-Kinns & Hamilton, 2012;Franco & Cataluna, 2009;Heye & Lamont, 2010;Juslin, LiljeströM, VäStfjäLl, Barradas, & Silva, 2008;Krause, North, & Hewitt, 2013;North & Hargreaves, 2008;Sloboda, Lamont, & Greasley, 2009;Tseng & Hsieh, 2018). As a consequence, the music listening scenario has completely changed and further investigation is needed on the effect of music listening on user emotions and mood in such mutated context. Randall and Rickard (2013) developed a Mobile Experience Sampling Method (M-ESM) to collect real-time data on personal music listening. In a subsequent work (2016), the same authors used this method to provide empirical data to further investigate the different motivations for music listening preferences and the corresponding affective outcomes. In both studies, the aim of the authors was to determine the conditions under which personal listening to some musical track, chosen by the participant, results in either short-term hedonic increase or decrease. The participants recorded their initial mood in terms of valence and arousal, then they chose and listen to a musical piece, and finally they recorded their final mood in terms of valence and arousal (Randall & Rickard, 2013).
The aim of our study is different: we are interested in understanding whether listening in ecological (mobile) contexts differs from listening in non-mobile contexts in terms of the elicited emotive response. As we are interested in providing an "emotional rating" of the musical track (in the case of mobile listening), we evaluate the emotional response to a randomly ordered sequence of specific classical music tracks, disregarding the initial condition of the listener and the reasons that could lead them to prefer or not a piece. To our knowledge in the relevant literature, no other studies focused on estimating the emotional response of users in the mobile context with a specific set of music excerpts.
To reach this objective, we firstly assessed the perceived pleasure, arousal and dominance levels induced by each track, secondly, we analysed how different factors, including gender, age, music preferences, place, time and modality of listening, can be related to the perceived emotions (Antonietti et al., 2015;Fabio, 2017;Fabio & Antonietti, 2012;Fabio & Caprì, 2015Fabio et al., 2018;Fabio, Castriciano, & Rondanini, 2015;Fabio & Urso, 2014;Martino, Caprì, Castriciano, & Fabio, 2017). We included gender because in Western culture women perceive more emotions compared to men and, in particular, are more reactive to unpleasant scenes, especially to those more aversive and traumatic (Bradley, Codispoti, Sabatinelli, & Lang, 2001). Also, based on the evidence, women tend to report experiencing more pleasure and joy (Deng et al., 2016). Age of participants is another important factor as participants of different generations might have different perceptions of musical genres and could perceive pleasure, arousal and dominance in different ways (Paule-Ruiz et al., 2017;Rentfrow, Goldberg, & Levitin, 2011). North and Hargreaves (2007a, 2007b, 2007c also underlined that, according to several studies in the literature, musical preferences provide a way to discriminate between social groups and that such studies suggest indirectly that musical preferences might correlate with a variety of different lifestyle choices. In their study, 328 participants responded to questions regarding their musical preferences and provided data on various social class-related aspects of their lifestyle (namely travel, personal finances, education, employment, health and drinking and smoking). The reported results show that there are numerous associations between musical preference and the considered aspects of participants' lifestyle, for example, melancholic situations called for sad and moody music, while an arousal situation would call for loud, strong rhythm and invigorating music. Differently from the work by North and Hargreaves, in our work the location in which participants normally listen to music is also recorded, thus taking into consideration the context (both activities and environment) for listening. To this end, for example, melancholic situations called for sad and moody music, while an arousal situation would call for loud, strong rhythm and invigorating music. In our study, the participants could choose where and when to run the tests and reply to the questionnaires, so that data was collected in an ecological setting.

The system architecture
We collected the data through an automatized procedure realized by a client-server mobile application. The mobile application (App) was made available for both iOS and Android platforms, through their respective official application stores.
The App is modular, each module dedicated to a specific task, so that the user can autonomously and easily access the system and fill in the questionnaires. On the server side, the data are automatically collected in a database and made available for analysis.
In this section, we present the architecture of the system implemented for data collection and storage. Figure 1 illustrates the architecture for data collection.
The App was developed using Cordova (cordova.apache.org), an open source Apache project that allows the development of mobile applications using languages that are typically used in the Web context, namely HTML, CSS and JavaScript. The advantage of this approach is that, starting from the same application project, Cordova provides different packages for the most common mobile application markets.
After the development, we published the App through the official app markets and made the respective links available, together with an adequate description, through the project website (nisclab.wordpress.com).
The App allows a new user to register or a registered user to login and choose between the two available languages (either English or Italian) for the user interface. The data are collected and stored anonymously, even though the user can optionally register his/her own email address to be contacted for further news about the research project. By accessing the application, the user can listen to different musical tracks and provide a self-evaluation of the emotions elicited while listening to the tracks.
In the registration page, the user can add information about her social condition and musical preferences. The registration questionnaire is composed of seven questions, as shown in Figure 2.  When clicking on the Register button, the app sends a SOAP (www.w3.org/TR/soap/) request to a Web service through a Php (www.php.net) page provided by the Web server. Subsequently, the server sends a SOAP message, alerting about the success of the registration. All the collected data are then stored in a MySQL (www.mysql.com) database hosted by the same server.

Participants
A total of 328 participants were enrolled, 176 women and 152 men. The participants were divided into three groups: 116 subjects under the age of 18 (35.4%); 88 subjects between 19 and 21 years old (26.8%); 124 subjects over 22 years old (37.8%).
All participants were asked to register through the mobile application and to provide personal and social information anonymously: gender; age; employment; place where the participant listens to music (where); time spent listening to the music per week (when); activities during the listening (how); what kind of music the participant usually listens to (what). A summary of the participants' data is shown in Table 1.

Materials
The set of songs used for the analysis was created by selecting 100 audio fragments, e.g. ten audio fragments for each of the following authors: W. A. Mozart, L. van Beethoven, F. Lizst, A. Vivaldi, S. V. Rachmaninov, F. Shubert, J. S. Bach, M. Ravel, P. I. Tchaicovsky, S. S. Prokofiev. The audio files were collected from www.musopen.org and are shown in Table 2.
To obtain an evaluation of the emotional state elicited by the music for each participant, we adopted the SAM (Self-Assessment Manikin) scale, a non-verbal pictorial evaluation technique that allows the participant to express degrees of pleasure, arousal and dominance feeling. Designed by Mehrabian and Russell (1974), this elaborated semantic differential scale is a widely used tool to evaluate the three-dimensional structure of objects, events and situations. The three dimensions of the scale refer to pleasure, arousal and dominance. The Pleasure-Displeasure Scale measures how pleasant or unpleasant one feels about something. The Arousal-Nonarousal Scale measures how energized or soporific one feels. The Dominance-Submissiveness Scale represents the controlling and dominant feeling versus the submissive feeling. The SAM scale was originally implemented to become an interactive computer program but later expanded to include a paper version. Figure 3 shows the SAM figure with pleasure, arousal and dominance scales on the top, middle and bottom rows, respectively.
As already mentioned, we used a mobile application to collect user data that includes the SAM scale questionnaires, to collect emotion-related data about participants. Figure 4 shows a screen page of the mobile application where the user is asked to respond to a SAM questionnaire. Every time a new user registers to the mobile application, the Web service is invoked to build a new random list of audio excerpts for that specific user, as we thought it might not be realistic that a single user could evaluate each of the audio fragments. We have therefore collected 2631 user evaluations over 100 audio excerpts; the mean of tracks evaluated for each user was 26.31 (SD = 5.06).

Data analysis
Data analysis was performed assuming gender, age, music preferences, place, time and modality of listening as independent variables, and the SAM subscales as dependent variables.
Firstly, we analysed the following: 1) the correlation of the subscales using two-tailed Pearson correlations (with an alpha value of 0.05) of the SAM subscales and 2) the general performances on the subscales for each track.
Secondly, we tested specific hypotheses regarding gender, age, music preferences, place, time and modality of listening. Independent samples t-tests (with an alpha value of 0.05) were used to determine differences in the SAM scores when the levels of the variable were two (as in gender: male and female) and F-tests (with an alpha value of 0.05) were used to determine differences in the SAM scores when the levels of the variables were higher than two.

Results
To assess the association among the three dimensions of the SAM scale (pleasure, arousal and dominance (PAD)), Pearson's correlation coefficients were calculated. There were strong and statistically significant correlations between the pleasure sub-scale and the arousal sub-scale (r = 0.594, p < 0.001), between the pleasure sub-scale and the dominance sub-scale (r = 0.837, p < 0.001) and between the arousal sub-scale and the dominance sub-scale (r = 0.614, p < 0.001). These results mean that the tracks that elicited a high feeling in one dimension also elicited high feeling in the other two dimensions. The strong covariation between reports of pleasure, arousal and dominance have been also reported in other studies (Bradley & Lang, 1994) and indicate that SAM is an effective method for measuring existing feeling states.
To identify those tracks that gave rise to significantly higher or lower scores we assessed pleasure, arousal and dominance scores across the 100 tracks. We report each track's mean in Figure 5, and we indicate the highest and lowest scores as follows: 'X's show track values that are below the mean minus one standard deviation (< M-1SD), and circles show track values are above the mean plus one standard deviation (> M + 1SD).
As such, this analysis identified the highest and lowest scores in terms of each of pleasure, arousal and dominance dimensions. These results demonstrate that the PAD dimensions can distinguish between the different tracks.   Table 2 shows the author and the title of each track presented in the X-axis of Figure 5. Tracks 3, 36, 84 and 85 were perceived as eliciting a high level of pleasure (see Table 2).
Tracks 18,22,26,34,62,78 and 93 were perceived as eliciting a low level of pleasure. Tracks 13, 26, 68 and 84 were perceived as eliciting a high level of arousal, while tracks 62, 66, 76 and 93 as eliciting a low level of arousal. Furthermore, tracks 84 and 85 were perceived as eliciting a high level of dominance, and track 22, 62, 93 were perceived as eliciting as low level of dominance. Table 2 shows the mean evaluation values, with their standard deviation, for the above-mentioned audio excerpts, together with the source musical pieces and their mode and tempo attributes. To explain these first results, we considered some musical characteristics of the audio excerpts. It is well known from the literature that musical mode (major or minor) and tempo (slow or fast) affect the user's emotional reactions (Antonietti, Cocomazzi, & Iannello, 2009;Colombo & Antonietti, 2017;Hunter, Schellenberg, & Schimmack, 2010;Husain, Thompson, & Schellenberg, 2002;Mehrabian & Russell, 1974;Stalinski & Schellenberg, 2012). According to these findings, as shown in Table 3, excerpts in major mode are associated with high levels of pleasure, while excerpts in minor mode correspond to low levels of pleasure. Moreover, tempo affects the activation of the users in the arousal and dominance scales. We considered that user's response is also related to past experiences about known music pieces.
All the considerations above agree with prior research on music listening in non-mobile context.
To address the second aim, we analysed how different factors, including gender, age, employment, music preferences, place and time may influence perceived pleasure, arousal and dominance. Owing to the high number of variables, a Bonferroni correction was applied to each analysis. With reference to gender, as the data were normally distributed, t-test for independent measure was applied. As in the study of Fricke and Herzberg (2017) women tend to report a higher level of pleasure with the musical tracks than men in tracks 10, 11, 12, 15, 26, 35, 48, 50, 62, 96 and 97. With reference to arousal, score t-tests were statistically significant in tracks 62, 44, 59, 68, 69, 81, 99: again, women report experiencing more arousal with the musical tracks than men. With reference to dominance, score t-tests were statistically significant in tracks 25, 35, 55, 73, 90, 97; in this case, women report experiencing more dominance in tracks 15 and 35 and men report experiencing more dominance in tracks 55, 73,90,97. For all the above-mentioned tracks we obtained p values of t-test lower than 0.01. With reference to age, F tests were significant in tracks 18, 22, 23, 42, 43, 56, 57 and 72. People with age higher than 22 report experiencing more pleasure, arousal and dominance than younger people do.  With reference to employment, employed people report experiencing more pleasure and arousal than students and the unemployed (tracks 22, 23, 28, 43, 56, 59, 72). Both these results are similar to the results of the work of Rentfrow et al. (2011).
We found that the results referred to the location are weak and not significant and with regard to the time spent in music listening we found that subjects who spend more time listening to music experience more pleasure and dominance than subjects that spend less time (tracks 14, 16, 23, 24, 25, 56, 42, 69, 73, 93). For all the above-mentioned tracks we obtained p values of F-test lower than 0.01.
As seen in the introduction of this work, the same variables that influence the perceived emotions such as gender, age, music preferences, place, time and modality of listening in nonmobile contexts influence also the perceived emotions in mobile context. Far from considering this as a negative result, we think that it suggests that the way to listen to music has a low impact on the perceived emotions and that the challenge could be to understand if it has a greater impact on other aspects of the social relationship such as communication or lack of communication.

Discussion
Music listening in the mobile context has grown in the past years together with the massive spread of mobile devices. This modality in music listening, and the enhanced technical capabilities of mobile devices, has led to new ways to conceive the fruition of music by users. Prior research focused on measuring the reasons that motivate users to listen to music in mobile context, and how this different context may change the listening experience (Randall & Rickard, 2013. Moreover, the possibility to propose musical playlists based on the mood of the user has also raised the interest of industry (U. S. Patent No. 9,788,777 B1, 2017).
However, to the best of our knowledge, the relevant literature lacks studies focused on measuring the specificity in emotional response of users in the mobile context with a specific set of music excerpts. On the basis of prior research, we considered the behavioural and social parameters that are known to influence the music listening, and measured the emotive response of the users within an ecological context, by the means of an original mobile application.
Our results show how, for each musical track, different sources, i.e. gender, age, employment, place, time and music preferences, differently affect the Pleasure-Arousal-Dominance measures. Thus, substantially agreeing with the findings of prior literature and suggesting that the mobile context does not sensibly affect the way music listening elicits emotions in the users.

Conclusion
In conclusion, the adoption of a mobile application has confirmed to be a valuable method to collect users' data in an ecological context, and we proposed an original research showing that also in a more ecological (mobile) context the music characteristics and the social data that affect emotive responses corroborate and generalize those already known from prior research conducted in non-mobile contexts.
One limitation of the present study is the low heterogeneity of users, being most of them students. Hence, as part of future work, we plan to extend the number of users, increasing their variety in age and employment, to study more deeply the discussed findings. In fact, our findings suggest that it would be possible to get more precise music emotional evaluation within a specific user group.