Sensory professionals’ perspective on the possibilities of using facial expression analysis in sensory and consumer research

Abstract The increase in digitalization, software applications, and computing power has widened the variety of tools with which to collect and analyze sensory data. As these changes continue to take place, examining new skills required among sensory professionals is needed. The aim with this study was to answer the following questions: (a) How did sensory professionals perceive the opportunities to utilize facial expression analysis in sensory evaluation work? (b) What skills did the sensory professionals describe they needed when utilizing facial expression analysis? Twenty‐two sensory professionals from various food companies and universities were interviewed by using semistructural thematic interviews to map development intentions from facial expression recognition data as well as to describe the established skills that were needed. Participants’ facial expressions were first elicited by an odor sample during a sensory evaluation task. The evaluation was video recorded to characterize a facial expression software response (FaceReader™). The participants were interviewed regarding their opinions of the data analysis the software produced. The study findings demonstrate how using facial expression analysis contains personal and field‐specific perspectives. Recognizability, associativity, reflectivity, reliability, and suitability were perceived as a personal perspective. From the field‐specific perspective, professionals considered the received data valuable only if they had skills to interpret and utilize it. There is a need for an increase in training not only in IT, mathematics, statistics, and problem‐solving, but also in skills related to self‐management and ethical responsibility.


| INTRODUC TI ON
The food industries are facing new demands on skills and knowledge due to an increase in digitalization. The employers have reported difficulties predicting the required skills and competencies, and graduates do not fully meet their expectations and requirements for a rapidly changing sector (Flynn et al., 2013;Weston et al., 2019).
When working with digital technologies and data rather than traditional measures of products, new expectations for sensory professionals are likely to emerge, which challenge the traditional skills in sensory and consumer research. However, there is limited knowledge on food sensory professionals' perspective of work changes and necessary skills. Since digitalization has widened the variety of sensory methods to be performed by the sensory professionals,

| Using facial expression measurement of emotion responses in sensory evaluation
Facial measurement of product-evoked emotions can provide useful and applicable data from consumer emotions (Abbasi et al., 2013;Meiselman, 2016). There are a few software packages to measure facial expression of emotions. Several studies have been published about the measurement of automated facial expression analysis (AFEA) tools (FaceReader™, PrEmo, Affdex), and a few of them using FaceReader™ technology (e.g., De Wijk et al., 2012;Danner et al., 2014;Zhi et al., 2018). The software measures facial expressions related to emotions based on standardized facial movement coding system, the Facial Action Coding System (FACS) developed by Ekman et al. (1978).
A Dutch company named Noldus has developed the FaceReader™ software, which analyzes facial expression patterns, whether from recorded videos or in a real-time video stream. The FaceReader™'s facial expression analysis is an emotion indicator, and it is noninvasive and acquired through video recording (Zhi et al., 2018). It also reconstructs the face in a three-dimensional space, based on a 500-point finite element model. Facial measurement is based on a discrete theory of universally recognized emotions (i.e., anger, disgust, contempt, fear, happiness, sadness, and surprise) (Ekman & Cordaro, 2011). It can be helpful in finding rapid, uncontrollable responses that are related to liking and preferences. Exploring emotion patterns elicited by varying stimuli provides additional value and insights in understanding emotions in relation to verbal expressions of emotion and acceptability (Leitch et al., 2015).
Facial expression measurement technology has been used for research in fields such as psychology, education, and consumer behavior (Rocha et al., 2019). Several studies of advantages and disadvantages, difficulties related to use, and the reliability and validity of the method have been conducted. The robustness and reliability of FaceReader™ have been tested. According to Terzis et al. (2010), the emotions it measured agreed with the judgments of trained observers on up to 89% of occasions. Measuring emotions evoked by a wide variety of eliciting conditions, including statements that referred directly to sensory properties and experienced consequences, and statements that referred to more indirect conditions, such as expectations, recognition, and associations have been reported. However, according to Hwang and Matsumoto (2016), there were difficulties in using FaceReader™ related to training, technical and methodological aspects and to the context of the situation. Their study pointed out that expressed emotional responses can occur from the product but also from the evaluation process, environment, and the evaluators' behavior and circumstances. Reasons for facial expression can vary and enjoyment of product is not always detectable on the face (Hwang & Matsumoto, 2016). Generally, a product can elicit many subjective emotions and multiple emotions, rather than eliciting a single emotion, and individuals differ in handling their emotional responses (Desmet & Schifferstein, 2008). More positive than negative emotions were experienced when the product is owned, is used precisely for pleasure, and if it fulfilled a consumer's goals (Schifferstein & Desmet, 2010). When measuring food-related emotions, the variability in emotional response across and within participants and incomplete understanding of the evoked emotion terms affected the consistency of the results (Köster, 2003).
During the FaceReader™'s video recording, staying still is recommended and participant's face should not be covered by anything (Danner et al., 2014). Moreover, awareness of video recording while experiencing the sample could introduce a bias among participants and they might try to limit their expressions. To avoid misinterpreting the results, extra care is needed when using the software and undertaking analysis (Rocha et al., 2019) and interpretations.
Arousal and valence are dimensions describing emotions evoked from human consciousness, and each emotion can be described according to its position on the pleasantness and arousal dimensions (Barrett, 2016;Russell, 1980). According to Russell and Barrett (1999), valence (positive or negative) and arousal (involves activation or deactivation) congregate emotions in two fundamental dimensions. FaceReader™'s circumplex model is based on Russell's (1980) "circumplex" model of emotions, in which all measured emotions are placed into a two-dimensional space.

| Skills of sensory evaluation and consumer research
Digitalized instruments and methods give rise to new requirements for skills. When the changes in the field continue to take place, the examination of skills needed allows to identify the new skills required by sensory professionals. Skills have a variety of definitions, referring to an acquired ability or capacity. According to Eraut (2000, p.114), skills are part of a particular type of knowledge, allowing representations of competence, capability, or expertise in which the use of such skills and propositional knowledge are closely integrated. Skills also refer to highly competent performance (Smith, 2002). In a general approach, skill means "the ability to apply knowledge and use knowhow to complete tasks and solve problems" (European Commission, 2008, p. 11). Effective and progressive problem-solving skills are an expert's capacity to strive continuously for a higher level where a problem can be approached. Adaptive expertise is a continuous, active search for opportunities to develop knowledge and understanding (Hatano & Inagaki, 1992). When new technologies and methods are introduced, they will always require employees to learn new skills.
Sensory evaluation requires individual physiological and cognitive skills. According to earlier studies, the skills required are sensory analysis methods and decision-making, teamwork capabilities, and problem-solving (Lawless & Klein, 1989;Savela-Huovinen et al., 2018;Stone et al., 2012). According to Flynn et al. (2013), the most desired food-sector skill in the field of research and development is product development. Communication skills have been defined as highly desired generic skills, followed by thinking and problem-solving and skills demonstrating positive attitudes and behavior (Flynn et al., 2013). In sensory evaluation, the interactions between novel methods and procedures with traditional product test variables and the number of new applications emphasize the skills required to understand the methodological issues and problems in product emotion research (Jager and Cardello, 2016). Levy and Murnane (2004) categorized occupational tasks as changing by digitalization where one category includes tasks that require problem-solving capabilities, intuition, creativity, and persuasion. These tasks are characteristic in professional, technical, and managerial occupations with high levels of education and analytical capability, and place a premium on inductive reasoning, communications ability, and expert mastery (Levy & Murnane, 2004). The ability to communicate, acquire information, and think critically was included in formal education programs in food science (Flynn et al., 2013;LeGrand et al., 2017).

| Research questions
The aim with the study was to obtain information from sensory professionals on how they experienced the use of FaceReader™ in their work. We wanted to gain an in-depth knowledge of this issue from the sensory professional's perspective. The objective was to gain understanding of how the sensory professionals, who are experts of their field, experienced the possibilities of the software in relation to their work and more broadly in relation to the field. And further, how facial expression analysis can be used in sensory and consumer research. The sensory professionals were not experienced with facial expression analysis; therefore, their responses can be understood as potential uses or expectations based on experiences gained through participation in the study. The specific research questions were: 1. How did sensory professionals perceive the option of using facial expression analysis in sensory evaluation? 2. What skills did the sensory professionals describe they would need when utilizing facial expression analysis?

| Overview
The qualitative study was based on a semistructured thematic in-

| Participants and recruitment
Twenty-two participants from Finnish food companies and universities were interviewed to gain an understanding of how facial expressions and analyses can be utilized in sensory and consumer research.
The invitation to take part in the research was sent out via email to the members of the Finnish Society of Food Science and Technology in May 2018 (n = 1,160). We estimated that about one-third of the email recipients work in the sensory science field. The study was carried out with the assessors who volunteered and were equivalent to standardized expert sensory assessors (ISO 5492, 2008). The criterion for selecting the participants was that the participant had to work in a sensory laboratory or product development department as an assessor. The participants who agreed with the requirements were involved in the experiments. They signed an informed consent form agreeing that the video data, FaceReader™ analysis data, selfreport, and interview data could be used for research purposes. No previous experiences with the FaceReader™ software were reported by the participants. Participants' demographic information is shown in Table 1.

| Samples and self-reports
In this study, facial expressions were elicited by odor samples. The participants evaluated the samples for pleasantness, intensity, and familiarity on visual analogue scales (10-cm line scales printed on paper). After scaling, the participants described the quality of the odor with their own words (they were asked to write 1-5 descriptive words/sample). The samples were: sea buckthorn juice (20% in tap water), rapeseed oil with garlic (100%), balsamic vinegar (100%), and l-carvone, d-carvone, and beta-ionone (10%), all diluted with propylene glycol (>99.5%; Sigma Aldrich). Selection of the flavoring ingredients and odorants was based on their ability to produce an adequate odor. Water was provided for the participants to neutralize the sense of smell before sniffing the sample. The order of the samples was randomized but it was kept same for all evaluators, because in this study our focus was on the experiences of the assessors, not on the properties of the samples. Self-reports were requested after evaluation of each sample.

| Procedure
First, the participants were instructed (but not extensively trained) for the evaluation. General knowledge about the technical specifications of the FaceReader™ software was provided to the participants, and also what kind of analyses can be carried out (e.g., facial expression classification, valence and arousal calculation).
Previous studies were mentioned in relation to the weaknesses and opportunities of the software. Detailed results were not explained because these could have influenced the evaluation process to the opinions of the facial expression analysis given by the professionals.
Before the experiment, they were informed about the general experimental purpose, the use of line scales, the camera recording, and characterization of a facial expression responses by the FaceReader™. The participants were instructed to sniff the sample at once after opening the cap and to remove the odor vial away from face immediately after sniffing. They received the following instructions before the evaluation task: 1. Adjust your position to make it comfortable, as you are not supposed to move during the experiments.
2. There are samples in front of you. Please sniff them one by one and keep facing the camera while sniffing.
3. Remove the vial from your face and look directly ahead. Do not cover your face during the experiments.
No timer was used, to ensure natural, spontaneous facial expressions and to keep the experiment as natural as possible. The selfreports were collected to a data spreadsheet for further analysis.

| Video recording
A high-resolution digital video camera (Logitech, Brio stream ultra HD4K, 5× optical zoom) was used for video recording. The camera was placed in front of the participants and adjusted to take a straight frontal face view. The participants were instructed to minimize head movement. The light was white and uniform, and the background was also white to ensure the video was in good light condition. The distance of participant from the camera was approximate 30 cm. The video was set at 5-30 frames/s with frame dimension 1,920 × 1,080, and the recordings were saved as AVI files. The recording was started when evaluation started (the evaluation of the last sample was recorded separately) to visualize the facial expressions elicited by the samples. All the recordings were analyzed with face model "Western," and "continuous calibration" was chosen for standardization.

| Semistructured interviews
The participants were interviewed individually and in a group of 2-3 people about their experiences of the use of facial expression analysis and the usability of data analysis for one sample (balsamic vinegar). They were asked to describe their experiences as well as the

| Data analysis
The data used in the study consisted of the FaceReader™ outputs and transcribed interviews. The self-reports on odor perception represented a normal evaluation practice and were not analyzed for the study. The interview data were analyzed by following the principles of inductive content analysis, with a focus on how the facial expression analysis could be utilized and which skills did the participants report to need when using facial expression analysis. The data were first analyzed inductively through repeated examination and continuous dialogue. We used ATLAS.ti software in the data analysis (Scientific Software Development GmbH).
The coding process consisted of three phases and 498 quotations were extracted from the data and categorized. In the first phase, we categorized inductively several text segments representing two broad categories of more specifically coded fragments: the  To analyze the inter-rater agreement of classification, an independent rater classified approximately 10%-15% of the analysis produced. The Kappa coefficient for rater agreement was 0.830 (Cohen's Kappa) for analysis of the participant perspective and the field-specific perspective, which was considered to represent excellent congruity between the raters (over 0.75 rated as excellent, see Fleiss et al., 1969, p. 281).

| RE SULTS
The data analysis included participants' perceptions of facial expres-  Tables 2 and 3.

| The perspective of participants
The recognizability of the odor (I) was the first part of the evaluation, and it was significant for the participants. That was also reported several times in the interview (19.6%). Thoughts whether the odor sample could or should be identified in the booth caused uncomfortable feelings for the participants. If the odor was not recognized, it was disturbing, while the odor recognition was considered to be a pleasant surprise. The participants tried to estimate the identification time when the odor expression occurred from the outputs of facial expression analysis. Second, the participants described that they added memories and associations (II) brought by the odor to the evaluation: the memories from childhood, significant events or the gracious association, and associations from some chemicals.
Association has connection for odor recognition and knowledge of the identification process is acquired through association.    Table 2.

| (I) Product
According to the analysis, the participants reported that the facial expression analysis could be beneficial for evaluating the following products (I): vinegar, sour confections, aromas, product from a same product category or with similar properties or evaluation of aftertaste (e.g., fatty products). The participants mentioned that even though they evaluated odor as unpleasant, vinegar was compatible with salad. They also revealed that unpleasant odors are common with various foods (e.g., some cheeses, yogurts, milky or earthy products), and the method is not suitable for evaluating preferences about these kinds of product. Subcategories and descriptions are presented in Table 3.

TA B L E 3
The field-specific perspective for using facial expression analysis

| (II) Practices
The facial expression analysis was perceived as a suitable method for measuring children' expressions because they are often incapable of describing products using words. The interpretation of practices (II) could be easier than with traditional methods (self-reports)  consumer does not necessarily buy any product if it does not arouse any emotions (negative or positive) but the feeling of happiness affects acceptance, participants said. Some consumers want to be surprised, while some of them avoid surprises. Generally, survey responses have a greater and more significant impact on the purchase decision than data of emotions. Geographical location and situations were mentioned as an important aspect when testing individual differences between consumers. Consumers do not always tell everything they know about the product, or they do not have knowledge or sufficient vocabulary (e.g., for descriptive analysis). Children were mentioned as a group in need of special arrangements.

| (V) Analytical sensory evaluation
The participants perceived that the facial expression analysis could be used in analytical sensory evaluation (V). The method could improve training and maintaining the expertise of trained panels. It was not considered suitable for creating vocabulary or a group test, but it could be useful for quick single testing if an evaluator was also a user of the software. According to the participants, facial expression analysis could offer additional value to the following tests: difference tests, quality control, shelf life tracking, seasonal tasting, taste profile creation, and trained panel evaluation.

| (VI) The Context
The participants mentioned that the video recording influences the emotions expressed. Sensory evaluation should be conducted in a genuine context (VI). If facial expression analysis is used in the context of group test at work, social pressure will influence evaluation, the participants said.

| (VII) The data
The participants pointed out how facial expression analysis data (VII) analysis could provide additional information, and additional training and knowledge is needed. This emerged in nearly one-quarter of the responses (24.1%). When analyzing the data, interpretation was mentioned as being significant. The participants considered and mentioned security-related issues because the analysis contained personal data. Numerical information directly from the software was mentioned as being easy to handle, to store, and to present to the clients, and receiving big data was considered valuable.
The participants brought up the benefits of the facial expression analysis for companies. In particular, they raised questions such as, what feelings are interesting from a commercial point of view?
Would the facial expression analysis be useful without a discussion session? What is the correspondence between consumers' selfreport and occurred facial expressions? Is it relevant or even possible to eliminate anomalies from the analysis? Is it possible to use it for learning about one's own expression of emotion?

| (VIII) Skills
According to the study analysis, the following skills (VIII) were reported in the interview: thinking and problem-solving (1), IT, mathematics, and statistics (2), language and communication (3) self-management (4), ethical responsibility (5), principles of sensory evaluation and consumer science (6), educational training and leadership (7). Subcategories and descriptions are presented in Table 3.

| Identified skills
During the interview, the participants were asked to describe the skills required when using facial expression analysis. The skills presented by the participants and the defined skills, frequencies, and examples of the quotations are shown in Table 4.
According to the study analysis, the participants mentioned  Table 4.

| D ISCUSS I ON
The findings demonstrate how using facial expression analysis contains personal and field-specific perspectives. Recognizability, associativity, reflectivity, reliability, and suitability were perceived as a personal perspective. From the field-specific perspective, professionals considered received data analysis valuable only if they had skills to interpret and utilize it. The study highlights the potential skills suggested and provided by sensory professionals. We conclude that there is a need to increase training not only in IT, mathematics, statistics, and problem-solving, but also skills related to self-management and ethical responsibility. The needed skills were raised by the sensory professionals based on their knowledge and experience of sensory evaluation work.
Odors can evoke associations and memories and enable samples to be identified.

| Methodological considerations
This qualitative study was done in a single country and with a nonrepresentative set of industries and participants. The results would have been more comprehensive if the study had also used experienced FaceReader™ users instead of beginners only. However, the study focus was on authentic professional reflection to evaluate the facial expression analysis results. We argue that the data collected were valid for studying the phenomenon and answering the study questions, as was the method of using inductive content analysis. As a qualitative study, the most significant results were the development of categorization and the distribution of skills. The conceptualization of the results can be generalized and exploited in other studies.
The approach was developed to measure the perception of odor sensory attributes and the participants were informed that the evaluation was being video recorded. From an ethical perspective, this was mandatory, but it could have introduced a bias with participants reacting according to the principle of social desirability. They may also have limited their expressions. It was also impossible to ensure that the participants were not affected by the laboratory environment or other people next to the booth. People are not always aware of their emotional responses or the conditions that underlie their emotions (Barrett et al., 2007). The eliciting conditions reported have been influenced by the participants' beliefs or expectations between the odor sample and emotions. The objective approach of this study could be seen as treating the observed participants as a type of a physiological bundle focusing on input and measuring outputs. This aligns with the utilitarian approach not just in the objectiveness of the approach, but also viewing participants as objects of a study to produce knowledge.
In an authentic context, food product would always be tasted, not only sniffed, in order to obtain an overall rating of the product (e.g., preference). In turn, product-evoked emotions are more difficult to evaluate than sensory attributes (Thomson et al., 2010).
In addition, the emotion ratings may have been influenced by the participants' initial emotional state or by the preceding task (in this case previous odor sample). Sensitizing respondents by having them imagine emotions may have stimulated them to report higher levels of emotional relevance in the actual test. The representativeness of the products tested can be questioned; other types of odor may yield different results. We agree with the literature that more research of testing procedures and the context of evaluation is needed (Danner et al., 2014;Hwang & Matsumoto, 2016).

| CON CLUS ION
The study contributes to the explanation of how the sensory professionals described and examined personal and field-specific perspectives of using facial expression analysis. Future research could examine evaluation context and the consequences for their use in product development. According to various affordances provided by changes experienced, the study results could be utilized by training and education providers in order to develop the profession. When developing their expertise and acquiring new skills and competencies, the professionals would be better prepared to adapt new methods and technologies and effectively manage their workplace challenges.

I N FO R M E D CO N S E NT
Written informed consent was obtained from all study participants.

E TH I C A L R E V I E W
This study was conducted according to the guidelines laid down in the Declaration of Helsinki, and this study was approved by the Institutional Review Board of Helsinki University (statement number 30/2018).

ACK N OWLED G M ENTS
This work was supported by The Finnish Work Environment Fund #190039, 2019-2020.

CO N FLI C T S O F I NTE R E S T
The authors declare that there is no conflict of interest that could be perceived as prejudicing the impartiality of the study reported. Writing-review & editing (equal).

DATA AVA I L A B I L I T Y S TAT E M E N T
The data that support the findings of this study are available on reasonable request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.