Towards Classroom Affective Analytics. Validating an Affective State Self-reporting tool for the medical classroom

Background: Emotion and cognition are closely interconnected. Specifically in learning engagement and motivation are crucial pillars of effectively absorbing the educational material and being able to utilize it. For this reason, one of the key challenges in the classroom, especially in pre-clinical medical education, is the assessment of the audience’s mood in order to apply effective engagement strategies in it. This work presents the first efficacy and robustness results of a subjective classroom affective state measurement instrument. Methods: Students recorded affective states using a range of positive to negative smileys in lightweight web based and mobile applications. A total of 225 pre lecture, 220 post lecture and 20 mid lecture recordings were taken throughout 5 lectures. In one of these (lecture 2), as an intervention, the mood of the students was positively altered using an impressive demonstration. Results: Kolmogorov Smirnov tests revealed samples not following normal distributions. Independent samples Mann-Wittney U Tests presented statistically significant variation between pre and post recordings only for lecture 2 at significance p<0.01 (U=1416, p=0.006). Kruskal-Walis one-way ANOVA for post lecture data revealed statistically significant difference lecture 2 at p<0.01 (p =0.000) with no variations presented in other post and all pre-lecture recordings, further establishing the robustness of the tool. Post-hoc pairwise test of post lecture tests revealed that the detected difference in lecture 2 was strongly positive (z between 4.495 and 2.039 between lecture 2 and all others). Conclusions: Demonstrated efficacy of such a simple and readily available tool for affective measurements opens new classroom strategies through easily accessible web based and mobile technologies. Selfreported affective classroom analytics can readily be aggregated from such instruments and used for even real time improvement of Antoniou P, Spachos D, Kartsidis P, Konstantinidis E, Bamidis P MedEdPublish https://doi.org/10.15694/mep.2017.000134 Page | 2 audience engagement.

At all levels of instruction, from apprentice-training craftsmen to academic teachers, emotional effect has been identified and used mainly for incentivizing and motivating learning. Teachers have recognized the crucial role of emotion in learning and the impact that its disturbance brings. The emotional life of the learner can interfere with her mental performance. Teaching becomes ineffective if the learner is anxious, angry, or depressed. Information uptake suffers, as well as the organization of knowledge in usable and sustained expertise is diminished [8], [9]. Impressive such variations, include even total inability to retain information [10], [11].
One emotion strongly linked to learning, Motivation, is defined as the condition of being eager to act or work [12]. This exact emotion was the first "countermeasure" used by teachers to overcome the learner's internal emotional weight when it prevented them from maintaining their learning performance. For that they always attempted to create an environment of active participation in order to motivate and engage their learner audiences. The importance of active participation has been proven from several research endeavors [13]. It has been documented that in private tutoring settings (one on one instruction) time is almost evenly split between linking learning with the student's motivational goal and the actual communication of information and formal instruction [14]. Highly motivated students routinely outperform their less motivated counterparts. Even a slightly motivated disposition of a learner is often accompanied by increased creativity, decision making and problem solving capacity [15]. Furthermore, if student motivation perseveres through periods of fatigue and disengagement, then the frustration from low learning performance can be overcome more efficiently [16]. Due to this important role of motivation, a cohort of studies regarding its impact on learning have been conducted. Themes explored include intrinsic versus extrinsic influences, past pleasurable experiences, feelings of importance and contribution, all considered aspects of self-efficacy ( [17]- [24]). disconnect from school [26]. Since student dropout is a gradual process, the engagement construct provides ways for detecting early signs of this process and intervening appropriately. This conceptual construct also provides a link to alterable underlying psychological variables, providing an additional intervention avenue for reform and towards increased learning adherence [27], [28]. Both this and other theories of motivation are often based on components of both cognition and affectiveness in goal oriented behaviors [29], [30].
This term, affectiveness, as it evolved encompasses a diverse mosaic of academic fields including psychology, cognitive science, neuroscience, engineering, computer science, sociology, philosophy, and medicine [9]. With such breadth, one of the main problems in its contemporary exploration is a definition of the term itself and its exact, even ontological relation to the overall educational research field.
On a pragmatic level affective interventions encourage learning, lessen student humiliation and provide support and motivation that outweighs or distracts from the unpleasant aspects of failure [16]. With theories of affect in learning needing evaluation and evolution, the most prudent approach suggested [9] at the current level of technological sophistication is to define measurable quantities from classical macroscopic observations. In order to approach the validation of such theories with evidence based rigor, the data mining and analytics' methodology for the consumption and extrapolation of conclusions from quantitative data enter the field of affective learning research.
Learning analytics has been defined as "…the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs." [31]. This definition is broad enough to include tools such as adaptive learning, learner social network analysis, predictive modelling, learner profiles and other quantifiers of learner activities in e-learning systems and face to face learning activities [32].
This quantification of learning activities facilitates data collection and measurement that then can be utilized for analysis and aggregation into actionable high level metrics [33]. The field has matured significantly, diversifying into two strands, core learning analytics and educational data mining. The first focuses on the actual data collection, analysis and reporting, while the latter focuses more on the computational and algorithmic solutions that facilitate the previous strand [33].
The core application of learning analytics is the monitoring of learner's performance and prediction of critical decision points that might lead to significant changes of a student's academic life (e.g. dropout or significant changes in chances of success [34], [35]. For that reason significant work has been done in learning analytics models in order to identify student risk level in real time, in order to facilitate their success with appropriate interventions [34], [36]. Significant results in data mining of student behavior has revealed the patterns of both successful and unsuccessful students with metrics such as mails sent, forum posts and number of tests completed [37]. These have been exploited by analytics' feedback loops that alter dynamically educational content (e.g. at the most basic level exercise or course order) in order to facilitate student improvement [38].
This real time nature of the analytics' feedback loop has resulted in the evolution of tools for the macroscopic assessment of such heterogeneous and dynamic data sets by human consumers [33]. These visual analytics tools are designed to tap into the ability of humans to see patterns and structure in complex visual stimuli [34].
Learning Dashboards, visual analytics applications embedded in learning management systems, have been established as a type of "personal informatics" applications, assisting users in aggregating patterns regarding their own habits, thoughts, behaviors and interests [39], [40]. Such personal informatics apps can provide valuable selfknowledge that can then be acted upon for tangible behavioral improvement, e.g. better self-control or overall Antoniou P, Spachos D, Kartsidis P, Konstantinidis E, Bamidis P MedEdPublish https://doi.org/10.15694/mep.2017.000134 Page | 4 socially responsible behavior [41], [42].
In the context of the previously described research environment this work aims to take the rigorous and quantified approach of learning analytics and bring it into the critical learning enabler of affectiveness. Specifically we present here a simple application that subjectively measures and stores learners' mood in a classroom. Additionally we present validation evidence regarding this tool's analytic value and its link with future affective analytics based class strategies.
The rest of the paper is organized as follows. After this introduction, in the materials and methods section the applications utilized are described. In the same section the description for the evaluation method, including survey details and statistical tools are described. In the results section the outcomes of the evaluation that validate the analytics capacity of this tool are detailed. In the discussion section, the capacities of this solution in the context of affective sensing and learning analytics are explored. Finally in the conclusion section some insights are offered regarding the place of this work in the overall context of the evolving field of quantified affective aware classroom strategies.

The Web application
A simple web application was developed for recording the subjective mood of each student entering the lecture hall (cf. Figure 1 for a screenshot of the user input screenshot). The Fast Emotion App is based on a server-client architecture. A web service, developed in c#, is located behind the functionality of the application. Along with a SQLite database, the web service manages the users and their authentication, the "sessions", the scoring procedure during the session and the statistical analysis' results when it is demanded by the client (Figure 2) with capacity for real time update to the statistics. On the client side, the web application was based on the angularJS engine [43]. A service of the angular solution is responsible for the communication with the web service (server side), while a controller provides the essential functionality of the user's interface.

Figure 2
The FastEmotionApp Settings and statistics display screen.
The application was utilized at 5 lectures of the formal curriculum of Aristotle University's medical school. Students entering those lectures were asked to select from a variety of "smileys" at 5 levels of positive or negative mood (again cf. Figure 1) the one that best represented their affective state. At the end of the lecture (and in one case at the middle of the lecture when the lecturer changed) the participants were asked to also record their affective state on this same application. Each user's selection was submitted to a database that recorded the response and its timestamp.
The database then was queried and data for each lecture were collected and split into pre and post lecture populations. The makeup of these populations is presented in Table 1. It must be noted that participation of the learners was 100%. Everyone entering the lecture hall before the lecture, as well as anyone leaving the lecture hall after the lecture registered their state. However due to the voluntary nature of these lectures some students arrived after the course started or left during the course. This explains the variation between the pre and post lecture populations. It also must be noted that the lectures were selected at various times without any provision for consistency in order to assess any possible interferences due to time of lecture to the analytics' power of this subjective tool.
To determine the analytics value of this subjective affective state measurement tool we chose to try and induce, in one of the lectures, a significant change in affective state. For that we attempted to motivate the students and engage them through an experiential demonstration. Specifically in Lecture 2 a live demonstration of a portable EEG device and real time description of the signal processing workflow was made. On all the other lectures the traditional routine of exposition with the help of slides was conducted.

The AffectLecture mobile application
The mobile application called 'AffectLecture' is currently available on Google Play [44] and soon will be available at Apple's App Store. It's a hybrid mobile application, developed using one of the most well-known frameworks, the Ionic framework [45], which is free and open source. From a technical point of view, it combines highly interactive native and hybrid components and focuses on speed and usability. The backend is developed using AngularJS, a framework well-suited for building rich and robust apps. The use of the mobile application is simple and straightforward. Once it's open, the user can see available lectures of the day, arranged in a view list and ordered by the beginning time of each lecture. By clicking on an item, the application navigates to the specific lecture. But in order to be restricted only to those students who they will attend the lecture, a 4-digit PIN is required (Figure 3). The PIN is defined by the teacher/instructor, when initially inserts' the lecture on the system. After successfully enter the PIN, the user is able to determinate his/her mood by clicking on a toolbar with icons, as show in the next figure: The user votes twice: the first time to determinate his/her mood before the beginning of the lecture and the second time after the end of the lecture. After each vote the toolbar is locked and the user cannot change its given information. Only two votes per lecture per user are allowed. Teachers and instructors can create lecture available to students by using the web AffectLecture application, available at: http://affectlecture.eu. After registering for an account (which is free), teachers can enter lectures in the system's database, by defining the title of the lecture, the university, the related course, the start and end time and, finally, the 4-digit PIN number. Later on, in the amphitheatre can give this PIN to their students, allowing them to access and vote for the specific lecture.

The Intervention
To determine the analytics value of this subjective affective state measurement tool, irrespective of deployment platform (web or mobile), we chose to try and induce, in one of the lectures, a significant change in affective state. For that we attempted to motivate the students and engage them through an experiential demonstration. Specifically in Lecture 2 a live demonstration of a portable EEG device and real time description of the signal processing workflow was made. On all the other lectures the traditional routine of exposition with the help of slides was conducted.

Results
For the statistical analysis of the data we tested them using Kolmogorov-Smirnov test for normality, rejecting it for all but one sample. ( Table 2) Therefore we utilized non-parametric tests to ascertain whether there are differences over the students' affective states. We tested for difference pre and post-lecture in each one of them utilizing Mann-Whitney U tests except for Lecture 3. The null hypothesis in all cases was that there was no difference pre to post-lecture in the recorded emotion data. Out of 4 lectures tested only lecture #2 exhibited difference in the pre and post lecture emotional status with p=0.006 (Figure 4). The p values of the Mann Whitney tests are tabulated in Table 3.

Figure 4
Pre-Post rank distribution for Lecture 2 from independent samples Mann-Whitney U test. For Lecture 3, since there were 3 independent samples (pre lecture, mid lecture, post lecture) performed an independent samples Kruskal-Walis one way analysis of variance test to ascertain whether there was statistically significant variation between these samples (p = 0.230). The test's results (cf. Figure 5 and Table 4) supported the null hypothesis and thus proved that there was no statistically significant variation between those samples.  data. Comparison of the pre-lecture data demonstrates no statistically significant variation between them (cf. Figure  6 and Table 5). That suggests a robustness of this tool to mood varying factors such as the time of the lecture. Post lecture results revealed statistically significant difference between the 5 lectures (cf. Figure 7 and Table 6).

Figure 6
Kruskal-Walis one way ANOVA for pre lecture emotion data from all the Lectures.   Post-hoc pairwise comparison of Lectures analysis presented in Figure 8 and Table 7 reveals that this difference was due to the highly positive emotional status reported after Lecture 2.

Discussion
Following on past activities by the authors on affective medicine [46], [47], this work is a proof of application for the capacity of the developed assessment tool in detecting affective state changes within a classroom. It was demonstrated that this tool is robust to minor mood impacting factors such as lecture time and lecturer changes while it is also ubiquitous since it is web based and thus accessible on any web enabled platform.
There is a lot of work done in the area of affective sensing in e-learning [2], [9], [16]. Also on the front of learning analytics there is a significant body of literature (cf. [48] as well as references therein) that has demonstrated their impact within the scope of learning management systems, distance and overall asynchronous learning. In the classroom, on the other hand, there have been initiatives with devices such as clickers [49] to increase student engagement and motivation through active participation. This work is a first step in the wider research environment of medical education by taking the outcomes of the first two topics (affect sensing in learning and learning analytics) and putting them in the classroom for pervasive and ubiquitous participation of the learners.
It has been postulated that: "too many educational institutions still lack serious leadership engagement with the innovative application of digital technologies". However there are bottom-up initiatives to change this with researchers striving to 'design for learning' and to engage and motivate learners [50]. This work aims to provide an assessment tool in this academic "crowdsourcing" of technological innovation in education, specifically in the aspect of affective self-reporting in the classroom. The overall aim is for the learners to become content co-creators and to participate in the feedback loop by either expressing their opinion or, equally important, by recording their experience's impact on them.
One of the reasons that has led affect's impact to be less explored than cognition, in the overall learning endeavor, is that it is intrinsically harder to measure. The assessment of a learner on whether she has managed to maintain recall of lists of learned items, or to persevere in the solution of even complex cognitive problems is something that can be tackled rather straightforwardly. On the other hand, it is much harder to measure a person's feelings during the learning process. A whole cohort of technologies, including sensors, robotics, machine learning and data mining are being aligned to objectively measure if the learner is pleased, engaged or on the verge of quitting. A different, more direct approach, one that we explored in this work is that of self-reporting or subjective assessment of affective states and motivation [9].
A whole arsenal of evaluation instruments has been developed in order to evaluate a lecturer's motivational characteristics in the classroom [51]. These questionnaires and other self-reporting instruments have come under serious criticism being characterized as unreliable when measuring emotion. Issues of self-reflection, fear of perceived weaknesses from exposing emotional context in adults, and intrinsic lack of emotional maturity in children makes self-reporting of emotion equally unreliable [9]. On the other hand, evaluation instruments, such as questionnaires have been assessed to disrupt the normal educational process and thus not being able to be used unobtrusively and continuously during the learning process. On the argument of affect sensing reliability, our approach offers some significant caveats to the aforementioned criticisms. The anonymity and the massiveness of the participating learners alleviate a lot of the stress related with declaring internal affective states. An additional enabler to that, is the generic declaration of positive or negative overall affective state with the simple "smiley" system. The learner is not asked to declare her innermost emotions but only a "hard and fast" metric regarding positive or negative affect. Both the actual and the perceived effect of this revelation is minimal on a personal level but valuable on the statistical level. On the obtrusiveness front, the simple act of clicking on a form responding to a simple question ("how do you feel") is as unobtrusive as can be achieved without sensor based data aggregation.

Conclusion
Modern medicine has been shifting from basing diagnoses on intuitive clinical practice to a more evidence based assessments Physicians are basing decisions less on their personal experience with earlier patient cases and more on rather quantified intervention protocols validated by rigorous statistical data [48]. This shift, with the subsequent accuracy and efficiency increase that characterizes more contemporary medical interventions provides an interesting parallel for the education space.
In that field, also, there are some rather contemporary and conceptually "exotic" approaches that have been explored. For example, taking queue from the rather unintuitive "Spectacular failure is better than moderate success" [16] a theme can be explored. Specifically the theme of transcending mediocrity. If the effort is not towards spectacular success this will never be achieved. Moderate achievement is one form of underachievement. In that context it has been postulated that "difficulty should be sought out, as a spur to delving more deeply into an interesting area. An education system that tries to make everything easy and pleasurable will prevent much important learning from happening." [52]. This could be construed as another example of putting the challenge up to the skill level towards achieving "Flow", the optimal experience [53]. Even more recent developments have demonstrated that [54] complex learning tasks demand from the learners to diagnose and solve problems, generate inferences, answer causal questions with several conceptual steps as well as diagnose and solve incomplete problems that require conceptual comparisons and cross theme transfer of acquired knowledge [55]. This form of "deep learning", as opposed to simple, shallow learning activities such as memorizing key phrases and facts is expected to lead to confusion. In fact confusion may be more of the norm in this kind of learning than the exception. However, the subsequent "eureka" moment that will emerge after negotiating the hurdles of this confusion state is far more likely to promote learning at deeper levels of comprehension under appropriate conditions [54].
Such "exotic" educational interventions, however, entail significant dangers. The negative impact of a fail or "stuck" state in learner engagement and motivation can be significant before the "eureka" moment anchors the knowledge. For that reason affective context awareness based on specific hard analytics instead of ambience and intuition becomes crucial for the success of such educational endeavors.
'Context-awareness', the awareness of surroundings and the potential of them to provide information becomes a starting point for effective learning. Personal and social technologies allow learners to use their learning environment, tapping on people, resources or even social ambience (e.g. other people's and object's presence nearby) as dynamic learning resources or factors that adapt their learning experience [48], [56].
In such environments, the learners are becoming increasingly self-sufficient. They are all the more often in positions to engage in educational activities according to their circumstance and need as well as to draw on formal or ad-hoc folksonomies of fellow learners [57]. In that environment, the ubiquitous nature of mobile devices being carried in the person of the learner almost at all times can acts as a real time open window into learner preference, study behavior as well as affective state [58].
In that context the tool presented in this work, utilized as a web based or mobile ubiquitous affect self-reporting tool is a step towards an integrated quantified lecturer tool. The overall aim of this work is to bring lecturer knowledge of the classroom beyond the insight and ambiance level to the quantified and analytics based level of efficiency and