The importance of sufficient feedback to foster skill acquisition has been widely demonstrated in decades of educational research [17][18]. Leveraging these insights, we developed a desktop application that can support people’s focused attention while working in computer-based environments.
2.1.1 Core functionality: goal setting and feedback
Following evidence from goal setting research (e.g., [19]), people are more likely to follow their goals when they are highly committed. Even though legitimate authorities or peers might exert a persuasive influence, commitment towards a goal is a personal choice, which increases with goal importance and the experience of being rewarded for it. Giving users agency to choose their own goal aligns with evidence from Self-Determination Theory (SDT; [20]), which emphasizes that learners’ perception of competence, relatedness or autonomy can increase their task-related motivation. The enhancing effect of autonomy on performance has also been demonstrated in instructional design research. For instance, [21] found not only less perceived effort in task execution but also increased learning performance when including a choice option in their learning task. Decades of research further emphasize the moderating effect of feedback on goal-related performance, resulting in improved performance when combining both feedback and goal setting [19]. Our application therefore guides the user to set a goal for what they want to do during a defined period of work, subsequently called focus session, and then gives them feedback on how well they are pursuing that goal. Concretely, our application gives the user two types of feedback. During a focus session, the user receives feedback on whether they are currently distracted (formative feedback). After the focus session, the user receives feedback on their overall level of focus during the focus session (summative feedback).
As Fig. 1A displays, a focus session starts with setting a goal for this session. In addition, the user can also select an individual focus session duration, aligning the software use with their own workflow. During a focus session, the software uses a keyword list derived from the selected activity to determine whether a user is in a focused or distracted state, that is, whether the programs and websites the user interacts with match their goal for the current focus session. Inactivity of mouse and/or keyboard for more than 2 min is also classified as distraction. However, for tasks such as reading or lecture video watching, including longer periods without mouse or keyboard activity can be defined as well by selecting the respective option in the settings menu (see Fig. 1C). Upon the first use of the software and/or the first time performing a new activity, this activity is planned down to the programs or websites required to complete the chosen task goal (see Fig. 1B–D).
Returning to our initial example of the journalist: If one – despite all good intentions – finds themselves watching an online video, our application would detect a deviation from your initially specified goal of writing, as you no longer use your indicated text editor, and display the negative feedback message shown in Fig. 2A. As you see, it reminds you of your own prior intention, holding you accountable for your individual achievements. As long as you stay distracted, our application will give negative feedback every 7 sec (see Fig. 2B). Now, if the software successfully manages to convince you to return your focus to your initially chosen task, the detected congruence between the specified focus and the actually performed task results in positive feedback, displayed in Fig. 2C. According to [22], feedback generally answers three questions: where I am going (feed up), how I am doing (feed back), and where to go next (feed forward). Our application’s formative feedback therefore reminds users of their original goal (feed up), indicating if the chosen focus is maintained or derailed (feed back), and hinting on the next action, i.e., getting back on track or staying on track (feed forward). It references the respective task by reminding the user of their initially set activity, provides information on the current state of process, i.e., staying focused vs. getting distracted, and fosters engaging in self-regulation activities in case of getting distracted.
Our rationale for designing this formative feedback was that choosing actions with higher long-term gains is much easier for people when those actions are also beneficial in the short run [2]. We therefore used a recently developed optimal feedback method [23] to compute feedback signals that align how the user experiences (not) investing the mental effort required to pursue their goal with the value of doing so (for more details see supplementary material, section A1). This feedback rewards each mental action (e.g., to return one’s focus to a goal-consistent activity) according to how much it helps or hinders the user in achieving their stated goal for the current focus session (e.g., to spend as much time on writing as possible). The resulting feedback messages therefore signal how many points the user lost or gained when they stayed focused (not displayed), got distracted (-2.5 points; see Fig. 2A), remained distracted (-1.3 points every 7 sec; see Fig. 2B), and regained their focus (+ 1.0 points; see Fig. 2C), respectively. To increase the effect of the feedback, we included visual cues related to color and facial expression and added auditory cues, resembling an alarm for negative feedback, and gaining points in a video game for positive feedback. Feedback messages were always displayed for 5 sec to allow the user to process the content but at the same time minimize distractibility of the feedback itself. By contrast, the feedback for staying focused was not shown immediately to avoid distracting the user constantly. However, it did inform the summative feedback that the user received at the end of a focus session.
The summative feedback, displayed in Fig. 3, is based on the points the user earned in the focus session. Our application sums up the points the user gained or lost to compute a score that measures for how long the user stayed focused and how often and for how long they got distracted. This so-called focus score is stored after each focus session. Our implementation thereby used an earlier version of Eq. 6 (see supplementary material, section A1), which builds on the maximally obtainable point score for a fully focused session and subtracts the point values of the user’s state transitions from focused to distracted, distracted to focused, and from distracted to distracted. We further assume that systematic changes in the focus score across sessions reflect changes in the person’s underlying ability to stay focused on the chosen task. However, the focus score can also fluctuate at random. Our software therefore uses a Kalman filter [24] to smooth out these random fluctuations and obtain a reliable estimate of practice-induced improvements and lapses in the user’s ability to stay focused. When the estimated change in the user’s ability to stay focused is positive, then the user receives positive summative feedback (see Fig. 3A); when their inferred ability remained the same, then the summative feedback is neutral (see Fig. 3B); and when the inferred change in the user’s ability to stay focused is negative then the summative feedback is negative (see Fig. 3C). For further details, see section A2 in the supplementary material.
2.1.2 Usability ratings
Software development followed an iterative user-centered design process with embedded user studies that focused on technical functionality, usability, and user satisfaction. A total of 35 volunteers (Mage = 27.75 years, SDage = 5.37, 43% female) participated in usability studies between April and December 2020. As part of the evaluation, they received defined use case scenarios, such as writing a text or creating a presentation. In addition, they could perform self-selected tasks with the software. We used think-aloud and screen recording and administered the meCUE 2.0 usability questionnaire [25][26].
The meCUE 2.0 comprises five modules that break down into ten subscales to assess defined qualities of experience related to using a technical product. The first module relates to perceived usefulness (e.g., “With the help of this product I will achieve my goals.”) and usability (e.g., “It is quickly apparent how to use the product.”), whereas visual aesthetics (e.g., “The product is creatively designed.”), status (e.g., “The product would enhance my standing among peers”), and commitment (e.g., “I could not live without this product.”) form inherent dimensions of the second module. Positive emotions (e.g., “The product makes me feel happy.“) and negative emotions (e.g., “The product annoys me.“) related to the product are addressed in the third module, and the fourth module involves the aspects of product loyalty (e.g., “I would not swap this product for any other.“) and the intention to use the product (e.g., “If I could, I would use the product daily.“). Module five involves a rating scale asking for an overall impression of the product (“How do you experience the product as a whole?“). The average scores across modules and sub-dimensions are displayed in Table 1.
Table 1
User evaluations with meCUE 2.0 combined over studies from April to December 2020.
Module
|
Subscale
|
M
|
SD
|
Module I
|
Usefulnessa
|
4.55
|
1.37
|
|
Usabilitya
|
5.44
|
1.04
|
Module II
|
Visual Aestheticsa
|
2.80
|
1.15
|
|
Statusa
|
3.35
|
1.46
|
|
Commitmenta
|
2.14
|
1.10
|
Module III
|
Positive emotionsa
|
3.27
|
1.45
|
|
Negative emotionsa, c
|
3.06
|
1.28
|
Module IV
|
Intention to usea
|
3.20
|
1.50
|
|
Product loyaltya
|
3.41
|
1.39
|
Module V
|
Overall evaluationb
|
15.33
|
4.40
|
Note.aLikert scale ranging from 1 to 7 in line with meCUE 2.0. bScale ranging from 0 to 20 due to technical implementation, deviating from scale range of -5 to 5 in meCUE 2.0. cHigher values are related to more negative emotions. |
Upon evaluating these scores with reference to the respective scale means, we find that overall, our application receives a quite reasonable evaluation score. Inspecting the subscales in more detail, we observe that our application is perceived as both useful and usable, obvious from above-average scores in both subscales. In addition, looking at both subscales in Module III, we find scores below average, indicating that our application does not elicit strong emotional responses that might interfere with performance. Improvements persist, for instance, regarding visual design, taken from the rather low score of visual aesthetics and the around average score of perceived status. We also conclude quite limited implications for commitment, intentions of sustained use and product loyalty due to the short and one-time setting of the user studies. Taken together, these findings suggest that our application could be useful for tackling the real-life challenge of staying focused on a chosen task. We therefore proceeded to evaluate our application in a field experiment.