AdaptiveVoice: Cognitively Adaptive Voice Interface for Driving Assistance

2.1 Adaptive Voice User Interface

Voice User Interfaces (VUIs) have gained widespread adoption in various domains such as smart home environments [32], driving assistance systems [25], accessibility applications [37]. However, it is important to acknowledge that VUIs must be calibrated to accommodate the diverse preferences and requirements of individual users. Substantial research has delved into the examination of several pivotal factors influencing the VUI user experience, encompassing voice familiarity [38], speaker gender [9], personality attributes of the voice agent [4], ages [1], voice input methods [51] and the physical capabilities of users [23].

Even for an individual user, the interaction requirements with Voice User Interfaces (VUIs) can vary depending on the context in which they are used [31]. Therefore, it is often essential for VUIs to adapt to users’ in-situ context, especially when information assimilation is impacted by a primary task. Previous work has primarily focused on adaptive auditory presentation in driving scenarios. Auditory interfaces can be essential in driving scenarios of high risks (e.g., rail crossing [39, 40]) to ensure efficient brake reaction and safe driving speeds. Previous work has explored various formats (e.g., speech vs. sound [53]) to deliver the message or encoding richer information via spatial auditory displays [3]. For instance, one system is designed to react to drivers’ emotional states using different sound effects to ensure they notice alerts [43]. Another example is Soundsride [26], which explores how to temporally and spatially align high-contrast events on the route, e.g., highway entrances or tunnel exits, with high-contrast events in music, e.g., song transitions or beat drops.

Nonetheless, instead of changing the sound effects, there is a gap in presenting voice messages in an adaptive manner by adjusting the presentation format while keeping the core information the same. As a starting point, we select the user’s cognitive load as the factor to adapt VUIs to. It is intuitive that users with varied cognitive loads have different capabilities and capacities for perceiving information from VUI. Especially in the driving assistance scenario, where users are focused on the cognition-demanding task of driving while receiving voice messages, adaptive VUIs can be crucial not only to improve the user experience but also to ensure safer performance.

Motivated by this, we propose AdaptiveVoice, an adaptive VUI for driving assistance, which jointly determines the amount of information to present, the voice speed, and whether to repeat the message, taking as input users’ current cognitive load.

2.2 Measuring Cognitive Load

Estimating users’ mental states (e.g., cognitive load) has been a longstanding challenge for researchers in HCI and Psychology [2, 14]. Currently, there are three main measurement methods. The first is collecting subjective measures from questionnaires such as the NASA TLX [20]. In addition, performance metrics such as response time have also been used as indicators of cognitive load. Lastly, physiological measures, such as electromyography (ECG), skin conductance, and respiration [21] also correlate to the cognitive load. Among the physiological estimation methods, the Index of Pupillary Activity (IPA) [11] captures the frequency of users’ pupil diameter oscillation and has been proven effective in discriminating task difficulty-induced cognitive load. Lindlbauer et al. [30] successfully applied IPA in the real-time estimation of users’ cognitive load into three discrete levels (low, medium, high). In a follow-up research, Duchowski et al. [10] derived a wavelet-based method for computing the low/high-frequency ratio of pupil oscillation, which leverages how the human autonomic nervous system functions and yields a hybrid measure based on the ratio of low/high frequencies of pupil oscillation. It was tested to be robust in its sensitivity to the measurement of cognitive load in response to the difficulty of a task.

In this work, we leverage the HP Omnicept ¹ for cognitive load estimation. As reported [46], it provides a robust cognitive load estimation based on behavioural and physiological measurements. In an experiment with 738 participants, it achieved an average classification accuracy of 79.08\(\%\) with a mean absolute error of 0.1106, which was a normalized score ranging from 0 to 1. The results also align with previous research [12], which shows that people’s pupil size measured by an off-the-shelf VR headset with an integrated eye tracker positively correlates with their self-reported cognitive load.

2.3 Reality Simulation in Virtual Reality

It has been common to simulate real-world tasks [34, 36] and augmented reality scenarios [5, 6] using virtual reality (VR). It allows researchers to gain complete control over the experimental environment and the tasks while avoiding various risks and inconveniences if the same study is to be conducted in a real-life setting. For example, Shi et al. [45] utilized a multi-user VR system with motion-tracking capabilities to simulate hazardous scenarios and study how social influence affects construction workers’ safety behaviors. Fernandes et al. [13] simulated realistic music concerts to host virtual concerts using VR. Wang et al. [50] used a VR system to simulate a marine environment and establish an ocean current interference model. This method has also been evaluated regarding the effectiveness of the simulation. Mathis et al. [34] evaluated and compared lab research in VR and the real world. Their findings demonstrate VR’s great potential to avoid potential restrictions and risks researchers experience when evaluating authentication schemes [35]. Regarding simulating driving scenarios, using VR as the simulator is also a common practice [47, 54]. As mentioned in previous research [54], VR simulators provide the unique advantages of being more realistic and immersive when compared with fixed-base simulators, and can also provide greater flexibility and safety than on-road testing environments. Goedicke et al. also combines immersive play with on-road haptic feedback to achieve even more realistic simulations. Beyond simulation, Jansen et al. also explored the use of VR as the visualization tool in analyzing users’ driving behaviors in a post-hoc manner. Based on these previous practices, we also decided to evaluate the adaptive algorithm by simulating a driving task in VR because it would allow us to control the experimental environment and minimize any risk to participants.

3.1 Participants and Apparatus

We invited twelve participants (7 males, 5 females; ages: \(M = 22.41\), \(SD = 1.23\)) with diverse educational backgrounds from the local university. They self-rated their familiarity score with AR/VR devices on a 7-point Likert scale (\(M = 3.75\), \(SD = 6.46\)). They all had normal or corrected-to-normal vision and could clearly see the objects in the scene during the experiment.

We used an HP Reverb G2 Omnicept VR headset for this user study. It has equipped the vertical field-of-view (FoV) of 90°, a horizontal FoV of 98°, and 2016 × 2160 per eye resolution displays. Cognitive load detection sensors that were supplied with the device were integrated into the headset, enabling us to collect data based on users’ mental effort, which reliability has been validated by previous work [46].

We developed the experimental program with Unity3D 2021.3.12f1 and Omnicept SDK 1.14. The program was run on a computer equipped with the CPU of Core (TM) i9-10900F and NVIDIA GeForce RTX 3090 graphics card. A USB cable was used to connect the computer and headset, allowing the real-time scene rendering in the HMD. Throughout the experiment, participants were seated in a stationary chair and interacted with objects within the VR system through two controllers provided with the headset.

3.2 Design

The study contained two sessions. In the first driving simulation session, the interface plays voice instructions in different formats, and participants need to select an operation to take for a driving task based on the information they interpreted from the instructions. At the same time, a secondary task was introduced to induce different levels of cognitive load in the participants. In the second reflection session, they revisited all the voice message formats and selected the best format for each level of cognitive load when they perceived it.

3.2.3 Secondary task in the first driving simulation session.

In the VR driving scenario, the participants’ cognitive load would naturally vary due to the complexities and unpredictability of the driving task. By using the adapted secondary task, the study could control for these variations and more accurately assess the impact of cognitive load on the processing of voice messages. The secondary task was designed to be cognitively demanding so that we could alter participants’ cognitive load (CL) by changing the task difficulty. Participants are required to memorize a set of numbers and determine whether the current stimulus is identical to a previous one (recalling a stimulus from n stimuli back. If the number showed up, they needed to press the "Y" button on the controller; otherwise, they should press the "X" button. Following each selection of X and Y by the user, a new set of numbers is presented, with each set being independent of the others. We designed the secondary task with short duration and high frequencies. Additionally, we adjusted the task difficulty by modifying the length of the numerical arrays, changed the display speed from showing each number for 0.5 to 2 seconds, and the number from containing a single digit to three digits. Table 2 shows how we adjusted these factors to induce five levels of cognitive load. These modifications enabled us to more precisely control the level of cognitive load, thereby spanning a broader range than traditional n-back (0-0.6), specifically from 0.2 to 0.8. To ensure the validity of the design, we ran a pilot study with five participants and recorded their estimated cognitive loads. We used the score for cognitive load obtained from the headset as the metric, which ranged from 0 to 1. The average scores were 0.28, 0.35, 0.47, 0.58, and 0.69 for different levels, which preliminarily showed the effectiveness.

Figure 2:

View Figure

Figure 3:

View Figure

Table 2:

Cognitive Load level	level 1	level 2	level 3	level 4	level 5
Interval	2	1.5	1	0.8	0.5
Number presented per round	2	3	4	5	6
Number range	single-digit	single-digit	single-digit	two-digit	three-digit

View Table

Table 2: Second task design that we applied to affect participants’ cognitive load with different difficulty levels.

3.3 Procedure and Measurements

After participants familiarized themselves with the tasks in a warm-up session, they completed two experiment sessions. In the first driving simulation session, they went through the dual-task experiment with 5 CL × 5 LoD × 3S × 2R = 150 conditions. In the second reflection session, participants selected one specific voice message format for each of the five levels of cognitive load, respectively. In both sessions, the sequence of their experience in the secondary task is determined by a Latin square. The task completion time and accuracy in the first driving simulation session and participants’ preferences in the second reflection session were recorded. After the experiment, participants joined a semi-structured interview where their thoughts and needs for voice messages were collected. The experiment lasted for around 45 minutes.

3.4 Objective Data Results

We conducted Bartlett’s test to assess the assumption of sphericity for our ANOVA analysis. The results confirmed that the assumption of sphericity had not been violated for the metrics we analyzed. Specifically, for the task completion time, the results were (χ² = 2.15, p > 0.05), indicating consistent variance across different conditions. For the accuracy, the test showed (χ² = 1.87, p > 0.05). These outcomes supported the use of repeated-measures ANOVA for our data. Further, We conducted repeated-measures ANOVA tests on the effects of four independent variables (CL, LoD, S, R) on the completion time and accuracy. For the independent variables showing significant effects (p < 0.05), we used Bonferroni-corrected post-hoc tests for pairwise comparisons. We performed our data analysis using spss.

3.4.1 Task completion time.

RM-ANOVA results showed significant effects of the cognitive load of participants \((F_{4, 44} =77.76, p \lt 0.05, \eta =0.87)\), LoD of voice messages \((F_{4, 44} =12.78, p \lt 0.05, \eta =0.53)\), and speech speed \((F_{2, 22} =3.42, p \lt 0.05, \eta =0.24)\) on task completion time. No significant interaction effect was found between the independent variables. Figure 4 shows the average completion time of LoD, Repetition, and Speech speed with standard error in different cognitive load levels.

Figure 4:

View Figure

3.4.2 Task accuracy.

RM-ANOVA results showed significant effects of cognitive load \((F_{4, 44} =18.36, p \lt 0.05)\), LoD of voice messages \((F_{4, 44} =9.97, p \lt 0.05)\), and speech speed \((F_{2, 22} =7.93, p \lt 0.05)\) on task accuracy. Figure 5 shows a summary of the results. No significant interaction effect was found between the independent variables.

Figure 5:

View Figure

3.5 Subjective Results

3.6 Discussion: Insights from the First Study

Based on the study results, we identify the following requirements for optimizing voice message systems in real time, particularly for drivers or individuals in multitasking scenarios. These requirements set the stage for developing an algorithm that can adaptively modulate the voice messages based on the cognitive load and context of the user.

These requirements provide a comprehensive framework for the forthcoming algorithmic implementation aimed at optimizing voice message delivery in real-time. By meeting these requirements, the algorithm could offer a personalized, effective, and less cognitively demanding experience for users.

4.1 Cognitive Load Estimation

We used the HP Reverb G2 headset to estimate the real-time cognitive load, leveraging the built-in gaze tracker and the photoplethysmogram (PPG) sensor. The real-time estimate of cognitive load as a continuous value ranging from 0.0 to 1.0 was obtained from the headset, with a higher score indicating a heavier cognitive load. The cognitive load prediction is a centered, normalized value, with a standard deviation describing the margin of error of \(+/- 0.05\). The sampling rate is 3 Hz. We applied the Hamper filter [42] to remove outliers before feeding the data into the algorithm model.

4.2 Input

We relied on designers to prepare different formats of voice messages as the pre-defined inputs to the model. In addition, we obtained estimates of the user’s cognitive load and their interaction history as real-time inputs while the model was running (see Table  3).

4.3 Output

At run time, the algorithm determines the format of the voice message, defined by the combination of (i, j, l), at the moment to deliver it to the user, based on the real-time estimates of the user’s current cognitive load and their previous interaction records. The binary decision variables denoted as v_i, f_j, s_l, all ranging between 0 and 1, indicate whether the respective levels (i, j, l) are rendered. A value of 1 means the level is rendered, while a value of 0 means it is not.

4.4 Optimization Scheme

The algorithm optimizes the LoD, repetition, and speech speed of the voice messages to maximize the utility score. We solve the optimization via integer linear programming. Our implementation was inspired by the insights obtained from the user study. When the cognitive load is low, the user appreciates more information at hand, delivered at a relatively slow speed, with potential repetitions to help digest it. When faced with a heavy cognitive load, users find a concrete format that efficiently delivers core information to be more practical. To encode these considerations into our model, we defined three sub-objectives: utility of LoD (u), utility of repetition (h) and utility of speech speed (w). Our optimization aims to maximize the overall weighted sum of the objective function. We empirically determined the weights \(\lambda _{u} = 0.48, \lambda _{h} = 0.25,\lambda _{w} = 0.27\)based on the frequency \(p _{u} = 0.58, p _{h} = 0.25, p_{w} = 0.16\), at which the participants in the user study rated the dimension as the most important. We refined the weightings at a step of 0.01 until the algorithm could generate the top choices that participants made in the second session of Study 1. This can be formulated as follows, where n, m, k denote the number of levels for each factor, which are 5, 2, and 3 in the current format: (1) \(\begin{equation} max(\lambda _{u}\sum _{i=1}^{n} v_{i}u_{i} +\lambda _{h} \sum _{j=1}^{m} f_{j}h_{j}+\lambda _{w} \sum _{l=1}^{k} s_{l}w_{l}) \end{equation}\) The real-time cognitive load is measured and sent to the utility function. The utility function definition assigns low-level utility values to the current cognitive load far away from the average cognitive load of all users and high-level utility values closer to the average cognitive load. In addition to measured and average cognitive load, we discovered that the utility of each level (i, j, l) should also be influenced by its corresponding probability of adaptive type and historical information. We calculated the proportion of choices (i, j, l) among the total choices and selected the one with the highest probability. The utility value of each level of the voice element is correlated with the user’s interaction history. We used the \(\frac{N_{i,j,l} }{N_{all} }\) to calculate the probability of (i, j, l) level occurrence. The more the level appears, the greater the probability it will appear later. The utility functions are displayed below: (2) \(\begin{equation} \begin{split} &u_{i} =\frac{1}{e^{\left| A_i-L_{cr} \right| } -1} +e^{\rho _{i} +\frac{N_{i} }{N_{all} } }\\ &h_{j}=\frac{1}{e^{\left| A_j-L_{cr} \right| } -1} +e^{\rho _{j} +\frac{N_{j} }{N_{all} } }\\ &w_{l}=\frac{1}{e^{\left| A_l-L_{cr} \right| } -1} +e^{\rho _{l} +\frac{N_{l} }{N_{all} } }\\ \end{split} \end{equation}\)

4.5 Constraints

The following restrictions determine whether items should be displayed and at what level while limiting the range of potential solutions.

5.1 Task Design

In each trial, the participant drove a virtual vehicle through thirteen intersections and received assistive instructions from the system 100 meters before these locations. To alter the task difficulty at each intersection and induce different levels of cognitive load, we change the intersection types (straight roads, turns, T-intersections, crossroads, and overpasses) see in Fig. 6, the level of information (low, high) as shown in Fig. 7 and designed three special cases containing incidents (in total \(5 intersection types\times 2 information level angles+3 cases=13 intersections\)). Three special cases [49] are designed as follows see in Fig. 8:

We compared four methods to present assistive instructions to the participants: AdaptiveVoice without HUD (A), Fixed-format baseline without HUD (F), AdaptiveVoice with HUD (HA), Fixed-format baseline with HUD (HF). Each participant drove four times with each method, in which the order was randomized.

Figure 6:

View Figure

Figure 7:

View Figure

Figure 8:

View Figure

5.2 Metrics

We recorded steering wheel angle variance, response time, average speed, and number of collisions to measure the participants’ driving performance. Additionally, we recorded the users’ cognitive load throughout the session and the types of voice instructions issued by the system to facilitate subsequent review and analysis of user behaviour. Among them, a higher variance in steering wheel angle signifies poorer driving performance, which is calculated as: \(\begin{equation*} \text{Variance} (\sigma ^2) = \frac{1}{N} \sum _{i=1}^{N} (x_i - \mu)^2 \end{equation*}\)

In addition, we also collected subjective feedback through three questionnaires: NASA TLX [19], TAM Model [8], and Emotional appeals (This section is tailored to capture the emotional responses of the users, an aspect often overlooked by other models [48]. It gauges factors like freedom [15], guaranteed [15], advance in technology [29], fun and surprise [15], and convenient [41].), which covered the functional and emotional aspects of user experience while performing the tasks. We have incorporated semi-structured interviews into our evaluation to enrich our data further and obtain nuanced insights into user experience. These include:

5.3 Apparatus

We implemented our software in Unity 2021.3.12f1 with Omnicept SDK 1.14 on HP Reverb G2 Omnicept. Gurobi 9.5.2 was used to solve the integer program formulated in the optimization scheme. We used Python 3.7 with Anaconda 2020 to develop the algorithm and process the collected data. The optimization results were sent to Unity through a local socket. We used Adobe Premiere Pro and Adobe Audition to generate different voice instructions. The same voice generation method was used in Section 3 and the evaluation study. We rendered a city-sized virtual environment with Unity for participants to drive around virtually. Participants sat in a simulated car mock-up. We utilized the Moza R9 racing simulation system, which features an R9 Direct Drive Wheel Base ³, RS V2 Steering Wheel ⁴, and MOZA SR-P Pedal ⁵.

Figure 9:

View Figure

5.4 Participants

We recruited 30 participants (16 males, 14 females; age: \(M = 23.83\), \(SD = 2.42\)) from the same local university through email solicitation and word-of-mouth. Among them, 60% were undergraduates and master’s students, 30% were doctoral students, staff, and faculty, and the remaining 10% were from outside the university community and lived near the campus. The 30 participants reported having used AR/VR devices before, rated (\(M = 3.95\), \(SD = 2.41\)). 26 participants reported having driving experience before (1 - poor; 4 - neutral; 7 - excellent) rated on a 1 - 7 Likert scale (1 - poor, 4 - neutral, 7 - excellent). Two participants rated themselves a 7, and six rated themselves below 3. We also collected data on how frequently participants used voice navigation (\(M = 4.67\), \(SD = 1.44\)), which averaged with a standard deviation of 4 on a 1 - 7 Likert scale (1 - never, 4 - neutral, 7 - almost every day). Participants reported having normal or corrected-to-normal vision. No participants reported elevated susceptibility to motion sickness when queried using the Motion Sickness Susceptibility Questionnaire Short-form (MSSQ-Short) [17].

5.5 Procedure

Participants read and signed an informed consent form. Subsequently, they completed the MSSQ-Short (Motion Sickness Susceptibility Questionnaire - Short), which was later compared with a post-experiment questionnaire to assess the occurrence of motion sickness symptoms such as oculomotor disturbances, disorientation, and nausea [18]. Following this, eye-tracking calibration was performed using the built-in eye-tracking function of the HP Reverb G2. Once calibrated, participants entered the driving simulator and were instructed to sit in the driver’s seat and adjust it for comfort. A 10-minute training session was then provided to familiarize them with driving in the simulator, including driving, braking, and overtaking. The training route was different from the route used in the formal experiment. After they became familiar with the simulator, a brief overview of the city map and their driving route was presented, allowing them to acquaint themselves with the driving tasks, including a city map with road names and a driving route showing the start and end points.

When participants felt comfortable to proceed, the experiment commenced. Overall, participants underwent the experiment under four different conditions in a within-subject design. After each condition, participants were required to complete an evaluation questionnaire for that specific condition. The study involved a route comprising 13 intersections, each characterized by varying levels of informational complexity of intersection types. These intersections were arranged in a random sequence along the route. Participants experienced the same route under four conditions. They were informed that potential incidents might occur at certain intersections, necessitating heightened driving caution. However, to emulate a realistic driving environment, specific locations where incidents might occur were not disclosed to the participants.

The order of conditions was counterbalanced among 28 participants following a Latin Square approach  [27], while 2 participants experienced conditions in a random order. Participants were invited for a semi-structured interview after completing all four conditions and the associated questionnaires. The entire study for each participant lasted approximately 60 minutes.

5.6 Quantitative Results

Upon collecting the data, preliminary tests were conducted to assess its suitability for advanced statistical methods. Specifically, the normality and homogeneity tests were run for Response time (χ² = 1.96, p > 0.05), Steering Variance (χ² = 2.41, p > 0.05) and Average speed (χ² = 0.54, p > 0.05). The tests confirmed that the data met the assumptions for normality and homogeneity, thus validating its distribution for subsequent analyses. Then, one-way ANOVA was conducted on Response Time, Steering Variance, and Average Speed. These analyses revealed significant main effects across different experimental conditions for key variables. No significant main effect was found for Average Speed. The non-parametric Kruskal-Wallis test finds that the number of collisions does not differ significantly across AdaptiveVoice and the Fixed both in HUD or Voice-only.

5.7 Qualitative Results

5.7.2 Enhanced Efficacy and User Engagement through AdaptiveVoice.

The superiority of the AdaptiveVoice in our study was markedly evident, with users expressing a distinct preference for its real-time adaptability and precision. This preference was particularly pronounced in scenarios requiring acute awareness of changing road conditions, such as navigating overpasses. One user commented,  "At places like the Overpass, I am not sure about the road conditions after making the turn. A longer message would be useful" (P17). This is backed by the statistical data where AdaptiveVoice had a higher perceived usefulness (\(M_{\text{HA+A}} = 5.95\), \(SD_{\text{HA+A}} = 0.81\)) and convenience (\(M_{\text{HA+A}} = 5.67\), \(SD_{\text{HA+A}} = 0.96\)) compared to the Fixed (\(M_{\text{HF+F}} = 4.63\), \(SD_{\text{HF+F}} = 1.37\)), (\(M_{\text{HF+F}} = 4.96\), \(SD_{\text{HF+F}} = 1.45\)) in Figure 12.

Additionally, in HUD, AdaptiveVoice was met with positive user reception. This was quantitatively supported by the data, showing that the ’Perceived Ease of Use’ for AdaptiveVoice (HA) scored significantly higher (M_HA = 6.12 , SD_HA = 0.75), compared to the Fixed (HF), which scored (M_HF = 4.89 , SD_HF = 1.22) in Figure 12. Participant 3 stated  "If I had to choose, I’d lean more towards adaptive. It makes navigation tasks more engaging." The AdaptiveVoice not only met users’ needs for clear and timely information but also markedly enhanced their overall interaction with the system.

Figure 10:

View Figure

5.7.5 Navigating Preferences between HUD and Voice-Only.

Participants who had prior experience with racing video games or Virtual Reality (VR) found the arrows to be more familiar and, therefore, easier to adapt to. One participant noted, "Although the turning arrow doesn’t exist in reality, in games like racing games, the direction is shown on the HUD, so I feel quite familiar with this environment"(P4). On the other hand, some users found the arrows distracting, as they appeared directly within their field of vision and felt "sudden and not easily ignored." One participant elaborated, "Although the arrows guide you to press the direction keys at the junction, they appear too suddenly and you are compelled to look, which I am not used to" (P21).

Figure 11:

View Figure

Figure 12:

View Figure

Figure 13:

View Figure

6.1 Applications Beyond Driving Assistance

We demonstrated that adaptive voice instructions help improve driving as the drivers’ cognitive load might cause them to react differently to the events happening on the roads. The differences in their capability of attending to voice information require adaptation. We expect AdaptiveVoice to be beneficial for a broader range of applications beyond driving assistance, as long as users’ cognitive load dynamically changes when they are immersed in a task and need voice instructions to be adaptive to changes in their mental load.

For instance, see Fig. 14, when the user is situated in a relaxed state near the window, the cognitive load can be at a low level. During such leisurely moments, if the user inquires about evening restaurant recommendations and their signature dishes, the smart home system can offer a comprehensive list. For example, it might suggest, "Certainly, here are some dining options: La Bella Italia is renowned for its authentic pasta and pizzas, with Fettuccine Alfredo as a standout dish". In contrast, when the user is engrossed in work tasks in the study room and the cognitive load level rises to a high level, a more concise response is appropriate. For instance, if the user queries about the necessity of carrying an umbrella the next day, the system might simply reply "No." without providing further information about the current weather.

Figure 14:

View Figure

Another example is the cooking voice assistant. When the user is engaged in cooking activities, such as preparing a Creamy Mushroom Pasta, they may require the assistance of a voice-activated system to guide them through the entire process. During periods of high activity, such as continuously adding seasoning to the ingredients, the user’s cognitive load can increase. In such instances, the voice assistant can modulate its delivery speed or repeat instructions to ensure clarity. This allows the user to accurately gauge the required amounts of milk and salt, for instance, without the need to consult a recipe manually. Conversely, during periods of lower activity, such as waiting for the microwave to heat the food, the user’s cognitive load may decrease. Under these conditions, the voice assistant can accelerate its delivery speed and provide an overview of the upcoming steps in the cooking process, thereby offering the user a more coherent understanding of the tasks ahead.

6.2 Extension to Voice Instruction Adaptation

In this work, we adapted three features of voice instructions: the level of detail, repetition, and speech speed, considering users’ cognitive load and the temporal consistency across messages. In the semi-structured interview, five participants mentioned that voice tone adaptation could be helpful and a useful feature, as they stated they prefer a humorous tone when having low cognitive load, so their driving is more interesting and not so monotonous. Moreover, when having a high cognitive load, they favour a serious and professional tone that can better able to deliver the information akin to an order. In addition, the tone can be useful to provide implicit information about how much focus and concentration they need to have in anticipation of what they could encounter soon (e.g., nearing a traffic jam or entering the highway). In short, more adaptive features, including volume, tone, and frequency of the voice instructions, can be adjusted in the optimization, and factors, including the users’ driving skills, traffic situations, and the strength of ambient noise, can also influence the optimal manner of presenting voice instructions. It is essential to explore considerations for these features and factors to extend the optimization algorithm in the future. To estimate users’ cognitive load, we used the HP Reverb headset and conducted the studies in VR. There are also alternative sensors and sensing techniques (e.g., Index of Papillary Activity [10]) for estimating cognitive load, and we believe AdaptiveVoice will benefit from more advanced techniques to provide a more accurate estimation.

6.3 Potential Challenges of Applying AdaptiveVoice in Real Life

We conducted both user studies in a VR environment as a first step in understanding users’ need for voice instruction adaptation and the effectiveness of the algorithm-optimized voice instructions. However, we acknowledge that applying AdaptiveVoice in a real-life setting may encounter several challenges caused by the difference between driving in VR and in real life. First, users may handle cognitive loads in different ways, which is highly correlated to how skillful they are at driving. More skillful drivers can spare more cognitive resources in consuming information from voice instructions. Therefore, they might prefer a high level of detail, fast speed, and no repetition in most cases. The results of Study 1 showed that only a small proportion of participants selected the more detailed formats. Thus, as a starting point, our current adaptive method follows a one-size-fits-all approach that looks for common behaviours amongst all users to balance performance and simplicity. We believe that it is an important future direction to explore personalization of AdaptiveVoice, taking users’ driving skills and experience together with their cognitive capacity into consideration. Another challenge is the real-time estimation of drivers’ cognitive load. We leveraged the built-in functionality of the VR headset in our user studies to estimate users’ cognitive load. In a real-life setting, we believe that it could be replaced by advanced computer vision-based technologies (e.g., IPA [11], LHIPA [10]), with wearable cameras attached to the glasses or in-car cameras located in front of the driver. We also expect that novel estimation techniques with biophysical sensors will be developed and equipped on cars of the future to further reduce privacy issues.

6.4 Transferring Headset-Based Cognitive Load Measurement to Behavioral Data-Based Methods

Currently, we use the HP Reverb G2 headset for real-time cognitive load estimation, leveraging its integrated sensors. However, we recognize the potential in exploring simpler, behaviour-based methods for estimating cognitive load, such as analyzing steering wheel angle as a proxy.

A future direction of this research could involve correlating headset-based cognitive load measurements with behavioural indicators observable in driving scenarios, like steering wheel movements. This approach might offer a more practical and less intrusive method of assessing cognitive load, especially in real-world driving conditions where the use of specialized headsets might not be feasible. The challenge would be to develop algorithms that accurately interpret these behavioural data to make reliable inferences about the driver’s cognitive state.

By investigating the relationship between direct cognitive load measurements (from the headset) and indirect behavioural indicators (like steering wheel angle), we could refine the adaptability of voice interfaces. This would not only broaden the applicability of our findings but also enhance the practicality of implementing adaptive voice systems in everyday driving scenarios. The goal would be to maintain the benefits of cognitive load-sensitive system adaptations while simplifying the means of cognitive load assessment.