"It Must Be Gesturing Towards Me": Gesture-Based Interaction between Autonomous Vehicles and Pedestrians

2.1 Autonomous Vehicles-Pedestrian Interaction Design

Pedestrians are one of the most vulnerable users on the road. Misjudgments between vehicles and pedestrians can lead to accidents and even cause casualties [68]. Compared to traditional cars, autonomous vehicles lack certain methods of communication [11]. Therefore, the interaction between autonomous vehicles and pedestrians will affect pedestrian safety [23]. The interaction issues between AVs and pedestrians have attracted a lot of attention in HCI research. Past research has explored various design approaches to facilitate AV-pedestrian interaction, such as audio signals (engine sounds, bell rings, and soft alarm sounds) [72], Light Signals [16, 17, 30], Augmented Reality (AR) projection on the road [63], Mobile Applications and Connected Devices [36, 69], Internet Vehicle-to-Everything (V2X) Technology [75], Interfaces on Street Infrastructure [45] and external Human-Machine interface (eHMI) [26, 42].

Empirical research on the aforementioned methods indicates that these approaches have their respective advantages in terms of clarity, familiarity, and politeness, yet they still present challenges. Regarding clarity, the effectiveness of Audio Signals is limited in conveying risk levels [72]; Light Signals are also inefficient in ambiguous situations and busy scenarios [16, 17, 30]; Street Infrastructure Interfaces may cause visual confusion and fail to indicate multiple pedestrians’ intents, affecting clarity [45]. Regarding familiarity, AR [63] offers novel interaction methods but its complexity can increase cognitive load, reducing familiarity; the effectiveness of Audio Signals in conveying risk levels varies across environments, limiting clarity; Mobile Apps and Connected Devices provide intuitive, personalized interactions for understanding AVs’ intentions but their effectiveness is contingent on widespread adoption and user familiarity [36, 69]; V2X Technology enhances safety through effective communication but faces challenges in network latency and reliability, reflecting limited user familiarity [75]. Regarding politeness, Audio Signals may be perceived as impolite, especially in varied environments where they can be intrusive [72]; Light Signals lack politeness in multi-vehicle scenarios, leading to visual confusion [16, 17, 30]; AR Technology may potentially be more polite by personalizing interaction information, minimizing pedestrian disturbance and misunderstanding [63].

From the studies mentioned above, it is evident that an effective pedestrian-AV interaction design should be taken its clarity, familiarity, and politeness into consideration. Therefore, in this paper, we also comprehensively take into account these factors for the selection and evaluation of gestures.

2.2 The State of the Art (SOTA) Dispaly-based eHMI.

Display-based eHMI has become a mainstream research direction. Display-based eHMI uses visual screens or projections that can display dynamic information. These display technologies have seen significant development in recent years and are being integrated into automobiles. For example, AUO showcased its Micro LED automotive display technology at CES 2023. This technology offers high brightness, high contrast, a wide color gamut, and rapid response times. The PML programmable smart headlights and ISD intelligent interaction lights equipped in HiPhi X can project interactions with pedestrians. These display technologies can be integrated inside vehicles, such as on windshields or the body of the car, or even projected onto the road or surrounding environment. In recent years, a large number of display-based eHMI design concepts have been proposed or applied. Examples include the Mercedes-Benz F015 concept car (Daimler, 2015)[4], the Nissan IDS concept car (Nissan Motor Corporation, 2015), the Volvo Concept 360 (Volvo Cars, 2018), the Smart EQ ForTwo concept (Daimler, 2017)[14], the BMW VISION NEXT 100 (BMW, 2016), and the Jaguar/Land Rover Virtual Eyes concept (Jaguar Land Rover, 2018) [53].

In addition to industrial products, the academic community has also conducted empirical research on various display-based eHMIs. Bazilinskyy et al. [3] found textual eHMIs in AVs clearer than non-textual ones, especially with zebra crossings, and recommended egocentric text for safety. Werner et al. [71] identified turquoise as the best color for AV external lighting, emphasizing visibility and uniqueness. Lau et al. [40] showed that dynamic eHMI enhances safety in yielding AVs but warned against mismatches between eHMI signals and vehicle actions. Dey et al. [18] discovered cyan with flashing or pulsing animations as effective for eHMIs in AVs, aiding pedestrian interaction.

While there has been considerable research on eHMI, it has been somewhat limited to interaction designs already present in traditional vehicles. Therefore, in this paper, we integrate the SOTA eHMI design and studies into an optimally effective eHMI to be used as a control group design (see details in Section 4.2, Fig. 6).

2.3 Driver Gestures and eHMI

In Katz et al.’s [38] study on driver-pedestrian interaction in cross-conflicts, it was found that after vehicle-pedestrian conflicts, drivers and pedestrians often communicate through non-verbal means like gestures and eye contact. Social norms or tacit communication play a significant role in predicting each other’s intentions  [48]. For instance, eye contact, gestures between pedestrians and drivers, or changes in vehicle speed can convey intentions  [1]. Stanciu et al. [59] summarized interactions between vehicles and pedestrians using gestures, eye contact, or headlights. Drivers use gestures to encourage other road users to pass or stop [61]. Pedestrians also use gestures to communicate with vehicles [32]. Research shows that gestures are a common method of traffic communication, improving driver-pedestrian interaction [13, 74]. Gestures (like waving) are common practices in negotiating right-of-way between pedestrians and drivers [20]. Similarly, eye contact is considered a social cue; it’s an indicative signal conveying to whom one is speaking and that subsequent actions or information will be meaningful [6]. Šucha [61] confirmed that both pedestrians and drivers use eye contact for communication, with pedestrians using it to encourage drivers to yield [72].

Fass et al. [26, 43] noted that with the increase of automated function vehicles, future traffic systems will be shared by autonomous and manually driven vehicles. In such scenarios, traditional communication methods between pedestrians and AVs, like eye contact and gestures, may no longer be effective [2, 34]. Hence, many studies have analyzed pedestrian-driver interactions at crosswalks in urban settings and how these interactions influence the behavioral design of future AVs. The focus is on understanding interactions at crossroads between drivers and pedestrians, deriving insights from these interactions to guide the development of autonomous driving functions [9, 45, 57, 62]

Ackermann et al. [1] believed that using existing communication methods (gestures, eye contact) by autonomous vehicles might be beneficial. These traditional and common behavior patterns of drivers expressing priority to pedestrians have been discussed between autonomous vehicles and pedestrians [73]. Risto et al. [52] emphasized the need for AV developers to recognize and understand the impact and importance of motion gestures in daily traffic interactions, and to design effective and scalable forms of human-vehicle communication in multi-party interactions. A simple gesture action performed well in comparisons across different interfaces, as it clearly conveyed intentions [45]. When gestures align with the information provided by eHMI, pedestrians’ willingness to cross the road in front of autonomous vehicles operating in yielding mode will increase. Conversely, it will decrease when gestures and eHMI information are inconsistent [12].

Figure 3:

View Figure

4.1 Scenario and Task Design

To explore the usability and performance of the selected gestures in facilitating pedestrian-AV interaction, we designed a VR experiment to compare all the gesture-based eHMIs along with baseline conditions. The VR experiment took place in a virtual scenario (See Fig. 9) where pedestrians and AVs met at uncontrolled crosswalks. The participant’s task is to control the virtual pedestrian’s behaviors (gaze and move) in response to AVs with different eHMIs. To address possible dizziness and nausea problems in pilot studies, we limited participants to sitting down and pressing two buttons that simulate walking in the virtual environment.

Before crossing, pedestrians usually observe vehicles around them and consider possible dangers [39]. We hypothesize that an effective eHMI should help reduce the amount of time pedestrians observe and hesitate. These times serve as indicators of the effectiveness of eHMI.[20]

Thus, a within-subjects design was utilized in this study. The usability of 12 eHMI designs was assessed, including eight gesture-based eHMIs (four yielding and four non-yielding gestures) , two SOTA eHMIs (light-based, one yielding and one non-yielding) and two AVs without any eHMI (one yielding and one non-yielding), as shown in Fig. 6. Each eHMI was presented to participants three times, and the sequence in which eHMI designs appeared was randomized. This randomization aimed to prevent subjects from discerning trends that could potentially influence the experimental outcomes. We have designed the system to randomly generate different types of AVs at each intersection. Each AV comes with its own eHMI and corresponding motion logic, all of which are programmed into an AV blueprint. In this experiment, there are 12 different blueprints. To ensure that each AV appears exactly 3 times, we will create two duplicates of each AV’s blueprint, expanding the blueprint library to 12 × 3 = 36 in total and assigning each blueprint a unique number from 1 to 36. As the pedestrian approaches the trigger point before an intersection, the program will randomly select an integer from this range and summon the corresponding AV. To ensure that each intersection corresponds to a unique blueprint without repetition, the system will exclude numbers that have already been randomly selected when moving to the next intersection.

Figure 6:

View Figure

Additionally, to encourage pedestrians to actively interpret eHMI signals and take action, a incentive mechanism is implemented in the experiment. Under conditions where the AV yields, a reward is triggered if the pedestrian’s hesitation time is less than 5 seconds. Conversely, when encountering a non-yielding AV, a reward is earned by successfully navigating the pedestrian through. No additional reward is given for failing to meet these conditions. Each time an additional reward is triggered, participants will receive extra money on top of base compensation.

4.2 Independent Variables

eHMI Designs: We assessed the 12 eHMIs designs. The SOTA eHMI design were labeled as Y0 and N0. AVs without any eHMI design were labeled as Y-base and N-base. The eHMI designs are summarized in Fig. 6.

Encounter Times: Each design was examined by each participant three times.

4.3 Dependent Variables

  The following measures were recorded or calculated to assess the effect of eHMIs:

Error Rate: Whether the subject misjudged the intent of the AV to calculate the Error Rate, the understanding of the intentions of AVs is reported by the participant orally. For non-yielding AVs, if the pedestrian makes an error in judgment and walks up to the crosswalk without waiting for the car to pass, he or she is bound to be hit by the car. The occurrence of a collision is recorded. The error rate for an eHMI design was calculated as the number of times that its intention was misunderstood, divided by the total times that the eHMI design was examined by all participants.

Difficulty in Comprehension: The score of the difficulty of understanding the shown eHMI on a Likert-type scale of 1 to 5, reported by the participant orally.

Perception of Danger: The score of the perceived level of danger when passed by AV with specific eHMI, on a Likert-type scale of 1 to 5, reported by the participant orally.

Duration of Observation: The time the participant spent observing the AV. Given that AVs always approach pedestrians from the left, pedestrians must turn their heads toward the left to observe the oncoming vehicle. In the VR, the camera’s field of view is 90°, therefore the AV only fully enters the pedestrians’ field of view when they turn their head 50° to the left. Once the observation is complete, pedestrians should then redirect their attention back to the road surface in front of them. The duration of observation could be recognized as the time when they turned their heads more than 50° to the left.

Duration of Hesitation: The time the participant spent hesitating to go forward. During the experiment, the participant was given a yellow button to press and hold to control walking. The time that the participant did not press the button while crossing the intersection was recorded to be the measurement.

Physiological Indicators: Electrodermal activity(EDA), heart rate (HR), and heart rate variability (HRV), measured by wearable monitoring device——Empatica E4 Wristband. Physiological indicators were derived :

SCR_Peaks_N: The number of peaks of the Phasic Skin Conductance Response (SCR).[7]

HR: The number of times the heart beats per minute.

HRV_RMSSD: The root mean squared differences of successive difference of intervals.[50, 56]

4.4 Apparatus

Using the Carla version of Unreal Engine[22], we designed a city scenery that closely resembles the real world. In order to simulate the real world as much as we could, we added streets, complex supermarkets, office buildings, residential architecture and plants, etc. The city street scenery is made by the software RoadRunner. As the dual carriageway, each lane is 5 meters wide, and both sides of the road have a sidewalk 3 meters wide.

Figure 7:

View Figure

Figure 8:

View Figure

Pedestrians will encounter AVs at crossroads. There are zebra crossings at the intersection, but there are no traffic lights. Pedestrians need to cross every crossroad. For a crossing, a starting point is marked 5 meters in front of the zebra crossing, and an ending point is marked at the end of the zebra crossing. Between the starting point and the ending point is the experimental stage, where pedestrians understand the intention of the autonomous vehicle and cross the road. The layout of the scenario in the virtual environment and the routes of AVs and pedestrians are illustrated in Fig. 7.

The generated positions and behaviors of AVs are standardized. We meticulously designed the motion behaviors and transition points of AVs, making it challenging for participants to infer vehicle intent through implicit information. The AV’s driving behavior consists of three stages: the first stage involves the AV traveling at a speed of 43.2 km/h for 2 seconds after being generated, during which the eHMI signals on the vehicle start playing. In the second stage, the AV undergoes a 2-second deceleration, reduces its speed to 14.4 km/h, and continues traveling for 9 seconds. In the third stage, the yielding and non-yielding AVs exhibit different motion behaviors. If the current AV decides not to yield, it continues at a constant speed of 14.4 km/h through the intersection. If the current AV decides to yield, it decelerates to a stop 4 meters before the crosswalk.

According to Fig. 8 A, the experiments are conducted in an empty indoor environment in which tables and chairs are placed. Beyond the desk, there is a PC (CPU: intel i5 13600kf, GPU: Nvidia rtx3080 10GB, RAM:32GB, ROM:4TB) with a virtual environment, an android cellphone (XIAOMI MIX 2s, CPU: snapdragon845, ROM:6GB), a physiological state monitoring wristband (Empatica E4 Wristband), a VR headset (Oculus Quest2), a display device(34-inch display or Redmi) and two physical buttons. The VR device was connected to the PC with a USB 3.0 cable. The two buttons connected to the computer are red and yellow, separately. While running, the PC will sync the picture to the monitor, and the E4 wristband is connected to an Android cellphone by Bluetooth. A paper answer sheet was used to record how participants rated their responses during the experiment.

4.5 Procedure

After welcoming the participants and introducing the research target briefly, we asked them to sign an informed consent form. Next, we explained experimental tasks and procedures in detail and told them how many buttons they could use and the data needed to be collected. The experimenter wore a VR headset to demonstrate the approaches to operating for participants. Before continuing, we ensured that we had solved any unclear problems posted by participants. Afterward, the researchers helped the participants wear E4 wristbands and launched the acquisition software running on the Android cellphone. Then, the participants were helped to put on the VR headset and adjust it to a position where it was comfortable to wear. Until confirming there were no problems, participants entered the virtual environment of the first round. To collect baseline physiological data for 30 seconds, participants were also asked to breathe steadily without any movement. 30 seconds later, the researcher issued a start command, and the participants took over the operation buttons to start the formal experiment.

Figure 9:

View Figure

During a round of the experiment, participants pressed and held the yellow button to control to walk to the starting point according to the instructions in VR. When the pedestrian in VR played by the participant arrived at the starting point of each crossroad, Unreal Engine started to record all the data required. In the meantime, AV was generated and approaching the crossroad by the left side of the pedestrian. Participants need to judge the vehicle intention according to the eHMI information designed in the experiment and decide whether to go before the AV. When pedestrians arrive at the ending point of the crossroad, a crossing is considered to be completed. Participants need to answer the questions at the pop-up windows orally. Below is the content of the question:

The researcher recorded the answers in writing. Participants can press the red button to close the pop-up window after finishing the questions and then use the yellow button to control the pedestrian to go to the next intersection. The procedure of the experiment is illustrated in Fig. 9.

Semi-structured interviews lasting no more than 30 minutes were conducted after the experiments. Participants need to review the process of the experiment together and the questions answers. If they were bumped/ got a wrong answer to question 1 in the experiment, they need to check the reason. During the process of answering, participants can view the video to review different kinds of gestures anytime. For questions 2 and 3, after review, participants need to re-evaluate and rank all different yielding or non-yielding eHMI effects and explain why. Finally, there were a few questions related to AVs and gesture-based eHMI at the end.

4.6 Participants

In August 2023, this study recruited adults between 18 and 55 years old in Beijing. The recruitment criteria were as follows: i) No cognitive disorder, basic social cognition, and judgment ability, normal mental state. ii) No perception disorders, visual, auditory, and sensory functions are normal. iii) Normal athletic ability without any physical disability. iv) No eye or skin infections. v) Participants have long-term residency in the North China region, being familiar with local traffic rules and cultural norms. vi) Participants have previous experience with VR devices and have not exhibited any adverse reactions. vii) Participants are in a normal state and have abstained from alcohol or similar substances before participating in the tests.

Finally, we selected 33 healthy adults with VR play experience from many applicants to participate in this experiment, and all the participants needed to voluntarily sign informed consent before the experiment. Participants played as pedestrians to interact with automatic vehicles in the VR world. Each participant received 60 RMB as compensation after the experiment finished. This experiment has been approved by the local university’s Institutional Review Board (IRB).

After being scrutinized, the data from 31 participants was considered reliable and analyzed. Due to equipment problems during the experiments with two participants, the data for them were incomplete. Consequently, the data from these two participants were excluded from the analysis.

4.7 Analysis

             We checked whether participants’ reported interpretations of the AVs’ intentions were consistent with the true intentions and recorded the times of errors. For a non-yielding vehicle, the time of collisions was counted as a supplement.

The error rate for an eHMI design was calculated as the number of times that the participants misunderstood the intention of AV with the eHMI design, divided by the total examined times (each design was examined by each participant three times, the total number N = 31 × 3 = 93), as aforementioned in Section 4.3.

Because “Error” is a categorical variable (error = 1, no error = 0), we used the chi-square test to evaluate whether there is an association between different eHMI designs and “Error”.

Given that each participant encountered AVs with each eHMI design three times, perceptions and behaviors of participants may vary with each time. To investigate whether there is an interaction between the different eHMI designs and the times of encounter, we conducted a two-way repeated measures ANOVA for analysis. We tested the data for normality of the residuals and sphericity assumptions with Shapiro-Wilk’s and Mauchly’s tests, respectively. If the assumption of sphericity was violated, we corrected the tests using the Greenhouse-Geisser method. For significance testing, we used two-way repeated-measures ANOVA to identify significant effects and applied Bonferroni-corrected t-tests for post-hoc analysis. Further, we report the generalized ETA squared \(\eta _G^2\) as a measure of the effect. If the homogeneity of variance assumption was violated, Welch’s ANOVAs were used. If normality was violated according to the results of Shapiro-Wilk’s tests, we applied the Aligned Rank Transform (ART) as proposed by Wobbrock et al. [24].

Further, we plotted the average data as line graphs, as shown in Fig. 11 and Fig. 13. In a line graph, the X-axis represents the 1st, 2nd, and 3rd time, and the Y-axis represents the average of the 31 participants’ performance.

Additionally, we applied the Student’s t-Tests for physiological data to examine the difference between the baseline and each level. If normality was violated, Wilcoxon Rank-Sum tests were used.

4.8 Results

4.8.1 Yielding Vehicles.

Error Rate: The error rates are summarized in Table.1 There is a significant association between different eHMI designs and “Error” (χ²(5) = 31.445, p <.001). The intentions of AVs without any HMI design were inaccurately understood 16 times. Y2, Y3, and Y4 were almost not misinterpreted by the participants (N=2, 1, 1). The frequency of misinterpretation of the intentions of Y1 and Y0 is relatively low (N=7, 7).

Table 1:

Yielding eHMI design	Y1	Y2	Y3	Y4	Y0	Y-Base
Error Rate	7/93	2/93	1/93	1/93	7/93	16/93

View Table

Table 1: Error Rate for Yielding eHMI

Difficulty in Comprehension: A significant main effect was observed (\(F(5, 150)= 22.098, p \lt 0.001, \eta _G^2=0.250\)) for eHMI designs on the difficulty in comprehension for yielding vehicles. There was no significant main effect for the times of encounter.

But the critical finding was the significant interaction effect for eHMI designs and encounter times (\(F(10, 300)= 2.182, p = 0.041, \eta _G^2=0.019\)). For Y0, there is a significant difference between the first time and the third time (p = 0.002), with a lower difficulty in comprehension for the third time.

Results of post-hoc tests indicate that the Y0, Y2, Y3, and Y4 are significantly easier to understand by participants than Y1 and Y-Base (p < 0.05). No significant difference is found between AV with Y1 and AV without any eHMI (p > 0.05). (Fig.10(a))

Among yielding vehicles, the average level of difficulty in perceiving the intentions of AVs without any eHMI design was found to be the highest and had slight variations among the three encounters. The yielding intentions of AVs with Y1 are relatively hard to understand either. As the number of encounters increased, the decrease in comprehension difficulty was not noticeable. The difficulty of obtaining yielding intentions from AVs with Y3 is the lowest, followed by Y2, regardless of the number of encounters. The level of comprehension difficulty for Y0 exhibits an approximately linear decrease, but for Y2 has an upward trend with increasing repetitions. (Fig.11(a))

Perception of Danger: There are significant differences (\(F(5, 150)= 10.989, p \lt 0.001, \eta _G^2=0.114\)) between different designs on the perception of danger for yielding vehicles. There was no significant main effect for the times of encounter and no significant interaction effect for eHMI designs and encounter times.

In contrast to AVs without eHMI applications, the applications of all five types of eHMI designs result in significant decreases in people’s sense of danger compared with Y-base, based on the results of post-hoc tests (p < 0.05). (Fig.10(b))

It was shown that participants perceived AVs without an eHMI design as the most hazardous, while AVs equipped with Y3 elicited the highest sense of safety among participants, regardless of the number of encounters. (Fig.11(b))

Figure 10:

View Figure

Figure 11:

View Figure

Duration of Observation: A significant main effect was found (\(F(5, 150)= 12.002, p \lt 0.001, \eta _G^2=0.092\)) for eHMI designs on the duration of observation for yielding vehicles. There was no significant main effect for the times of encounter.

We found a significant interaction effect for eHMI designs and encounter times on the duration of observation for yielding vehicles (\(F(10, 300)= 2.900, p = 0.010, \eta _G^2=0.029\)). For Y4, there is a significant difference between the first time and the third time (p = 0.003), with a lower duration of observation for the third time. For Y0, there is a significant difference between the first time and the third time (p = 0.027), with less duration of observation for the third time.

In comparison to AV without eHMI design, the implementation of Y0, Y2, Y3, and Y4 demonstrates significant benefits in terms of decreasing the observation time (p < 0.05). However, the application of Y1 takes even longer than the time spent observing when compared to AV without eHMI design (p > 0.05). (Fig.10(c))

Participants need to spend a greater amount of time to observe the intents of AVs equipped with Y1, compared to those without an eHMI design, during their initial encounter, and the duration of observation indicates a slight reduction as the number of repetitions increases. For AVs without an eHMI design, observing time demonstrates an upward trend with increasing repetitions. The duration required to figure out the intentions of AVs with Y2 is minimal during the initial encounter, but it gradually lengthens during subsequent encounters. The time required to observe Y3, Y4, and Y0 shows a noticeable decrease as the number of encounters rises. During the last meeting, it was found that the participants were able to comprehend the intentions of AVs equipped with Y0 in the shortest amount of time. (Fig.11(c))

Duration of Hesitation: A significant main effect in the duration of hesitation time was observed (\(F(5, 150)= 23.674, p \lt 0.001, \eta _G^2=0.202\)). There was also a significant main effect for the times of encounter (\(F(2, 60)= 12.560, p \lt 0.001, \eta _G^2=0.0.028\)).

The critical finding was the significant interaction effect for eHMI designs and encounter times (\(F(10, 300)= 3.352, p = 0.003, \eta _G^2=0.0.023\)). For Y3 and Y4, there is a significant difference (p < 0.05), with less duration of hesitation for the second time and third time.

The utilization of Y0, Y2, Y3, and Y4 in comparison to AV without eHMI design proves to be significantly advantageous in terms of reducing the amount of time of hesitation (p < 0.05). Nevertheless, the application of Y1 fails to produce any significant distinction when compared to AV without eHMI design (p > 0.05). (Fig.10(d)) Post-hoc analysis revealed significant differences in the duration of hesitation between the first and second encounters as well as between the first and third encounters (p < 0.05). It indicates that participants were able to respond more quickly when encountering an eHMI design that they had previously encountered.

For yielding vehicles, we found that pedestrians reacted worst to AVs without any eHMI design (Y-base). The hesitation time for the next two encounters rises compared to the performance for the first encounter. The performance of participants when confronted with Y1 is similar to that of Y-base, exhibiting a little improvement in the subsequent two encounters. However, it remains the worst among all eHMIs. The duration of hesitation indicates a discernable decreasing trend for Y2, Y3, Y4, and Y0. The pedestrians performed best in deciding their actions when they met AVs with Y0 in the first encounter. When considering the overall performance of the three encounters, it can be observed that Y4 shows the most pronounced Learning during experiencing. (Fig.11(d))

4.8.2 Non-Yielding Vehicles.

Error Rate: The Error Rate and Collision Rate were summarized in Table.2. There is a significant association between between different eHMI designs and “Error” (χ²(5) = 28.049, p < 0.001). There is a significant association between different eHMI designs and “Collision” (χ²(5) = 12.638, p = 0.027). When answering question No. 1, some participants tend to report the observed behavior shown by the AVs after crossing the roads rather than their true thoughts before crossing. This may lead to a possible situation where the collision rate is greater than the error rate.

The intentions of AVs without any eHMI design were incorrectly interpreted 17 times, which was similar to the yielding AVs. The N4 eHMI design had the lowest number of misunderstandings, only eight times, compared to other eHMI designs. The frequency of misinterpretation of the intentions of N1, N2, and N0 is relatively low (N=12, 17, 11). Particularly, N3 was the worst among all eHMIs, and even more difficult to accurately understand than AV without any eHMI (N=29).

Table 2:

Non-yielding eHMI design	N1	N2	N3	N4	N0	N-Base
Error Rate	11/93	12/93	23/93	8/93	11/93	17/93
Collision Rate	12/93	17/93	29/93	7/93	7/93	13/93

View Table

Table 2: Error Rate for Non-Yielding eHMI

Difficulty in Comprehension: A significant main effect was found (\(F(5, 150)= 14.818, p \lt 0.001, \eta _G^2=0.169\)) between designs on the difficulty in comprehension for non-yielding vehicles. There was no significant main effect for the times of encounter and no significant interaction effect for eHMI designs and encounter times.

Results of post-hoc tests indicate that the N0, N1, N2, and N4 are significantly easier to understand by participants than N-Base (p < 0.05). As compared to AV without eHMI, N3 has no advantages in terms of cognitive comprehensibility (p > 0.05). (Fig.12(a))

Figure 12:

View Figure

Among non-yielding vehicles, the average level of difficulty in perceiving the intentions of AVs without any eHMI design was found to be the highest and has an upward trend with increasing repetitions. AVs with N0 and N3 show similar levels of understanding difficulty. The non-yielding intentions of AVs with N2, N1, or N4 can be relatively easy to perceive by participants among which, with N4 being the most easily understandable. The level of comprehension difficulty for N2 tends to diminish with increasing repetitions. (Fig.13(a))

Figure 13:

View Figure

Perception of Danger: A significant main effect was found (\(F(5, 150)= 9.703, p \lt 0.001, \eta _G^2=0.078\)) between designs on the perception of danger for non-yielding vehicles. There was no significant main effect for the times of encounter and no significant interaction effect for eHMI designs and encounter times. The use of N0, N1, N2, and N4 on AVs results in significant improvements in participants’ perception of safety when compared to AV without any eHMI design (p < 0.05). The application of N3 does not have a significant impact on the perceptions of risk (p > 0.05). (Fig12(b))

It was shown that participants perceived AVs without an eHMI design as the most hazardous. It has been observed that specifically for N1, the perception of danger of participants tends to decrease as the number of encounters increases. For the 2nd and 3rd encounters, N1 is regarded as the most secure eHMI design for pedestrians. Participants experience a similar sense of danger when they encounter AVs with N2 and N4. (Fig.13(b))

Duration of Observation: A significant main effect was found (F(5, 150) = 3.504, p = 0.011) between designs on the duration of observation for non-yielding vehicles. There was no significant main effect for the times of encounter and no significant interaction effect for eHMI designs and encounter times.

As compared to AV without eHMI design, the implementation of N2 and N4 is significantly helpful in shortening the observation time (p < 0.05). The utilization of other eHMI designs does not provide any significant advances (p > 0.05) towards reducing the time spent observing. (Fig.12(c))

Among non-yielding vehicles, the average duration of observing the AVs without any eHMI design was found to be consistently the longest, irrespective of the number of times it is encountered. The time needed to understand the intentions of AVs equipped with N3 is shortest on the first encounter but increases when encountered again. The duration required to observe AVs with N1 or N4 decreases with increasing repetitions. When met for the second or third time, N4 requires the shortest duration for comprehension. (Fig.13(c))

Duration of Hesitation: A significant main effect in the duration of hesitation time was observed (\(F(5, 150)= 8.000, p \lt 0.000, \eta _G^2=0.049\)) between designs for non-yielding vehicles. There was no significant main effect for the times of encounter and no significant interaction effect for eHMI designs and encounter times.

The duration of hesitation when meeting with Avs equipped with N3 is noticeably lower compared to other counterparts. Nevertheless, it does not imply that N3 is the most effective. In contrast, pedestrians often misinterpret the non-yielding intentions of AVs equipped with N3, leading them to initiate crossing the crosswalk before AVs arrive. (Fig.12(d)

4.8.3 Physiological State.

Figure 14:

View Figure

SCR_Peaks_N (the number of SCR peaks) can indicate the number of times the participant was stimulated over a period of time [7]. The participants were not exposed to any external stimuli during the collection of their baseline physiological data. During the experiments, participants were stimulated in each trial. ANOVA reveals no statistically significant differences between designs. For all eHMI designs, the SCR_Peaks_N is significantly different from the baseline (p < 0.05). (Figure 14)

HR and HRV_RMSSD have no significant differences, neither between designs nor between each design and baseline.

4.9 Qualitative Data

In addition to quantitative data, qualitative feedback also enables us to gain insights from the subjective reasoning of participants. We conducted a thematic analysis of the responses to semi-structured interviews, which required participants to recall the process of answering questions and explain the reasons for ranking eHMI. At the end, there will be several questions about trust in autonomous vehicles and gesture interaction. The insights generated from this are discussed here, and some relevant participant quotes are provided. When recalling the situation of their experiment, participants often use various words to indicate gestures and lights, such as "driver", "red light", "green light", etc., to refer to the eHMI of the vehicle they saw and even imitate the gestures in memory during the interview to represent the eHMI they saw at the time. For the convenience of discussion, the numbering of gestures will be used in subsequent quotations to clarify the gestures mentioned by the participants, such as Y1, N1, etc. All quotes used in the paper were translated into English by the lead author and checked by other co-authors.

Recipient and content: "It must be gesturing towards me"

The key to whether pedestrians can effectively interact with autonomous vehicles is whether pedestrians can receive and understand the information conveyed by the vehicle eHMI. Before receiving information, pedestrians need to know that they are the Receiver of eHMI information. However, some participants mentioned that they did not believe that the car was sending messages to themselves during the experiment. As P27 said:

In addition to lights, some gestures have the potential to confuse people.

Nonetheless, the majority of participants stated that when they saw the gesture, they immediately recognized that the car or virtual "driver" was interacting with pedestrians.

On the other hand, the content of the information conveyed by eHMI is also very important. The utilization of lights, specifically red and green, for conveying information is characterized by its simplicity, as it is limited to the binary options of "yes" or "no", "go" or "stop". In contrast, gestures possess the capacity to convey a greater amount of complicated information due to their intricate series of motions, encompassing factors such as the direction of gesture movement, frequency of repetition, etc.

Nevertheless, the complexity may not always be advantageous for all types of information required to be conveyed, as P17 told us:

When indicating the intention not to yield, several participants agreed on the use of a static gesture rather than a dynamic one on an eHMI system. They felt that a dynamic gesture could result in confusion, especially in the case of N3, which shows a waving motion closely resembling that of Y2.

Interaction distance and the observability of eHMI: "It was just too little to notice from a distance"

Through conducting experiments and interviews, it was observed that pedestrians consider "distance" to be an essential factor, which is consistent with the conclusion of Dey et al. [19]. The identification of a vehicle’s intent at an early stage allows pedestrians to cross the street at a safer distance from the vehicle, resulting in increased safety and time efficiency. Nevertheless, it is hard to achieve in both virtual reality (VR) tests and real-world scenarios——the human eye has visual limitations in seeing objects that are far away clearly. When driving close, their specific intentions become evident within a few seconds, either by coming to a stop or passing by. Some of the participants still judge the vehicle’s intention by observing its movement behavior, such as waiting for the vehicle to either pass by or come to a stop before proceeding to cross the street. They said:

Avoiding this quandary is also challenging for gesture-based eHMI. Gestures without a sufficient range of motion exhibit visual similarity when observed from a distance. In cases where these gestures have distinct intentions, their possible consequences can be quite severe.

It can be found that an extended visual distance enhances pedestrians’ ability to comprehend the intentions of a vehicle, regardless of whether it involves a gesture or any other form of eHMI. Hence, for gesture-based eHMI, the significance of a wider range of motion, increased surface area of the arm exposed, and an adequately discernible interface can not be ignored.

Unfamiliar traffic signal: Green indicates “Go” ? Red indicates “Stop” ?

Although traffic regulations in China have been regulated and people have become accustomed to the fact that green indicates ’go’ and red indicates ’stop’, participants’ performance on red and green lights was not at the expected levels, and participants’ ratings of red and green lights varied widely. Initially, the majority of the people involved in the experiment would guess as to whether the color of the eHMI light strip on the car corresponds with the color of the usual traffic lights, specifically green indicating ’go’ and red indicating ’stop’.But they still need to try empirically before they can confirm the facts. It is consistent with the conclusion of Dey et al. [17].

Participants who took a conservative approach to crossing the street found it easier to understand that the red light signals ’Stop’ rather than the intention represented by the green light:

Furthermore, we discovered that in addition to their inconsistent meanings, red and green lights appeared to many participants to be very different in other ways. The participants had different levels of difficulty comprehending the red and green lights:

Comprehensibility of gestures:"I immediately understood its meaning"

Participants explaining the reasons for the ordering in the interviews often mentioned that they often encountered a certain gesture in their own lives, which could either make the gesture simple to understand or cause confusion due to the common intent of the gesture in their lives. For example, Y1 can also indicate you’re awesome, while N3 can mean hello, farewell, and so on. Participants said:

Although some of the gestures were not familiar to the participants, they were easy to learn and effective:

Participants largely felt that the ineffective gestures well enough were the lowest-ranked of all eHMIs. However, when compared to an AV without any eHMI designs, the attitudes of participants varied.

Although some gestures are ineffective at helping pedestrians understand the intent of the vehicle, these gestures will increase walkers’ sense of security when crossing the street.

Communicating in a more friendly manner: "I am communicating with the vehicles"

Information conveyed through gestures is rich. In addition to the original meanings and directions, gestures also contain the driver’s feelings and attitudes in daily life. In the experiment, although only the gesture animation was displayed on the external interface of the autonomous vehicle, the participants could still feel the "driver" emotion and think that they are communicating with a "person". According to the participants:

In the last questions of the interview, a majority (N=19) of the respondents indicated an increased level of trust in the autonomous vehicle after the experiment. Additionally, they expressed confidence in the vehicle’s ability to effectively convey its intentions to pedestrians. The majority of participants (N=26) showed agreement with the potential use of gestures as a means of facilitating communication between autonomous vehicles and pedestrians in future scenarios. This tendency was primarily influenced by good experiences during the experiments.

To understand the influencing factors of participants’ ranking of gestures, We adopted grounded theory in the qualitative analysis[60], reviewing and labeling the emerging codes in an iterative process. The lead author conducted inductive coding of the data and discussed the resulting themes with other co-authors to identify principal themes from the data. The primary codes included: Clarity of gestures, familiarity with gestures, politeness of gestures, and so on. The whole research team reached a consensus on the final themes.

5.1 Survey Design

This questionnaire research categorizes gestures into three dimensions for evaluation: clarity, familiarity, and politeness. Clarity refers to how clearly the gesture expresses intentions. Familiarity refers to how familiar the gesture is in daily life, and politeness refers to how polite the gesture is in expressing intentions. The Gesture Research Questionnaire collected basic demographic information of participants alongside evaluations of three gesture dimensions in a two-by-two comparison format. Participants were also asked to rate various gestures on the three dimensions (clarity, familiarity, politeness) using a two-by-two comparison rating. Technical term abbreviations were explained upon first use. The text adheres to all given principles and requires no further revision.

The Analytic Hierarchy Process (AHP) is a renowned method for decision-making, initially proposed by American operations researcher Saaty in the early 1970s. It involves decomposing decision-making elements into levels such as objectives, criteria, and options, which are then qualitatively and quantitatively analyzed through pairwise comparisons [28, 54, 66]. In this questionnaire study, an evaluation model was developed using the AHP method. Participants’ scores for three-dimensional weights and multidimensional scores for each gesture were obtained by the two-by-two comparison method. To construct a unified judgment matrix, group opinions were harmonized by calculating the geometric mean. Additional analysis provided the three-dimensional scores for judging the gestures, as well as the scores of different gestures on the three dimensions. The results of the specific data analysis will be presented in the following sections. Additional information on questionnaire design and analysis can be found in the attached appendixA.

Figure 15:

View Figure

5.2 Respondent Recruitment

The survey has been approved by the local university’s Institutional Review Board (IRB). The survey was distributed via the Internet and was accessible to all respondents through an anonymous link. Since respondents played the role of pedestrians in the study, there was no screening for categories such as gender, age, and occupation, which allowed a wider range of people to participate in this survey. As an incentive to participate in the survey, all respondents whose questionnaires were verified as valid were rewarded with about $0.7.

5.3 Data Quality

In this questionnaire study, several measures were taken to ensure data quality. Questions were phrased in everyday language to facilitate comprehension. For better visual understanding, moving images with gestures were included in the questionnaire. Moreover, participants were informed that data collection was completely anonymous to prevent bias. Two questions were included in the questionnaire to eliminate the possibility of machine answers. Additionally, any questionnaires that were completed too quickly or took too long were deemed invalid (Less than 2 minutes or over 30 minutes).

5.4 Respondent Demographics

We distributed 394 questionnaires to the general public via the Internet. We removed 41 invalid questionnaires, leaving 353 valid responses after a consistency test. The participants included 164 males and 189 females from various regions of mainland China. They ranged in age, with a larger proportion falling in the 18-25 age group. See the table for precise demographic information.

5.5 Result

After analyzing the data, we obtained the participants’ ratings for each gesture. In the following, we will discuss the ratings of the different gestures and the differences between the three rating dimensions of the gestures in the different age groups.

5.5.1 Total score for gestures.

For yielding gestures, the gesture with the highest score was Y2, followed by Y3, while Y1 and Y4 scored lower and closer, combining the weights of the three dimensions as well as the scores. In terms of clarity, gesture Y2 performed the best, followed by Y3, Y4, and Y1. The gesture with the best familiarity was Y2, followed by Y3, Y4, and Y1. Y4 scored highest on politeness, followed by Y1, while Y2 and Y4 scored lower and closer. Combining the above data, Y2 receives the highest overall score and ranks first in both clarity and familiarity but falls short in politeness and ranks third. The yield gesture Y3 comes in second and has an obvious edge in politeness. The Y1 yield gesture ranks third. It scores poorly in the clarity and familiarity categories but has some advantage in politeness. Yielding Gesture Y4 ranked fourth and did not perform well in any of the three dimensions.

Figure 16:

View Figure

For the non-yielding gestures, the gesture with the highest score was N1 (T-gesture). N4 and N2 received the second and third highest scores, respectively, while N3 was the non-yielding gesture with the lowest total score. The clarity scores for N1 and N4 were higher and closer, and the clarity scores for N2 and N3 were lower and closer, with N1 gesture having the highest clarity score and N3 the lowest. Regarding familiarity, N1 and N4 scored the highest, while N2 and N3 scored lower and similar. Regarding politeness, N1 performed best, while the other three gestures showed little difference in this level. Summarizing the data, the highest total score was obtained by the N1, which performed best in three dimensions, with a more obvious advantage. The non-yielding gesture N4 ranked second, showing better performance in the dimensions of clarity and familiarity but no advantage in politeness. The non-yielding gestures N2 and N3 ranked third and fourth, respectively, and did not perform well in all three dimensions.

5.5.2 Age differences in three evaluation dimensions.

Other noteworthy phenomena were discovered during the data analysis. When evaluating the importance of the three gesture dimensions, the 12-25 age group emphasized familiarity, while the 25-45 and 45+ age groups emphasized politeness. In the case of yielding gestures, Y2 and Y3 were found to be dominant, with only a small difference between them. The gesture Y3 is more popular among the older age groups because the older age groups place more value on the politeness of the gesture, and Y3 has a clear advantage in terms of politeness. Thus, the phenomenon is observed that younger people prefer Y2 and older people prefer Y3 as the optimal gesture. Opinions on the non-yielding gesture were consistent across age groups, with N1 considered optimal and specific rankings agreed upon. No statistically significant differences in the gender dimension were found in the study results.

Figure 17:

View Figure

6.1 Specific receiver and clear content

Our findings show that the clarity regarding the recipient of the signal is paramount in the interaction between pedestrians and AVs. Among various eHMI methods, gesture-based eHMI stands out as a more effective means of communication. Pedestrians can promptly discern that the AV engages with them through gestures.

Firstly, it is crucial to address the issue of some participants needing help to grasp the role of lighting signals. This could stem from the deeply ingrained habits formed throughout pedestrian-driver interactions since the automobile’s invention. Alternatively, the default understanding in traffic scenarios might be that only humans can interpret gestures. Consequently, when a gesture-based eHMI initiates a signal, it instantly captures the attention of other nearby individuals. In contrast, the lighting of the eHMI lacks clarity for the message receiver. This is due to the ubiquity of lighting applications in contemporary life, resulting in light signals lacking distinct receivers. Lights assume various roles in artificial objects, such as status indicators or decorative elements. Consequently, pedestrians might mistake the car’s lights and light bars for mere adornments. Some participants even perceived the light eHMI on the AV as an indicator of the vehicle’s status or a means of signaling to other vehicles. In real-world scenarios, a vehicle’s lights 10 meters away struggle to draw pedestrians’ attention.[58] Additionally, it is crucial to ensure that pedestrians can perceive the signals the eHMI emits after drawing their attention to them. In the case of gesture-based eHMI, the recognition of actions plays a pivotal role in its effectiveness. The design should make gestures with distinct intentions easily distinguishable, especially from a distance. Beyond a specific range, the eHMI may fail to convey the message, regardless of the gesture. Nevertheless, adjusting factors such as the gesture range, repetition frequency, display position on the car body, and skin color may alleviate this issue. In the future, designers should prioritize these factors when developing gesture-based eHMIs.

6.2 Overlapping semantics

The importance of avoiding pedestrian misinterpretation of vehicle intentions stems from the potential overlap in semantics between gestures and lights. In specific scenarios, people associate a particular signal with a specific meaning over an extended period. Consequently, introducing new semantics for the same signal in a different context can lead to confusion and resistance among individuals. Furthermore, the correspondence between colors and intentions only sometimes aligns with common expectations, and participants explained these discrepancies. For instance, in conventional vehicles, red lights have traditionally been used as brake lights, leading people to associate red lights with braking. However, red lights on other artificial objects often signify standby or inactivity. Thus, using red lights in vehicles, eHMI may convey contradictory meanings, potentially baffling participants.

Figure 18:

View Figure

Some gestures exhibit more complex overlapping semantics compared to lights. For instance, Y1 might convey "ok," Y2 could signify "get out of the way," Y3 may denote "please," while N1 could express both "pause" and "technical foul." N3, on the other hand, encompasses meanings such as "hello," "goodbye," and "no," and N4 may convey "reject," "error," "warning," and so on. Notably, not all gestures with overlapping semantics are equally prone to misunderstanding by pedestrians. For gestures like Y2, Y3, N1, and N4, their multiple meanings are often interconnected, making it easier for individuals to relate them to the vehicle’s intent in traffic scenarios. This could explain why certain gestures perform as well as, or even better than, light-based eHMI. However, the overlapping semantics of some gestures can be exceptionally confusing, as seen with the N3 gesture. Drawing parallels between the semantics of "hello," "goodbye," and "do not" is challenging. The meaning of this gesture varies substantially across different scenarios and lacks a fixed interpretation in traffic contexts. Consequently, pedestrians frequently struggle to correctly interpret this gesture, sometimes even more so than an empty vehicle. They tend to misinterpret the intent to yield as the intent not to yield, leading to increased observation and hesitation times when encountering such gestures on two occasions, further exacerbating their confusion. Conversely, gestures like Y4, which are less common in daily life, are quickly understood by pedestrians once they learn their meaning. Therefore, when designing gestures for vehicle eHMI, it is crucial to consider whether a gesture already carries widely accepted meanings, whether these meanings share commonalities, and whether it might be necessary to create entirely new gestures. Please do so to ensure clarity among pedestrians.

6.3 A polite gesture or a commanding light?

Gesture-based eHMI holds a significant advantage over light-based eHMI due to the different modalities, as it can convey a broader range of information. While lights are limited to just two colors, red and green, and offer relatively simple messages, such as "yes" or "no," "go" or "stop," gestures can convey semantic, directional, emotional, and attitudinal information. Participants who prefer light-based eHMI argue that it is more efficient. They contend that they can quickly discern the car’s intentions by simply remembering the meanings of cyan and red lights. They spend more time observing gesture-based eHMI, as extracting visual information from gestures is perceived as a slower process. AV lighting eHMI, with its color information and flashing patterns, allows pedestrians to grasp the situation rapidly. In contrast, processing additional information from gesture-based eHMI takes more time. It is acknowledged that simplicity trumps complexity in this context, making gestures less efficient. Data from Study 1 show that the hesitation time for Y0 and N0 gestures is notably low. However, it is essential to recognize that pedestrians are not machines but humans with cognitive processes. Combining quantitative and qualitative data, we discover that eHMI not only objectively influences pedestrians’ success rates when crossing intersections but also affects their perception of risk. Specifically, gesture-based eHMI tends to make pedestrians feel safer during intersection crossings. Nevertheless, when it comes to the "better" gestures, there is no significant difference in understanding difficulty, observation time, or hesitation time compared to light-based HMI. Study 2 reveals that politeness is crucial, particularly for individuals aged 25-45 and those over 45. The politest gestures, "Y3" and "N1," received high ratings in both studies. The utilization of gesture-based eHMI facilitates the interaction between pedestrians and AVs through the involvement of an anthropomorphic "agent", conveying the intentions of AVs to pedestrians, as well as engaging in mutual exchanges with them through gestures. This is similar to the real traffic scenario in daily life. Pedestrians will not consider the action directed towards them as coming from an inanimate device, namely an AV. Instead, they will view the virtual "driver" as a human being who is carrying out interaction and providing responses to the information transmitted by the "agent". Furthermore, gesture-based eHMI will enhance the safety of pedestrians and gain their trust due to the trust people have in real drivers.

6.4 Learning during experiencing

When comparing pedestrians’ performance in encounters with different eHMIs over three instances, we discern some differences in their interactions with autonomous vehicles equipped with various eHMIs. Participants tended to learn during experiencing. Different eHMIs exhibit varying learning costs, manifesting in distinct performance levels during initial encounters. Moreover, there are discernible differences in learning rates, with pedestrian performance improving as the number of encounters increases.

Regarding learning costs, the study suggests that the more familiar an eHMI is in daily life, the lower the associated learning cost [33]. Pedestrians perform better during their initial encounter with eHMIs resembling familiar displays. Light-based eHMI, a relatively standard and recognizable format, yields shorter hesitation times when it appears for the first time, as exemplified by "Y0." Its resemblance to everyday traffic lights contributes to this familiarity. Similarly, when pedestrians encounter an unfamiliar gesture-based eHMI for the first time, gestures commonly used in daily life incur lower learning costs and are grasped more swiftly. For instance, in Study 1, pedestrians exhibited the shortest observation time when first encountering "Y2," a gesture frequently used in daily life. The findings in Study 2 corroborate this observation, particularly with "Y2" performing exceptionally well in a survey based on a questionnaire, which entails only one repetition.

Concerning learning rates, it is evident that different eHMIs exhibit distinct tendencies. Light-based eHMI mainly conveys intentions through colors and logo animations, fostering stable learning and memory of abstract graphics associated with vehicle intentions. For instance, the decrease in understanding difficulty and hesitation time for "Y0" follows a nearly linear trend. Even unusual gestures have the potential to be rapidly learned and accepted. Take "Y4" as an example: despite its initial lack of familiarity, it impressed pedestrians due to its straightforward and easily recognizable characteristics, coupled with instructions that lacked overlapping semantics with the direction of pedestrian movement. As a result, "Y4" exhibited that it is learned the best by participants, performing exceptionally well by the third encounter.

6.5 Future work and limitations

This research has several limitations. Firstly, our VR experiments focused only on the interaction between autonomous vehicles and pedestrians, excluding other traffic participants. Future research needs to investigate the interaction effects of autonomous vehicles using gesture-based eHMI in more complex mixed traffic scenarios, involving pedestrians, cyclists, and other non-autonomous vehicles. Additionally, one of the challenges in the development of autonomous vehicles is the "open set" nature of the environment, with countless factors that could pose challenges. Pedestrian behavior varies widely, and future work should explore whether different factors, such as different speeds, day or night conditions, single or multiple pedestrians, impact the interaction with autonomous vehicles.

Furthermore, we were interested in combining lighting and gestures to see if there would be better results. However, due to the eight different gestures used, combining them with lighting or other types of eHMI would result in a large number of experimental conditions. This would make the participation time in VR experiments very long and could easily lead to dizziness or nausea. Attempting to combine gestures and lighting and testing their effects is undoubtedly a promising avenue for future research.

Similarly, during the internal testing phase of the experiment, we attempted to have participants stand and wear VR headsets, physically walking in the real world to control the motion of their virtual pedestrian avatar. However, we found this to be severely limited by physical space and hardware constraints, leading to significant discomfort and difficulty for participants to complete the entire experiment. Therefore, we ultimately modified the experiment design to have participants sit and observe approaching autonomous vehicles, using buttons to control their pedestrian avatar. We raised the initial height of the first-person camera to ensure consistency between the seated perspective and the normal walking perspective. In future research, the use of VR walking simulators, such as flat-based or bowl-based omnidirectional treadmills [37], may yield better results to ensure immersion and measure changes in pedestrian walking speed and direction.

Secondly, we were unable to address the issue of gesture-based interaction interfaces being difficult to identify and understand when the vehicle is far from the pedestrian. In VR environments, due to issues such as resolution and texture, it is challenging for the human eye to discern signals from several tens of meters away. This problem may improve in the real world, but it remains a focal point for future work. If interactions between pedestrians and autonomous vehicles can occur at greater distances, it would provide more space and reaction time for all traffic participants.

Since this study was conducted only in China, some of the chosen gestures may not be applicable in other countries or regions. Future research could further explore cultural differences and broaden the scope of experiments. In the survey study, we only selected age and gender as common indicators to analyze preference differences among different demographics. However, education, driving qualifications, and other indicators could also lead to preference differences. Due to the questionnaire of study 2 being distributed on internet platforms, participants may tend to be relatively young, and the gesture score may not represent the preferences of a more diverse population. Nevertheless, our investigation into age differences provides insightful findings. Future work could explore the impact of different gestures in a more diverse population.

In future work, to further validate the effectiveness of gesture-based eHMI, it is inevitable to work within actual vehicle systems. Nowadays, automotive manufacturers are already experimenting with augmented reality projections to display content on car windows, achieving high visibility, as demonstrated by the BMW i Vision Dee at CES 2023 ¹. However, accepting gestures that prohibit crossing for many accustomed to crossing in front of vehicles may be challenging. It will take time for people to adapt to observing signals conveyed by the eHMI of AVs at intersections to understand the vehicle’s intent and take action. Of course, this also requires cooperation with local policies and laws. Nevertheless, this is undoubtedly a trend in the future development of AVs, providing a means for AVs to communicate with pedestrians.

A.1 Introduction

This document provides further explanation of the design and analysis concerns in the STUDY2, questionnaire research section.

A.2 Questionnaire Structure

The questionnaire initially obtained basic demographic information from participants, including age and gender. Subsequently, the questionnaire classified the factors for evaluating gestures into clarity, familiarity, and politeness. The questionnaire requested participants to assess these three dimensions of evaluating gestures by comparing them in two-by-two scenarios and by rating various gestures on the three dimensions (clarity, familiarity, and politeness) in two-by-two comparisons [29, 55, 67].

Figure 19:

View Figure

The questionnaire participants were given clarification on the three dimensions beforehand. Clarity, which measures the clarity of the gesture in expressing intention. Familiarity, which measures the degree of familiarity with gestures in everyday life. Politeness, which measures the politeness of the gesture in expressing intention.

A.3 Questionnaire Design

A.3.2 Topic Design and Analysis Methods.

Analytic Hierarchy Process (AHP) is a renowned method for decision-making, initially proposed by American operations researcher, Saaty, in the early 1970s. It involves decomposing decision-making elements into levels such as objectives, criteria, and options, which are then qualitatively and quantitatively analyzed through pairwise comparisons. The initial step involves structuring the problem into a hierarchy (Figure 21). The highest level entails selecting the optimal yielding and non-yielding gesture. The second level comprises three criteria that contribute to achieving this goal. The third level consists of eight gestures that need to be evaluated considering the criteria of the second level.

Figure 20:

View Figure

The second step is to construct a comparative judgment matrix. A comparative rating matrix is created by comparing factors two-by-two, using the criteria from the previous level. A_ij may be utilized to express the relative importance of the ith and jth elements, with a typical value range of 1-9 and their reciprocal. The resulting A_ij values constitute the judgment matrix, which is also known as the comparison matrix. For instance, the second layer of the three evaluation dimensions will be compared in pairs to obtain the relative importance of the three dimensions. Additionally, the gestures are compared in pairs again, this time under the clarity dimension, to determine the relative importance of each gesture under that dimension. To make it easier to understand, each comparison is designed as a slider input.

Figure 21:

View Figure

Figure 22:

View Figure

The third step involves conducting hierarchical single ranking and its consistency test. By obtaining a characteristic equation for each comparison judgment matrix, the equation can then be solved to give the solution vector and regularized, forming an importance ranking of factors at the same level with comparable factors at the previous level. For example, after determining the relative importance of the three dimensions at the criterion level, the judgment matrix is processed to obtain the final solution vector which reflects the weight share of each dimension for selecting the optimal gesture. The process is then repeated to determine the score of each gesture in terms of clarity within the clarity dimension.

The fourth step involves conducting a hierarchical total ordering and a consistency test. This requires calculating the relative importance of all elements within the same level to the target level. The resulting hierarchical total ranking must be tested for consistency to guarantee the reliability of the findings. After completing the aforementioned steps, the calculation process yields the weights of the three dimensions, scores for each gesture in each dimension, and, ultimately, the overall score for each gesture. Prior to score calculation, it becomes necessary to unify the group opinion by processing the geometric mean to establish a unified judgment matrix.

Figure 23:

View Figure

A.4 Questionnaire Design

The survey was formulated using the AHP analysis technique, resulting in questions being presented in a two-by-two comparison format. This approach obtains the assessors’ assessment of the significance of the three dimensions and the effectiveness of each gesture in each dimension, allowing the researchers to achieve a more exhaustive analysis of the gestures. This approach obtains the assessors’ assessment of the significance of the three dimensions and the effectiveness of each gesture in each dimension, allowing the researchers to achieve a more thorough analysis of the gestures.