Work with AI and Work for AI: Autonomous Vehicle Safety Drivers’ Lived Experiences

5.2.1 Forming and calibrating perceptions of AV.

Based on the qualitative data, we identified factors that influence driver perceptions of AV capabilities and classified the calibration process into three stages: preparatory, initial, and regular, as shown in Figure 1. We found that during the preparatory phase, due to a lack of knowledge about AV, their perception was formed based on outside sources such as news media, journalists, social media, etc., which often contain incorrect information. And there was a higher likelihood of technophobia [9] and technopraise [35] among this group, which may result in over-trust or mistrust about AV capabilities. In the initial phase, as they gain access to AV and receive training and guidance from their companies, their perceptions will be quickly calibrated.

"At first, I was skeptical. I couldn’t wrap my head around how a few tons of metal could drive on its own. I was even scared to get into the car at the start of my work. But after a few days, I started to feel more comfortable, and it exceeded my expectations." (P10)

"I used to think self-driving cars were all they were cracked up to be, but after driving it during training, I found that it wasn’t like what I had seen on TV. I found that it wasn’t like what I had seen on TV. I just couldn’t relax and trust it to drive itself." (P26)

In the regular phase, safety drivers continuously calibrated their perceptions through their work practices, causing their mental models to converge with the actual capabilities of AV over time. "It took me about a year to really get to know the car, and I feel like I’m getting better at making predictions now", P1 said. 

We found that company-level factors such as training and guidance had a significant impact on safety drivers’ perceptions in the initial phase. However, in the regular phase, their AV perceptions became more based on their hands-on practices than company-level factors. Many participants reported that the theoretical pipe-lined training they received didn’t help much with calibrating their mental models, and it was difficult to apply that theoretical knowledge into real-world driving practices, especially in high-risk and emergency situations. Instead, they found that constantly trialing and erroring in their work practices was a better way for them to explore the boundaries of AV capabilities and calibrate their perceptions accurately. Also, they reported that the lessons gained in these processes would be more impressive.

"Although the company has informed us that the radar often misses low objects, if you suddenly encounter obstacles in the road, it’s tough to act on that information right then and there. But after being startled, you can remember this lesson very well and handle it better next time." (P12)

Additionally, participants reported that after the "novice period," their companies usually paid less attention to the accuracy of their mental models about AV and didn’t provide adequate training and assessments in a timely manner. There was also a lack of official methods for continuously calibrating their perceptions while working with AV on a long-term basis. With each update to the algorithms, system iteration, or hardware change, safety drivers had to form new mental models. However, they didn’t know what the "standard answer" was. They had to test and adjust their perceptions in high-risk environments, which resulted in their being potentially exposed to safety risks resulting from misperceptions, and they had to assume responsibilities for those risks.

"The engineers usually only give us a rough idea of version characteristics when the version is updated, and they don’t know when the car might have a malfunction. We must concentrate our attention while driving to try and identify any potential problems with the car." (P22)

5.2.2 Takeover in real world.

Figure 2:

View Figure

According to the interviews, takeover was the most important operation for safety drivers to interfere with AV, including braking, throttleing up, and turning the steering wheel, which often took place in risky and unexpected situations and would be recorded by the AV system for problem analysis. For the majority of the systems our participants worked with (N=23), once the driver took control of the vehicle, the automation system would shut down and transfer control to the human.

From the qualitative data, we identified three sets of factors that impacted safety drivers’ takeover decision-making and vehicle control authority: organizational factors, personal tendencies, and real-time situations (including the autonomous vehicle and the external environment), as shown in Figure 2.

We found that in low-risk and non-emergency situations, the safety drivers were more likely to be influenced by company factors when making takeover decisions. But in emergency situations, participants expressed a desire to take control of the vehicle and made more subjective and intuitive takeover decisions, which may go against the company’s requirements, guidelines, and advocacy.

Organizational tendencies vs. individual tendencies. Many participants reported that their companies usually guide the driver’s takeover criteria based on corporate strategies. Companies that prioritized efficiency in testing and the quality of takeover data were more likely to ask their safety drivers to relax their takeover criteria. P23 mentioned,

"My company wants us to be less sensitive to takeovers so we can better know the reaction of AV. If you feel something wrong and take over it immediately, you won’t know what its abnormal reaction exactly is." 

While companies that prioritized test safety were more likely to ask their safety drivers to take over in advance. P19 said,

"My leader always emphasizes, ’if you feel something wrong, you must take-over promptly, and you should not tempt the reaction boundaries of the car, otherwise, you are risking the company’s property and your personal safety,’ so we rarely tempt the limits of the car."

Moreover, safety drivers’ takeover attitudes also fluctuated according to their subjective tendencies. P24 said,

"Occasionally, the car will act strangely, and if there are no other cars on the road, I will not take over control purposefully to see how the car will behave. Obviously, I shouldn’t do that, and my actions may be recorded by the system. I only do this on a few rare occasions out of curiosity."

Sometimes, there were conflicts between the company’s tendencies and individual tendencies. In non-emergency situations, these tensions were not obvious. Most safety drivers tended to follow their companies’ guidelines to adjust their takeover attitudes and tighten or relax their takeover criteria. As P8 mentioned, "This is my responsibility to meet my company’s requirements. I need to get my job done well."

However, in high-risk emergencies, participants reported that they would disregard the company’s guidelines and rely on their intuition due to concerns about the consequences of risk, such as their personal safety, liability for accidents, and traffic laws. They explained that they wanted to take control of the vehicle and felt that having more control would give them a greater sense of safety.

"If I feel danger, I will take over control. In that moment, I am unable to give any thought to the company’s requirements." (P12)

"After taking over and getting the wheel in my hands, I feel relieved and safer." (P10)

Some participants said they had more faith in their own decisions and believed that human decisions were better than those made by machines, especially in emergency situations. It is worth mentioning that the majority of the participants (N=21) stated that they preferred to take control of the vehicle themselves in risky situations, even if the self-driving decision might be the correct one. As P16 stated,

"Even if it happens many times that the autopilot is right and my judgment is wrong, the next time there is a conflict between our decisions, I would still choose to make it listen to me. Although its ability is excellent, I don’t think I can rest assured to give all of myself to it."

For real-time situation assessment. Most safety drivers (N=24) reported they needed to pay much more attention when supervising the self-driving vehicles compared to when they were driving manually. Extensive lab-experiment-based literature describes that high-level autonomous driving may bring distraction [16], loss of attention [59], decreased situation awareness [30], etc. to drivers. However, few participants mentioned that their level of attention decreased when supervising AV, even for long periods of time. There were even some reported that their level of attention increased with driving time.

"There are so many unpredictable factors involved. When driving for a long time, I feel that my focus improves." (P22)

"The more time you spend driving, the more tense you become." (P1)

Safety drivers were required to take regular breaks, and their driving time typically did not exceed one hour at a time. This was perhaps one reason why there were no reports of decreased attention caused by long-time automation. Also, participants interviewed in this study were urban road safety drivers, who encountered driving scenarios usually with low monotony and high uncertainty. They had to deal with the uncertainty of self-driving hardware, software, and external environments and making quick decisions when something wrong. The unpredictability factors in the real world made them continuous high attentions. "I need to pay more attention to the surroundings and detect them more frequently when monitoring AV. I will be very alert and my attention will be more focused," P7 said. Meanwhile, long periods of concentration also lead to fatigue in safety drivers’ work. P2 said, "It’s very brain-intensive, and I feel very tired at the end of the day."

Moreover, participants reported that, due to the fact that their personal safety and work performance were closely tied to the safety of autonomous cars in the real world, they were unable to let their guard down, which was different from the safety-guaranteed laboratory-based studies. Additionally, upon the company’s request, the perception and decision-making of safety drivers needed to be paralleled with that of autonomous vehicles. Hence, reducing attention and handing over part of the driving perception tasks to AV would have been negligent for safety drivers. We inferred that one of the possible reasons for increased attention was that our participants were professional safety drivers, had passed related training, and would increase their attention proactively at work. But we did not exclude the possibility that the sensitivity of their work might have deterred them from speaking more critically and admitting their dereliction of duty.

5.2.3 Cognitive Preferences and Characteristics of AV Technologies.

Know little but want to know more. The close interaction between safety drivers and AV made exposure to AV technologies in their daily work inevitable, such as learning codes to assist engineers in testing, detecting, and fixing minor AV problems, which reduced their sense of separation and awe about the technology while also stimulating their curiosity. Despite the fact that most our participants had limited knowledge of AV technologies and no technical background, they expressed a desire to learn about how AV works and AV technical principles. P14 said,

“I spend every day with this car, but I don’t know what it’s thinking. I want to know why it makes certain decisions and behaves the way it does, and I want to know the logic behind it.”

Learning from interacting with engineers. We found that most AV companies believed that it was sufficient for safety drivers just to be drivers for AV and did not invest in technology-related training for them. Participants reported that most of their understanding of AV technologies gained through their interactions with engineers. P13 mentioned,

"When testing with the engineer, he is too busy to handle all the tasks himself. He usually teaches me some codes so that I can assist him in tuning the program, and show me how to query the database and fix some bugs. I often ask him why certain things can be tuned in certain ways during the test, and he will explain what the code stands for and what else it can do."

P17 mentioned that he would establish a good relationship with engineers who were willing to share their knowledge, in order to facilitate future consultation and learning.

"If I build a good relationship with the engineer, he may request that I be paired with the captain so that I can spend more time communicating with him and learn something."

Although communication with engineers can help safety drivers understand technology to a certain extent, this knowledge transfer was limited, and the information obtained might be incomplete. Engineers were not always concerned with how much safety drivers had mastered or whether their knowledge mastered was accurate. "The company will not teach us these, and the engineers only speak these to us briefly, so we can only understand the superficial aspects," P15 said.

Invisible algorithms and visible hardware. We found that most safety drivers tend to associate invisible algorithms with visible hardware. When we asked them about their understanding of AV, most answered by talking about their knowledge of AV hardware. P19 said,

"I don’t know much about software algorithms, but I do understand some basic hardware concepts like lidar, millimeter-wave radar, cameras, and so on. I understand their perception range, parameters, and simple hardware debugging methods."

We also found they would naturally associate the level of AV capabilities with the vehicle’s hardware systems. When there was a problem with the AV, they would first assume it was a hardware issue, and then consider the possibility of an algorithm problem. "If the car suddenly stalls, I will first check its radar, camera, and other sensors," P15 said. Participants also reported their greater interests in understanding hardware knowledge than software algorithms. They believed hardware knowledge was easier to comprehend and more useful in their daily work. P14 and P16 mentioned that they would like more training on hardware knowledge from the company. P10, P18, and P21 expressed their desire to switch to a career as hardware engineers.

"I prefer purely mechanical things." (P14)

"I have difficulty understanding those red and green codes, but I’m more familiar with how the hardware works." (P16)

5.3.1 Work Experiences.

From technology experiences to work experiences.

Safety drivers need to work in close contact with AV in their daily tasks. In such a work context, they were the supervisor and also the experienced user of AV, and their happiness and sadness at workplaces were closely linked to the AV they worked for.

As supervisors, participants reported that they were able to gain a great sense of effectiveness when the issues they reported were resolved, and they could feel firsthand the progress of AV. "When we report a problem and it is suddenly resolved after a few days, I feel a sense of satisfaction knowing that I was able to contribute to the improvement of the vehicle," P24 said. Some indicated that they feel neglected and unappreciated when they do not receive feedback or the issues they report are not resolved. "We’ve reported this issue several times, but it still happens as soon as the car approaches this intersection. It’s been a long time, and the problem has not been resolved. I feel like they just don’t want to pay much attention to me," P15 said.

As users of AV, participants reported that they have positive experiences when the vehicles behave in accordance with human thoughts in their daily driving.

"I want it to go faster, and it does; I want it to slow down, and it does. It’s as if it knows what you’re thinking, and it feels like I’ve become integrated with it." (P11)

Some indicated that the inconsistency between the vehicle’s behavior and the human’s intentions and expectations might trigger their negative experiences.

"It has its own thoughts, and I have mine. Sometimes it doesn’t inform me, it just goes right by and it just lets me agitated." (P21)

Some expressed disappointment that the limitations of algorithms sometimes override human intentions. "I like the aggressive driving style, but the car just drives very conservatively," P23 said. Participants also mentioned they would prefer to work with the "perfect machine".

"I’ve worked for two different companies. The technology of the first company was not very developed, as the autonomous vehicle would frequently brake abruptly and cause conflicts with other road participants. When I finish my daily work, I’m often in a bad mood. But now, the technology of my current company is much more developed, and the autonomous vehicle is much smoother and more comfortable to ride. I feel much better now." (P9)

Technology mediates social interactions in the workplace. Autonomous driving technology also had an impact on the social interactions among safety drivers at workplaces. We found that AV technologies improved communication and collegiality among them. P9 mentioned that when she first started at the company, she was unfamiliar with everything. AV became an ice-breaker topic between her and her colleagues, hastening their acquaintance. Safety drivers were willing to discuss their guesses about AV with others and share problem solutions, which gave them a sense of accomplishment and self-efficacy and strengthened their colleague relationships. "We often discuss common problems together, such as navigation and perception issues, and we are willing to help each other," P11 said.

We also found that safety drivers’ interactions with engineers significantly enhanced their desire to learn more about AV technologies and their self-efficacy, which in turn impacted their career planning.

"The engineers have taught me a lot, and I feel like I can do the same things they do. I’m thinking about becoming an engineer in the future." (P21)

While we observed positive effects of technology on workplace interactions, we also found that technology hindered some safety drivers’ self-expression to some extent. Some participants mentioned that their lack of technological knowledge made them hesitant to express their opinions. They tended to suppress their expression in certain situations because they feared being denied or ridiculed.

"When I encounter some simple problems that I don’t know how to solve, I’m afraid to ask my leader because I don’t want them to think I’m incompetent. I often ask my colleagues, but they don’t always know the answers to my questions." (P11)

"Sometimes I have some guesses about AV problems. I usually wait until I’m absolutely certain before sharing them with others. After all, I’m not a professional engineer, so I’m concerned that my ideas may be incorrect." (P23)

Real world driving experiences. When safety drivers drove the "strange-looking black box" on public roads, they attracted more attention from the outside world and had novel experiences with AV, which gave them the pride of being noticed, unnecessary annoyances, and risks caused by other traffic participants’ curiosity and low acceptance of new things. Some participants expressed a very positive feeling about the curiosity and attention of the outside world caused by autonomous driving and felt proud and happy.

"When I’m waiting for the traffic light at the intersection, people often take pictures of the autonomous vehicle, which gives me the illusion that they are paying attention to me." (P2)

Some also mentioned that the outside attention would cause them unnecessary distress.

"Because this vehicle is too conspicuous, the traffic police sometimes specifically spot-check me." (P6)

"When the vehicle breaks down on the road, some passers-by may go near and take a look, and even stop to take a picture, which made me very embarrassed." (P16)

Furthermore, participants stated that they had to accept the risk of low acceptance of AV by other road participants, such as malicious behavior caused by curiosity.

"There are many cars on the road that often deliberately come to a halt in front of you because they are curious how the self-driving car will react." (P20)

"A few years ago, the news media widely reported that AV would replace human drivers. During that time, I was often bullied by taxis while conducting road tests. But it’s much better now, they have become accustomed to our vehicles." (P1)

Moreover, due to their limited capabilities, autonomous vehicles may not be able to handle all road test conditions perfectly, which may result in exclusion by other road participants. P23 mentioned that his vehicle sometimes has to bear the consequences of mistakes made by other autonomous vehicles while driving:

"The cars in our team often make mistakes on the road and affect others (road participants). Those drivers probably hold a grudge, and the next time they encounter our team’s autonomous vehicles, they will deliberately bully them, even if the car they retaliate against is not the same one that disrupted them last time. I am often wrongly accused."

5.3.2 Taking Risks Accumulated from upstream of AV Industry.

Forced to expose themselves to accumulated risks from AV upstream. During road tests, safety drivers, as the downstream workforce of the AV industry, were forced to face the accumulated risks from multiple upstream links, including algorithm development, hardware manufacturing, assembly, etc. Unlike traditional testers [29], they needed to verify and test AV in high-risk real-world environments, where a minor omission from other links could expose them to great risks. Based on the interviews, developers in other upstream links of AV were not required to participate in road tests like safety drivers. In such a workflow, these stakeholders were unable to predict precisely how the AV they developed would perform in the real world, which might reduce their sense of responsibility and increase the likelihood of negligence.

"I remember a remote debugging session. The engineer told me the car had been fixed and let me test it. But when I tested it, I found that the problem had not been set up right, and I almost collided with an obstacle." (P7)

"There was a system version update, but the new version had many strange problems, and the engineers didn’t know what was going on. We felt very unsafe while driving and had to pay very close attention." (P14)

Although it was the duty of the safety drivers to look for defects and intervene in AV to prevent risks, relying solely on the safety drivers to identify accumulated problems and avoid accumulated risks is difficult and immoral.

P3 said, "There are some unexpected situations where it’s too late for you to takeover, and no one can change the outcome in such a short amount of time." P7 expressed his concerns about safety, "Every morning when I go out for work, I pray to come home safely," and thoughts about changing jobs.

5.3.3 Ambiguous Responsibility Assignments and Moral wrinkles.

Responsibility assignments. In addition to bearing the accumulated risks, safety drivers also faced ambiguous and undue assignments of responsibility. China’s traffic laws did not consider self-driving cars as responsible entities. When safety drivers conducted road tests with AV, they were held legally responsible. Even though the majority of the time the car was being driven by AI and not the safety driver, the safety driver was still held accountable for the vehicle. For traffic violations, even those caused entirely by the self-driving system, the safety driver was still punished as the primary responsible party and their driver’s license points would be discounted. A poor driving record would be recorded in the driver’s file and might impact their driving qualifications or limit their opportunities for other driving-related jobs. P8 said, 

"If we receive a citation for breaking traffic laws because of the self-driving car, the company will compensate us with some money. However, there is no way to regain the license points that have been deducted, and we have to accept the loss."

Participants said they do not have to take responsibility for passive accidents, but if the accident was caused by the fault of the safety driver or the autonomous vehicle, they may face consequences such as a warning or penalty from the company. Two-thirds of the safety drivers reported that the company’s responsibility allocation system was not fair to them.

"The company has a zero-tolerance policy for accidents. Because safety drivers are hired for the purpose of ensuring safety. If an accident occurs, it is considered the responsibility of the safety driver." (P15)

Some participants reported that their companies implemented a shared responsibility system as a way to warn safety drivers not to have an accident.

"The training of new safety drivers is conducted by old safety drivers, and if the new safety driver causes an accident, the old safety driver will take responsibility for the new safety driver." (P21)

It is undeniable that this stringent responsibility system can improve the safety drivers’ sense of responsibility at work, but it also infringes on their rights and leads to a negative work experience to some extent. There were also some companies that differentiated the responsibilities between human and autonomous vehicles.

"If accidents happen while on autopilot mode, your responsibility may be reduced, but if you were distracted, fell asleep, drank water, used your phone, etc., that would be your responsibility." (P25)

However, due to the lack of clarity on the boundaries of responsibility, safety drivers may take on more responsibility than they are supposed to.

"In the moment before an accident, you must have taken control instinctively. It can be difficult to determine whether you were involved in causing the accident or not." (P17)

"Although it is said that the company will analyze whether the accident was caused by AV or the safety driver through monitoring data and system records, but then again, if there is an accident and you take over, then you may be responsible for involving the accident. If you don’t take over, that means you have not performed the duties of the safety driver, and then you are still responsible." (P13)

Operator or passive observer. As a new type of occupation, the current corporate systems, laws, and regulations are not fully equipped to guide safety drivers in all situations, leading to ambiguities in road testing. As described in the previous section, some companies adopt a safety-driver-friendly allocation of collision responsibility. If a collision occurs while the autonomous system is running without any human interference, the company will assume full responsibility for the autonomous system they developed. However, this regulation raises another question of whether the safety driver should intervene in the AV system at the moment of an upcoming accident. Participants reported that they would not stand by and watch an accident happen to reduce their responsibility. They would instead instinctively take over the autonomous vehicle, even if their actions would not save anything.

"Takeover better than an accident." (P19)

"I’ll take over even if I can’t save it. I can’t watch the car go into an accident." (P11)

"I wouldn’t allow an accident to happen for fear of liability, but I don’t know what the rest of my colleagues think." (P8)

"Even though not intervening could absolve me of responsibility, there’s no guarantee that the company wouldn’t hold me accountable if an accident occurs. So I believe it’s more important to prevent an accident from happening." (P16)

Collision leaded by avoid-collision. According to our interviews, AV companies usually adopt very cautious algorithmic strategies to avoid active collisions. However, these overly conservative driving strategies made self-driving cars slow and easy to stop, increasing the chances of being rear-ended by other vehicles. Participants reported that they have to put in extra effort and take on additional risks because of the shortcomings of AV strategies.

"When the self-driving car falls short, it’s up to us safety drivers to fill in the gaps. But sometimes the car’s capabilities are just too limited and we are also very helpless. But after all, this is our job." (P22)

"The safety driver must pay more attention to the rear of the vehicle to avoid being rear-ended, which requires a high level of attentiveness from the safety driver. Sometimes some novice safety drivers are not able to effectively consider the surroundings, so it is quite prone to accidents." (P25)

5.3.4 "I can see the future of AV, but not mine".

As described in Section 5.1, safety drivers typically relied on their driving skills for their livelihood and had limited career choices. Although they had the opportunity to be exposed to emerging technologies and witness changes in the autonomous driving industry as safety drivers, it was challenging for them to achieve career growth through this job.

"You can’t learn anything just by staring at this car every day. I feel as if I’ve been extinguished after working for a long period of time as a safety driver." (P9)

Participants expressed their concerns about their future developments.

"As long as you can drive, you can be a safety driver. There is nothing irreplaceable. Who knows, I might be laid off one day." (P5)

They also reported an age crisis among safety drivers. Some companies set age requirements due to the high levels of endurance, sensitivity and responsiveness required for the job. Despite many studies [19, 34, 38] showed that there was no significant difference in takeover response ability between young and middle-aged people, the inherent impression can also expose this group to age discrimination.  "Younger drivers may have better response abilities, and the older drivers may be gradually eliminated from the company," P10 said.

Moreover, participants also recognized that AV technology is moving towards unmanned operation, and they feared that the role of safety driver may become an overstaffed position that is vulnerable to being eliminated. P18 said, 

"Safety driver is only a transitional position in AV development. With the progress of AV technology, safety drivers may become obsolete. When that happens, I don’t know what the company’s plan is, and I am not sure about my future."

P9 expressed his ambivalence between his expectations for the development of AV and his future development as a safety driver:

"I wish for the technology to progress faster to make my job easier, but I don’t want it to develop very well because it may jeopardize my employment."