Introduction

Our cities are complex social-technical systems in which people continuously interact with infrastructures and technologies as they travel, work, or go shopping. Confronted with climate changes, cities around the world are under pressure to reach sustainability targets, and transform their infrastructural systems, ranging from the energy transition to urban gardening for local food production or electric vehicles and ride sharing. Complex systems like our cities are always difficult to control and steer, especially during transitions.

A major challenges is that our transitioning cities are increasingly confronted with crises: the 6th Assessment report of the IPCC once more issues stark warnings: climate change is confronting our cities with an increasing number and severity of extreme events, ranging from extreme heatwaves to heavy precipitation, rising see levels and droughtsPersson and Tinghög (2023); (Seneviratne et al., 2021) sho In their 2023 Global Risks report, the World Economic Forum refers to the global “new normal” as a situation where in our cities basic needs, such as food, energy, and security, remain often unmet (WEF, 2023). Combined with the increasing interconnectedness of socio-technical systems this trend implies that decision-makers need to make increasingly complex and far-reaching decisions at an accelerating pace. In other words: crises fundamentally transform the nature and timescapes of planning and decision-making: the proliferation and volatility of information (Höchtl et al., 2016; Tsoukias et al., 2013) combined with the urgent need to alleviate human suffering pressure decision-makers to rapidly respond to the increasing number of crises and disasters worldwide.

Recognizing this great complexity of (urban) crisis response, AI is frequently portrayed as a way ahead to improve rapid information collection, analysis, and decision support. Facilitated by advances in (remote) sensing and crowd sourcing, increasingly available open data and accelerating computational capacities, the hope is that AI can help humans to better prepare for, respond to and recover from crises (Sun et al., 2020). At the same time, reliance on AI - and the introduction of digital technology more broadly - have added a layer of interdependencies, and therefore introduced additional vulnerabilities. How, for instance, can we live in smart homes during a power or internet outages? Sure enough, critiques of AI have pointed to the potential downsides of experimentation with AI and digital technology in a very vulnerable context (Sandvik et al., 2017); the need for accountability and the lack of adequate legal frameworks (Coppi et al., 2021); as well as the tremendous carbon footprint of AI, especially for training machine learning models (Van Wynsberghe, 2021).

To guide the use of AI amidst the promises and perils, a series of guidelines, standards and principles have been put forward. Some of the most prominent ones include guidelines by regulatory bodies such as the United Nations Educational Scientific and Cultural Organization (UNESCO) recommendations on the Ethics of AI (UNESCO, 2022), the Organisation for Economic Co-operation and Development (OECD) Recommendation of the Council on Artificial Intelligence (OECD, 2019), the European Commission’s recommendations by the High-Level Expert Group on AI (EC, 2019) or the Institute of Electrical and Electronics Engineers’ (IEEE) standards for Ethically Aligned Design of Autonomous and Intelligent Systems (IEEE, 2019). For an overview, see (Jobin et al., 2019). All of them embrace the idea of an AI that needs to be designed to be explainable, accountable, trustworthy, and fair – corresponding to the central tenets of human-centred AI (Shneiderman, 2020). Human-centered AI should be “amplifying, augmenting, and enhancing human performance in ways that make systems reliable, safe, and trustworthy” (Shneiderman, 2020). However, putting forward fundamental principles leaves many questions about the concrete design requirements and implementation (Mittelstadt, 2019).

Therefore, if the aim is to design a human-centered AI that supports, facilitates or even makes decisions in crises, an understanding human decision-making is crucial (Miller, 2019). Decision theory, as a field, is concerned with both human decision behaviour, cognition and interaction (descriptive) as well as with how optimisation and how we should make decisions (normative) (Tsoukiàs, 2008). Importantly, decision theory stresses the prominence of the context (French & Geldermann, 2005), which also finds an echo in the calls for ‘contextual transparency’ for automated systems (Sloane et al., 2023). Especially the context of crises is crucial, since crises have been shown to change human information processing and decision behaviour (Klein et al., 2010; Paulus et al., 2022; Weick et al., 2005). Nevertheless, decision theory has remained largely disconnected from the discourse around human-centered or explainable AI (Främling, 2020). This oversight leaves a gap in our understanding of the interactions between humans and AI (Rahwan et al., 2019), and unclear or inadequate design requirements (Mittelstadt, 2019).

To start exploring this gap, this paper aims to establish a research agenda around the decision-theoretical implications for AI in the context of crisis response. The underlying research question is: what are research directions for a decision theoretical approach to support the design and development of human-centred AI for crisis management? As indicated before, contextualisation is central to both decision theory and AI. Therefore, this article first explores the context of crisis decisions as super-wicked problems before unpacking the specifics of AI for crisis response. By taken a decision-theoretical perspective, I will analyse the implications of increasingly automating information acquisition, analysis, or decision-making. From there, I will highlight the differences between crisis and conventional planning decisions and argue for dedicated design principles for AI that are tailored for crises. The paper concludes with a discussion of important research directions for crisis management as an example case of super-wicked urban decisions.

Crises as super-wicked decision problems

Why is it worthwhile to analyse decision-making in crises, in addition to the many problems that decision-makers are already confronted with? In this section, I will argue that crisis decisions are an ideal testing ground to understand the intricate feedback between human decision-making and AI-systems in urban contexts, given the time pressure and the value-laden context. While the Covid-19 pandemic has fueled interest in urban resilience (Champlin et al., 2023), there remains limited research on crisis management in the smart city – for a recent review, see (Alshamaila et al., 2023). Generally, the focus on the smart cities crisis literature is exploiting sensing technologies for real-time decision support (Wang & Li, 2021; Yang et al., 2017), but there is limited reflection on how people interact with technology in the smart city.

In his article about AI and smart cities, Batty (2018) argued that many of the challenges around the smart city are “peculiarly human”, or intrinsically related to human reasoning and the dilemmas or “hard choices” that planners need to make. The combination of urgency and complexity that we observe in crises is characteristic for many super-wicked problems that urban planners are confronted with (Levin et al., 2012), ranging from climate change to sustainability or migration. Such urgent and complex problems are known to change information processing and decision behaviour (Weick et al., 2005). So, if AI in smart cities is indeed promising to address problems in healthcare, mobility, or energy (Herath & Mittal, 2022), then I argue that analysing (AI for) urban crisis management can help us make headway in the crucial area of understanding the interaction of human decision-makers and machines.

Decisions in (urban) crises are complex since crises affect virtually all aspects of our societies (Comes et al., 2022). A crisis occurs when people perceive a severe threat to the fundamental values or functioning of a society or system, requiring an immediate response despite the prevailing uncertainties (Boin et al., 2016; Rosenthal et al., 1989). Even though many taxonomies exist that distinguish crises according to their scale, the underlying hazard (natural vs. human-made), time (sudden vs. slow onset) or geographical scale (from local to global), traditionally, crisis management has focused on actual or proverbial firefighting in a clearly defined region or sector. Not surprisingly then, also design of coordination mechanisms and decision support systems or AI to support crisis management has focused primarily on providing rapid information to people in various – yet shifting – roles (Comfort, 2007; Quarantelli, 1988; Turoff et al., 2004).

Today, driven by the proliferation of AI, crises are harder to manage as they evolve within complex and deeply interconnected systems (Renn & Lucas, 2021). This implies that crises increasingly cascade into distant regions or other sectors (Boin, 2019). Amplifying feedback loops at different timescales may eventually result in largescale disruptions (Helbing, 2009). For instance, the war in Ukraine has caused food insecurity and inflation across the globe (Arndt et al., 2023). The response to the Covid19 pandemic, initially thought of as a ‘health crisis’, has led to a loss of income, food insecurity, limited access to education or the ability to access and purchase basic necessities – especially in low-income countries (Josephson et al., 2021).

Further, AI and digital technology act as an amplifier of the emergent behaviour of individuals that can fundamentally impact the impact of crisis decisions. From fear to ‘pandemic fatigue’ driving non-compliance to Covid19 restrictions and political discontent (Jørgensen et al., 2022) to the outpour of solidarity and help after natural disasters such as the floods that affected Western Europe in 2021 (Bier et al., 2023), the spontaneous responses by self-organizing individuals, emergent or organized voluntary groups are a common feature of crises, especially in urbanized areas (Twigg & Mosel, 2017). And there are many calls to localize response and leverage self-organization to create participatory resilience (Mahajan et al., 2022; Nespeca et al., 2020).

The initial phase of a crisis response is often heavily resource-constrained, leading to a combination of uncertainty, cognitive and moral overload and fragmented sensemaking (Comes et al., 2020; Ishmaev et al., 2021). Here, sensemaking is the cognitive process by which decision-makers interpret the situation, and constructing meaning from data or experiences in order to understand and respond to complex, ambiguous, or novel situations (Weick et al., 2005). The messy characteristics of a crisis pose a double challenge (Paulus et al., 2022) (a) data may be unavailable, uncertain, conflicting or biased, given limited access or data collection regimes – with limited options to collect additional data because of the time constraints; and (b) the cognitive processes of analysis or decision-makers is under strain, given the urgency and high stakes of the situation, leading to biased decisions.

Despite the highly volatile, complex, and uncertain environment, the time pressure of crises forces decision-makers to make choices that have far reaching consequences in a much shorter time than traditional models require (French & Geldermann, 2005). Time is of the essence in urban search and rescue (Kleiner & Dornhege, 2007): after the earthquake that hit Turkey and Syria in spring 2023, the world followed the desperate race of urban search and rescue teams against time, and their attempts to locate and reach victims under the collapsed buildings and infrastructures and bring them to safety. Logistics and planning in these humanitarian crises are time-compressed (Holguín-Veras et al., 2012) On the other end of the spectrum, Covid19 has brought about a rapid fire of restrictions and measures ranging from tracking apps to lockdowns, from travel bans to large scale investment programs – all of which were made rapidly despite having a profound impact that was at times poorly understood (Atkinson et al., 2020; Sigala et al., 2022). The war in Ukraine led to fast measures to compensate the lack of Russian gas by a turn to coal (Aitken & Ersoy, 2023), despite its potential impact on greenhouse gas targets. And while conventionally urban planning decisions are made in structured planning processes that take years, in the recovery phase to an urban disaster, planning is focused on rapidly rebuilding the status ex ante, even though it is this very status that may have led to the disaster in the first place (Krishnan et al., 2024).

These decisions have in common that despite their profound impact on human lives and livelihoods, the conventional time frames to make decisions with far reaching consequences collapses. We observe fundamentally different timescapes of decision-making as highlighted in Fig. 1. For conventional problems in planning and decision-making (in blue), the further the decisions reach – and the more uncertain therefore is related to future predictions – the longer the time to make the decision corresponds. For crisis decisions (red in Fig. 1), however, the window to decide dramatically shrinks. Since inaction or waiting until further information becomes available is not possible, decision-makers need to rapidly act. This leads to a time compression of complex problems with far-reaching consequences (red arrows).

Fig. 1
figure 1

The shifting timescapes of decision-making

Full size image

In sum, decision-makers in crises are confronted with a combination of complexity and urgency that is typical for super-wicked problems (Levin et al., 2012). The phenomenon of complexity and time compression is further exacerbated characterised by decision density, by which – especially in the early phases of a crisis – an exceptional number of decisions need to be made rapidly (Baharmand et al., 2019).

AI in crisis decisions

Artificial Intelligence is becoming increasingly prominent as a vehicle to support or replace human decision-making in crises and disasters. With the increasing prominence of AI, there is also a plethora of definitions that aim to characterise the behaviour or function of what constitutes an ‘AI’ (Krafft et al., 2020). Since this research is situated at the interface of science and policy, I follow here the OECD definition “An AI system is a machine-based system that can […] make predictions, recommendations, or decisions influencing real or virtual environments. AI systems are designed to operate with varying levels of autonomy.” (OECD, 2019).

There is a broad range of applications from automating early warning to damage detection and response and recovery planning. Often, AI systems are used to improve the predictive accuracy or speed of prediction as compared to traditional models (Guikema, 2020) to predict hazards, model human mobility or analyse supply chains. There are recent reviews that provide a full account of the different applications for crises and disasters (Munawar et al., 2022; Sun et al., 2020) or risk communication (Ogie et al., 2018), therefore I provide here only a short overview of selected applications, examples and data requirements in Table 1. Applications are mapped on the crisis management cycle of preparedness, response, and recovery to indicate the different timescales and actors involved in the application of the AI system.

Human control and oversight are at the heart of human-centred AI (Shneiderman, 2020) and instrumental to the discussions around AI, see e.g., the UNESCO guidelines (UNESCO, 2022) or the EU’s guidelines on trustworthy AI (EC, 2019). Yet, increasingly there is a push for automation in crisis management (Coppi et al., 2021). Therefore, Table 1 includes automation. Automation can be defined as “a device or system that accomplishes (partially or fully) a function that was previously, or conceivably could be, carried out (partially or fully) by a human operator” (Parasuraman et al., 2000). While colloquially, automation is understood as ‘the machine taking over’ a task at which it is ‘better’ (Bradshaw et al., 2012), there is more nuance to the degree of automation and control. One of the first frameworks to discuss automation was established by Sheridan et al. (1978) to understand the control of underwater vessels. Later, their work was refined to study automation of air traffic control, resulting in a 10-point scale to measure automation of decision making that describes the interaction between humans and machines in decision-making. The scale reaches from no interference of a computer via the computer making suggestions, to the computer ignoring - not even informing - the human (Council, 1998). Building on these initial ideas, Parasuraman et al. (2000) put forward a framework that establishes four axes of autonomy or control related to the full cycle of decision-making: (1) information acquisition, (2) analysis of information, (3) decision or choice of action, and (4) execution of action. From there, Endsley (2017) makes the link to situational awareness stressing that automation can actually impede situational awareness by adding a layer of complexity and the need to continuously monitor the machine. Thereby, instead of increasing efficiency and effectiveness, automation actually leads to lower performance (Strauch, 2017). To mitigate the effects, adjustable autonomy refers to a system where the level of automation can be dynamically adapted to the context (for a review, see (Mostafa et al., 2019).

However, all these automation studies assume – often implicitly - that there is a single decision-maker or operator, guiding a vessel or an aircraft. However, the pervasiveness of AI has introduced more decentralized or hybrid cases. For instance, if a script is used to acquire and download data from Twitter, it is obviously the machine that acquires data, but the data itself is provided and fed into the machine by thousands of humans. Similarly, the execution of an action may require the interplay of humans and machines, for instance in the case that unmanned aerial vehicles (UAVs) are used to transport vaccines or medication to those in need. The ‘many hands problem’ provide a framework to think about whom to attribute responsibility to – also in the context of AI (Coeckelbergh, 2020). The challenge here is the double complexity and emergence: in the context of a crisis, the roles and responsibility of human actors emerge, and are continuously adjusted to the situation at hand (Mendonça et al., 2007; Turoff et al., 2004). Technology and AI need to be designed to be adaptive to both the changes in roles and tasks, as well as to the situational and cognitive context of (human) decision-making to decide what needs to be assigned to whom in which role and in what situation exactly. Predefining all the possible contexts and requirements then become a challenge. The potential irreversibility of action, the high stakes and moral values at play further complicate the situation.

To address this complexity, human-centered AI suggests a variety of design principles to ensure to combine high levels of automation with human control (Riedl, 2019; Shneiderman, 2020). However, these design principles are not tailored to the crisis management context. For instance, the need for “incremental, and reversible action” (Shneiderman, 2020) may not be achievable in the context of crises where many actions lead to irreversible consequences (Pauwels et al., 2000).

Table 1 Examples of applications of AI in crisis and disaster management
Full size table

Even though Table 1 just provides selected examples of uses of AI for crisis management, it shows applications across all phases of the crisis management cycle, ranging from preparedness to response and recovery. Data sources combine from contextual data (e.g., socio-demographics, climate, or risk profiles) and data about the impact of the hazard (e.g., aerial imagery) or the reaction of the population to it (e.g., mobility or social media data). By its very nature, contextual data is rather static, whereas data about the evolving disaster is highly volatile. Importantly, a recent review has shown that – driven by the differences in available data – there is a lack of standardization in terms of data sets, even for the very same problems (Casali et al., 2022), leading to a lack of comparability and standards. At the same time, for wicked and complex problems it is – by definition – impossible to guarantee that a model is comprehensive and captures all required variables. For instance, (Martini et al., 2022) excluded pandemics from their food security assessment – despite just having lived through the raging impacts of the Covid-19 pandemic. The need to a priori collect data and train the models may hamper their applicability and contextualization in crises.

For automation, not surprisingly, the focus is on automating information analyses, which is common to all examples. Further, AI is used to automate the acquisition of large amounts of volatile – or Big – data. This includes aerial and satellite imagery collected by drones; the using police robots to sense and control traffic; or sensors to detect and predict earthquakes. Hybrid examples include social media data as well as the combination and collation of different human and machine acquired data sets as is the case for early warnings.

Further, Table 1 presents two distinct cases for the automation of decision-making and implementation. First, as may be expected, the time pressure is the driver behind the use of AI and the push for fully automated decisions, spanning the full cycle of information acquisition to implementation. As there are conventionally only seconds between the detection of an earthquake and its onset. Similarly, the break of a levee or a nearing flash flood require extremely rapid intervention. Second, in other cases, such as forecast-based financing and early action, or automated decisions of food insecurity (and declaration of famine), the main argument to push for an AI is to make the decision more transparent and replicable, thereby removing it from the realm of political decisions and motivational biases (Lentz et al., 2019). Here, the ‘black box’ nature of political decisions is replaced by potential ‘black box’ AI models. These models also present a way to escape the cognitive (Van de Walle et al., 2016) or moral overload (Ishmaev et al., 2021) that is typical for crises.

To be sure, the move to an automated decision support system shifts the power towards the design of the protocols, and the many decisions about the data sets, algorithms, or thresholds that trigger assistance. However, in crises, we are not just ‘replacing’ the bias of a human by a biased machine. Rather, we need to take into account the interaction of humans with the machines and consider important path-dependencies that might emerge from this interaction. Research on the interaction of humans with information has shown that the very push for more ‘evidence-based’ decisions amplifies political biases (Paulus et al., 2023), and that there are strong path-dependencies by which initial biased decisions are not corrected (Paulus et al., 2022).

AI for crisis decisions? Towards decision-centred AI

Designing for crisis decisions – contextualised AI

Understanding the differences between crisis decision-making and conventional complex planning (e.g., in climate adaptation) needs to build on the recognition that planning and decision-making is increasingly a socio-technical process that AI systems. A social network of stakeholders and decision-makers from various organisations (e.g., NGOs, volunteers, police, fire fighters, businesses) make choices (e.g., if and where to evacuate, how to behave in a city in extreme heat, or how to accommodate a surge of incoming refugees) that depend on the availability and accessibility of urban infrastructures (e.g., hospitals, transportation), and increasingly depend on artificial intelligence (e.g., traffic jam predictions, or forecasted extreme events).

As diverse as the impacts of crisis decisions are therefore the actors and stakeholders involved. From professional crisis managers (e.g., police, fire fighters, health professionals) to policy-makers; from citizens and to specific professional interest groups or NGOs – the broad impact of choices that are made in crises implies any actors with different interests and preferences that have to be aligned, ranging from the (potentially) affected population to private and public sector, to the many volunteers that often engage in crisis response. While it may be difficult to activate stakeholders in the preparation for a crises, response will flip the situation into a messy multi-actor context characterised by ambiguity, fragmentation and emergence (Nespeca et al., 2020; Wolbers et al., 2018). The crux in crises is the interplay of professional teams with the affected actors and stakeholders that self-organize and act to respond to a crisis (Mahajan et al., 2022), where the actions, roles and responsibilities of individuals can therefore not be predetermined (Turoff et al., 2004). Because of the lack of pre-established rules, norms or processes that bind all actors and groups, information (sharing) becomes central to make sense of and coordinate in crises (Bharosa et al., 2009; Nespeca et al., 2020). In essence, information allows different actors to form a shared mental model of the situation and to translate this mental model via sensemaking trajectories into activities and tasks. While much focus in the discussion around the application of AI in crises evolves around automation of decisions, we also need to recognize that if an AI that steers or influences information sharing or analysis, it will have an influence on human sensemaking trajectories and network.

Therefore, we need to reconsider the current focus of much of the work on human autonomy to integrate the impact of automation on the broader sensemaking or meaning-making process. Rather than focusing on the ability of machines to carry out a specific task such as identifying damage or triggering financial assistance, we also need to understand in how far the automation of information collection and analysis leads to a removal of the emergency management authorities from context. van Wynsberghe and Comes (2020) point out by using the example of drones that such an automation step “may push to de-contextualize care, also threaten to de-skill aid workers”.

To analyse the challenges for decision-making and the use of AI, we synthesize here various frameworks on decision-making in crises and disasters (Comes et al., 2011; French & Geldermann, 2005; Holguín-Veras et al., 2012; Paulus et al., 2022). We consolidate these characteristics with the value-driven character of crises that has been emphasized throughout the paper. While much literature on disaster and ethics primarily focus on health crises and duty of care (Leider et al., 2017), here we stress the need for principles around information and AI that have been forward by the UN (Van de Walle & Comes, 2015). We combine these aspects into a framework that summarizes the key differences between crisis decisions and complex planning (Table 2). The framework distinguishes nine dimensions: (i) uncertainty, (ii) stakes, (iii) complexity, (iv) cognitive load, (v) environment, (vi) actors and social networks, (vii) objectives, (viii) principles that guide the decision logic, and (ix) timescapes.

In a nutshell, crisis decisions combine the great complexity and uncertainty that is typical for planning with emergent and shifting objectives, volatile social networks, and great time pressure. This time pressure induces or amplifies cognitive and moral overload: in situations that combine the complexity with urgency, human decision-makers are far from rational and discount important cues, especially the distributive and long-term implications of their choices. Previous research has indeed shown that in such situations, decision-makers tend to focus on operational and local decisions that have an immediate impact on their environment, and tend to discard the long-term, strategic implications of their decisions (Comes et al., 2020).

If we design AI that increasingly automates information collection, processing, and decision-making, there are two major challenges: first, as discussed, human decision-making behaviour changes, and thereby also the interaction with the AI. Therefore, automated AI systems must be tuned to the changes in information processing and enactment that occurs in crises. Second, Table 2 shows that crisis management authorities are called to abide to specific principles and values, ranging from solidarity to humanity. However, given the lack of a clear operationalization of many of these principles, it is unclear how these principles operate if decision-making is automated (Coppi et al., 2021).

Table 2 Characteristics of crisis decision-making
Full size table

From principles to trade-offs

Crisis decisions affect the lives and livelihoods of many. Because of this broad impact, decisions in crises often cause moral dilemmas (Crawford & Finn, 2015; Qadir et al., 2016). In the broadest sense, the well-known ‘do no harm’ principle that guides crisis response can be viewed as an analogue to the principle of human beneficence and the need to mitigate risks that is widely promoted in the AI community (IEEE, 2019). Acknowledging that crisis decisions are always implicitly or explicitly guided by values, the European Working Group on Ethics in Science and New Technologies has advocated to prioritise the value of solidarity as a lens and guideline for ethical, and therefor also politically and socially more acceptable decisions (Prainsack et al., 2023). In contrast, in the humanitarian domain, the principles of humanity, neutrality, impartiality, and independence are thought to be the fundamental values to guide all information processing and decision-making (UN-OCHA, 2016). However, these principles are not conventionally considered in the literature on trustworthy or human-centred AI, which rather focuses on features such as accountability, explainability and transparency (Riedl, 2019). While these are undoubtedly crucial for AI in crisis response, what is missing is a discussion on what it will mean for an AI to uphold a value like solidarity or a principle like humanity. How can these principles be translated into protocols for data acquisition, or into an algorithm for information analysis or decision-making?

The principles that guide crisis response do not fall under the routinely used economic logic of costs and benefits that is typically used for planning. Even though broad welfare economics principles or distributive justice (via proxies such as access or distribution of wealth) are now increasingly advocated for in the realm of crisis response and resilience (de Bruijn et al., 2022; Holguin-Veras et al., 2013), these obviously do not correspond to principles such as ‘solidarity’ or ‘humanity’. Because we are missing frameworks to translate these principles into machine readable formats, recommendations by an AI may be misleading and therefore discarded or lead to potentially harmful choices.

Further, crises also may bring about trade-offs between high-level principles that have been developed. Mittelstadt (2019) already points to the challenge that “principles do not automatically translate to practice”, arguing for the need to develop methodologies to design concrete requirements and practices. However, the underlying assumption still seems to be that all principles can all be always maintained. Undoubtedly, the hallmark if crisis response is the do no harm principle (van Wynsberghe & Comes, 2020), which is translated into the beneficence of the technology in AI recommendations (UNESCO, 2022). However, it is not clear what the relation is of this principle with others, or how potential trade-offs should be made. For instance, in Covid19, the argument for increased surveillance (and potentially exposing vulnerable populations) was the safety and health of the population. Other questions include: would we sacrifice transparency or explainability for rapid action? And in how far would we relax accountability (as for the Covid19 vaccines) if we could potentially alleviate human suffering via an AI? Importantly, humans have been shown to change their preference for specific values under time pressure. For instance, Persson and Tinghög (2023) show that time pressure induces a preference for egalitarian over efficient solution. However, an AI may not be adaptive to the context. Therefore, what is needed are methodologies to decide a priori which principles dominate, and what may need to be sacrificed, and under which circumstances. Here, decision theory provides empirical methods to understand how humans make trade-offs (Keeney, 1996), and also analytical methods to formalize preferences and moral trade-offs into machine-readable formats by using preferential models (Holguin-Veras et al., 2013) or methods such as taboo trade-offs (Hanselmann & Tanner, 2008).

All in all, the urgency of a crisis clearly brings about challenges for the ethical design and oversight of an AI: crisis are often a space for experimentation (Sandvik et al., 2017). Given the urgency to alleviate human suffering, however, risk standards can shift, and what is acceptable in a crisis, and to whom, changes. An open question is: how can the tools that are rapidly developed and deployed in response to a crisis be tested and validated (Tzachor et al., 2020)? And who should set the guidelines and thresholds of what is acceptable, and to whom?

These questions also relate to the context of smart cities. According to Batty et al. (2012)’s seminal paper, smart cities entail both the planning, control and optimisation as well as “technologies that ensure equity, fairness and realise a better quality of city life [and] ensure informed participation and create shared knowledge for democratic city governance” (Batty et al., 2012). So far, however, in the development of AI for crisis management in the smart city, the focus has been overwhelmingly on sensing, analysing, and optimizing the response to crises (Yang et al., 2017). Limited attention has been paid to the potential downsides of the turn towards AI, such as introducing new vulnerabilities (Kitchin & Dodge, 2020), or the potential amplification of injustices (Kong & Woods, 2018). Because crises are known to especially affect the most vulnerable populations, it is critical to integrate the aspects of equity, fairness, participation into the design of technology for crisis management in the smart city. With respect to the sustainability of the technology (Van Wynsberghe, 2021), addressing the different timescapes of decision-making (cf. Table 2) is crucial: as crisis decisions tend to focus on the here and now, they tend to neglecting the contribution of the technology used to solve one crises to amplifying the climate crisis. Figure 2 highlights these interdependencies: on the short term both AI and crises are a driver of inequalities, increasing vulnerability a thereby leading to new crises and calls for (AI) technology. At the same time, AI may introduce more technology-dependence, complexity and vulnerability (orange loops). On the longer term (blue loop), the combination of increased vulnerability combined with the emissions of unsustainable AI, both effects may drive future crises.

Fig. 2
figure 2

Interdependencies of AI in the smart city

Full size image

I argue that to move forward, a decision-theoretical perspective is helpful that clearly distinguishes the problem(s), for which an AI is designed, and allows to specify the decision context along the dimensions as laid out in Table 2, and explicitly considers trade-offs, human decision behaviour and preferences. This approach emphasizes that no two crises or cities will ever be identical. Rather, they differ problem context, social networks, and actors (with their cognitive abilities and resources), and the guiding values, norms, and principles.

Discussion & conclusions

AI systems have tremendously influenced our lives. As our cities will be confronted with an increasing number and severity of crises, decision-makers are confronted with highly dynamic, volatile, and interacting processes and phenomena. Because of this complexity, especially in urban crisis response, there is widespread hope that AI can help us deal with the combination of complexity and urgency that is critical in crisis management. At the same time, this paper argues that AI introduces new dependencies and vulnerability, and may deepen or shift the crises that it is designed to resolve. I also argued that the insights from the area of crisis management in increasingly smart cities can be used for other contexts of ‘super-wicked problems’, ranging from adapting to a changing climate to pandemics, migration or poverty.

To guide the design and use of AI in society, various high-level AI guidances put forward a series of principles that AI should adhere to (for a review see (Jobin et al., 2019). Similarly, in the context of human-centred AI (Ozmen Garibay et al., 2023; Shneiderman, 2020), principles are set out to define the design process. Yet, in both cases, challenges persist in translating the principles to concrete design requirements or implementation for the purpose of urban crises. This paper has shown that crises decisions combine the well-known urgency and complexity with values, leading to a combination of cognitive and moral overload that AI is not yet adequately equipped to handel. First, while there is a lot of work on principles such as transparency or explainability, the fundamental principles that guide information management and decision-making in crisis response – such as humanity or solidarity – have not yet been formalized into machine-readable formats. Second, the urgency and high stakes of crises fundamentally change human information processing, sharing and decision-making – and therefore needs to be considered when designing for an AI that advises or interacts with humans in crises. Third, traditional models of automation need to be reconsidered that largely study one human interacting with one machine (Endsley, 2017; Parasuraman et al., 2000). In a crisis, there is no single decision-maker or ‘operator’. Especially in densely populated urban areas, many different citizens, volunteers, emergency management professionals, businesses, and policy makers all act and decide. Decisions are therefore often informal or improvised, and far from unified, bringing about thousands of independent actions, and attempts to coordinate that may overlap, compete, interfere, cooperate, or even battle for the scarce resources available (Holguín-Veras et al., 2012). To be effective, AI systems have to be adaptive to the different constellations of people, machines resources, and information that come together in an urban crisis, and operate in a coordinated, yet flexible fashion (Jennings et al., 2014).

Throughout the paper, several questions have arisen, and below, I am briefly characterising the most urgent research needs for the responsible design of AI for crisis decision-making. As such, the paper aims to inspire decision-makers, researchers and AI developers and aims to make headway in understand how AI can be designed for the shifting timescapes that characterise crises and other super-wicked urban problems.

Designing for Human Control in volatile networks

The urgency of crisis response, and the fear of human bias have led to a push for automation in crisis response. Yet, the volatile nature of crises often calls for improvisation and creativity, and machines need to be designed to be flexible adaptive to the changing contexts, and the emergent preferences and objectives. A critical issue is the balance of agency between the humans and the AI systems involved, especially under the time pressure of a crisis, when given the cognitive and moral overload, it may not be possible for humans to control or potentially override each AI system they interact with.

Even though there is a push to embed morality into machines (Arkin et al., 2012), acknowledging the context, the concepts of adjustable autonomy (Mostafa et al., 2019) and human moral autonomy, which defines the conditions that need to be fulfilled for human decision-makers to be able to maintain moral agency when interacting with an AI. Crisis AI needs to be designed to respect the cognitive strain on decision-makers while maintaining their moral autonomy, and they need to be dynamic, i.e., they need to ‘learn’ and adjust to the volatile roles and responsibilities as well as the changing situation at hand (Mendonca et al., 2006; Wallace & De Balogh, 1985). Besides theoretical work on defining how autonomy can be adjusted and upheld, interaction models and empirical research is needed on how humans (inter-)act with AI in crises. Given the possible pitfalls of experimentation in crises that were mentioned before, simulation games are a promising research methodology here, since they combine a safe space for trying and testing technology with relatively high fidelity and realism (Lukosch & Comes, 2019). To scale the insights from one or several humans to a whole city, agent-based models have been shown to help formalize and analyse the dynamic emergence of patterns (Helbing, 2012).

Designing AI for principles + urgency

While remains a debate in crisis management if response should be guided by a rights based approach that focuses on capacities (Gready, 2008; Nussbaum, 2007), or by standards of care and care ethics (Leider et al., 2017; van Wynsberghe & Comes, 2020), there are several principles that have been suggested to guide crisis response, including solidarity and humanity. While there has been much work on areas such as explainability, accountability or fairness in AI that is translated into guidance, standards and recommendations (e.g., (EC, 2019; IEEE, 2019; OECD, 2019; UNESCO, 2022), these high-level principles are thus far neglected by the technology that mostly focuses on control and optimisation. An open question is how crisis-related principles relate to other principles (such as trustworthiness), and how they can be translated into protocols for data acquisition, or into an algorithm for information analysis or decision-making.

For the well-established principles of trustworthy or human-centred AI (Riedl, 2019), such as explainability or accountability, their definitions do not account for the emergent and networked nature of social-technical systems in crisis response. To ensure for instance explainability in urban crises, the high level principles will need to be refined to also take into account trade-offs with explanations vs. timeliness. By using a decision theoretical approach to explore trade-offs and preferences, crucial questions can be answered that lead to more concrete requiements, such as: what needs to be explainable and to whom in the heat of a crisis, or how to define accountability if negative effects arise from the interaction of many machines with many humans. This step requires a combination of empirical and theoretical work, ensuring that the behavioural insights on how humans do behave can be linked to the theoretical understanding of how they should behave.

Designing for the most vulnerable

Crises affect all of society. Therefore, decisions around AI in crisis management are necessarily social choices. Yet, in crises, often those that are already vulnerable or marginalized are affected most. How can we ensure that AI is sensitive to their needs? Here, social choice ethics for AI design (Baum, 2020) advocates to consider questions around standing (who is heard?), measurement (what are the objectives, and how are they translated to data?), and aggregation (how do we form a choice based on potentially diverging preferences and attitudes?). This paper stressed that implementing ethical codes, values, norms and preferences of many stakeholders within AI systems is challenging (Sun & Medaglia, 2019). A major obstacle herein is creating a consensus across different viewpoints, given the difference in preferences and the potential power dynamics (Lewis et al., 2020). To ensure that there is true engagement, human-centred design approaches are needed that are fully inclusive. Besides the need to avoid biases in the way that damages are detected or resources are prioritized (Shams et al., 2023), this also means that vulnerable communities need to have a say about the fundamental principles of the technology that is designed to protect them. Here, methodologies are needed that help translate especially technical requirements into understandable language, and to ensure that people that may depend on assistance have the full autonomy to state their preferences.