A Digital Twin-Based Platform towards Intelligent Automation with Virtual Counterparts of Flight and Air Traffic Control Operations

Abstract

Automation technologies have been deployed widely to boost the efficiency of production and operations, to trim the complicated process, and to reduce the human error involved. Nevertheless, aviation remains human-centred and requires collaboration between different parties. Given the lack of a collaborative decision-making training platform for air traffic operations in the industry, this study utilises the concept of cyber-physical systems (CPS) to formulate a system architecture for pilots and air traffic control officers training in collaborative decision making by linking and integrating the virtual counterparts of flights and air traffic control operations. Collaborative decision-making training and the corresponding intelligent automation aids could be realised and supported. A performance analysis via a flight task undertaken with different computational load settings was prepared to evaluate the platform’s latency and integrity. The latency is presented using its 95% confidence interval, and integrity is presented using the percentage of data loss during wireless transmission. The results demonstrated convincing performance and a promising system robustness in both domains.

1. Introduction

In the 21st century, various intelligent automation technologies have been deployed widely to boost the efficiency of production and operations [1]. Complicated and time-consuming processes could be shifted to automated processes so that the time and effort spent on those tasks could be trimmed. The gradual technological growth of artificial intelligence made adaptive decision aids possible, thereby reducing the occurrence of human errors. Data sharing and open data initiatives have been proposed by Airbus in their Skywise application—with the use of the full potential of aviation operational data with artificial intelligence-driven data analytics. These intelligent approaches raise the operational efficiency, prevent delays, provide additional support to human intelligence, and reduce an individual’s mental workload. Numerous real-world applications of automation technologies can be found for daily life usage, from the development of intelligent transportation systems to the transformation into a smart city. Yet, the potential of digital twin-based automation technologies is yet to be unleashed in numerous industries, such as for air traffic operations in the aviation industry.

Aviation remains human-centred and requires collaboration between different parties. Through the continuous feeding of coordinates data from the aircraft, the air traffic control officers’ (ATCOs) main responsibility is to ensure safety in the regulated airspace by resolving potential conflicts and perform efficient air traffic management (ATM) in the terminal manoeuvring area [2]. The ATC decision-making process is a multicriteria problem which considers numerous aspects, such as the environmental aspects, safety aspects, and efficiency aspects [3]. The work for both flight crews and ATCOs can have no room for errors as their decisions have a direct implication on the safety of the passengers on board [4]. For instance, a mid-air collision might result if the flight crews in charge do not follow the appropriate instructions from the ATCOs, or if the ATCOs issue an instruction that is not feasible from the pilots’ perspective. The accidents resulted would transfer to injuries and casualties; thus, a zero-tolerance policy to safety regulation violations is enforced mandatorily in air-transport operations. Nevertheless, human error is inevitable during the flight crews’ and ATCOs’ decision-making processes, but the corresponding risk should be mitigated or minimised to a level that is generally acceptable by the industry. Human error is prevalent in every industry, and, in principle, 90% of workplace accidents are caused by human errors [5]. In aviation, up to 80% of accidents result from human errors [6]. With the increased workload of the operators, such accidents are more likely to happen in dense air traffic environments.

To minimise the potential faults from happening in real ATC situations, decent training, licence renewals, annual reassessment under normal and emergency procedures [7], and decision aids are common strategies that are proven effective and are widely adopted in the airline and flight training industry. Most airlines and flight training schools have adopted the digital twin of a real aircraft (high-fidelity flight simulators) as the key equipment to train pilots with a lowered risk of accidents [8,9]. According to the AIAA Digital Engineering Integration Committee [10], a digital twin is defined as “a set of virtual information constructs that mimics the structure, context, and behaviour of an individual/unique physical asset, or a group of physical assets, is dynamically updated with data from its physical twin throughout its life cycle and informs decisions that realise value”. Digital twins are also integrated for offline training with real-time monitoring, decision making, data-driven modelling and analytics [11], and human–computer–machine fusion [12]. The high-fidelity flight simulator software (mimicking the actual flight operations) in commercial settings is connected with third-party physical assets to formulate a digital twin environment, with the virtual training scenario being dynamically updated on the basis of actual physical inputs from assets. On the ATC side, the training of ATCOs is also done via digital twin with pre-set scenarios [13]. The complexity of ATC tasks is a result of its multimodality and the task nature as the coordinator to control the operations of different aircraft in the airspace using dissimilar technologies [14]. In academic research, digital twins are also widely used to analyse human performance in ATC [15,16,17]. For instance, Trapsilawati, Herliansyah, Nugraheni, Fatikasari, and Tissamodie [17] investigated the situational awareness and stress level of ATCOs via ATC simulators with the aid of electroencephalography (EEG). Kästle et al. [18] correlated situational awareness and signals from EEG with assigned tasks to subjects, and this could be implemented in the ATC context with ATC digital twin by assigning context-related tasks. Yet, existing training which focuses on analysing human performance only cannot cope with the swelling air traffic volume. The use of digital twin plays a crucial role in driving industrial skill training and academic research, and proper decision aids could minimise human errors in aviation.

As operations in aviation require collaborative actions from different stakeholders, every collaborating party could be jointly considered as a team [19,20,21]. In air traffic operations, the team comprises at least the flight crews and the ATCOs. Hence, their performance and effectiveness should be well managed in order to ensure the team’s ability to safeguard the passengers’ wellbeing. However, the end goals for flight crews and ATCOs might differ, and flight performance might deteriorate if both sides are unable to establish a common understanding. Nguyen et al. [22] reviewed the current assessment approaches to measuring situational awareness in the aviation context and denoted that individual situational awareness might not be adequate for complex environments such as arrival and departure management under extreme weather conditions. The literature suggests that measuring shared situational awareness is more important than realising individual situational awareness. The current practice in shared situational awareness training for ATCOs is to provide them with flight training, such that they will be able to understand the roles of pilots through acquiring basic flying knowledge. Therefore, a suitable digital twin should be adopted to train shared situational awareness between pilots and ATCOs. Such a simulation platform can permit real-time interaction between pilots and ATCOs in order to develop a shared situational awareness, as well as serving as a solid research support platform for academic researchers to study shared situational awareness via experiment for the development of intelligent automation aids to pilots and ATCOs.

Despite well-developed commercial flight simulators and ATC simulators being readily available, they are standalone systems and not integrated with each other. There is a lack of a collaborative decision-making training platform for air traffic operations in the industry. Therefore, the core contribution of the study is a cost-effective platform for pilot and ATCO training in collaborative decision making with high popularity. By linking and integrating the two digital twins for flights and ATC operations, i.e., the flight simulator and the ATC simulator, collaborative decision-making training and the corresponding intelligent automation aids could be realised and supported. In particular, as the flight simulation software is publicly affordable and the ATC simulation software is open-sourced, this provides a highly affordable and feasible solution for academic research and training. Therefore, the solution proposed in this paper serves as a solid foundation for developing advanced expert systems that strengthen future aviation safety. Furthermore, regarding the feasibility of collaborative decision-making training, the current study fosters a new direction for education and research in the aviation industry and academic community. Previously, most research only focused on either one of the parties but did not consider both roles. With the utilisation and integration of information and communications technology (ICT) resources for aviation operations, more work could be done to further strengthen the literature on teamwork in aviation. Most importantly, airline management and official bodies might consider incorporating this system into the current training plan for achieving a safer airspace in the future. Managerial insights could be gained from the current research to turn research ideas into real practice in the industry. Therefore, the core research questions of the current study are as follows:

• How could one incorporate the available digital twin into a suitable platform for shared situational awareness training?
• Could the proposed platform reach a satisfying or even promising performance for shared situational awareness training?

The remainder of the paper is organised as follows: Section 2 reviews the theoretical background and related work on collaborative decision making in air transport operations, the use of the digital twin-based platform in aviation vocational training, and the application of intelligent automation in aviation. Section 3 illustrates the system architecture of the digital twin-based platform. Section 4 presents the performance analysis and the discussions of the results. The conclusions, limitations of this study, and potential future work are presented in Section 5.

2. Theoretical Background and Related Work

To start, this section reviews the state-of-the-art research in the areas of collaborative decision making in air transport operations, as well as the use of digital twin-based training and intelligent automation in aviation.

In complex environments and real-life situations, it is inevitable that most of the processes cannot be achieved by a “one-man band”. Multiple stakeholders might be involved in the decision-making process according to their roles. In the context of air transport operations, pilots who operate the aircraft need to consider the potential effect of their operation on neighbouring traffic, the comfort of their passengers, the environmental impacts, the overall safety, etc. Therefore, the role of ATCOs is to regulate all of these factors from a macro perspective to provide a safe and efficient airspace. The complex nature of such a role demands high levels of human intelligence. As Salas, Sims, and Burke [19] defined a team as two or more people possessing a set of shared goals with multiple information resources, the flight crews and the ATCOs could be considered as a team. Hence, there is a need for collaboration and teamwork among every contributing party to act together to achieve a set of shared goals. A shared mental model among collaborating parties should be achieved to ensure a promising team performance [23].

Nguyen, Lim, Nguyen, Gordon-Brown, and Nahavandi [22] reviewed the situational awareness assessment approaches in aviation. The review suggested that shared situational awareness could be attained starting with the pilots. Flight crews first need to build and retain their situational awareness, as well as to notice any relevant events during their flying task. Then, they have to share their situational awareness and the relevant events with other members of the team and, most importantly, communicate with the ATCOs. While pilots usually are just concerned with their own aircraft, ATCOs can provide them with a more comprehensive understanding of the dynamics within the current airspace with the use of a time-dependant radar map.

Several studies also attempted to examine the mental model disconnects between the pilots and the ATCOs. For instance, Lai et al. [24] empirically investigated the mental model disconnects in a contextual perspective using a crew-based critical decision method (CDM) interview. The interview was conducted with pilots and ATCOs in Singapore with probing. Data were analysed qualitatively. It was discovered that flight crews might operate taking the passenger’s comfort into account, while ATCOs expect flight crews to strictly comply with their instructions. In particular, during abnormal operations such as adverse weather and dense traffic, a shared situational awareness between the two might be harder to achieve. In addition to qualitative analysis, simulation techniques were also used to model and validate real-world collaborative decision-making processes. Agent-based model simulation was adopted by Lai et al. [25] and Lai et al. [26] to study the air transportation system’s reliability and the mental model disconnects, respectively. In particular, the previous studies considered the aircraft as the agent as a function of the interactions between pilots and ATCOs.

To eliminate potential human error, vocational training is one of the conventional strategies with extensive applications in the aviation industry. From pilots to ATCOs and aircraft maintenance professionals, on-the-job training is a crucial part towards building a solid foundation of required technical and nontechnical skills. Qualified pilots have to go through knowledge-based and practical-based training in a designated organisation approved by the relevant civil aviation authorities. The practical-based training for pilots usually starts with a digital twin-based platform before moving on to a real aircraft. These training also enables trained pilots to learn the procedures and operations during exceptional circumstances such as emergency landing and engine failure. Therefore, FSTD-based training is a key element in student pilots’ education and renewal of pilot licences [7]. In terms of human-centred studies for pilot training, a flight simulator is also a promising offline environment to carry out experiments without interfering with real-world air transportation. Villafaina, Fuentes-García, Gusi, Tornero-Aguilera, and Clemente-Suárez [9] carried out a flight simulator-based study to collect psychophysiological responses during take-off, landing, air–air attacks, and air–ground attacks for military pilots with EEG and heartbeat rate measurement devices. Shao et al. [27] deployed 16 pilots for a simulator-based take-off task and established the relationships among attention span, instrument reading, and eye movement mode using an eye-tracking system with behavioural responses. Koglbauer and Braunstingl [8] evaluated a training method aiming to improve the situational awareness for ab initio student pilots by inviting the student pilots to complete a flight task in a flight simulator. Wojcik et al. [28] assessed the metacognition of student pilots during flight simulation tasks with demand resource evaluation scores (DRES) and revealed that individual metacognitive evaluations might be crucial in stressful events in aviation. From the above, it is apparent that the virtual counterpart of flight operations serves as a crucial digital twin-based platform for learning and practising flight operations.

In addition to pilots, the ATCOs need to undergo training in the digital twin-based platform before moving on to on-the-job training with real equipment located in the operations control centre [29]. Various commercial ATC simulators have been developed for this purpose, such as the NLR Air Traffic Control Research Simulator (NARSIM). To evaluate the human performance in aviation operations and facilitate visualisation of flight data offline, different scholars also built different ATC simulators. For instance, Fothergill et al. [30] developed the ATClab-Advanced simulated platform for conducting human-centred research with a limited geographical scope. Hoekstra and Ellerbroek [31] developed an open-sourced Python-based BlueSky ATC simulator to facilitate research works on air traffic management and air traffic flow. BlueSky incorporated the OpenAP developed by Sun et al. [32] to achieve automation in simulation. Compared to ATClab-Advanced, the geographical scope of BlueSky is global but it is under heavy development. However, regardless of the purpose of the simulators, the majority of ATC training employed pseudo-pilots to simulate the aircraft movements and conversations. In other words, the communication element, which is the key contributor to efficient teamwork, cannot be realistically simulated by the ATC simulators. Nonetheless, this is also the case for pilot training, with the instructions provided during simulation being imitated by pseudo-ATCOs. Therefore, it is evident that there is currently a lack of a platform for nurturing shared situational awareness.

With technological advancement, intelligent automation decision aids are becoming more feasible in the 2020s. Therefore, on top of appropriate training, proper intelligent automation decision aids can also be designed and implemented to minimise human errors occurrence within aviation operations. Trapsilawati et al. [33] first examined the autonomous components of the human–automation interaction within conflict detection and resolution advisory systems, revealing that utilising such autonomous systems could improve the performance and reduce workload of the operators. Trapsilawati, Chen, Wickens, and Qu [16] further adopted intelligent automation decision aids such as conflict resolution aids and vertical situation display for ground control operation. The corresponding experimental evaluation was also deployed in the NARSIM with various air traffic scenarios. Li, Ng, Fan, Yuan, Liu, and Bu [4] also proposed a human-centred data-driven and adaptive decision-making methodology by providing real-time feedback to the ATCOs for decision making. The workload of ATCOs could then be trimmed as part of their tasks were shifted to automated processes. A simulated environment was also deployed in E-prime 2.0 for demonstrating the framework, as well as for experimental data collection with follow-up analysis. The noticeable potential of future decision aids incorporated with intelligent automation has yet to be unleashed. In short, a promising virtual counterpart is crucial to driving industrial skill training and academic research in improving human performance in air transport operations.

From the above, it is observed that most current studies used a qualitative and simulation method to study the mental model disconnects between the pilots and the ATCOs. This is most likely a result of the technical infeasibility to have an offline training and experimental platform for nurturing the shared situational awareness between pilots and ATCOs. Moreover, one could also identify the demand for the digital twin-based training platform for vocational tutoring in aviation. Despite the well-developed existing digital twin for pilot and ATCO training, the only current methodology to foster shared situational awareness between pilots and ATCOs is to require the ATCOs to undergo fundamental pilot training [29].

However, it should be emphasised that the current strategies are insufficient in training efficient communication and cooperation between pilots and ATCOs, thus compromising aviation safety. Such events should only be permitted under extreme circumstances. Therefore, a digital twin-based platform for collaborative aviation vocational training should be developed on the basis of existing excellent standalone work for flight training and ATC training. Collaborative decision making could then be enhanced, thus reducing the possibility of mental model disconnects. It is, therefore, of paramount importance to develop real-time collaborative decision-making training with a digital twin-based architecture featuring low latency and high integrity.

3. System Architecture and Methodology

To realise real-time collaborative decision-making training and research, a digital twin-based platform was designed. The platform should perform with full integrity in order to ensure information is delivered to ATCOs within an acceptable range of latency with minimal data loss. We designed the platform using digital twin due to the ease of offline training in real-time monitoring, decision making, and data-driven modelling and analytics [11]. Moreover, Sepasgozar [12] provided a few future directions for developing a digital twin. One of those directions is to ensure the data generated and digital entity can be shared among various stakeholders, which supports human–machine, machine–machine, and human–computer–machine interactions. This is also the aim of the current platform—to train the flight operators and ATCOs to have a shared understanding of the environment through the training on the proposed digital twin-based architecture. This section illustrates the system architecture, the data flow within the network, and a description of the operations of the digital twin-based platform.

3.1. System Architecture & Physical Layout

To begin with, the digital twin-based platform was conceptualised and formulated via a local area network (LAN). Therefore, to present the system architecture effectively, a network diagram was deemed appropriate. Transmission Control Protocol (TCP) and Internet Protocol version 4 (IPv4) were applied in the transport layer to support the flow control in the LAN and the internet layer to formulate the LAN, respectively.

Figure 1 shows the network diagram of the proposed system architecture for shared situational awareness and collaborative decision-making training and research.

The platform was built with the hardware and software located in the Human Factors and Ergonomics Laboratory, Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University, Hong Kong SAR. The LAN framework consisted of two workstations and an integrated router with a wireless access point, as bounded by a purple rectangle in Figure 1; separate workstations were provided to the ATCOs and pilots. The workstation dedicated to the ATCOs (W-ATCOs) was a computer with Intel Core i7-10700 2.90 GHz CPU and 32 GB RAM in a Windows 10 Enterprise operating environment. It was equipped with a modified version of the BlueSky ATC simulator (BlueSky) [31]. The BlueSky ATC simulator is an open-sourced ATC simulator that allows modification to suit the user’s need. Its open-sourced design permits the integration of pre-set scenarios, which are suitable for scenario training, as well as real-time communication with the pilots operating the flight simulator. Therefore, taking into account BlueSky’s extensive scalability, we employed it to serve as the ATC simulator with a real-time ATC radar map. The modification of BlueSky for real-time flight status surveillance of the flight simulator is discussed in detail in Section 3.2.

The workstation on the pilot’s side consisted of two computers. Among the two computers, W-Visual was used for deploying the flight simulator software and W-Flightdeck was adopted to connect with the flight simulator hardware, as further explained in Section 3.3. The Prepar3D flight simulator software (version 3) was installed on W-Visual (specification: Intel Core i7-7700K 4.20 GHz CPU, 16 GB RAM, Windows 7 Home Premium), and a physical asset CockpitSonic A320 FTD was connected to W-Flightdeck. The CockpitSonic A320 FTD is full-scale simulator that can accommodate operations with two pilots, similar to that in real-life scenarios. Like W-ATCOs, a permanent private IP address at 192.168.0.130 was assigned to W-Visual to facilitate real-time data transmission. As W-Flightdeck is only responsible for coordinating the user inputs to manoeuvre the aircraft in the flight simulator, no permanent private IP address needs to be assigned. A well-developed application programming interface (API) SimConnect provides a solid interface to request data from parameters in Prepar3D. Hence, it was adopted to support the linkage between W-Visual and W-Flightdeck, as well as between W-Visual and W-ATCOs.

Lastly, the connection between the workstations was made wireless to facilitate the experiment’s settings. In real situations, ATCOs would not be able to observe and hear the pilot’s voice directly as they are physically separated. Therefore, to maintain the highest possible level of fidelity, the two workstations were not placed in the same room. Nonetheless, it is impossible to have a lengthy wired connection to formulate the LAN. Therefore, a router that served the role of a wireless access point was deployed to formulate the LAN and provide connections to the internet when necessary.

Figure 2 shows the two-dimensional physical layout of the digital twin-based platform from the top view. Being placed in two separate rooms, it was infeasible to have wired connections between the workstations. Intuitively, there would be a slight trade-off in real-time performance, but we believed that a wireless connection remained a promising solution. Moreover, a wired connection is impossible in real ATC scenarios. Therefore, it was reasonable to deploy a wireless network for connection. A detailed analysis of performance in latency is presented at a later stage of this paper.

3.2. Data Flow and the Realisation of Real-Time Transmission

Standalone simulators for flight operations and ATC operations are widely available in the commercial market. Nonetheless, the core aim was to establish a communication channel between simulators so that a real-time representation of the flight operations could be realised in a radar map. The current focus was to provide a communication channel for flight data to transmit from the flight simulator to the ATC simulator. Therefore, a diagram indicating the data flow from data requesting to data responding is presented in Figure 3. Figure 4 supplements Figure 3 by demonstrating the application of the TCP/IP model.

In the data flow, W-ATCOs could be considered as the client requesting data and W-Visual could be considered as the server responding to said request. It should be noted that W-Flightdeck is not involved in the data flow. A standalone data request application that can be called via BlueSky is used to support the data flow. It was coded in C++ language using Microsoft Visual Studio Community 2013, with the aid of the SimConnect API. The application calls the SimConnect API to return five crucial flight parameters, including the latitude, the longitude, the altitude, the ground speed, and the heading of the aircraft simulated in Prepar3D at a specified frequency $f$ (either 1 Hz or 6 Hz). In other words, the refresh rate of the data acquisition application allowed receiving data every second or every 0.1666 s, according to Equation (1).

$$T=\frac{1}{f}.$$

(1)

The data transmission process is first initiated from the client side by activating the data request application in BlueSky. Then, the SimConnect API is regularly called at specified intervals to send a request for obtaining the required flight parameters. A detailed description of the process is given below. Initially, the data request is generated in the application layer by the data request application. It is then passed to the transport layer for data block division and TCP encapsulation. A TCP header consisting of a random source port number and a designated destination port number 500 is joined with the data blocks of the data request to compose TCP segments. After that, the TCP segments are passed to the internet layer to form IP packets. An IP header is added to each TCP segment with the source IP address of 192.168.0.247 (W-ATCOs) and a destination IP address of 192.168.0.130 (W-Visual). The destination IP address and the communication protocol (IPv4) are set in a configuration file located in the client. The IP packets eventually reach the host-to-network layer, and a frame header and trailer are added to form Wireless LAN (WLAN) frames. They are then emitted as frames from the client through wireless media to the wireless access point. The WLAN frames of the request would first reach the wireless access point for routing. They subsequently depart from the wireless access point to reach the server. Upon receiving the request, the server decapsulates the WLAN frames to IP packets and further to TCP segments. The arriving TCP segments are rearranged in an appropriate order and joined together to return to the original request transmitted. With the aid of the destination port number, the TCP also allows exploring a suitable application for reaching the application layer. The port number 500 on the server side is the port number for the SimConnect API to connect with Prepar3D, as specified in an XML file located in the server. Lastly, in the application layer of the server side, the request is executed to retrieve the instantaneous flight parameters.

After the retrieval of the required flight parameters, a similar process is conducted but in reverse order. The response to the data request can be treated as a new data transmission process independent from the request. The process starts from the server side. The data response from the application layer is first divided into data blocks and encapsulated with TCP. The resultant TCP segments are equipped with a source port number 500, and the destination port number is identical to the random source port number from the data request. Then, the IP packets are formed in the internet layer. The source and destination IP addresses are the reverse of those from the data request. New WLAN frames are formed to send the reply from the server with the requested flight parameters to the wireless access point, and then routed to the client physically in the host-to-network layer. Upon the reception of the WLAN frames, the decapsulation process is undertaken to unpack WLAN frames to IP packets and TCP segments. Similarly, the TCP segments are ordered and embedded to form the original data. Due to the limitation of having to cross platforms, the arriving data are then passed to BlueSky in the client side by the data request application. They are also saved in a CSV file for easier reference. A sample of the outputted data is shown in

Table 1. CSV data format with sample data.

Variable	Latitude (°)	Longitude (°)	Altitude (Feet)	Ground Speed (Knots)	Heading (°)
Data	22.2222	113.749	2163.44	148.713	16.1816

. The units of the flight parameters are as follows: latitude and longitude, degrees (positive denotes east/north, negative denotes west/south); altitude, feet; ground speed, knots; heading, degrees. Therefore, this example implies that the aircraft position is at 22.2222° E, 113.749° N at the altitude of 2163.44 feet, with a ground speed of 148.713 knots and a heading of 16.1816° at the snapshotted instance of the data request.

The bridge from the data request application to BlueSky terminates the data request application in a request. The remaining process is done using BlueSky in a Python 3.7 environment. The original version of BlueSky supports the use of user inputs to alter the movements of the aircraft shown on the radar map, allowing for the imitation of ATCOs providing instructions to flight crews. As simulated aircraft operations involving verbal command are not currently possible, a command console approach is adopted in BlueSky, which recognises and accepts inputs with the syntax used in NLR’s TMX Traffic Manager. Therefore, users can enter commands according to the syntax used in NLR’s TMX Traffic Manager to instruct the aircraft to fly according to the desired operations. A simulated aircraft can be created by entering the following command: “CRE [callsign], [latitude], [longitude], [heading], [altitude], [speed]”, and it can be subsequently moved by “MOVE [callsign] [latitude] [longitude] [altitude]”. Speed and heading of the simulated aircraft can also be set via “[callsign] SPD [speed]” and “[callsign] HDG [heading]”, respectively. For instance, if there exists an aircraft with callsign AAE123, the ATCO could input the command “AAE123 HDG 300” to change the aircraft’s heading to 300°. The user interface (UI) was modified to utilise this user input feature.

Figure 5 shows the modified UI of BlueSky. The details of the UI modification and real-time radar map updating process are provided below. In terms of UI modification, a menu “SimConnect” consisting of two buttons “Initialise” and “Load Flight Data” was added to the existing UI. The first action done by “Initialise” is to call and activate the data request application. After that, the view of the radar map is panned to the desired airport. The Hong Kong International Airport (ICAO: VHHH, IATA: HKG) is adopted via a command “PAN VHHH” in our case. Then, as the aircraft synchronising with the flight simulator is not initially available on the radar map, the initialisation also creates the aircraft via the CRE command described in the previous paragraph. The checkable button “Load Flight Data” then follows. When “Load Flight Data” is checked, another thread is activated to run the sub-process of retrieving the flight parameters exported by the data request application. Multithreading is utilised to tackle the problem of having two simultaneous processes being executed by the same thread. The main worker remains responsible for executing BlueSky, while the sub-worker is deployed to import the flight parameters. The retrieval rate of the flight parameters is identical to the writing interval. Whenever the flight parameters are read successfully into BlueSky, the sub-process emits said data into the main process. The main process continues to set the corresponding command according to the syntax previously described. The constant radar map update in BlueSky then includes the command entered automatically when flight parameters are received. Eventually, an intelligent and automated data transfer and visualisation process is realised to support real-time displays of the situation in the flight simulator on BlueSky.

3.3. Operational Description of the Digital Twin-Based Platform

To illustrate the operations of the digital twin-based platform, a conceptualised framework for training on collaborative decision making is presented in Figure 6.

The training based on this digital twin-based platform comprises at least one pilot and one ATCO. The pilot is required to operate a flight task in the region that the ATCO is managing. As with BlueSky’s universality, both a macro view for tower/en route cruising operations and a micro view for ground operations can be supported. Therefore, multiple ATC tasks can be assigned to different ATCOs if more W-ATCOs are connected to the system architecture. The training begins with executing Prepar3D and BlueSky on W-Visual and W-ATCOs, respectively. A predefined air traffic scenario using text-based scenario files that can be read by BlueSky is also generated. Past flight trajectory data can also be adopted to formulate these scenarios, or self-designed scenarios can be utilised.

After loading an appropriate scenario, the button “Initialise” is first clicked. As previously described, the function allows moving the radar map in BlueSky to a predefined airport. The data request application is called, and the required data are sent through wireless media. The aircraft is then created accordingly. This can be demonstrated by the flight shown in Figure 5, which is at 2163 feet with a speed of 149 knots. The heading is shown via an arrowhead. By using the flight simulator hardware located in the laboratory, pilots can then operate the aircraft according to the flight task assigned to them. During the flight task, the ATCO-in-charge is assigned to monitor the operations of the concerned flights and other pre-set surrounding flights in their area of control.

To train for different abnormal situations in the corresponding flight information region (FIR), a tailor-made scenario can be adopted. For instance, heavy traffic scenarios would usually cause a disturbance and raise the stress faced by the ATCOs. More human errors are expected in heavy traffic scenarios. Therefore, the real flight data with dense air traffic can be converted into training materials, or an imaginary heavy traffic scenario can be defined. To imitate the air–ground communication, a wireless microphone is placed in the cockpit for the pilots to communicate with the ATCOs. The ATCOs are required to listen to the pilots’ requests/reports with an earphone and provide instantaneous verbal instructions to the pilots through a microphone. Their transmission is done through a video conferencing tool. The communication history, their behavioural responses, and the psychophysiological responses of the ATCOs can be recorded and analysed, such that possible intelligent decision or communication aids can be proposed to relieve their workloads. These decision aids can utilise the recorded behavioural reaction time in responding to the pilot’s requests/reports and provide a real-time alerting mechanism to ATCOs regarding their potential lack of situational awareness. Furthermore, online electroencephalography (EEG) data-based classification of mental workload state can aid in improving communication performance between ATCOs and pilots. When the threshold of automatic decisions is reached, the ideal mechanism is to entertain the pilots’ request via computational intelligence until the ATCOs are free to entertain the request. Most importantly, an experiential learning approach is always useful as ATCOs could understand the constraints of the pilots more deeply via discussions on the communication history and vice versa.

In addition to abnormal airspace situations, various abnormal flight operations, such as engine fire and landing gear failure, can be triggered through the instructor’s panel next to the captain’s seat. To simulate such situations, the scenarios in the flight simulator can be modified via the instructor’s panel. It is hypothesised that the effect of abnormal flight operations might be more apparent on pilots than ATCOs, who are only receiving second-hand information and are not members onboard the concerned aircraft. Nevertheless, ATCOs should still be informed instantly in order for them to propose solutions for safeguarding the wellbeing of those onboard, and the use of intelligent automation methodologies can be applied in providing assistive instructions on the basis of surveillance from the ground. Similarly, the responses from both sides can be recorded for analysis, as well as for future decision aid development. This further shifts the human intelligence of aviation to machine intelligence and reduces the possibility of human errors. Most vitally, the ATCOs can also learn the way that pilots usually respond to these situations, and human errors can be further minimised with experiential learning.

4. Performance Analysis and Discussion

4.1. Latency Measurement

The work and instructions issued by the ATCOs indirectly affect the safety of the flight crews and passengers on board. As pilots need some time to execute the instruction from the ATCOs, the response time of humans should be considered during decision making. In addition, aircraft are usually travelling at a high speed; thus, consequences might still occur even when correct decisions are made. If the decisions are delayed, the consequences might be hard to reverse. Therefore, despite it being essential to make error-free decisions, timely decisions should also be made. Hence, the decisions should be supported by instantaneous/near-real-time data to ensure validity. Given this, the latency of the data transmission process should be adopted to evaluate the performance, especially if the network is wireless.

Figure 7 shows the method utilised to measure the latency of the system. As the data transmission process is mainly conducted through SimConnect, it is more appropriate to snapshot the time of data request at the client side instead of the time of request arriving at the server. This can be accurately timestamped in the data request application, and it is denoted as “Timestamp 1”. The second timestamp snapshotted is when the data are received by W-ATCOs, denoted as “Timestamp 2”. Therefore, the difference between “Timestamp 1” and “Timestamp 2” is the time elapsed for data request and data return transmission through the wireless network. As the ATCOs only read the radar map instead of directly interpreting the raw data, it is essential to have the third timestamp snapshotted. Moreover, it is crucial to evaluate the time elapsed between data reception and data visualisation. “Timestamp 3” is the time that the data is visualised in the radar map. Hence, the elapsed time from “Timestamp 1” to “Timestamp 3” is the total time used to complete one request, while that from “Timestamp 2” to “Timestamp 3” is the time spent on the flight parameter passing procedure and visualisation.

Table 2. Timestamp snapshotted for latency measurement.

Timestamp	Notation	Description
1	$t_1$	The flight parameter data request is sent out by W-ATCOs
2	$t_2$	The flight parameter data are returned to W-ATCOs
3	$t_3$	The flight parameter data are visualised and displayed on the radar map

and

Table 3. Key performance indicators of system’s latency.

KPI	Description	Equation
$t_{1,2}$	Time elapsed for wireless data transmission	$t_2-t_1$
$t_{2,3}$	Time elapsed for flight parameters passing procedure and visualisation	$t_3-t_2$
$t_{total}$	Total time elapsed for a data request to be processed	$t_{1,2}+t_{2,3}$ or $t_3-t_1$

show a summary of the timestamps selected for latency measurement and the key performance indicators (KPIs) of the system’s latency, respectively.

The KPIs are the latency performance indicators defined by the time difference between every two timestamps that are recorded. The experimental evaluation was conducted using the platform described in Section 3.1. The evaluation was done using a flight task consisting of the three major flight phases, classified as take-off and climbing, en route cruising (with autopilot enabled), and descent and landing. It is expected that the traffic density in the flight simulator will affect the computational performance, and hence result in latency. Therefore, for each flight phase, an experiment was designed and implemented for different traffic densities, i.e., light traffic load at 10%, medium traffic load at 50%, and heavy traffic load at 100%. The light traffic load was set to 10% instead of 0% as having 0% air traffic density is unrealistic. The entire flight task lasts around 60 min from Hong Kong International Airport to Kaohsiung Siaogang International Airport. Furthermore, the difference in graphics between daytime environment and night-time environment can affect the latency. Therefore, the time setting, i.e., day (10:00 a.m. local time, start time of the flight) and night (8:00 p.m. local time, start time of the flight) in the flight simulator was another independent variable in the experiment. A trial of the flight task was conducted for each experimental setting, i.e., each set of independent variables. The acceptance criterion of satisfactory performance was 500 ms (one-way) as suggested by Baek, et al. [34] for ADS-B-Out message transmission. As $t_{1,2}$, $t_{2,3}$, and $t_{total}$ involve two-way transmission, the acceptance criteria were defined as a double of the threshold set by Baek, Hableel, Byon, Wong, Jang, and Yeo [34], i.e., 1000 ms. The boxplots of the latency values are provided in Figure A1, Figure A2, Figure A3, Figure A4, Figure A5 and Figure A6 in Appendix A.

A 95% confidence interval of the results with specified lower bound and upper bound was constructed to demonstrate the system robustness. As such, the 95% confidence intervals of the KPIs for the take-off and climbing phase, en route cruising phase, and descent and landing phase are presented in

Table 4. Descriptive statistics of the system latency performance ¹.

Time Setting	Frequency of Request	Heavy Traffic			Medium Traffic			Light Traffic
		${\mathit{t}}_{\mathbf{1}\mathit{,}\mathbf{2}}$	${\mathit{t}}_{\mathbf{2}\mathit{,}\mathbf{3}}$	${\mathit{t}}_{\mathit{t}\mathit{o}\mathit{t}\mathit{a}\mathit{l}}$	${\mathit{t}}_{\mathbf{1}\mathit{,}\mathbf{2}}$	${\mathit{t}}_{\mathbf{2}\mathit{,}\mathbf{3}}$	${\mathit{t}}_{\mathit{t}\mathit{o}\mathit{t}\mathit{a}\mathit{l}}$	${\mathit{t}}_{\mathbf{1}\mathit{,}\mathbf{2}}$	${\mathit{t}}_{\mathbf{2}\mathit{,}\mathbf{3}}$	${\mathit{t}}_{\mathit{t}\mathit{o}\mathit{t}\mathit{a}\mathit{l}}$
Daytime	Take-off and climbing phase
	1 Hz	[15.8, 16.2]	[493, 521]	[509, 537]	[15.8, 16.2]	[468, 502]	[484, 518]	[15.7, 16.3]	[478, 516]	[494, 532]
	6 Hz	[15.9, 16.1]	[84.9, 87.1]	[101, 103]	[15.9, 16.1]	[85.9, 88.1]	[102, 104]	[15.8, 16.2]	[86.6, 89.4]	[103, 105]
	En route cruising phase
	1 Hz	[14.7, 15.3]	[500, 538]	[516, 554]	[15.8, 16.2]	[508, 538]	[524, 554]	[14.8, 15.2]	[487, 513]	[502, 528]
	6 Hz	[14.9, 15.1]	[86.5, 89.5]	[101, 105]	[14.8, 15.2]	[83.3, 86.7]	[98.0, 102]	[14.8, 15.2]	[85.9, 91.0]	[101, 105]
	Descent and landing phase
	1 Hz	[15.8, 16.2]	[470, 500]	[486, 516]	[16.7, 17.3]	[473, 515]	[490, 532]	[16.7, 17.3]	[414, 466]	[431, 483]
	6 Hz	[17.8, 18.2]	[86.8, 91.2]	[105, 109]	[16.9, 17.1]	[84.7, 87.3]	[102, 104]	[15.9, 16.1]	[85.9, 88.1]	[102, 104]
Night-time	Take-off and climbing phase
	1 Hz	[18.4, 19.6]	[484, 516]	[503, 535]	[15.7, 16.3]	[504, 534]	[521, 551]	[15.8, 16.2]	[481, 515]	[497, 531]
	6 Hz	[14.9, 15.1]	[84.8, 87.2]	[101, 103]	[16.9, 17.1]	[86.9, 89.1]	[104, 106]	[16.9, 17.1]	[84.9, 87.1]	[102, 104]
	En route cruising phase
	1 Hz	[17.4, 18.6]	[495, 531]	[513, 549]	[15.8, 16.2]	[459, 493]	[475, 509]	[15.8, 16.2]	[489, 517]	[505, 533]
	6 Hz	[15.9, 16.1]	[84.6, 87.4]	[101, 103]	[15.9, 16.1]	[87.0, 89.0]	[103, 105]	[15.8, 16.2]	[84.0, 88.0]	[100, 104]
	Descent and landing phase
	1 Hz	[16.6, 17.4]	[493, 523]	[511, 541]	[16.7, 17.3]	[497, 539]	[514, 556]	[16.6, 17.4]	[497, 549]	[514, 566]
	6 Hz	[15.9, 16.1]	[84.9, 87.1]	[101, 103]	[15.9, 16.1]	[85.6, 88.4]	[102, 104]	[15.9, 16.1]	[85.1, 86.9]	[101, 103]

¹ Units: milliseconds (rounded off to three significant figures). Data are presented with their 95% confidence interval in the format of [lower bound, upper bound].

. In general, the results demonstrated convincing performance in latency as the upper bound of the 95% confidence interval of $t_{total}$ did not exceed the acceptable threshold at 1000 ms. This indicates that one can have 95% confidence that the latency value will not exceed the upper bound specified in Table 4. Furthermore, as $t_{1,2}$ accounts for nearly negligible impact in latency, it is evident that the use of wireless connection settings is acceptable. Most importantly, the connection between aircraft and ground ATC centre is also wireless, and the current setup also retains such a feature. Interestingly, compared to the data visualisation delay $t_{2,3}$, the network delay $t_{1,2}$ was nearly trivial. Therefore, we can prove that the major source of latency is not a result of network delay. This phenomenon may be caused by the data processing time and refresh rate constrained by the user interface of BlueSky. The data request process is only a two-way byte stream transmission process without complicated manipulation. Compared to the data request process through SimConnect, the data parsing process from the data request application to BlueSky incorporates data input from the data request application, passing the work to the second thread, control command executing, parsing data back to the main thread, and eventually executing the command. Therefore, the instantaneous computational load of the client for the second process would be significantly higher than that of the data request process. Hence, with the raised computational load for the client, the time spent on data visualisation is longer. Additionally, the refresh rate of BlueSky would limit the data input process from the data request application. It is noted that the ranges of mean values of $t_{total}$ for 1 Hz and 6 Hz were 457–540 and 100–107, respectively. This implies that the range for the 1 Hz data request frequency was around four to five times that of the range of the 6 Hz data request frequency. As the data request application would only trigger the data input, the visualisation refresh rate is independent of the data request application. In addition to the refresh rate, the executing time of the code can cause a minor deviation of the refresh rate timer of BlueSky. As the final visualisation time is measured, the deviation may be coupled with the latency resulted from the data processing time. This phenomenon was more commonly seen when the data request application was running at 1 Hz. Therefore, it is likely that the refresh rate constrained by BlueSky is causing such a phenomenon. Nonetheless, the overall time $t_{total}$ remained in the acceptable region. Coupled with the 95% confidence interval, the system’s robustness was, hence, substantiated. Thus, it is proven that the platform has promising performance in latency.

In addition to the 95% confidence interval of the latency KPIs, a desired real-time platform should also be stable, i.e., having consistent performance throughout similar settings. All the latency KPIs $t_{1,2}$, $t_{2,3}$, and $t_{total}$ do not vary dramatically across different traffic loads, flight phases, and time settings at the same frequency of request. In particular, the maximum absolute differences between the highest and lowest values of the mean of $t_{total}$ at 1 Hz and 6 Hz were 83 and 7 ms, respectively. This suggests that the real-time performance can likely be maintained throughout most scenarios, and the effect of the computational load on the server side does not significantly affect the data request. Therefore, as the platform performs similarly well in various possible scenarios that vary the server’s computational load, the platform can be regarded as having a stable performance in real-time visualisation. Given the promising data for on-time performance (OTP), the platform was able to fulfil the requirements of an online platform, with the data being transmitted in real time/near-real time to provide up-to-date information for the ATCOs to perform proper decision making.

4.2. Integrity Measurement

In addition to the on-time performance of the platform, integrity is another key factor in aiding ATCOs to carry out ATC tasks. In particular, as our network is wireless, it might be possible for data loss in the atmosphere to occur during data transmission. Nevertheless, every bit of data is crucial to ATC decisions and situational awareness. Therefore, high integrity should be attained. With the same experimental setting in Section 4.1, each request is paired with a return value to indicate whether data loss during wireless transmission is minimised. The KPI adopted to measure integrity is the percentage of data loss during transmission ${\%}_{loss}$, regardless of the cause of data loss.

$${\%}_{loss}={\displaystyle \sum }_i^N\frac{Number of transmission loss}{Total number of transmission}\times 100\%.$$

(2)

Ideally, it should be kept at 0% so that all the requests can be successfully transmitted to the server and the required data are returned to the client. The performance is regarded as satisfactory if ${\%}_{loss}$ is below 10%. Nevertheless, it is more promising to achieve a ${\%}_{loss}$ that is less than 5% and ideally less than 1%. The interval of the data output is measured and analysed for deducing ${\%}_{loss}$. For instance, when the data request application is running at 1 Hz, the interval between two data should be maintained at the same rate (with minor tolerance for computational load) to demonstrate no data loss during transmission.

Table 5. Percentage of data loss resulted over the specified flight task period ¹.

Frequency of Request	Day			Night
	Heavy Traffic	Medium Traffic	Light Traffic	Heavy Traffic	Medium Traffic	Light Traffic
Take-off and climbing phase
1 Hz	*** (0.976%)	* (6.27%)	** (2.71%)	** (2.82%)	** (2.04%)	** (1.44%)
6 Hz	* (5.45%)	* (5.63%)	* (5.20%)	* (5.07%)	* (5.64%)	* (5.39%)
En route cruising phase
1 Hz	*** (0.499%)	* (9.37%)	** (1.07%)	** (2.99%)	** (1.04%)	** (1.27%)
6 Hz	* (5.23%)	** (4.17%)	* (5.11%)	** (4.32%)	* (5.31%)	** (4.15%)
Descent and landing phase
1 Hz	** (1.05%)	** (1.46%)	** (1.62%)	** (1.47%)	* (5.07%)	*** (0.420%)
6 Hz	* (6.12%)	** (3.13%)	** (3.61%)	** (3.59%)	* (5.25%)	** (4.98%)

¹ Rounded to three significant figures. The notations ***, **, and * denote values less than 1%, 5%, and 10% respectively.

shows the evaluation results of integrity measurement. The results demonstrated an acceptable performance throughout all the experimental settings. Out of the 36 KPIs measured for different scenarios, three of them (8.33%) achieved less than 1% data loss, 22 of them (61.1%) achieved less than 5% data loss, and all of them (100%) achieved less than 10% data loss. The performance was convincing with more than 60% of the test scenarios that could benchmark performance having less than 5% data loss. It can be observed that the average data loss percentage for the 1 Hz frequency of request (2.42%) was lower than that for the 6 Hz frequency of request (4.85%). This is likely due to the huge number of requests and data flowing between the client and the server at the 6 Hz request frequency. During large instantaneous data flow, data loss in wireless media is possible and reasonable. Even the transmission of ADS-B data involves a large proportion of data loss. The original ADS-B transmission interval should be 0.5 s. Nonetheless, it is often seen to receive a data stream with double the interval. In the controlled experimental platform evaluation, the amount of data loss should be minimised as much as possible, and the criteria should be stricter than in the ADS-B transmission. With the reduced frequency of transmission, the network traffic load is alleviated. The alleviated workload can reduce transmission errors caused by the overwhelming flow of data. Therefore, it is reasonable to have a small percentage of data loss as it is difficult and unrealistic to reach ideal conditions. Hence, with the high integrity achieved, fewer information errors would disrupt the ATCOs’ decision-making process. In short, the proposed platform is practical for real-world applications in shared situational awareness training.

4.3. Future Roadmap in Shared Situational Awareness Training and Research

Given the low latency and high integrity achieved in our proposed platform, the digital twin approach to provide real-time collaborative decision-making training is highly viable through the above-proposed setup. The practical implication of this paper is to contribute a feasible and economical (partially open-source-based) solution in enhancing human factors training in air traffic operations. Multiple stakeholders in air traffic operations have been considered instead of only some. The communication and mental model disconnect issues are expected to be addressed through a thorough and extensive use of the proposed platform. On the academic research side, future studies can attempt to use the platform to conduct digital twin-based experiments. The subjects’ performance can be evaluated online such that data-driven, adaptive, and intelligent automation aids can be further designed to address human factors involved in aviation operations. Therefore, the current solution serves as a cornerstone, as well as a starting point, for further investigation regarding the shared situational awareness issue in an objective approach.

5. Concluding Remarks

In summary, the current study attempted to integrate the virtual counterparts of different applications of simulation technology in aviation for collaborative decision-making training and research. Previously, standalone simulators were well developed with high fidelity, but there was a lack of an integrated platform to unleash the potential of a well-developed virtual counterparts for aviation team training applications. This research contributes by proposing such an integrated system to benefit aviation training and further human performance research in aviation. For aviation training, the platform can be deployed in a collaborative exercise between pilots and ATCOs. In addition to the present curriculum, training on shared situational awareness and collaborative decision making can be included in the standardised vocation educational procedures. In terms of human-centred research, this platform provides a solid foundation to support such research. Experiments via offline simulated settings can be conducted through this platform with the aid of psychophysiological measurement devices and the feasibility of using user-defined scenarios. This facilitates researchers in setting up different scenarios and compares the human performance between them accordingly. Therefore, as aviation remains highly dependent on human intelligence, this platform further contributes to the world of human factor research in aviation. Eventually, with a comprehensive understanding of interacting human factor contribution to air transport operations, more intelligent automation aids that facilitate decision making can be built using models trained through physiological and behavioural responses.

Regarding the performance and the robustness of the platform, in Section 4, the two main dimensions, i.e., latency and integrity, were measured through a flight task experiment repeated for different settings. Different variables were utilised as independent variables to form matrices to present the results. For latency, three KPIs in terms of the time elapsed were presented in the form of their 95% confidence interval. The average values of all the KPIs were less than the threshold specified. Additionally, the upper bound of the 95% confidence interval of the system’s latency did not exceed said threshold. For integrity, one KPI in terms of data loss percentage was measured during the experiment. The results demonstrated all the test scenarios resulted in less than 10% data loss, and over 60% of the test scenarios resulted in less than 5% data loss. These convincing results indicated that the platform performed well and robustly in these two domains. Yet, these KPIs can indirectly influence the human performance of the ATCOs as latency and data loss might result in a lack of situational awareness, particularly the shared situational awareness with pilots. Hence, one should avoid compromising safety, and further enhancement can be made to mitigate and minimise the undesired consequences via synchronisation of refresh rate.

Despite the current research supporting the feasibility of deploying collaborative decision-making training, the platform is offline and delinked from real-time air traffic. Therefore, as there is only one flight simulator connected with one set of flight crews operating the simulator, the effect of communication among pilots from other flights cannot currently be realised. In other words, there is a lack of pseudo pilots that can communicate with ATCO trainees/subjects regarding the motion of other nonuser aircraft shown on the radar map. As the environment is offline, controlled, and simulated, such limitations are inevitable. Thus, it is suggested to connect real-time flight data and a pseudo pilot’s voice to further raise the fidelity of the platform. Further developments can be built on the current architecture for enhancing the performance and the fidelity of the platform via commercial simulators and advanced networking/telecommunication systems. Lastly, if the pseudo pilots can be further developed using artificial intelligence with the ability to respond to ATCOs effectively, the digital twin can be improved to further reduce the human factors involved in aviation operations.