Challenges and Solutions for Service Continuity in Inter-PLMN Handover for Vehicular Applications

The reliability and availability of network connectivity, which significantly varies with mobility, is crucial in Connected, Cooperative and Automated Mobility (CCAM). Handover and roaming are the most challenging situations in terms of connectivity of cellular networks, which require switching across cells of the same cellular network or between Public Land Mobile Networks (PLMNs). This paper proposes a set of solutions for vehicular applications to mitigate the impact of mobility in service continuity, including a dual modem solution that reduces the interruption time when switching PLMNs, an adaptive bitrate mechanism for media streaming that increases reliability, a WebRTC server acting as a gateway in media streaming sessions between vehicles, and a MEC discovery and handover method. The proposed solutions have been evaluated executing an Extended Sensors application in several commercial and experimental 5G Non-Standalone (NSA) and Stand Alone (SA) setups with different Multi-access Edge Computing (MEC), edge-cloud and cloud infrastructures to host services. It can be concluded from the results obtained that 5G networks have not yet achieved the required performance for CCAM, and that practitioners need to implement solutions and workarounds, such as the ones proposed in this work, to mitigate the issues. As lessons learnt from the deployment and experimentation, this paper also overviews a detailed set of problems and the proposed solutions that CCAM industry and cellular network stakeholders need to consider.


I. INTRODUCTION
Vehicle sensors enable the onboard systems to understand better the environment to assist the driver or even take control of the vehicle. Advanced Driver Assistance Systems (ADAS) are proliferating in the automotive market, turning them into a key marketing claim and increasing the vehicles' added value and profit margin. As the connectivity comes to the onboard systems, it opens unlimited possibilities to consume Connected, Cooperative and Automated Mobility (CCAM) services. First, the vehicles can better understand the environment from external sensors, extending the capacity of in-The associate editor coordinating the review of this manuscript and approving it for publication was Eyuphan Bulut . vehicle devices. In this sense, Computer Vision (CV) assisted applications have a prominent role, and the availability of data beyond the Field of View (FoV) of the ego-vehicle makes the difference.
These applications need to discover relevant sensors or vehicles in a Region of Interest (ROI), to connect participants performing required handshakes and to exchange data streams across systems through the cellular networks. In this context, the edge plays an essential role by acting as a common entry point for service participants and a trusted system where to find others.
It becomes evident that the connectivity of the vehicle and onboard systems fosters a new generation of CCAM services. The next generation of cellular networks (5G) promises some Key Performance Indicators (KPIs) that are supposed to satisfy the performance and capacity requirements of those CCAM services. In some CCAM services, the response time is not required to be real-time, and the communication is mainly transactional, such as parking lots services or route planning solutions. In these cases, temporary network under-performance or short interruptions do not mean a critical impact on the service. However, some critical CCAM services require real-time connectivity with continuous availability, so a best-effort approach is not enough [1]. These needs are more likely as far as the session implies several parties with different vehicles moving and potentially using different networks that need to establish a session between them.
CCAM applications bring some challenges to be managed and mitigated linked to the availability of external sensors, which heavily rely on network connectivity. The impact of mobility in application continuity is evident, specially in safety applications where packet losses corrupt the sensor images and high latencies and jitter force frame dropping, producing shortages or even blackouts of CV techniques. The most challenging situations, in terms of continuity of connectivity in cellular networks, are the handover and roaming. A handover basically means switching across serving cells of the same cellular network, while roaming involves registering in a new Public Land Mobile Network (PLMN).
This paper focuses on the evaluation of the impact of handover and roaming situations on CCAM services such as extended sensors to understand better the severity of the conditions applicable to such applications. Specifically, the CCAM application includes Transmission Control Protocol (TCP) traffic of Message Queuing Telemetry Transport (MQTT) messages for communication with the discovery service and TCP and User Datagram Protocol (UDP) packets for Web Real-Time Communication (WebRTC) sessions of video streams. Note that 3rd Generation Partnership Project (3GPP) Release 18 has identified WebRTC as the suitable protocol for real-time collaborative applications [2].
On top of these technologies, we propose the application of a set of solutions to mitigate the impact on service continuity. First, a MEC discovery and handover service is proposed, which provides the MEC endpoints for the CCAM services based on the geolocation of the vehicle. It also enables the vehicle to get the endpoints for a new serving MEC as it approaches a new area.
Second, a Quality of Service (QoS) adaptation mechanism is proposed, which passively monitors the network KPIs from the WebRTC-generated reports. This mechanism can adapt the traffic demand to track the network performance and boost the start-up time needed for spontaneous data sessions. It not only upgrades or downgrades the employed bitrate but also tunes the Group of Pictures (GoP) size and the framerate to favour the correct inputs to CV-assisted systems.
Third, a WebRTC gateway is proposed to be deployed in the MEC. This gateway performs the WebRTC signalling between the vehicles that want to establish a video streaming session. In addition, when UEs are not allowed to open sockets between them, the gateway is used as an intermediary that forwards the traffic to the peers.
Finally, a dual modem solution is proposed to boost the switch time in a multi-PLMN scenario. This solution has a backup modem connected to another network ready to be used in case problems with the first modem arise. The applications are unaware of the coverage areas or the performance issues in some specific areas. So, in the same way, some traffic signs provide awareness to drivers on road conditions or driving events, networking signals could be provided by Multi-access Edge Computing (MEC) servers or roadside infrastructures to CCAM systems that rely heavily on connectivity. The beacons of the unfavourable conditions in a local ROI would notify the CCAM applications of activating the countermeasures or mitigation actions.
These solutions have been tested and validated in different 5G non-standalone (NSA) and standalone (SA) networks and locations, including commercial and experimental setups. Additionally, several server options are evaluated: experimental MEC, edge-cloud with shortcuts in the network backhaul but far away from the Base Station (BS), and commercial cloud.
From all the issues raised during deployment and testing, an analytical classification has been produced to provide awareness of cellular network limitations to CCAM research and development (R&D) community and to the 5G standardization and industry communities.
The paper is organized as follows. Section II reviews the networking requirements coming from representative use cases compiled from 3GPP documents; the handover and roaming procedures according to the standards; and the literature on solutions using MEC architectures in CCAM applications. Section III introduces specific networking aspects relevant for CCAM application developers when operating services in cellular networks. Section IV describes the solutions proposed in this paper to the extremely challenging mobility conditions targeted in the paper. Section V details the different experimentation setups employed for the tests. Section VI provides impact assessment values and quantitative solutions' scores in terms of application continuity. Based on the variety and long test runs done in the different setups, a list of issues and challenges is presented in Section VII. Finally, conclusions are in Section VIII.

II. BACKGROUND AND RELATED WORK
A. VEHICULAR NETWORKS AND USE CASES 3GPP identified a set of Vehicle-to-everything (V2X) scenarios grouped into five categories [3]: Advanced Driving, Vehicles Platooning, Extended Sensors, Remote Driving and Vehicle Quality of Service Support. The service continuity is challenging in these use cases where different 5G technologies are relevant: roaming, multiple Radio Access Technologies (RAT), multi-Subscriber Identity Module (SIM), New Radio (NR) sidelink, mmWave communications, network VOLUME 11, 2023 slicing and Edge Computing [4]. Specifically, in terms of continuity: • Coordination techniques which boost handover and roaming are imperative for the seamless migration of vehicle connectivity to speed up data sessions across different systems and domains.
• Multiple radio technologies are crucial to providing substitute connectivity when cellular radio technology fades out.
• Redundancy provided by multiple SIMs enables traffic to be delivered through two available networks, enforcing communication reliability.
• NR Sidelink allows opportunistic communication with entities in the surrounding area, which is essential for decentralized communication areas where ad-hoc infrastructure is unavailable.
• Terahertz bands for mmWave communications have a limited range but are suitable for infrastructure checkpoints when vehicles accumulate data upload/download traffic packets.
• Provisioning virtual/logical network assets devoted to specific traffic isolates the performance from concurrent data flows, preventing the traffic from potential bottlenecks or transitory outages damaging continuity.
• Edge computing resources need a seamless MEC handover mechanism to migrate vehicle sessions on mobility. The session data must be transferred to the following MEC node as the devices physically navigate along them, including the MEC nodes located in different PLMNs. The main aspects to be considered by each of the previously described 5G features in terms of service continuity are summarised in Table 1.

B. HANDOVER PROCESS IN CROSS-BORDER SCENARIOS
Handover is the process carried out in mobile communications when an ongoing connection is transferred from one network or base station to another. This process is usually triggered when the link quality is not good enough at the source base station and is better at the destination base station. This mechanism guarantees the mobile service when it moves out of the coverage area of the source base station. The handover procedure for 5G networks is described in [11].
In the case that the handover process involves crossing an operator's boundary, the process carried out is called inter-PLMN handover. This process happens when crossing a country boundary. This means that the source (home network) and destination (visited network) networks have a different Mobile Country Code-Mobile Network Code (MCC/MNC). Therefore, when a UE crosses to another country, the neighbouring Mobile Network Operator (MNO) with its MCC/MNC will serve the UE. So, inter-PLMN handover can be considered as a specific roaming scenario.
5G communication systems can enable inter-PLMN handover using the N14 interface. This interface will connect the home network's Access and Mobility Function (AMF) to the visited network's AMF to minimize the interruption time and provide session continuity. However, despite the required inter-PLMN support in 5G networks, the current commercial 5G networks do not support the inter-PLMN handover using the N14 interface. The current 5G Core procedure specifications [11] do not support seamless handover with N14 interface. In this case, the PDU session must be re-established, potentially causing service interruption.
Thus, the process currently carried out by commercial networks when performing inter-PLMN handover, called roaming, involves the UE keeping connected to its home network until it completely loses connectivity to the home network. Then, the UE scans available networks and connects to the preferred network in the visiting country, according to contractual agreements with the home network provider.
From a user experience point of view, there are two types of handover, the so-called hard-handover or breakbefore-make and the soft-handover or make-before-break. The hard-handover means that the communication between the source BS and the UE is interrupted before the connection to the destination BS is established. The UE is therefore momentarily disconnected from the network, resulting in packet loss. In contrast, a soft-handover means the connections to the source and destination BS coexist during the transition, so the connection of the UE with the network is uninterrupted along the migration avoiding packet loss.
The handovers carried out in current 5G and 4G networks are of the hard-handover type, which causes an interruption in the range of a few tens of milliseconds in the communications [12], [13]. This interruption time will further increase in inter-PLMN handovers due to increased latency caused during the handover between MNOs [14]. In future implementations of 5G networks, the standard considers keeping the UE connected using Session and Service Continuity (SSC) mode 3, which follows a make-before-break paradigm in which a new data session is set up before the old one is released. However, even in this advanced SSC mode 3, not provided by any network or vendor yet, the change of IP address forces the applications to rerun the handshake for ongoing sessions as the sockets involve IP address pairs. So, considering that the execution of a transparent and low-latency inter-PLMN handover is one of the most challenging situations for automotive applications [15], some actions must be taken to reduce communication interruption during the inter-PLMN handover process used in today's 5G networks.

C. MEC ARCHITECTURES
European Telecommunications Standards Institute (ETSI) envisions the MEC technology as a core component that will be integrated into future 5G systems [16]. MEC infrastructures favour the synergy of services and networks to boost traffic and enhance applications to specific users or groups. It opens networking infrastructures to host virtualized microservices of third parties. Publishing radio reports of monitored link status, microservices can get adapted on top of a summarised view of the cell's traffic activity and performance [17]. Furthermore, MEC brings proximity to the users, boosting latency, and it supports a distributed/capillary processing, enforcing privacy by limiting the data range to local infrastructures [18].
The MEC technology plays a core role in CCAM applications where multi-party sessions are performed. Here, several functions are needed to be provided: • Selection. As the MEC is mainly used by users on the move, it is important to declare the coverage area of each MEC server. It enables the user to ask for the serving infrastructure depending on its geo-location to use the closest available server. The service handover between servers can be done by the user polling for the suitable serving system in its new position or by the networked MEC infrastructures if a transparent continuity is needed [19].
• Subscription. Users need to enrol and register in a list to provide presence awareness, available published sensors and their metadata [20]. This local inventory is needed to be indexed and updated by a trusted system which also acts as the main entry point for data producers and consumers.
• Browsing. This geo-binned index can be exploited by a discovery service to accelerate the selection of candidates in a ROI, discarding irrelevant data sources [21].
• Meeting. Participants need a trusted system that supports the handshake and negotiation. From that moment on, the participants can communicate with each other without any intermediary unless support for monitoring, renegotiation or closing the bridge is required.
• Monitoring. The fusion of network performance reports with application-level information can help services to adapt individual flows in a more coordinated manner, including the ongoing service flows and the background traffic.
• Accelerator. In the context of video sensors for CVassisted systems, the mechanisms to adapt traffic to the network performance or cache media for multicasting data streams can help to make more efficient use of network assets and to enforce service continuity.
Scientific works using MEC for vehicular applications are mainly focused on the theoretical and mathematical analysis of the allocation of resources to offload heavy processing tasks over a pool of computing infrastructures to boost latency and reliability [22], [23], [24], [25], [26], and save energy and costs [27]. More practical ones explore the actual limits of these architectures, studying the limits on reducing the latency by the MEC services when compared to cloud infrastructures [28], analysing the interoperability when using multi-Radio Access Technologies (RAT) to grant universal access [29], and reviewing the versatility of MEC infrastructures when using virtualization and containerization technologies to automate the management of MEC services' lifecycle [30], [31]. However, the literature mainly targets services for individual vehicles using a collective view of the data [32]. In this paper, we focus on MEC as a service to connect vehicles and Road Side Units (RSUs) not as connectivity gateways [33] but to share information to external onboard systems favouring a distributed cooperation.
Another aspect to consider about MEC in CCAM applications is user mobility. MECs must support application mobility as a consequence of user mobility. Therefore, dynamic and transparent management of MECs must be performed to mitigate potential latency and loss of reliability due to MEC switching [19], [34], [35], [36].
Nowadays, canonical MEC, deploying infrastructures located at each Base Station (BS), has not been commercially deployed yet by MNOs, which are conservative to open network infrastructures to host microservices from third parties. In the meantime, some edge-cloud infrastructures such as MobilEdgeX, recently acquired by Google, or Amazon Web Services (AWS) for the Edge are exploring the niche that the MNOs are not filling. VOLUME 11, 2023

III. KEY FACTORS FOR MOBILITY CHALLENGES
In order to use 5G networks and MEC infrastructures for a multi-party CCAM application, when mobility events apply, there is a set of factors that come into play that are summarised in Table 2. Under the network domain category, we find three different factors to consider: • As the different participants could be subscribed to different networks having their own network domains and functions, such as Network Address Translation (NATs), firewalls, etc., getting public IPs and ports is essential to allow participants to reach each other. Furthermore, as the participants perform inter-PLMN migration, their IP addresses will change as soon as the connectivity is back.
• In the context of spontaneous sessions with surrounding vehicles, the distributed systems need to be supported by a common system. In this case, the MEC can act as a trusted and known infrastructure to bridge participants. However, the MEC provided by MNOs usually has many connectivity restrictions disabling access from the Internet or from UEs subscribed to other MNO. This imposes some impassable barriers. In this case, some alternatives such as cloudlets or edge-clouds can be used where the subscriber restrictions are removed, allowing participants to connect independently on their carrier provider.
• Even when participants belong to the same carrier and they are in the same cell, some MNOs apply policies to block sockets opened between UEs. This means cellular network only allows connection from UEs to the MEC or to internet servers. This situation forces the utilization of a Gateway at the MEC, turning the signalling or handshake service into a Gateway receiving and delivering all the messages to be exchanged across participants. The main challenge to consider in the network stack category is the possibility that each participant could use a different addressing version, where some participants can have IPv4 addresses and other IPv6 ones. This means that any connection between the endpoints needs to deal with the potential problems raised when unicast sockets are opened between two different systems.
In the network availability category, the main issues to manage are related to: • A known inventory service in the cloud needs to provide users with the serving MEC infrastructure, acting as a Level 7 (application) Domain Name System (DNS) which provides the server endpoint (IP address and port) to use.
• The service hosted in the MEC needs to register all the served users to connect them when applicable, working as a trusted intermediary. The MEC service should provide a mechanism to quickly provide participants in a specific ROI discarding surrounding irrelevant vehicles.
To this end, the MEC service needs to track updated geo-locations of vehicles. This also means dealing with credentials, tokens or any security handshake needed.
• Monitoring applications for debugging or reporting usually send through internet the captured metrics. They are not usually ready to lose internet connectivity, unable to flush unsent data and resume monitoring when connectivity is restored.
• Applications need to deal with the make-before-break situations or mitigate its impact on the application continuity as long as neither the network interface in the Operating System (OS) nor the network is performing seamless connectivity restoration when the connectivity blackout comes.
• Linux OS systems widely employed in the automotive, industrial and IoT markets apply uncontrolled penalties to network interfaces as far as the connectivity is lost and prioritize the one to be used based on the signal strength. This introduces some uncertainty as the OS applies its own rules to automatically use one network interface or another when routing packets based on non-real-time monitored metrics.
• MEC platforms providing hosting infrastructures for third parties are not co-located with the BSs, ideally being as closer to the user as possible. Thus, they are not local making it hard to provide a real-time view of the radio stats. Instead, MEC infrastructures are closer to the Evolved Packet Core (EPC) for NSA setups and the 5G Core (5GC) for SA networks, in the backhaul or even the backbone of the Core Network. This provides MNO's MECs lower latency compared to cloud services, but they are far away from the KPIs expectations released by stakeholders and industrial communities [37]. Furthermore, this distant and centralized architecture simplifies the need to operate sophisticated techniques for multi-MEC collaboration as the cardinality of MEC infrastructures is really low and serves a wide area.

IV. PROPOSED SOLUTIONS
The use case that this paper is targeting falls under the Extended Sensors category according to 3GPP taxonomy [3].
In the use case, two vehicles driving in the same way want to do the same overtaking manoeuvre, but the one in the rear loses visibility of what is ahead because the front vehicle blocks its FoV. The front vehicle, equipped with cameras capturing a 360 • view around the vehicle, is able to provide a wide view of the whole environment to the rear vehicle. The use case is depicted in Figure 1. Several industry technologies and standards are widely used or recommended in this context: • MQTT is an OASIS standard messaging protocol for the Internet of Things (IoT) valid for exchanging JavaScript Object Notation (JSON) formatted information on top of TCP or UDP connections and including credentials to protect the access. It also includes QoS modes that generate receipts for sent messages when they are received or lighter and faster modes where no acknowledgement is produced.
• WebRTC is a complete stack specification for video, audio and data streaming in real-time with low latency. It includes several standards such as Datagram Transport Layer Security (DTLS) for security, including encryption and credentials handshake, Session Traversal Utilities for NAT (STUN) to get public IP addresses and ports; Interactive Connectivity Establishment (ICE) protocol to negotiate endpoints across NATs; Real-time Transport Control Protocol (RTCP) for monitoring network performance stats; Traversal Using Relays around NAT (TURN) for gatewaying all the packets through a server bridging the peers when a direct socket connection is blocked; and WebSockets for secured data sending in real-time. Furthermore, WebRTC has been proposed in the 3GPP Release 18 as the core technology for real-time communications [2]. On top of these technologies and protocols, several systems come into play to establish a video streaming session between the vehicles providing the view of the leading car to the rear car: • Discovery service at the cloud as part of the Extended Sensors service to forward the service client to the service delegate running in a serving MEC. This includes a database with the list of MEC infrastructures and their serving area. The interface is MQTT receiving messages, including the geo-location and answering the services' endpoints running in the closest MEC.
• Edge Dynamic Map (EDM) service at a MEC, compiles and fuses the received Local Dynamic Maps (LDMs) information to produce a wider and deeper environment description. The interface is MQTT receiving: geolocation updates and LDM data of each vehicle registered in the service ready to consume or produce an external sensor; and requests to get a peer in the ROI to establish a video streaming session.
• Video service at the MEC to allow peers to perform the signalling to start or stop a video session. Furthermore, when peers are not able to communicate with each other due to MNO networking policies, it provides a gateway to deliver the video streams between the producer and the consumer.
• CCAM application in the vehicle can identify an upcoming FoV blockage, trigger the request for an extended sensor peer and perform the video stream session. In the case of the producer role, the same application is listening for any consumer request to fire the sensor sharing through a video stream. The Figure 2 provides a sequence diagram summarizing the communication performed between the involved systems.
As we have explained in previous sections, continuity in these multi-party CCAM applications is a critical aspect to consider, especially in inter-PLMN handover situations raised on cross-border roads. To address the challenges for service continuity, four different solutions are identified, designed and implemented: • S1: MEC discovery and handover. This includes a service that, based on the geo-location of the vehicle, provides the suitable serving MEC endpoints for the EDM and the video services. This is performed on top of MQTT messages enabling the vehicle to poll for a new serving MEC as it moves to a new area.
• S2: Adaptive Bitrate. It is evident that bitrate adaptation mechanisms are essential in the field of MEC services for video streaming-based applications [38]. In our case, an active monitoring of RTCP reports sets a three-level based configuration of the video encoder, enabling live adaptation to the network conditions [39]. As depicted in Figure 3, the bandwidth, jitter and round-trip-time (RTT) statistics are monitored to upgrade or downgrade the bitrate, resolution and frame rate. Two additional policies are applied. First, any change in the frame rate also alters the Group of Pictures (GoP) size to keep the consumer's capacity to process the received video stream quickly. Second, the session starts from the lower bitrate, resolution and frame rate to foster prompter startup time. This is especially useful in the situations under VOLUME 11, 2023 study in this paper, related to handover and roaming situations where connectivity is lost, continuity is broken and the recovering time elapsed from the last received frame to the moment a new frame comes from a new streaming session is crucial.
• S3: WebRTC Gateway. This is a lightweight server performing the WebRTC signaling between the peers connecting them. It is deployed at the MEC to favour real-time and low-latency communication. This server provides two communication modes between peers. First, as a WebRTC server, it performs the signalling, including the security (DTLS) and communication stack handshake (ICE), to use IPv4 or IPv6 UDP endpoints. Then, when limitations blocking sockets between the UEs are detected, making peers not allowed to open sockets between them, the server is used as an intermediary for all the data. This means the peers open sockets to the server, which forwards the traffic to the peers, acting as a gateway for the actual media streams sent from the data producer to the consumer destination. When limitations do not take place, the gateway is not in the middle, and the data producer peer opens sockets to the data consumer directly.
• S4: Dual modem. A mechanism to improve the reliability of the network is based on multi-modem devices with multi-sim docks able to connect to different networks simultaneously and boost the inter-PLMN handover. The goal is to perform the subscription in the network providing better power or coverage to have the backup modem and network ready to use as soon as possible while the current network connection is fading.
Here, we have explored three different triggers that can be used to fire the modem/network switch. First, hlthe vehicle triggers the connectivity switch based on the geo-position. Second, a daemon could actively monitor the signal strength/coverage of the network to trigger a switch from the one fading to a powerful one. Third, in a similar way some traffic signs alert the driver to be careful in adverse weather conditions, road managers could beacon through C-V2X PC5/sidelink areas with bad network coverage. This way, onboard CCAM systems would know that the network is not reliable there. These solutions try to provide some time for applications to support make-before-break mechanisms, as the cellular networks are not currently providing any solution for this. RTCP monitors network statistics providing passive reports about transmitted media packets. This way, applications can adapt the intended resolution, framerate and bitrate to be sent to the effective network performance. Different adaptive reactions are depicted in Figure 4, where a) and b) imply a gateway at the MEC to bridge peers which cannot directly communicate and c) means no intermediaries. There, a) means detecting an underperforming path from the producer to the gateway, while b) implies the gateway actively forwarding to the producer any detected issue in the path from the gateway to the consumer. This is done in this way to enable video encoder at the producer to adapt to any issue in the end-to-end path. Otherwise, the gateway should include transcoding abilities pushing heavy processing tasks to the MEC and making it more difficult to scale.

V. EXPERIMENTATION SETUPS
In order to evaluate the proposed solutions, several setups have been used that include multi-PLMN environments created by two networks with overlapping coverage. Both experimental and commercial 5G networks have been taken into account in these setups.
The tests have been carried out on top of four scenarios: i) scenario combining a commercial 5G NSA network with a commercial 5G NSA network both with outdoor coverage and in a highway environment with high traffic intensity, ii) scenario combining a commercial 5G NSA network with a commercial 5G NSA network both with outdoor coverage and in an urban environment with high traffic intensity, iii) scenario that combines a commercial 5G NSA network with an experimental 5G SA network both with outdoor coverage and in an urban environment with low traffic intensity and iv) scenario that combines an experimental 5G NSA network with an experimental 5G SA network both with indoor coverage. The first scenario (i) is settled on one of the two cross-border corridors defined in the 5G-MOBIX project. It covers the way between the city of Vigo (Spain) and the city of Porto (Portugal). For the Spanish side, the operator Telefónica provides 5G coverage through a commercial NSA network and for the Portuguese side, the operator NOS provides 5G coverage through a second commercial NSA network.
The border between Spain and Portugal in the corridor area is naturally divided by the Minho river and linked by several cross-border bridges. Specifically, the tests have been performed on the New Bridge E01, which is a highway environment with a high concentration of vehicles. It connects the cities of Tui and Valença on the Spanish and Portuguese sides, respectively. Figure 5 shows the test area for this scenario and the coverage area of the used 5G networks.
In this scenario, the MEC infrastructures provided by Telefónica and NOS operators have been used. They are located at Vigo (Spain) and at Riba d'Ave Braga (Portugal), respectively. These MEC servers are only accessible from the 5G network deployed in the described cross-border corridor. Finally, in this scenario, a custom trigger provided by a PC5/sidelink-compatible RSU has been used to switch to the backup modem connected to the second 5G network.

FIGURE 5. Test area of scenario (i).
The second scenario (ii) is seated in one of the trial sites operated in the 5G-MOBIX project. Specifically, the German trial site is located in the city centre of Berlin. This trial site covers an extension of 4 km (from Ernst-Reuter-Platz to Brandenburger Gate) in an urban area with high traffic density during working hours.
As in the first scenario, two commercial networks providing 5G NSA deployments have been used for the tests. These networks are operated by Deutsche Telekom and O2. Figure 6 shows the test area for this scenario. In this case, both network operators offer coverage throughout the test area.
In this scenario, the MEC infrastructures are provided by MobiledgeX, recently acquired by Google, and by the Technical University of Berlin (TUB). They are located at the latitude and longitude positions (52.49655, 13.35653) and (52.51286, 13.32002) respectively. Unlike the ones employed in scenario (i), these MEC servers are publicly accessible by any network. A geo-trigger has been used in this case to make the change between the modems.
The third scenario (iii) covers a smaller coverage area due to the use of an experimental 5G SA network deployed by us using an AMARI Callbox Mini provided by Amarisoft. Specifically, this network has been deployed on the roof of one of the buildings that Vicomtech has in San Sebastián (Spain). This experimental network has a coverage of about 100 m radius and serves an urban area with low traffic density. In addition to this experimental network, a commercial 5G NSA network deployed by the operator Euskaltel has been used. Figure 7 shows the test area for this scenario and the coverage area of the employed 5G networks.
In this scenario, we have used a local MEC connected to the BS for the experimental 5G outdoor network and a server at Amazon Web Services (AWS) infrastructure for Euskaltel's network, which is deployed on Amazon's servers in Paris (France), as this network lacks a MEC infrastructure. In this case, the servers employed for the MEC services are accessible from any network. The trigger to make the change between the modems is based on signal strength thresholds. Thus, the RSSI value has been used in this case. Finally, several tests have been done in a fully-controlled laboratory environment combining the deployment of two experimental 5G networks, one SA and another NSA, building the last scenario (iv). For this purpose, the two experimental 5G networks have been deployed using Amarisoft systems. One of the experimental 5G networks deployed uses the AMARI Callbox Mini device, which, despite offering a reduced coverage of around tens of meters, is a very compact solution that allows the deployment of a 5G SA core and base station. On the other hand, an experimental 5G NSA network has been deployed with a high-power semi-professional base station. In this case, the deployment has been carried out using several high-power Remote Radio Heads (RRH), which offer greater coverage and performance, along with a server on which the 5G stack software provided by Amarisoft is deployed.
In this last scenario, we have used both a local MEC deployed at Vicomtech and a server deployed at AWS, again deployed on Amazon's server in Paris (France). In this case, the servers employed for the MEC services are accessible from any network. In order to do the change between the 5G network, a trigger based on the RSSI of the received signal has been employed again in this case.
It should be underlined that since we can only physically access one of the MEC infrastructures in the scenario (iv), the synchronization of the clocks of the participant systems has been carried out through Network Time Protocol (NTP). In the case of the transmitter and receiver vehicles, synchronization has been carried out via Global Positioning System (GPS). Table 3 summarizes the main parameters of the network deployed in the different scenarios. The same CCAM use case was executed in all scenarios. This use case was previously described in Section IV and involves two cars driving one after the other. The front car transmits a video stream to the rear car. In order to establish the video streaming session, the consumer car queries the EDM server located in the MEC, edge-cloud or cloud, to discover the video producer vehicle. The sequence is described with more detail in Figure 2. In scenarios (i) and (iii), the producer vehicle was a Toyota Prius equipped with four Full HD cameras, which is used as CCAM test vehicle by Vicomtech. In scenario (ii), a sensorised Volkswagen Tiguan owned by the Technical University of Berlin was used as the video producer. In all cases, the video consumer was a regular car, equipped with a laptop and 5G connectivity. 5G modems have been used as UEs to evaluate the proposed solution in each scenario. The modems used, Quectel RM500QAE, Quectel RM500Q M2 and Quectel RG500QEA, are connected to Linux computers running the CCAM application.
In all the scenarios, exactly the same EDM and WEbRTC services are executed as the MEC, edge-cloud and cloud infrastructures allow the deployment of Docker containers. In all the scenarios, a video stream composed of four 1920 × 1080 camera images is transmitted from one vehicle to the other.
The MEC setup applicable in each scenario is as follows: i) The WebRTC Gateway and the MQTT Broker were deployed in the MEC at Telefonica and NOS. The measurements were captured in real driving scenarios crossing the border between Spain and Portugal. ii) The WebRTC Gateway and the MQTT Broker were deployed using MobiledgeX Edge-Cloud services in Berlin. Data was captured in real drive tests in the road depicted in Figure 6. iii) In this case, the data has been divided into two different setups. In our local Amarisoft network, the WebRTC Gateway and the MQTT Broker were deployed in the MEC in our local 5G SA network. As a MEC is not available in the 5G NSA Euskaltel network, we used Amazon Web Services (AWS) Cloud infrastructure instead to deploy the WebRTC Gateway and the MQTT Broker and test its performance in real driving scenarios. iv) The WebRTC Gateway and the MQTT Broker were deployed in our local MEC. The full control of the network in this setup enables the execution of quick roaming between two PLMNs when using a single modem. The driving conditions, in this case, are indoor and static.
In the scenarios (ii), (iii) and (iv) the gateway feature from the WebRTC service is not mandatory. Specifically, in the scenario (ii), the availability of an Access Point Name (APN) providing public IPs enables the tests without the gateway. Furthermore, the connection of sockets between UEs in the network is not blocked in the scenarios (ii), (iii) and (iv). This means that except from scenario (i), in the rest of the scenarios, the WebRTC server can support the peer handshake and then there is no need to gateway all the data flows through the server.

A. METRICS
The following metrics have been considered in the measurements taken during the tests: reliability, network latency or level 3 (L3) latency according to OSI Model, application latency or level 7 (L7) latency according to OSI Model, RTT, and application/service interruption time. To obtain these measurements, some custom and proprietary logging tools have been used, some embedded and others external to the applications under test. Moreover, both the sender and receiver equipment clocks are synchronized by GPS.

1) VIDEO STREAMING
Different metrics are captured when performing spontaneous video sessions between two vehicles. Here, the leading car, in the ROI of the following car, produces a video stream consumed by the following car to extend the range of onboard sensors. Those metrics are the packet reliability, showing the percentage of packets received, the latency of the cellular network (L3), the end-to-end application latency (L7) and the interruption time when inter-PLMN situations come into play in the different scenarios compiled in Table 3. For the packet reliability and the L3 latency, the logs provided by the WebRTC stats [40] and the RTCP reports are used. However, the recommended report interval is 5 seconds, meaning laggy visibility of quick network performance variability [41]. So, we have reduced the reporting rate to 1 second to avoid this situation. In terms of the L7 latency, the synchronization of all the involved systems from the GPS signal establishes a common clock. The producer watermarks each video frame with the system clock, and the consumer retrieves the transmitter clock when the video frame is received. The drift of the received timestamp with the local clock is the end-toend application latency (L7). For the interruption time, the timestamped logs trace the elapsed time from the moment the stream is not received by the consumer to the instant the stream is back and resumed in the onboard systems.
In all the scenarios described in Table 3, the proposed adaptive bitrate mechanism has been executed to check its benefits to accommodate the traffic demand to the actual network performance. It also speeds up the WebRTC startup when the communication is broken in an inter-PLMN switch.

2) MQTT
A test topic has been implemented in the MQTT Brokers deployed at the MEC infrastructures. When a standardized Cooperative Awareness Message (CAM) [42] is sent by a vehicle, the MEC instantly replies to it. In order to test the performance of the MQTT messaging, we have measured the RTT from the vehicle to the MEC Broker for the four scenarios summarized in Table 3.

1) VIDEO STREAMING
First, the tests assess the impact on the reliability when the proposed solution of adaptive video bitrate (ABR) is enabled in contrast to constant video bitrate (CBR) in all scenarios.
As we can see in Table 4 and Figure 8, reliability is improved when using ABR in all scenarios. ABR gets less packet loss rate under the unstable bandwidth conditions of commercial networks where available bandwidth depends on multiple factors, such as the number of clients connected, bandwidth used by clients or shadowing and reflection effects. It is also evident that ABR brings higher benefits as the reliability gets low, where the speed and density of vehicles have a significant impact. The more dynamic demand and resulting reliability when ABR is enabled are evident  in the standard deviation and more visually represented in Figure 8.
The second comparison focuses on measuring the impact in L3 latency depending on the type of deployment for the MEC. An Edge-Cloud server deployed in MobiledgeX has been used to host the WebRTC gateway in scenario (ii). While, in the scenarios (iii) and (iv), both a MEC server and a Cloud infrastructure have been employed. The results obtained are consistent with what was expected. The lowest latencies are obtained with a MEC server and the highest ones with a Cloud server. The Edge-Cloud performance is in between the MEC and the Cloud (see Table 5). Beyond the good scores on average values when a MEC is employed, it is clear that the deviation is much lower, providing a steady connection performance. These measurements were obtained with the WebRTC Gateway installed in the MEC, Edge-Cloud or Cloud virtualization platforms.
The third test targets the impact of the intermediary gateway when this feature is performed from the WebRTC server. As the intermediary WebRTC server operates the gateway feature, it performs not only the signalling but also forwards the data streams to the peers, adding L7 latencies. As expected, L7 latency is higher when WebRTC gateway is used (see Table 6 and Figure 9). However, it is important to underline that the impact is high, doubling the end-to-end latency when the intermediary gateway is employed. This penalty is present and similar when the WebRTC server is deployed in a MEC or in a cloud. Furthermore, when the gateway feature of the WebRTC server is not required, the video streams would not pass through the servers. So, Table 6 and Figure 9 show aggregated values for the different server types.   Once the different setups have been characterized in terms of QoS impact for CCAM applications, some tests have been performed to measure the impact of the proposed solutions on service interruption time.
The tests to measure the benefits of the dual modem on the service interruption are shown in Figure 10 and Table 7. Scenario (iv) is the only one where roaming can be operated, performing an inter-PLMN switch with a single modem. The service interruption time is then compared to the dual modem solution. The 5G network where the modem is connected is instantly powered off to trigger the roaming event in scenario (iv). This situation does not happen in a real scenario where the signal strength of the connected network decreases gradually. However, in a highway or in a cross-border context, the connectivity changes abruptly according to our tests. Both Figure 10 and Table 7 show a slightly enhanced interruption time when dual modem is employed, especially with a more steady performance.
When the WebRTC gateway solution for video streaming is used, it is expected to increase service interruption time.    As we can see in Figure 11 and Table 8, this penalty affects all the scenarios. In the scenario (ii), we can see that the service interruption time is also very high when the WebRTC gateway is used. In the rest of the scenarios, the longer interruption time when the gateway is employed is more evident. This is mainly related to the worse performance of commercial and open cellular networks compared to experimental ones.
Then, the impact of the ABR mechanism on the service interruption time is observed. We can see in Figure 12 and Table 9 that the interruption time is a bit lower using adaptive video bitrate in scenarios (iii) and (iv). On the other hand, it is bigger in scenario (ii). Here, the bandwidth was   highly unstable in the place where the dual modem switch was performed. The network variability was higher than the one-second metric monitoring frequency, causing erratic adaptation decisions.
Finally, the interruption time also increases as the distance to the server gets longer, as shown in Figure 13 and Table 10. In this case, the MEC server means lower latency between participants and less interruption time.
Thus, Tables 10, 7 and Figures 13, 10 conclude that the setups including MEC servers and the dual modem solution provide less interruption time.

2) MQTT
In contrast to Real-Time Protocol (RTP) based video streams, MQTT messages are transactional, not requiring any continuity, buffering or order on the individual messages. Furthermore, the small message size of these messages fits into the payload of a single TCP/UDP packet.  Accordingly, the tests performed for MQTT messaging are focused on the RTT from the client to the EDM service. As previously explained, the RTT from the client to the MQTT broker has been captured for each of the presented scenarios in Table 3. More details of the performed tests can be found in Table 11 and Figure 14. According to the results, the MEC scenarios (i) and (iv) provide significant lower RTT values. In the case of the MEC scenario (iii), it brings low median and high mean RTT values due to the outliers captured in the tests. This is evident in the high standard deviation value. Those results can be easily justified, as the setup was an experimental outdoor Amarisoft network locally deployed by us, acting as an edge setup and not a real commercial edge service. For the MEC results in scenarios (i) and (iv), we can see that we obtain similar results in both mean and median values. The standard deviation is lower in the experimental testbed as the setup brings deeper control, while in the first scenario, the tests were recorded in a highway environment with dense traffic. Concerning the Cloud scenarios, the scenario (iii) performed under real driving conditions using a commercial network and cloud brings the worst results in terms of latency but the best for network stability according to its standard deviation. When using a commercial cloud in the scenario (iv), shipping a local experimental setup, the latency values are better in comparison to scenario (iii). So, the 5G network performance has a big impact on the latency. In the case of the Edge-Cloud scenario, if we compare the obtained results with the commercial cloud setups, the obtained results are better. However, MEC scenario stands out in terms of latency. So, the obtained results corresponds with the expectations, where the MEC setup gets the best results followed by the Edge-Cloud and the Cloud infrastructures.

C. DISCUSSION
It becomes evident that KPIs provided by the commercial 5G NSA networks are far away from the expectations to interconnect future CCAM services across vehicles. Furthermore, the metrics are unsteady and highly dynamic. The experimental 5G SA networks and testbeds are closer to the industry requirements, but they still need some updates and tuning to achieve the challenging KPIs of applications such as extended sensors.
The MEC is a key element to get enhanced performance. However, it brings some limitations in return, making more complex the connection of services involving several vehicles from different mobile operators. Moreover, the mobile network operator often decides to deploy the edge infrastructure in the backhaul, close to the network Core, instead of co-located in each base station, losing the ideal latency performance and providing a performance similar to a regular cloud infrastructure.
It is also evident that the service continuity, latencies and reliability of CCAM services in cross-border environments, meaning multi-PLMN setups, bring extreme network conditions. CCAM applications need to include workarounds and solutions to reduce the high values of service interruption time, which are incompatible with safety-critical systems.
The solutions proposed in section IV are essential enablers in inter-PLMN handover situations such as S3: WebRTC Gateway in the scenario (i), while others can improve the service continuity in the other scenarios such as S2: Adaptive Bitrate and S4: Dual modem. The major benefit of S2: Adaptive Bitrate is the reduction of packet loss.
It is important to underline that the inter-PLMN testing with a single modem is limited to experimental setups as it requires specific configurations in the present networks. However, the single modem case brings a very high interruption time, even higher than the one provided by the S4: Dual modem.
The different networks and setups bring their own peculiarities, making CCAM application developers struggle with different issues and limitations. The more commercial the testbed, the more restrictive, unsteady and difficult to log. On the other hand, the more experimental the testbed, the more configurable, sparse and traceable. Thus, the scenario (ii) brought a more predictable context for the inter-PLMN handover when compared to scenario (i). The scenario (iii) brings a complete MEC infrastructure and SA performance in comparison with scenario (ii). And the scenario (iv) enables full control on inter-PLMN switching, also for single modem, not possible to be performed in any of the other scenarios.
As the main drivers of networks investment from MNOs are the social networks, the web-browsing and the consumption of on-demand video content from Over-The-Top (OTT) services, the deployed cellular networks bring added values limited to the nominal radio KPIs with a prominent asymmetric configuration without any dynamic management or smart operation of assets. The 5G networks and equipments have not yet implemented the full potential of 5G in terms of network slicing and self-organized networks to tune network assets to heterogeneous requirements from concurrent traffic. Furthermore, the specification of 3GPP's SSC mode 3 to accomplish the make-before-break paradigm in the 5G Core, providing a seamless migration across PLMNs, is not yet implemented or provided by any network [11]. The limits and barriers to play a multi-party CCAM application become clear from the set of issues faced across different commercial and experimental 5G networks. Specially, the network problems for multi-party sessions go beyond the connectivity provision for transactional communications between clients and servers in the cloud or the edge, with a wide set of issues hurdling peer communications. VOLUME 11, 2023

VII. LESSONS LEARNED
This section provides a summary of all the issues raised during the testing and experimentation in the different testbeds. Some of them apply to any scenario on top of cellular networks, while others manifest in inter-PLMN environments meaning extreme connectivity conditions that the CCAM applications need to manage to mitigate the impact. Table 12 overviews the issues arranged around different categories underlining the impact and proposing workarounds to be considered and implemented in CCAM applications.
All these issues raised and faced during the different experiments make us consider an experimental setup where we have full control and visibility of the network logs and the loaded configuration, providing a managed and controlled environment (scenario iv). It allowed us to tune and iterate on the solutions to try to accomplish the solutions for makebefore-break connectivity situations required by the targeted CCAM applications.
It is also evident that some technology gaps in the cellular networks are not in the hands of CCAM developers. So, CCAM developers need to implement some workarounds to avoid or mitigate the negative impact of these challenging situations.

VIII. CONCLUSION
The connectivity of the vehicles will open a new spectrum of innovative CCAM applications. New generations of cellular networks promise unseen performance and capacity to enable those applications. However, the 5G networks have not yet addressed the unsteady or even the outage of connectivity on mobility, especially in handover and inter-PLMN situations. The impact of these scenarios is heavier in CCAM applications where multi-party sessions are established, degrading QoS and breaking continuity.
CCAM application developers need to take care of network issues as the 5G networks are not targeting their needs. Additionally, the performed roaming completely breaks the CCAM sessions when considering mobility across borders.
We have proposed several solutions and workarounds that the CCAM application developers should consider to mitigate or avoid some of the problems raised. We have tested the solutions and the impact on continuity in four different setups. From the experimentation, it can be concluded that: 1) the dual modem solution stands out in keeping service continuity in mobility scenarios, 2) the adaptive bitrate mechanism mainly improves reliability reducing packet loss, 3) the WebRTC gateway should be avoided when possible by using an APN that provides public IPs, and 4) MEC infrastructures achieve lower latency and interruption time than edge-cloud or cloud infrastructures, however, for some applications, the performance of edge-cloud or cloud could be enough.