A Comprehensive Survey on Resource Allocation Strategies in Fog/Cloud Environments

The growing number of connected objects has allowed the development of new applications in different areas. In addition, the technologies that support these applications, such as cloud and fog computing, face challenges in providing the necessary resources to process information for different applications due to the highly dynamic nature of these networks and the many heterogeneous devices involved. This article reviews the existing literature on one of these challenges: resource allocation in the fog–cloud continuum, including approaches that consider different strategies and network characteristics. We also discuss the factors influencing resource allocation decisions, such as energy consumption, latency, monetary cost, or network usage. Finally, we identify the open research challenges and highlight potential future directions. This survey article aims to serve as a valuable reference for researchers and practitioners interested in the field of edge computing and resource allocation.


Introduction
According to the United Nations, in its review of global urbanization prospects, by 2018, 55% of the population will reside in urban areas, and by 2050, this percentage will grow to 68% [1]. In addition, Gartner [2] estimated that, by 2020, around twenty billion connected devices would be deployed, from end-user equipment, such as phones, tablets, and computers, to municipal systems for traffic management, climate, and health services [3]. There is also an increasing number of smart city initiatives being promoted to address urbanization challenges, including, for example, the European Commission's Digital Agenda that focuses on the deployment of energy-efficient cities [4], the i-Japan strategy [5], Singapore's Smart Nation 2015 plan [6], and the investment conducted by China and India in the implementation of more than 300 smart cities [7]. These are all clear examples of the interest of nations in not only the implementation but also in researching the deployment of intelligent territories, with a large number of connected objects (IoT) and technologies supporting applications that serve different actors. A key part of the operation of these solutions is the data life cycle, which starts and ends in the layer of interconnected objects. This requires the construction of a flexible architecture that allows data flow processing coming from different sources with different characteristics and particular QoS requirements.
In order to provide suitable solutions to the different requirements of the applications, new computing paradigms have been adopted in these architectures. Previously, sending data to applications located in the cloud using the Internet was a standard scheme. However, new IoT applications and the increasing data generated by users have shown the limitations of this alternative, especially when dealing with delay-sensitive applications [8]. Therefore, deploying computing resources closer to the sources could lead to achieving adequate service levels for every application. In this way, new paradigms such as mobile edge computing (MEC), mobile cloud computing (MCC), edge computing, cloudlets, and fog computing have shown that it is possible to reduce response time and bandwidth consumption, improve privacy and security conditions, facilitate the deployment of location-based services, and alleviate connectivity problems in environments that are hostile to connected objects, compared to systems deployed in the cloud [8].
The scope of each of these paradigms can be seen in Figure 1, where, for example, MEC tries to assign tasks to end devices, something that could be extended not only to mobile phones but also to vehicles or IoT devices. Another common option is exploiting resources on the first hop from the IoT layer in devices such as access points (APs) or access gateways, which generally provide more stable connections and higher computational resources. Some recent alternatives have implemented small servers near the cellular base stations to deploy content delivery networks (CDNs) in order to provide multimedia with lower latency to end users. Finally, there is a growing interest in deploying small data centers closer to the IoT layer, known as cloudlets; several works have included real implementations with commercial software where authors analyze the performance of solutions that perform resource allocation in these environments [9,10]. In this work, we will use the term fog for all the nodes that lay between the IoT layer and the cloud layer (including IoT nodes). Due to the growing interest in developing solutions that include intermediate computation paradigms, this work devotes its attention to analyzing resource allocation problems in architectures that have computational resources distributed between the cloud and connected objects. Different articles [11][12][13][14] denominate these nodes as fog nodes (FNs) and are associated with a hierarchical architecture known as fog computing, which serves specific problems that cannot be solved using only traditional cloud-based solutions [15]. According to the Industrial Internet Consortium [16], these are nodes that have enough computational and network capacity to run advanced services, represented in edge devices such as connected vehicles, surveillance cameras, industrial controllers, switches, access points, cellular network base stations, and specialized edge routers, among others.
Fog computing does not replace the functions of cloud computing but complements them, and both coexist and cooperate with each other. These interactions are of different types since they can be between the same or different layers of the architecture or between different operators [11], which also brings challenges towards the establishment of a federation between both paradigms [12,14,17,18].
Integration between fog and cloud resources presents some challenges [15,18]: • Scalability: The constant growth of geographically distributed devices represents a large amount of data that needs to be processed and analyzed. In this case, when IoT infrastructures exist, the distributed nature of fog computing is better suited than a centralized scheme such as cloud computing. • Interoperability: This aspect has two major challenges. The first is related to a large number of protocols available in a heterogeneous IoT environment. The second is associated with the existence of several fog and cloud computing operators and their interaction. In this case, these computing paradigms are usually based on virtualization techniques deployed on platforms designed to run different applications in a homogeneous way, which can be an advantage when dealing with different applications, information sources, and operators. • Real-time response: Bringing the computational resources that process data closer to the information sources is more crucial to carrying out activities that do not require in-depth analysis or high computing resources thanimproving the response time. Different surveys that address several aspects of fog computing and its integration with IoT and cloud computing have already been published. Surveys about the state of the art and research challenges can be found in [17][18][19][20][21], where several issues about fog computing are discussed, including architectural and security concerns and challenges. However, these articles do not focus on particular topics about fog computing. Instead, they include a review of several aspects related to these kinds of deployments, such as different types of architectures and integration with other paradigms. In [14], the authors considered the integration of fog-related technologies with smart cities; here, different use case scenarios are reviewed. Other initiatives collected alternatives for fog-cloud integration [22]. Further surveys [20] analyzed the implementation of new paradigms such as software-defined networking and network functions virtualization on IoT, and fog computing scenarios [23][24][25].
The resource allocation problem is a common challenge addressed in different fog computing surveys. However, not many of them analyze this issue in depth. In [15], the application placement problem was briefly described, and several options were classified according to the elected optimization metrics and the algorithms used to solve the problems. In this survey, there is no particular interest in analyzing the characteristics of each solution or the complexity of the considered architecture. In [26], the authors also provided an overview of the application placement problem in a fog environment and proposed a classification based on different scenarios and optimization strategies.
The contribution of this paper is twofold: first, we give a general definition of the application placement problem, focusing on the resource allocation aspect. We pay special attention to the resolution strategies and technical formulation of the problem, incorporating recent works that are not added in other surveys. Second, two recent research directions not covered in the reviewed works are included: the possibility of allocating resources in mobile computational nodes (e.g., vehicles) and the consideration of a federation between different fog operators and cloud operators.
This survey is structured as follows: Section 2 explains the methodology used to elaborate this review. Section 3 introduces the resource allocation problem in fog and cloud environments. Section 4 presents a taxonomy of different resource allocation alternatives according to control strategies, optimization objectives, resolution strategies, mobility support, and the inclusion of a federation between fog and cloud. Section 5 includes a discussion about the found research directions and challenges.

Methodology
This review concentrates on surveying different allocation strategies for applications deployed in nodes installed between the IoT and cloud layers, paying specific attention to strategies that assign resources in an online manner over dynamic topologies. Following the PRISMA guidelines, the methodology for building this report consists of three main phases:

Identification
We searched among the following databases: IEEE Xplore, ACM Digital Library, ScienceDirect, MDPI, SpringerLink, Google Scholar, and other engineering journals. We used the following keyword combination: (resource allocation or resource management) AND (fog computing or edge computing). This action delivered 1135 articles. In this phase, we removed 303 duplicated articles. The term "fog computing" was first coined by Cisco in 2012 [27], and the search conducted in the elected databases did not provide results focused on fog computing published earlier than the year 2016; articles before this year were oriented to cloud computing environments, a concept that is closely related to our research, but out of the scope of this survey. In this sense, articles published before 2016 were removed. We also removed articles that were written in languages different from English.

Screening
In this phase, three different researchers read the abstract and conclusions of each record. We excluded articles without experimental results and studies that did not evaluate at least one strategy for resource allocation in fog environments. Finally, we only included extended versions of the articles, removing, for example, conference papers that were further explored in their journal versions. After these actions, we retrieved 356 articles, obtaining 313 full texts to be read. At least two of the three reviewers read each article. Here, the researchers considered two reasons for final exclusions: the first one related to the characteristics of the strategy and the experimental scenario, and the second one associated with the experimental procedures conducted in the research. We considered distributed or centralized strategies that execute online resource allocation over dynamic topologies in works with detailed descriptions of their experimental environment. This is further explained in the following sections. Dissents were resolved by agreement.

Inclusion
A total of 94 articles were classified according to the defined criteria and included in the survey. The methodology is summarized in Figure 2.

Resource Allocation in Fog/Cloud Environments
There are different problems related to the resource allocation problem in fog environments that are considered in the literature. They are usually known as the service or application placement problem [10,15,26]. This problem is quite similar to the problem related to virtual network functions embedding and can be described following the representation in [28].
The terminology used in the following section is better described in Table 1. Let PR = (V, E) be a set of available physical resources where V represents the computational nodes and E the links between those nodes, and let AR i = (V i a , E i a ) be an application in a set of n application requests, where V i a represents the set of functions of the application i, consuming computational capacity, and E i a the flow of information between those functions, that consumes bandwidth. LetṘ = ∏ m j=1 R j be a set of resource vectors, and let cap : V ∪ E →Ṙ be a function that assigns available resources to elements of the physical nodes. Ultimately, let req i : V i a ∪ E i a →Ṙ be a function that assigns requests to elements of all application requests. Then, an application placement consists of two functions f i : V i a → V and g i : PR is a subset of PR that represents a path inside the graph. Finally, we can define the residual physical resources, RR = (V r , E r ), that contain the available resources that are left after placing the set of applications AR i in PR. RR satisfies the following conditions: ∀v / ∈ ( f i : V i a → V) : cap(v) = cap(v r ), v r ∈ RR and ∀e / ∈ g i : E i a → PR ⊆ PR : cap(e) = cap(e r ), e r ∈ RR. This means that the residual resources represent the remaining resources after application requests are placed. If RR has enough resources, then another application can be placed, repeating this process until no physical resources are left. Taking this into account is desirable to maximize the number of applications that are placed. In a more explicit way, the resource allocation problem consists in finding a representation of AR in PR in the presence of the defined constraints, as shown in Figure 3.

Term
Description PR is a set of V nodes and E links that hold the physical resources, representing the computational resources and connectivity between nodes.
AR is the i th application request consisting of V i a functions that take part in the information processing of the application (that consume computational capacity, e.g., RAM memory, CPU, or storage), and E i a represents the flow of the information (in bits per second, for example).
Represents the nodes that hold residual resources after placing the application requests in the available physical resources. R = ∏ m j=1 R jṘ contains the resource vectors for all resources (physical and logical).
The function cap assigns a capacity to an element of the physical resources (node or link).
The function req i assigns an application request to an element of AR i (node or link).
that maps a part of the i th application to a physical node.
function that maps a link of the application request to a path of PR.  Virtual network functions embedding includes the joint placement of virtual services and traffic flows and has been thoroughly examined. Placing applications in the fog differs from VNF embedding since the latter assumes the possibility of programming network flows from a centralized controller. A fog application deployment can span through different providers that may not support centralized network management [15,29].
Some strategies concentrate on the distribution of resources in a single layer of the architecture [30,31]; however, in this work, we address strategies that consider resources in fog and cloud layers, including nodes with different characteristics, from those with low computing capacity up to micro data centers. The resource allocation strategies for the location of an application in an infrastructure can be classified according to the criteria used in [26]: • Centralized (C) or distributed control (Di). • Online (On) or offline resource assignment (Off). • Static (S) or dynamic topology (Dy). • Mobility supported (M) or mobility not supported (nM) (of fog or end nodes).
Strategies that are centralized rely on one entity that holds information about the whole topology. In contrast, distributed strategies consider that the decisions could be made by different controllers that manage groups of physical resources (see Figure 4). Another criterion is to consider that the decisions can be made with prior knowledge of the applications that must be deployed (offline) or the services will be provisioned as soon as they are demanded (online) as shown in Figure 5; the second alternative is far more difficult to implement and will sometimes lead to suboptimal solutions. In addition, the introduction of mobility of fog nodes (e.g., vehicles that act as fog nodes) or end users (e.g., information collected on smartphones) suppose that some physical resources would not always be available, creating a dynamic topology, forcing the controller to reassign the application to a new resource in order to maintain the service (see Figure 6, a representation of a dynamic topology).   In this survey, we concentrate on solutions of type [C|Di]/On/Dy/M and pay particular attention to multiobjective optimization strategies since they constitute a closer representation of a real deployment that includes an IoT/fog/cloud environment, where you can find online requirements of several applications that can be allocated in mobile fog nodes, showing a dynamic topology that can be affected when a node ends up without network coverage.

Taxonomy of Resource Allocation Strategies in Fog/Cloud Environments
This survey classifies the resource allocation strategies according to different criteria. As was stated before, we concentrate on strategies that consider online resource assignments on dynamic topologies. From this, we address different alternatives and analyze how each one of them assigns resources when placing applications.

Control Strategies
One of the first issues to address is the mapping coordination strategies. Two options are the most common along the different alternatives: centralized or distributed (see Figure 4). A centralized mapping requires the concentration of the global information in one point of the architecture and also the communications infrastructure to disseminate the decisions made by this entity. It could lead to finding a globally optimal solution. However, there could be issues related to scalability and computational complexity due to the amount of information that is exchanged. Most of the reviewed alternatives rely on centralized strategies. For example, the authors in [32] propose a unique entity that manages the fog and cloud resources in order to find the optimal distribution of the VM that will execute a part of a distributed application. This centralized entity implements a genetic algorithm in order to solve a multiobjective optimization problem, minimizing cost and latency and maximizing the number of applications deployed.
On the other side, a distributed control strategy allows multiple entities to control service mapping. These entities make decisions based on local information and use computational resources in their vicinity. These kinds of solutions are more flexible and probably more suitable to fog computing environments since they could be resilient to local changes and also can increase the system's scalability. Still, they would probably lack information on the global system. These kinds of alternatives are often used in MEC environments [33][34][35], since particular devices make decisions related to computational tasks offloading in the architecture. As shown in Table 2, most of the reviewed papers implement centralized solutions. Still, in some works [36,37], the fog layer deals with tasks related to real-time processing and intermediate control entities that redirect the applications to the cloud (as shown in Figure 4). They propose a heuristic that takes local decisions and migrates the less requested services to cloud devices in the shortest path. An important thing to take into account when deploying a distributed control strategy is the communication protocol used for the information exchange between the fog controllers and, for example, a main controller in the cloud. Table 2. Classification according to control strategies.

Optimization Objectives
The service placement problem is normally addressed from different optimization objectives, using different types of formulations and a variety of solving algorithms. Most of the papers in this survey try to optimize only one metric, while few alternatives propose simultaneously optimizing a group of metrics. Normally, when more than one metric must be optimized, the most adopted option is to include the other objectives as constraints. The most common optimization metrics are cost and latency, both desirable to minimize in most fog contexts used for real-time applications, where particular fog or cloud operators will compete to host IoT applications. In addition, it is common to find that latency is included as a constraint [60] since critical applications expose a time limit for completing a task.
Other habitual objectives are to minimize the computational resource utilization [60] while deploying the maximum number of services over the fog nodes; also reducing the monetary cost (for end users or for service providers), [36,51] associated with the data transmission or the computational consumption of the fog nodes and also to the deployment of a node; and even minimizing the energy consumption [40,80], which is one of the main concerns in IoT systems. The optimization objectives are summarized in Table 3. The resource allocation problem is normally formalized using integer programming (and its variants: integer linear programming, integer nonlinear programming, mixedinteger programming, and mixed-integer quadratic programming) or general constraint programming. Recently, novel approaches have considered formulations related to game theory, Markov decision processes, and reinforcement learning, an issue that is analyzed in the following section.

Resolution Strategies
Optimal resource allocation in fog environments is an NP-hard problem [26]. There are many issues that entangle this task; first, the heterogeneous nature and limited capacities of fog nodes, and also the dynamicity of the environment since the resources could appear or disappear instantly, and the sparse distribution of the fog nodes makes the resource allocation problem in fog networks a challenging task. In order to solve this problem, different types of solutions have been implemented, and five main approaches to solving the optimization problem are identified [26]: Other alternatives: machine learning (DRL), game theory.
Exact solutions are the least common and focus primarily on small instances of the problem [36,68].
Because of the size and the various aspects related to fog infrastructures, heuristics and metaheuristics are often used; in works such as [19], a metaheuristic-based on Tabu Search was used in order to avoid suboptimal placements when minimizing the makespan and the communication cost. Several solutions implement genetic algorithms in order to define the best solution regarding a particular metric. For example, in [77], the authors compared the performance of a greedy first fit heuristic against a genetic algorithm where the chromosomes are vectors that represent a service placement plan containing the total number of services of all applications; they also contrasted scenarios where different combinations of resources (only fog, only cloud or fog, and cloud) are used, finding smaller delays when using the genetic algorithm.
Some recent papers include the implementation of machine learning and deep learning algorithms, such as in [45], where deep reinforcement learning (DRL) was used to adaptively allocate resources in order to reduce the average service time under an MEC environment that includes an SDN controller that runs the DRL agent. Here, the authors compared the performance of the DRL algorithm against a typical open shortest path first alternative, with different amounts of fog entities and applications to place. The DRL strategy obtained smaller service times and load distribution. A summary of the used resolution strategies can be found in Table 4.

Mobility Support
Mobility management is a major issue in fog computing. Solutions that consider the mobility of end users are usually related to MEC environments. However, the mobility of fog nodes is not commonplace among the reviewed papers and is normally related to vehicular fog networks. Frequent changes in the position of fog nodes (see Figure 7) could lead to an excessive delay or packet loss or even service outages for an end user. In those situations, the orchestrator must be able to migrate the application to a new fog device. Some strategies consider the mobility of end users, but few authors tackle the issue of mobile fog nodes. In this case, considering mobile fog nodes implies that the physical topology where the applications are deployed is dynamic since, in some situations, some service (fog) nodes will be out of coverage. In [55], the authors introduce the concept of foglets, a programming infrastructure that provides an API for managing the application components deployed in the fog nodes. In this case, they consider geodistributed fog resources that can be managed by a centralized entity in charge of allocating resources according to a particular QoS requirement. The alternatives that explicitly state the use of mobile fog nodes in their implementations can be found in Table 5.

Federation between Fog and Cloud
The different needs held by heterogeneous applications could be fulfilled by a combination of resources in the fog and the cloud. For example, an application that needs more computational resources could be redirected to the cloud environment, whilst a routine that must be executed in a shorter time can experiment with better results if it is assigned entirely to fog nodes. Here, several configurations can be considered where, for instance, different operators on the cloud or the fog infrastructure, and even, in some cases, different service providers could manage their own fog infrastructure.
In every case, there is a challenge if the interaction between operators is needed. Federation among cloud operators is already a well-investigated issue [92,93], and usually presents problems related (and not limited) to the lack of formalization, to the deployment of architectures that can act in real-time and dynamic, online and distributed scheduling [94], topics that are already highlighted in this survey. Not many alternatives consider a federation between fog operators. However, in [32], the authors proposed an architecture that includes three different layers (cloud, federation, and application management) where a multiobjective optimization task was made, trying to minimize the cost, latency, and user's footprint. Monetary cost is the most common metric to minimize in this kind of alternative since it is important to choose a provider that fulfills the application requirements at a minimal cost [38,48].
A summary of selected reviewed works is included in Table 6. They are classified according to their resolution and control strategies. All of them implement schemes of the type [C|Di]/On/Dy/M. The mobility parameter includes strategies that consider either mobile end users or mobile nodes. However, as seen before, few alternatives consider the latter. Most of them consider only one metric to be optimized, and initiatives that consider several optimization objectives usually choose one in the objective function and include the remaining metrics in the constraints of the problem. Many solution strategies are heuristics, and few authors have considered alternatives such as game theory or machine learning techniques. In addition, few authors have tackled the problem where multiple cloud or fog operators can host the same application. Table 6. Reviewed works on resource allocation.

Related Work
Objective Control Mob.

Strategy Fed Contribution
Dynamic application deployment in federations of clouds and edge resources using a multiobjective optimization AI algorithm [32] Cost, latency, users C Genetic algorithm Presented a resource allocation on a federated architecture based on a genetic algorithm.

A Dynamic Service-Migration Mechanism in Edge Cognitive
Computing [57] Cost C Reinforcement learning Proposed a dynamic service migration and deployed over a physical scenario and solved the service placement using RL.
A proximal algorithm for joint resource allocation and minimizing carbon footprint in geodistributed fog computing [80] Energy consumption D Heuristic Considered the problem of joint resource allocation and minimizing carbon footprint problem for video streaming service in fog computing. Developed a distributed algorithm based on the proximal algorithm and alternating direction method of multipliers (ADMM).
Towards dynamic resource provisioning for IoT application services in smart cities [95] Users, bandwidth, hops C Exact methods Developed a resource discovery service to exchange resource allocation information between the fog and the cloud layer.

Efficient Placement of Multicomponent Applications
in Edge Computing Systems [56] Cost C Heuristic Addressed the problem of placement of multicomponent applications in MEC, and developed a heuristic algorithm based on an interactive matching process.

MigCEP: Operator Migration for Mobility-Driven Distributed Complex Event
Processing [36] Cost Di Heuristic Presented a method to reduce cost in a federated environment (multiple fog operators), where the migrations are planned ahead of time.
Incremental Deployment and Migration of Geo-Distributed Situation Awareness Applications in the Fog [55] Latency C Heuristic They propose foglets, which are APIs for storing and retrieving data on the local nodes and enabling communication among the resources in the fog and cloud.
Service placement for latency reduction in the Internet of Things [54] Hops Applied Lyapunov optimization to decompose the long-term optimization problem into a series of real-time optimization problems to handle user mobility.

Related Work
Objective Control Mob.

Strategy Fed Contribution
Dynamic service migration and workload scheduling in edge-clouds [52] Latency C MDP -Lyapunov opt.
Applied a new approach to solving constrained MDP to a dynamic service migration and workload scheduling in edge-clouds environments.
Elastic urban video surveillance system using edge computing [50] Distance, latency C Heuristic Designed a three-tier edge computing system NFV-SDN architecture to elastically adjust computing capacity and dynamically route data to proper edge servers for the real-time surveillance applications. Cost, latency C Heuristic Formulated the problem of resource provisioning and workload assignment for IoT services to jointly decide on the number and the location of edge servers and applications to deploy, using a decomposition approach to divide the original problem into two subproblems.

Optimal Placement of Cloudlets for Access Delay Minimization in SDN-Based
Internet of Things Networks [48] Latency C Heuristic Investigated the optimal placement of cloudlets using SDN to minimize the average access delay.

UCAA: User-Centric User Association and Resource Allocation in Fog Computing
Networks [47] Latency, energy consumption C Heuristic Presented a user-centric resource allocation scheme, trying to minimize a utility function that depends on several parameters.
Optimizing QoS-Assurance, Resource Usage and Cost of Fog Application Deployments [15] Latency, cost, QoS C Heuristic Developed a prototype that runs a multiobjective optimization framework to determine the deployments of the application that provide the best tradeoff among optimization objectives.

Task-Driven Resource Assignment in Mobile Edge
Computing Exploiting Evolutionary Computation [46] Latency, energy consumption C Genetic algorithms Proposed a joint optimization problem for task-driven resource assignment based on evolutionary computation over three typical task-driven cases. A cost-and performance-effective approach for task scheduling based on collaboration between cloud and fog computing [43] Cost C Heuristic Proposed a scheduling algorithm to achieve the balance between the performance of application execution and the mandatory cost for the use of cloud resources.
Energy and time efficient task offloading and resource allocation on the generic IoT-fog-cloud architecture [42] Energy consumption C Heuristic Proposed a general IoT-fog-cloud architecture, and resource allocation was formulated into the energy and time cost minimization problem.
Application Component Placement in NFV-Based Hybrid Cloud/Fog Systems With Mobile Fog Nodes [19] Latency, cost C Metaheuristic Used the random waypoint mobility model for fog nodes and proposed a Tabu-Search-based component placement (TSCP) algorithm to find suboptimal resource placements.
A hybrid approach to scheduling real-time IoT workflows in fog and cloud environments [41] Cost C Heuristic Proposed a hybrid fog-and cloud-aware heuristic for the dynamic scheduling of multiple real-time Internet of Things (IoT) workflows in a three-tiered architecture.

Workload Allocation in
IoT-Fog-Cloud Architecture Using a Multiobjective Genetic Algorithm [90] Latency, energy consumption C Genetic algorithms Formulated an alternative to maintain a trade-off between energy consumption and delay in processing workloads in fog.
Energy-efficient task allocation and energy scheduling in green energy powered edge computing [90] Energy consumption C Heuristic Investigated the energy cost minimization problem with joint consideration of VM migration, task allocation, and green energy scheduling and proved its NP-hardness. Proposed a joint optimization framework for fog nodes, service providers, and users to achieve the optimal resource allocation schemes in a distributed fashion.

A Hierarchical Game Framework for Resource
Management in Fog Computing [30] Cost, latency C Game theory Proposed a three-layer hierarchical game framework to solve the problem related to the resource allocation in the virtualized network, the asymmetric information problem, and the resource matching in the physical network. Cost Di Heuristic Presented an NFV-based high-level architecture for a system that enables the deployment of IoT services across multiple edges and clouds.
Towards Network-Aware Resource Provisioning in Kubernetes for Fog Computing Applications [64] Latency C Heuristic Studied the VNF optimal placement problem in NFV-based edge cloud systems with IoT nodes. Considered IoT service chains composed of multiple VNFs deployed on edge clouds.  [58] Throughput C

Lyapunov optimization
Jointly optimized the service placement, data admission, and resource allocation of an edge server to maximize the time-average service throughput of the server.
Near Real-Time Optimization of fog Services placement for responsive Edge computing [59] Latency C Heuristic Presented a service scheduling algorithm for fog and edge networks containing hundreds of thousands of devices, which is capable of incorporating changes in network conditions and connected devices.
Resource Allocation in 5G IoV Architecture Based on SDN and Fog-Cloud Computing [59] Delay, stability, energy consumption, load balancing C Genetic algorithm Proposed a multiobjective optimization problem solved via a modified GA in order to improve resource allocation in a vehicular network combined with cloud resources.
A Micro-Level Compensation-Based Cost Model for Resource Allocation in a Fog Environment [59] Cost C Heuristic Proposed a heuristic for resource allocation trying to minimize the cost of placing applications and compared its performance against the best-fit algorithm, obtaining better results in terms of cost, successful placements and delay.

Resilience in Fog and Cloud Environments
Resilience can be defined as the ability of a system (network) to provide and maintain an acceptable level of service in the presence of failures and different challenges during operation. Two principal fields concentrate the research on resilient networks: the first one tries to deal with how to design systems that, in the presence of problems, can keep the provision of a service, and the second one is related to guaranteeing that a system will behave as expected, taking into account measurable properties of the network (trustworthiness) [97]. Normally, cloud-based environments offer properties such as elasticity, virtualization, scalability, and geodistributed services that are often troublesome when trying to implement standard resilience alternatives [98]. In [99], some key aspects on which the resilience of fog computing systems depends were remarked upon. In our case, we concentrate on those points that relate to the resource allocation problem: • Complexity: When network complexity increases, there is a higher probability of experimenting with unexpected failures. In fog computing systems, complex envi-ronments are common due to the number of devices and the heterogeneity of the hardware and functionalities. In addition, the chosen topology of a particular solution can lead to situations that are prone to random errors or security vulnerabilities. Therefore, the alternatives used for assigning resources in this kind of environment must consider how the architecture's complexity can affect the availability of physical resources. In [100], the authors considered the existence of dependencies in complex systems and tried to characterize these relationships in order to reduce the level of intricacy of the network. • Redundant resources: When assigning resources in fog computing systems, installing more nodes and reserving additional capacity to maintain the service in case of failure (without violating delay constraints) could increase the resilience of the network. In [101], the authors worked on a scenario of a vehicular network trying to minimize energy consumption. For this, they separated the fog nodes into clusters and enhanced energy saving by considering some collaboration between vehicular nodes. • Deploying of agents: Distributed systems that implement different controllers can carry out actions to enforce problem mitigation. It also can improve system scalability in situations of high demand. In [102], a large-scale IoT network was managed by a distributed controller enhancing the recovery mechanism from a failure, supported by a distributed decision algorithm that rerouted the traffic to available nodes.

Research Directions and Challenges
This survey paper explored the existing literature on resource allocation in fog computing, including its various techniques and challenges. In this discussion section, we summarize our findings and offer insights into the current state of research and future directions in this field.
The proposed taxonomy classified resource allocation strategy according to different categories. We found that fewer solutions explore strategies based on distributed control and rely primarily on assigning this task to a central node. In addition, there is a growing interest in exploring scenarios of distributed applications over mobile fog nodes (similar to problems explored in vehicular networks). In terms of resolution strategies, tools such as ML and DL are paving the way toward becoming reliable alternatives for solving multiobjective optimization problems or working in more complex topologies; however, data gathering is still an issue to be attended to. Finally, there is still room for exploring resource allocation problems in environments with different operators. We will try to elaborate in a more detailed way on these aforementioned points.
Exploration of different control strategies: There is a lack of solutions that address the resource allocation problems in cloud/fog environments in a distributed way, relying on a single central node. This is mainly related to the fact that distributed algorithms are difficult to construct and implement due to the complexity of communication and synchronization. A distributed resource management strategy would need the implementation of a communication protocol between the control nodes [103]. Distributed control could be a well-fitted option to overcome issues related to scalability and reliability. For example, in SDN there is still a lot of research on designing efficient distributed control platforms since the structure of the control plane and the number and placement of multiple controllers critically impact the performance of a system [104].
Mobility of fog nodes: Due to the high mobility of end users and fog devices, the solutions must ensure the continuity of the service and consistent performance of the infrastructure, for example, by migrating in an appropriate way the allocated application to another chosen resource. This situation could become problematic when the researchers consider distributed applications instead of monolithic ones since the migration must consider communication between the parts of the application and latency between the components. In these cases, an approach with a predictive model for the users' mobility could present advantages. Several studies propose models to predict the mobility of wireless sensor nodes [105]. In [106], a hidden Markov model is used to forecast the movement of the IoT devices in order to improve the handoff process and energy consumption.
Federation and multiple operators: Few alternatives consider the federation between fog and cloud or the possibility of having different fog operators, a situation that could increase the operational cost of the solution. This could be a common scenario since different actors could provide fog services; in this case, a solution that considers the monetary costs combined within a tolerable latency for a particular service is necessary. As stated before, there are several studies where a mesh of different cloud operators is considered. For example, one cloud operator that needs additional resources sends a request to foreign operators to enlarge its infrastructure elastically [107]. This is also an issue considered in the VNF placement problem, similar to in [108], where the authors implemented a multidomain orchestrator that manages resources in multiple operators. In [109], a deep reinforcement learning alternative was implemented in order to find an adequate VNF embedding in a noncooperative domain, where the network operators hid their infrastructure from other competing counterparts. In this sense, researchers could explore strategies that have proven their effectiveness in similar distributed problems such as VNF resource allocation.
Resilience in fog environments: Most of the reviewed alternatives do not consider failure protection in the proposed strategies. Most of them are related to architectures where vehicular networks are part of the edge and fog layers [66]. This is a crucial issue to be addressed due to the characteristics of fog nodes, which are typically energy-and availability-constrained. Resilient and survivable networks are points that have been exhaustively explored in NFV resource allocation problems and, in conjunction with SDN, are important alternatives to implement solutions that work in favor of the resilience of a fog computing system. Finally, we found issues related to the experimentation and the formulation of the resource allocation problem, for which improvement could lead to a better understanding and tackling of the problem. Most of the reviewed papers are oriented to address the problem in a specific context and to be evaluated on different platforms, ending in the formulation of solutions that could not be compared. The configuration of nodes (number of fog nodes, cloud nodes, IoT devices) and characteristics of the deployment seem somewhat arbitrary. They could range from a few devices up to thousands of them. In addition, a variety of tools are used in a simulation of fog environments, where iFogSim [110] is probably the most common. Other simulators developed in common computational tools, such as Python or Matlab are also used. There is still room for working on a baseline in order to facilitate comparisons of different alternatives. Few alternatives implemented their solutions on real testbeds in order to analyze their execution and behavior. There are some efforts in implementing, for example, resource allocation strategies on commercial software such as Apache Spark or Hadoop [10], but in controlled environments with specific solutions. We also found that there is a lack of generality in formulating the resource allocation problem. Some papers, the majority found on MEC environments, consider monolithic applications, whilst the most elaborated alternatives represent the deployment of an application or a service as a directed acyclic graph, where the nodes are functions that take part in the processing of the information or can also be sources or destinations of the data. Edges represent the flow of information between nodes. The latter representation could be a better method of representing the problem.

Conclusions
This paper focuses on the resource allocation problem in the IoT and cloud layers continuum, an issue with open challenges and discussions. This paper collected several works on this topic, proposing a taxonomy to classify the different alternatives according to the control strategies, optimization objectives, resolution strategies, mobility support, and federation between fog and cloud paradigms. This article focuses on strategies that implement online resource allocation on dynamic topologies.
There are several approaches to resource allocation, including centralized or distributed control and online or offline resource assignation, and even strategies that seek to improve resilience on the network. In addition, different environmental assumptions could be considered, such as dynamic topologies that include mobile nodes or scenarios that pose integration between fog and cloud operators. These resource allocation problems are usually solved using exact methods in smaller scenarios or heuristics and metaheuristics on larger layouts. Recent works have focused on proving the effectiveness of game-theoretic models or deep reinforcement learning alternatives.
Despite significant progress in resource allocation research, many challenges and open research questions still need to be addressed. For example, one crucial issue we seek to address in the future is evaluating different algorithms in the scenarios considered in this research.
This paper also formally defines the resource allocation problem in these environments. In conjunction with the assembled taxonomy, we aim to ease access to works developed in this specific context and to identify the challenges in this topic. Funding: This work was partially supported by the Science, Technology, and Innovation Fund (FCTeI) of the General Royalties System (SGR) under the project identified by code BPIN 2020000100044, also by FRIDA (Fondo Regional para la Innovación Digital en América Latina y el Caribe), and by the project "Red temática Ciencia y Tecnología para el Desarrollo 519RT0580" by the Ibero-American Science and Technology Program for Development CYTED.

Data Availability Statement:
No new data were created or analyzed in this study. Data sharing is not applicable to this article.

Conflicts of Interest:
The authors declare no conflict of interest.