Mean-Field-Type Games in Engineering

With the ever increasing amounts of data becoming available, strategic data analysis and decision-making will become more pervasive as a necessary ingredient for societal infrastructures. In many network engineering games, the performance metrics depend on some few aggregates of the parameters/choices. One typical example is the congestion field in traffic engineering where classical cars and smart autonomous driverless cars create traffic congestion levels on the roads. The congestion field can be learned, for example by means of crowdsensing, and can be used for efficient and accurate prediction of the end-to-end delays of commuters. Another example is the interference field where it is the aggregate-received signal of the other users that matters rather than their individual input signal. In such games, in order for a transmitter-receiver pair to determine his best-replies, it is unnecessary that the pair is informed about the other users' strategies. If a user is informed about the aggregative terms given her own strategy, she will be able to efficiently exploit such information to perform better. In these situations the outcome is influenced not only by the state-action profile but also by the distribution of it. The interaction can be captured by a game with distribution-dependent payoffs called mean-field-type games (MFTG). An MFTG is basically a game in which the instantaneous payoffs and/or the state dynamics functions involve not only the state and the action profile of the players but also the joint distributions of state-action pairs. In this article, we propose and analyze engineering applications of MFTGs.


I. INTRODUCTION
The article is structured as follows.The next section overviews earlier works on static mean-field games, followed by discrete time mean-field games with measure-dependent transition kernels.Then, a basic MFTG with finite number of agents is presented.After that, the discussion is divided into two illustrations in each of the following areas of engineering (Fig. 1): Civil Engineering (CE), Electrical Engineering (EE), Computer Engineering (CompE), Mechanical Engineering (ME), General Engineering (GE).
• CE: road traffic networks with random incident states and multilevel building evacuation.• EE: Interference field in millimeter wave wireless communications and distributed power networks • CompE: Virus spread over networks and virtual machine resource management in cloud Networks • ME: Synchronization of oscillators, consensus, alignment and energy-efficient buildings • GE: Online meeting: strategic arrivals and starting time and mobile crowdsensing as a public good.The article proceeds by presenting the effect of time delays of coupled mean-field dynamical systems and decentralized information structure.Then, a discussion on the drawbacks, limitations, and challenges of MFTGs is highlighted.Lastly, a summary of the article and concluding remarks are presented.

A. Mean-Field Games: Static Setup
This subsection overviews mean-field games in a static and stationary setting.Mean-field games have been around for quite some time in one form or another, especially in transportation networks and A. Tcheukam   in competitive economy.In the context of competitive market with large number of players, a 1936 article [1] captures the assumption made in mean-field games with large number of players, in which the author states: "each of the participants has the opinion that its own actions do not influence the prevailing price".
Another comment on the impact on the population mean-field term was given in [2] page 13: " When the number of participants becomes large, some hope emerges that of the influence of every particular participant will become negligible . . ." Since the population profile involves many players for each type or class and location, a common approach is to replace the individual players' variables and to use continuous variables to represent the aggregate average of type-location-actions.In the large population regime, the mean field limit is then modeled by state-action and location-dependent time process (see Figure 2).This type of aggregate models are also known as non-atomic or population games.It is closely related to the mass-action interpretation in [3], Equation (4) in page 287.
In the context of transportation networks, the mean-field game framework, underlying the key foundation, goes back to the pioneering works of [4] in the 1950s.Therein, the basic idea is to describe and understand interacting traffic flows among a large population of agents moving from multiple sources to destinations, and interacting with each other.The congestion created on the road and at the intersection are subject to capacity and flow constraints.This corresponds to a constrained mean-field game problem as noted in [5].A common behavioral assumption in the study of transportation and communication networks is that travelers or packets, respectively, choose routes that they perceive as being the shortest under the prevailing traffic conditions.As noted in [6], collection of individual decisions may result to a situation which drivers cannot reduce their journey times by unilaterally choosing another route.The work in [6] such a resulting traffic pattern as an equilibrium.Nowadays, it is indeed known as the Wardrop equilibrium [4], [7], and it is thought of as a steady state obtained after a transient phase in which travelers Fig. 2: Each agent with its own state and own mean-field interacts with the aggregates from the population.The population meanfield is formed from the reaction of the agents and affects the behavior of the individual agents and their own mean-field.
successively adjust their route choices until a situation with stable route travel costs and route flows has been reached [8], [9].In the seminal contribution [4], p. 345 the author stated two principles that formalize this notion of equilibrium and the alternative postulates of the minimization of the total travel costs.His first principle reads: "The journey times on all the routes actually used are equal, and less than those which would be experienced by a single vehicle on any unused route." Wardrop's first principle of route choice, which is identical to the notion postulated in [6], [10], became widely used as a sound and simple behavioral principle to describe the spreading of trips over alternate routes due to congested conditions.Since its introduction in the context of transportation networks in 1952 and its mathematical formalization by [5], [11] transportation planners have been using Wardrop equilibrium models to predict commuters decisions in reallife networks.
The key congestion factor is the flow or the fraction of travelers per edge on the roads (see Application 1).The above Wardrop problem is indeed a mean-field on a discrete space.The exact mean-field term here corresponds to a mean-field of actions (a choice of a route).Putting this in the context of infinite number of commuters results to end-to-end travel times that are function of own choice of a route and the mean-field distribution of travelers across the graph (network).
In a population context, the equilibrium concept of [4] corresponds to a Nash equilibrium of the mean-field game with infinite number of players.The works [7], [12] provide a variational formulation of the (static) mean-field equilibrium.
The game theoretic models such as global games [13], [14], anonymous games, aggregative games [15], population games, and large games, share several common features.Static mean-field games with large number of agents were widely investigated (see [16]- [21] and the references therein).

B. Mean-Field Games: Dynamic Setup
The next section overviews dynamic mean-field games and their applications in engineering.The key ingredients of dynamic meanfield games appeared in [22], [23] in the early 1980s.The work in [22] proposes a game-theoretic model that explains why smaller firms grow faster and are more likely to fail than larger firms in large economies.The game is played over a discrete time space.Therein, the mean-field is the aggregate demand/supply which generates a price dynamics.The price moves forwardly, and the players react to the price and generate a demand and the firm produces a supply with associated cost, which regenerates the next price and so on.The author introduced a backward-forward system to find equilibria (see for example Section 4, equations D.1 and D.2 in [22] ).The backward equation is obtained as an optimality to the individual response, i.e., the value function associated with the best response to price, and the forward equation for the evolution of price.Therein, the consistency check is about the mean-field of equilibrium actions (population or mass of actions), that is, the equilibrium price solves a fixed-point system: the price regenerated after the reaction of the players through their individual best-responses should be consistent with the price they responded to.
Following that analogy, a more general framework was developed in [23], where the mean-field equilibrium is introduced in the context of dynamic games with large number of decision-makers.A meanfield equilibrium is defined in [23], page 80 by two conditions: (i) each generic player's action is best-response to the mean-field, and (ii) the mean-field is consistent and is exactly reproduced from the reactions of the players.This matching argument was widely used in the literature as it can be interpreted as a generic player reacting to an evolving mean-field object and at the same time the mean-field is formed from the contributions of all the players.The authors of [24] show how common noise can be introduced into the mean-field game model (the mean-field distribution evolves stochastically) and extend the Jovanovic-Rosenthal existence theorem [23].
Continuous time version of the works [22], [23] can be found in [25]- [28].We refer the reader to [29]- [32] for recent development of mean-field game theory.The authors [27], [33]-[37] have developed a powerful tool for modelling strategic behavior of large population of agents, each of them having a negligible impact on the population mean-field term.Weak solutions of mean-field games are analyzed in [38], Markov jumps processes [39], [40], and leader-followers models in [41].Finite state mean-field game models were analyzed in [42]- [49].Team and social optimum solutions can be found in [41], [50]- [53].The reader is referred to [54], [55] for mean-field convergence of McKean-Vlasov dynamics.Numerical methods for mean-field games can be found in [56]- [59].

Limitations of the existing mean-field game models
Most of the existing mean-field game models share the following assumptions: • Big size: A typical assumption is to consider an infinite number decision-makers, sometimes, a continuum of decision-makers.The idea of a continuum of decision-makers may seem outlandish to the reader.Actually, it is no stranger than a continuum of particles in fluid mechanics, in water distribution, or in petroleum engineering.In terms of practice and experiment  [78], [79] electrical vehicles [80], [81] cloud networks [83], [136], [137] auction [84], [85] cyber-physical systems [86], [87] airline networks [88] sensor networks [89] traffic networks [90] small cell networks [91], [92] D2D networks [93], [94] multilevel building evacuation [123] power networks [81], [95]-[106] HVAC [107]- [112] TABLE I: Some applications of MFTGs in Engineering however, decision-making problems with continuum of decisionmakers is rarely observed in engineering.There is a huge difference between a fluid with a continuum of particles and a decision-making problem with a continuum of agents.Agents may physically occupy a space (think of agents inside a building or a stadium) or a resource, and the size or number of agents that most of engineering systems can handle can be relatively large or growing but remain currently finite (see the nonasymptotic analysis in [113]).It is in part due to the limited resource per shot or limited number of servers at a time.In all the examples 3-4 and applications 1 to 10 provided below, we still have a finite number of interacting agents.Thus, this assumption appears to be very restrictive in terms of engineering applications.• Anonymity: The index of the decision-maker does not affect the utility.The players are assumed to be indistinguishable within the same class or type.The drawback of this assumption is that most individual decision-makers in engineering are in fact not necessarily anonymous (think of Google, Microsoft, Twitter, Facebook, Tesla, . . .), the classical mean-field game model is inappropriate, and does not apply to such situations.In mean-field games with several types (or multi-population mean-field games), it is still assumed that there is large number of agents per type/class/population, which is not realistic in most of the engineering applications considered in this work.• NonAtomicity: A single decision-maker has a negligible effect on the mean-field-term and on the global utility.
One typical example where this assumption is not satisfied is a situation of targeting a room comfort temperature, in which the air conditioning controller adjusts the heating/cooling depending on the temperature in the room, the temperatures of the other connecting zones and the ambient temperature.It is clear that the decision of the controller to heat or to cool affect the variance of the temperature inside the room.Thus, the effect of the individual action of that controller on the temperature distribution (mean-field) inside the room cannot be neglected.
To summarize, the above conditions appear to be very restrictive in terms of engineering applications, and to overcome this issue a more flexible MFTG framework has been proposed.

Area
Anonymity Infinity Atom population games [4], [5] yes yes no evolutionary games [114] yes yes no non-atomic games [23] yes yes no aggregative games [15] relaxed global games [13], [14] yes yes no large games [16] yes yes no anonymous games [23] yes  MFTGs not only relax of the above assumptions but also incorporate the behavior of the players as well as their effects in the mean-field terms and in the outcomes (see Table II).
(i) In MFTGs, the number of users can be finite or infinite.
(ii) The indistinguishability property (invariance in law by permutation of index of the users) is not assumed in MFTGs.(iii) A single user may have a non-negligible impact of the meanfield terms, specially in the distribution of own-states and own mixed strategies.These properties (i)-(iii) make strong differences between mean-field games and MFTGs (see [115] and the references therein).
MFTG seems to be more appropriate in such engineering situations because it does not assume indistinguishability, it captures the effect of each agent in the distribution and the number of agents is arbitrary as we will see below.

C. Background on MFTGs
This section presents a background on MFTGs.Definition 1 (Mean-Field-Type Game): A mean-field-type game (MFTG) is a game in which the instantaneous payoffs and/or the state dynamics coefficient functions involve not only the state and the action profile but also the joint distributions of state-action pairs (or its marginal distributions, i.e., the distributions of states or the distribution of actions).A typical example of payoff function of player j has the following structure: with rj(x, u, D (x,u) ) where (x, u) is the state-action profile of the players and D (x,u) is the distribution of the state-action pair (x, u), X is the state space, and U is the action profile space of all players.
From Definition 1, a mean-field-type game can be static or dynamic in time.One may think that MFTG is a small and particular class of games.However, this class includes the classical games in strategic form because any payoff function rj(x, u) can be written as rj(x, u, D).
When randomized/mixed strategies in the von Neumann-type payoff, the resulting payoff can be written as E[rj(x, u)] = rj(x, u)D (x,u) (dx, du) = rj(D).Thus, the form rj(x, u, D) is more general and includes non-von Neumann payoff functions.
Example 1 (Mean-variance payoff): The payoff function of agent i is E[ri(x, u)] − λ var[ri(x, u)], λ ∈ R which can be written as a function of ri(x, u, D (x,u) ).For any number of interacting players, the term D x i ,u i ) plays a non-negligible role in the standard deviation var[ri(x, u)].Therefore, the impact of agent i in the individual mean-field term D (x i ,u i ) cannot be neglected.
Example 2 (Aggregative games): The payoff function of each player depends on its own action and an aggregative term of the other actions.Example of payoff functions include ri(ui, j =i u α j ), α > 0 and ri(xiui, j =i xjuj).
In the non-atomic setting, the influence of an individual state xj and individual action uj of any user j will have a negligible impact on mean-field term D (x,u) .In that case, one gets to the so-called mean-field game.
Example 3 (Population games): Consider a large population of agents.Each agent has a certain state/type x ∈ X and can choose a control action u ∈ U(x).Let the proportion of type-action of the population as m.The payoff of the agent with type/state x, control action u when the population profile m is r(x, u, m).Global games with continuum of players were studied in [13] based on the Bayesian games of [14], which uses the proportion of actions.In the case where both non-atomic and atomic terms are involved in the payoff, one can write the payoff as rj(s, u, D, D) where D is the population state-action measure.User j may influence Dj (distribution of its own state-action pairs) but its influence on D may be limited.The next section presents dynamic MFTGs.

II. A BASIC DYNAMIC MFTG: FINITE REGIME
Consider a basic MFTG with n ≥ 2 agents interacting over horizon [0, T ], T > 0. The individual state dynamics of agents is given by and the payoff functional of agent i is where the strategy profile is u = (u1, . . ., un), which also denoted as (ui, u−i).The novelty in the modelling (1)-( 3) is that each individual agent i influences its own mean-field terms D x i (t) , and D (x i (t),u i (t)) independently on the total number of interacting agents.In particular, the influence of agent i on those mean-field terms remain nonnegligible even when there is a continuum of agents.The distributions Dx i and D (x i ,u i ) represent two important terms in the modeling of MFTGs.We refer them as individual mean-field terms.In the finite regime, the other agents are captured by the empirical measures . We refer these terms to as population mean-field terms.
Similarly, a basic discrete time (discrete or continuous state) MFTG with individual state dynamics of agents, is given by xi,t+1 ∼ qi .|xi,t,ui,t, D (x i,t ,u i,t ) , xi0 ∼ Di,0 where qi(.|.) is the transition kernel of agent i to next states.Meanfield-type control and global optimization can be found in [29], [116], [117].The models (1) and ( 4) are easily adapted to cooperative and coalitional MFTGs and can be found in [118].Psychological MFTG was recently introduced in [94] where spitefulness, altruism, selfishness, reciprocity of the players are examined by means empathy, other-regarding behavior and psychological factors.Definition 2: An admissible control strategy of agent i is an Fi−adapted and square integrable process with values in a non-empty subset Ui.Denote by Ui = L 2 F i ([0, T ], Ui) the class of admissible control strategies of agent i.
Definition 3 (Best response): Given a strategy profile of the other agents (u1, . . ., ui−1, ui+1, . . ., un), with uj, j = i that are square integrable and the mean-field terms D, the best response problem of agent i is: To solve problem (7), three methods have been developed: • Direct approach which consists to write the payoff functional in a form such that the optimal value and optimizers are trivially obtained, and a verification follows.• A stochastic maximum principle (Pontryagin's approach) which provides necessary conditions for optimality.• A dynamic programming principle (Bellman's approach) which consists to write the value of the problem (per player) in (backward) recursion form, or as solution to a dynamical system.
Definition 4: A (Nash) equilibrium of the game is a strategy profile (u * 1 , . . ., u * n ) such that for every agent i, for all ui ∈ Ui.
Example 4 ( Network Security Investment [68] ): A graph connected if there is a path that joins any point to any other point in the graph.Consider n ≥ 2 decision-makers over a connected graph.Thus, the security of a node is influenced by the others through possibly multiple hops.For simplicity, we consider only an additive noise in the state model.The effort of user i in security investment is ui.The associated cost may include money (e.g., for purchasing antivirus software), time and energy (e.g., for system scanning, patching).Let The best-response of user i to (u−i, E[x]) := (u1, . . ., ui−1, ui+1, . . ., un, E[x]), solves the following linearquadratic mean-field-type control problem where, qi(t) ≥ 0, i(t) ≥ 0, ρi(t) ≥ 0, ri(t) > 0 and a, ā, bi, c are real numbers and where E[x(t)] is the expected value of network security level created by all users under the control action profile (u1, . . ., un).Note that the expected value of the terminal term in Ri can be seen as a weighted variance of the state [112] since E[(x(t)− E[x(t)]) 2 ] = var(x(t)).The optimal control action is in state-andmean-field feedback form: with βi(T ) = 1, η1i(T ) = −1, η2i(T ) = 0. Figure 3 plots the optimal cost trajectory with the step size 2 −8 , the horizon is [0, 1], the other parameters are b = 5, r = 1, q = 1, ρ = 0.0001, = 0.1.Figure 4 plots the optimal state vs the equilibrium state.As noted in [119], the security state is higher when there is a cooperation between the users and when the coalition formation cost is small enough.
The following example solves distributed variance reduction problem in discrete time using MFTG.
Example 5 (Distributed Mean-Variance Paradigm, [120]): The best response problem of agent i is given the strategy (uj) j =i of the other agents.

A. Civil Engineering
This subsection discusses two applications of MFTG in civil engineering.
Application 1 (Road Traffic over Networks ): The example below concerns transportation networks under dynamic flow and possible stochastic incidents on the lanes.Consider a network (V, L), where V is a finite set of nodes and L ⊆ V × V is a set of directed links.n users share the network (V, L).Let R be the set of possible routes in the network.A user with a given source-destination pair arrives in the system at source node s and leaves it at the destination node d after visiting a series of nodes and links, which we refer to as a route or path.Denote by c w i (xt, uit, mt) the average w−weighted cost for the path uit when mt fraction of users choose that path at time t and xt is the incident state on the route.The weight w simply depicts that the effective cost is the weighted sum of several costs depending on certain objectives.These metrics could be the delayed costs, queueing times, memory costs, etc and can be weighted by w in the multiobjective case.Again, the weight w could be different for different users due to their objectives.Henceforth, we omit w and work with generic cost ci(xt, uit, mt) for simplicity of notation.We assume that the cost is non-decreasing in the variable mt (congestion effect).We define two regimes for the traffic game: a finite regime game with n drivers denoted by Gn and an infinite regime game denoted by G∞.The basic components of these games are (N , X , R, I = {x}, ci(x, .)).A pure strategy of driver i is a mapping from the information set I to a choice of a route that belongs to R. The set of pure strategies of a user is R X .An action profile (route selection) (u1, . . ., un) ∈ R n is an equilibrium of the finite mean-field-type game if for every user i the following holds: for the realized state x.
The term + 1 n is the contribution of the deviating user to the new route.When n is sufficiently large the state-dependent equilibrium notion becomes a population profile m(x) = (m(x, u))u∈R such that for every user i m(x, u) > 0 =⇒ ci(x, u, m(x, u)) ≤ ci(x, u , m(x, u )), for the realized state x and for all u ∈ R. We refer to the equilibrium defined above as 0−Nash equilibrium.Note that the equilibrium profile depends on the realized state x.We now discuss the existence conditions.The equilibrium conditions can be rewritten in the form of variational inequalities: for each state x, ( * ) u∈R [m(x, u) − y(x, u)]c(x, u, m(x, u)) ≤ 0, for all y.Hence, the existence of an equilibrium is reduced to the existence of a solution to the variational inequality (*).By the standard fixedpoint arguments, we know from [121] that for each single state, such a population game has an equilibrium if the cost functions are continuous in the second variable m.Moreover, the equilibrium is unique under strict monotonicity conditions of the cost function ci(x, u, .).Note that uniqueness in m does not mean uniqueness of the action profile u since one can permute some of the commuters.We use imitative learning in an information-theoretic view point.We introduce the cost of learning from strategy mi,t−1 to mi,t as the relative entropy dKL(mi,t−1, mi,t).
Then, each user reacts by taking a myopic conjecture given by where ĉi,t is the estimated cost vector, βi,t is a positive parameter, dKL is the relative entropy from mi,t−1 to mi,t.dKL is not a distance (because it is not symmetric) but it is positive and can be seen as a cost to move from mi,t−1 to mi,t.We use the convexity property of the relative entropy to compute the strategy that minimizes the perturbed expected cost.
Proposition 1: The minimizer of ĉi,t, mi,t u ∈R mi,t−1(u )e −β i,t ĉi,t−1 (u ) By direct computation, one obtains that the minimizer strategy can be written as multiplicative weighted imitative Boltzmann-Gibbs strategy.
Proposition 2: Let βi,t = log(1 + νi,t) for νi,t > 0.Then, the imitative Boltzmann-Gibbs strategy becomes a multiplicative weighted imitative strategy: The advantage of the imitative strategy is that it makes sense not only in small learning rate but also in high learning rate.When the learning rate is large, the trajectory gets closer to the best reply dynamics and for small learning it leads to the replicator dynamics [122].One useful interpretation of the imitative strategy is the following: Consider a bounded rationality setup where the parameter νi,t is the rationality level of user i.Then, a large value of νi,t means a very high rationality level for user i, hence user i will use an almost "best reply" strategy.Small value of νi,t means that user i is of a low rationality level and is described by the replicator equation.It is interesting to see that both behaviors can be captured by the same imitative mean-field learning.Note that the logit (or Boltzmann-Gibbs) learning does not cover the low rationality level case.
Proposition 3: As νi,t goes to zero, the trajectory of the multiplicative weighted imitative strategy is approximated by the replicator equation of the estimated delays Each driver knows the current state and employs the learning pattern.Each driver tries to exploit the information on the current state and build a strategy based on the observation of the vector of realized delays over all the routes at the previous steps.Then the Folk theorem for evolutionary game dynamics states: • When starting from an interior mixed strategy, the replicator equation converges to one of the equilibria.• All the faces of the multi-simplex are forward invariant.In particular, the pure strategies are steady states of the imitative dynamics.
• The set of global optima belongs to the set of steady states of the imitative dynamics.The strategy-learning of user i is given by where ci,t(x, u) is the time-average delay (up to t) in route u and state x.
The imitative mean-field learning above can be used to solve a long-term mean-field game problem.We observe in Figures 5-6 that the imitative learning converges to one of the global optima.However, the exploration space grows in complexity.We explain how to overcome to this issue using mean-field learning based on particle swarm optimization (PSO).In it each user has a population of particles (multi-swarm).The particles within the same population (coalition) may pool their effort to learn faster and exploit better the available information.
The next example concerns multi-level building evacuation using constrained mean-field games.
Application 2 (Multi-level building evacuation [123]): A typical mean-field game model assumes that players have unconstrained state dynamics.This has been, for example, the case with most of the existing mean-field models developed in the last three decades.Such models may not however be useful in practice, for example in a context of building evacuation.Evacuation strategies and values are designed using constrained mean-field-type game theory.
Particle-based pedestrian models have been studied in [124], [125].Continuum approximation of theoretical models have been proposed in [124]- [129].Recent mean-field studies on crowd and pedestrian flows include [130]- [134].Below a mean-field game for multi-level building evacuation is presented.Consider a building with multiple floors and resolutions represented by a compact domain D in the m−dimensional Euclidean space R m .The number of floors is K.The domain at floor k is denoted as D k .For 1 < k < K, the floor k is connected to the higher floor k + 1 using the intermediary domain I + k but also the lower floor k − 1 using I − k .The sets I k can be elevator zones or stairs.n ≥ 2 agents are distributed in a multi-level multi-resolution building with stairs, exit doors, sky-bridges.Each agent knows her current location in the building.The state/location xi of an agent i changes depending on her control action ui.The agent is interested in a safe evacuation from the building.This means that she is interested in the minimal exit time that avoid huge crowd around her.The problem of the agent i is equivalent to where ci is a positive increasing function, with c2(0) = 0. T > 0 is the exit time at one of the exits.The final exit cost is represented by c3 which can be written as c3+ h(x) where c3 > 0 captures the initial response time of an agent (without congestion around), Gn(xi(t)) = 1 vol(B(x i (t), )) , represents the number of the agents around the position xi except i within a distance less than > 0, vol(B) is the m-dimensional volume of the ball B(xi(t), ) which does not depend on xi(t), due to translation invariance of the volume measure.When the number of agents grows, one obtains a mean-field game with several interacting agents.The state dynamics must satisfy the constraint xi(t) ∈ D at any time t before the exit.The nonoptimized Hamiltonian in macroscopic setting as where p is the adjoint variable.
The Pontryagin maximum principle yields The Hamiltonian H 0 (., ., G, p(t)) is concave in (x, u) for almost everywhere (a.e.) t ∈ [0, T ].Then, for convex function c3, u * is an optimal response if H 0 (x * (t), u * , G * , p * (t)) = maxu H 0 (x * (t), u, G * , p * (t)).The (optimized) Hamiltonian as , and the optimal strategy is in (own)state-and-mean-field feedback form: The dynamic programming principle leads to the following optimality system: The development of numerical result, simulation and a validation framework can be found in [123].Figures 9 and 10 show the application to a two floors evacuation building where 500 agents are spatially distributed.Next, two applications of MFTGs in electrical engineering are presented.

B. Electrical Engineering Application 3 (Millimeter Wave Wireless Communication):
Millimeter wave (mmWave) frequencies, roughly between 30 and 300 GHz, offer a new frontier for wireless networks.The vast available bandwidths in these frequencies combined with large numbers of spatial degrees of freedom offer the potential for orders of magnitude increases in capacity relative to current networks and have thus attracted considerable attention for next generation 5G communication systems.However, sharing of the spectrum and the available infrastructure will be essential for fully achieving the potential of these bands.Unfortunately, rapidly changing network dynamics make it difficult to optimize resource sharing mechanisms for mmWave networks.MIMO mmWave wireless networks will rely extensively on highly directional transmissions, where both users, relays and base stations transmit in narrow, high-gain beams through electronically steerable antennas.While directional transmissions can improve signal range and provide greater degrees of freedom through spatial multiplexing, they also significantly complicate spectrum sharing.Nodes that share the spectrum must not only detect one another, but also search over a potentially large angular space to properly steer the beams and reduce interference.Thus, the interference reduction is particularly important when multiple operators and users share the spectrum.
Power allocation, angle optimization and channel selection algorithms should consider the possible interference field and reduce it by adjusting the angles.This can facilitate rapid directional discovery in a dynamic and mobile environment as in Figure 11.A fundamental challenge in spectrum sharing is the coordination of transmissions amongst primary users, secondary users and relays to mitigate the effects of interference.Sometimes jammers and malicious are involved in the interactions.Beams adjustment and Interference coordination are central problem for users within the same network, or between users in different networks sharing the same spectrum.When multiple operators own separate core network and radio access network (RAN) Fig. 9: Spatial distribution of agents at different times.Fig. 7 represents the initial density of the agents in the building.Agents are represented by small circles in the map.Agent in the higher floors will be evacuated using the bridge (blue rectangle) on floor 2.
There is one exit door in the ground floor.The exit door is in green-color code in the ground floor.Fig. 8 represents the spatial distribution of agent at time t = 5.Notice that each agent chooses the shortest and less congested path and decreases its velocity according to its own congestion measure.nodes such as base stations and relays, but only loosely coordinate via wireless signaling, it is essential to use incentive mechanisms for better coordination to exploit the available resources.Cost sharing and pricing mechanisms capture some of the fundamental properties that arise when sharing resources among multiple operators.It can also be used in the uplink case, where users can select their preferred services and network provides and have to find tradeoffs between quality-of-experience (QoE) and cost (price).As an illustrative example, we use a particle swarm learning mechanism in which the particle adapt the parameters such as angle and power such that the satisfaction of the users is improved.Here the key mean-field term is the interference field (per angle).Since users are carrying smartphones with limited power consumption, it is crucial to examine the remaining energy level.As in [79] the energy dynamic can be written as de = −udt + vdt + σdW, subject to e(t) ≥ 0, and e(0) = e0, and u(.) ≥ 0 is the transmission power and v(.) is the energy harvesting rate (for example with distributed renewable energy sources).Users move according to a mobility dynamics (which imay not be stationary).The channel state can be modeled, for example using a matrix valued Ornstein-Uhlenbeck process processes dHj = Γj[ Ĥj − Hj]dt + dWj where Γj, Ĥj are matrices with compatible dimensions of antennas at source and destination.The (unnormalized) distribution of the triplet (position, energy, channel) of the population at time (or period) t is ν(t, e, x, H) = n j=1 δ {e j (t),x j (t),H j (t)} , and the one within a beam A(s, d) Compared to other wireless technologies, mmWave may generate less interference because of reduced and optimized angles.However, interference may still occur when several users and blocking objects fall within the same angle as depicted in Fig. 12.The success probability P(SINRi ≥ βi) from position xi(t) to destination di for both LoS and non-LoS can then be derived.The quality-of-experience of users can be termed as function as the sectorized interference field, satisfaction level and user-centric subjective measures such as MOS (mean opinion score) values.
Application 4 (Distributed Power Networks (DIPONET)): Distributed power is a power generated at or near the point of use.This includes technologies that supply both electric power and mechanical power.The rise of distributed power is also being Fig. 12: Interaction model driven by the ability of distributed power systems to overcome the energy need constraints, and transmission and distribution lines.Mean-field games theoretic applications to power grid can be found in [81], [95]- [106].We study distributed power networks using MFTG.A prosumer (producer-consumer) is a user that not only consumes electricity, but can also produce and store electricity.Based on forecasted demand, each operator determines its production quantity, its mismatch cost, and engages an auction mechanism to the prosumer market.The performance index is Lj(sj, ej) = ljT (e(T )) Each producer aims to find the optimal production strategies: infs j ,e j Lj(sj, ej, T ) d dt e jk (t) = c jk (t) − s jk (t) c jk (t) ≥ 0, s jk (t) ∈ [0, sjk ], ∀j, k, t s jk (w) = 0 if w is a starting time of a maintenance period.
where Dj(t) is a demand at time t, lj(Dj(t) − S(t)) denotes the instant loss where S(t) = S producer (t) + Sprosumer(t), S producer (t) = n j=1 sj(t) = n j=1 K j k=1 s j,k (t) , where s j,k (t) is the production rate of plant/generator k of j at time t.Kj total number of power plants of j.The loss lj is assumed to be strictly convex.The stock of energy at time t is given by the classical motion d dt e jk where c jk (t) is the maintenance cost of plant/generator k of j when it is in the maintenance phase.The optimality equation of the problem is given by Hamilton-Jacobi-Bellman: where Hj is the Hamiltonian function is The first order interior optimality condition yields −l j (Dj − Sj) − y jk +ρs jk = 0.By summing over k one gets an equation for the total production quantity S * j solves −Kjl j (Dj −Sj)− K j k=1 y jk +ρSj = 0 and the optimal supply of power plant k is ).
The solution of partial differential equation ( 18) can be explicitly obtained and it is given by the Hopf-Lax formula: where H * j is the Legendre transformation of Hj, given by  The mean-field equilibrium is obtained as fixed-point equation involving S * and D * .When l j is continuous and preserves the production domain [0, s] one can guarantee the existence of such a solution by using Brouwer fixed-point theorem.One can use higher order fast mean-field learning to learn and compute of such a meanfield equilibrium.Figure 13 illustrates the optimal supply based on an estimated demand curve.Figure 14 represents an allocation of the producer with two power stations.

C. Computer Engineering
This section provides applications of MFTG in computer engineering.It starts with an application of MFTG with number finite state-actions and then focuses on continuous state-action spaces.
Application 5 (Virus Spread over Networks): We study a malware propagation over computer networks where the nodes interact through network-based opportunistic meetings (see Fig. 15 and Table III).
The security level of network is measured as a function of some key control parameters: acceptance/rejection of a meeting, opening/not opening a suspicious e-mail, file or packet.We model the propagation of the virus in network as a sort of epidemic process on a random graph of opportunistic connections [135].A computer/node can randomly get online an infected or non infected data from other computers.An infected computer can be in two states: dormant or fully infected.The non-infected computers are susceptible to be approached by virus coming from infected ones.The possible states are therefore denoted as Dormant (D), Infected/corrupt(C) and Susceptible/Honest (H).The set of types is 1 or 2, also denoted generically as θ, θ .For each type the state may be different except for honest state where it is considered as honest in both regimes of the network.The network size is n ≥ 1.The repartition of the nodes at time step t is denoted as n = D θ (t) + D θ (t) + C θ (t) + C θ (t) + H(t).
The frequency of the states θ is called occupancy measure of the population and is denoted as ) is a random process and its limit measure corresponds to the mean field term.
The goal is understand the impact of the control action on combatting virus spread, which is the minimization of proportion O n (t) := 1 − H n (t)).The interaction is simulated using the following rules: Changes from Dormant States: A node in dormant state (transient) with type θ may become honest with probability δD ∈ (0, 1).A dormant with type θ may opportunistically meet another dormant of type θ , and both become active.This occurs with probability proportional to the frequency of other dormant agent at time t.For type θ, the probability is λ(D n θ (t) − 1 n 1l {θ=θ } ).Note that the dormant can decide to contact the other dormant or not, so there are two possible actions: {m, m} (to meet or not to meet).Those events will be modeled with a Bernoulli random variable with success (meeting) probability δm, which represents u(m|D, θ).
Changes from Corrupt States: A corrupt node may become honest with probability δC .A corrupt node of type θ may become dormant with probability β D n θ (t) q θ +D n θ (t) at time t.Here is assumed that, at high concentrations of dormants, each corrupt node infects at most a certain maximum number of dormant nodes per time step.This reflects the fact a corrupt has a limitation in terms its power, domination and capabilities.The parameter 0 ≤ β ≤ 1 can be interpreted as a maximum contamination rate.The parameter 0 ≤ q θ ≤ 1 is the dormant node density at which the infection spread proceeds.
Changes from Susceptible/Honest states: An honest node may become infected with probability δH + (1 − δH )C n (t).An honest node may become dormant via two ways.First, δSm is the probability of getting corrupt by the network representative node.In this case, the honest node can decide share or not, so there are two possible actions: {o, ō}.This case will be modeled using a coin toss with probability δe ∈ (0, 1).Second, η(D n θ (t) + D n θ (t)) models the probability of meeting a dormant node.Here η ∈ (0, 1).In this case, the dormant node can decide to contact the honest node or not, and it is modeled analogously to the other two cases.
The payoff function is the opposite of the infection level.Each transition described above has a certain contribution to be infection level of the society, which could be 0 if no corrupt or dormant node become honest, −1/n if there is a node which become honest and +1/n if one node is corrupt (D or C).In Table III are the transition probabilities, the contribution to M n (t + 1) − M n (t), the set of actions, and the contribution to information spread in the network.
The drift, that is, the expected change of M n in one time step, given the current state of the system is: Ĥ = h+f1p1 +f2p2 +f3p3 +f4p4 +f5p5.This is a twice continuously differentiable function in m, and ∂m j Ĥ = 5 i=1 [∂m j fi]pi for j ≤ 4. The optimum control strategies at time t are the ones that maximize Ĥ.

Combatting Virus Propagation by means of Individual Action
Let S(t) be the random variable describing the individual state at time t of a generic individual and assume that a generic individual is in a state s at time t.Then S(t+ 1 n ) is independent of previous values (S(t ) : t ≤ t) and as n goes to infinity for all state s .The reward of a generic individual payoff is defined as follows: p θ (s, u, m) = 0 if the individual state s is different than H, and equals 1 if the state s = H.By doing, each individual tries to adjust its own trajectory.
People in honest state will accept less meeting and will set their meeting rate δm to be minimal, and the other individual with state different than H will try to enter to H as soon as possible.As in a classical communicating Markov chain, this is the entry time to state H.
Figure 16 reports the result of the simulation with the following 3 starting points: (d, c) = (0.2, 0.6), (d, c) = (1/3, 1/3) and (d, c) = (0.2, 0).In the three cases, the system converges to the same steady state which is around (d, c) = (0.38, 0.6). Figure 17 plots the reward (honest people) as a function of time for two different control parameters δm = 0.9 and δm = 0.1.We observe that the reward is greater for δm = 0.1 than the one for δm = 0.9.Fig. 16: Proportion of dormant, corrupt and honest (followed by the corresponding time-average trajectory).As time increases, the system approaches a steady state.Network effect The primary advantage of network models is their ability to capture complex individual-level structure in a simple framework.To specify all the connections within a network, we can form a matrix from all the interaction strengths which we expect to be sparse with the majority of values being zero.Usually, for simplicity, two individuals (or populations) are either assumed to be connected with a fixed interaction strength or unconnected.In such cases, the network of contacts is specified by a graph matrix G, where Gij is 1 if individuals i and j are connected, or 0 otherwise.A connection could be a relationship between the two nodes.It may be represent an internet, social network or physical connection.They may not be close in terms of location.The status of an node will be influenced by the status of its connection following the rules specified above.The resulting graph-based mean-field dynamics is illustrated in Figure 18.

Case
Transition proba.(θ, θ ∈ {1, 2}).Application 6 (Cloud Networks): Resource sharing solutions are very important for data centers as it is required and implemented at different layers of cloud networks [83], [136], [137].The resource sharing problem can be formulated as a strategic decision-making problem.Lot of resources may be wasted if the cloud user consider an economic renting.Therefore a careful system design is required when a several clients interact.Price design can significantly improve the resource usage efficiency of large cloud networks.We denote such a game by Gn, where n is the number of clients.The action space of every user is U = R+ which is a convex set, i.e., each user j chooses an action uj that belongs to the set U. An action may represent a certain demand.All the actions together determine an outcome.Let pn be the unit price of cloud resource usage by the clients.Then, the payoff of user j is given by rj(x, u1, . . ., un) = cn(x) h(uj) if n i=1 h(ui) > 0 and zero otherwise.The structure of the payoff function rj(x, u1, . . ., un) for user j shows that it is a percentage of allocated capacity minus the cost for using that capacity.Here, cn(x) represents the value of the available resources (which can be seen as the capacity of the cloud), h is a positive and nondecreasing function with h(0) = 0. We fix the function h to be x α where α > 0 denotes a certain return index.x is the state of cloud networks which is a random variable on the availability of the servers.The cloud game Gn is given by the collection (X , N , U, (rj)j∈N ) where N = {1, . . ., n}, n ≥ 2, is the number of potential users.The next Proposition provides closed-form expression of the Nash equilibrium of the one-shot game Gn for a fixed state x such that cn(x) > 0, pn(x) > 0, and for some range of parameter α.It also provides the optimal price p * n such that no resource is wasted in equilibrium.
Proposition 4: By direct computation, the following results: (i) The resource sharing game Gn is a symmetric game.All the clients have symmetric strategies in equilibrium whenever it exists.(ii) For 0 ≤ α ≤ 1, and x ∈ X , the payoff rj is concave (outside the origin) with respect to own-action uj.The best response BRj(u−j) is strictly positive and is given by the root of where z uj and there is a unique equilibrium (hence a symmetric one) given by z α−1 αcn(x) npn(x) n 2 pn(x) .
It follows that the total demand na * N E (x) at equilibrium is less than cn(x) pn(x) which means that some resources are wasted.
The equilibrium payoff is rj(x, a * N E ) = ujpn(x) n < α one gets that the total demand at equilibrium is exactly the available capacity of the cloud.Thus, pricing design can improve resource sharing efficiency in the cloud.Interestingly, as n grows, the optimal pricing converges to α.
We say that the cloud renting game is efficient if no resource is wasted, i.e., the equilibrium demand is exactly cn(x).Hence, the efficiency ratio is na * N E cn(x) .As we can see from (ii) of Proposition 4, the efficiency ratio goes to 1 by setting the price to p * n .This type of efficiency loss is due to selfishness and have been widely used in the literature of mechanism design and auction theory.Note that the equilibrium demand increases with α, decreases with the charged price and increases with the capacity per user.The equilibrium payoff is positive and if α ≤ 1 each user will participate in an equilibrium.In the Nash equilibrium the optimal pricing p * n depends on the number of active clients in the cloud and value of α.When the active number of clients varies (for example, due to new entry or exit in the cloud), a new price needs to be setup which is not convenient.where θi is the phase of oscillator i, ωi is the natural frequency of oscillator i, n is the total number of oscillators in the system and K is a coupling interaction term.The objective here is to explore phase transition and self organization in large population dynamic systems.We explore the mean-field regime of the dynamical mean-field systems and explain how consensus and collective motion emerge from local interactions.These dynamics have interesting applications in multi-robot coordination.Figure 19 presents a Kuramoto-based synchronization scheme [138].The uncontrolled Kuramoto model can lead to multiple clusters of alignment.Using mean-field control, one can drive the trajectories (phases) towards a consensus as illustrated in Figure 20   Application 8 (Energy-Efficient Buildings): Nowadays a large amount of the electricity consumed in buildings is wasted.A major reason for this wastage is inefficiencies in the building technologies, particularly in operating the HVAC (heating, ventilation and air conditioning) systems.These inefficiencies are in turn caused by the manner in which HVAC systems are currently operated.The temperature in each zone is controlled by a local controller, without regards to the effect that other zones may have on it or the effect it may have on others.Substantial improvement may be possible if inter-zone interactions are taken into account in designing control laws for individual zones [107]- [111].The room/zone temperature evolution is a controlled stochastic process where c1, c2ij, c3 are positive real numbers.The control action ui in room i depends on the price of electricity p(demand, supply, location).The cost for driving to comfort temperature zone (see Figure 21) is (Ti − T i,comf ort ) 2 + var(Ti − T i,comf ort ).The payoff of consumer is a sort of tradeoff between comfort temperature and electricity cost uip.The electricity Fig. 20: A controlled Kuramoto-based synchronization scheme with 500 agents .A mean-field-type control helps to reach a consensus and an agreement independently of the initial distribution of the phases.
price depends on the demand D = I consumption(i)m1(t, di) and supply D = J supply(j)m2(t, dj).m1(t, .) is the population mean-field of consumers, i.e., the consumer distribution at time t.Note that m1 is an unnormalized measure.m2 is the distribution of suppliers.The building is served by a producer whose remaining energy dynamics is Explicit solutions can be obtained using the framework developed in [117].

E. General Engineering
Application 9 ( Online Meeting): Group meeting online, even over video, is much different than sitting in a boardroom commu-nicating face-to-face with someone.But they something in common: deciding to join Early or on Time the group meeting.In the context of online video group meeting, since the communication is over video, the opportunity for miscommunication is much higher, and thus, one should pay close attention to how the group meeting is conducted.Each group member aims to heighten the quality of her online meetings by acting professionally and by signing early or on time: Nothing throws off a meeting worse than scheduling woes.This is in particular widely observed for online group meetings (see Figure 22).Scheduling and synchronization is probably the hardest job in these meetings.The help scheduling groups from different sites can login to the meeting space at their convenience makes it easier to get meetings started on time.However, it does not mean the meeting will start exactly at scheduled time.The group members can decide to be at convenient place early and prepare for the meeting to start, giving you time to settle down and get acquainted with the interface.We examine how agents decide when to join the group meeting in a basic setup.We consider several industry and academia aiming to collaborate on a research development.The companies are located at different sites.Each company from each site has appointed work package leader.In order to improve savings from long business trips, hotels/ accommodation and to reduce jet-lags effect the companies decided to organize an online meeting.After coordinating all the members availability, date and time is found and the meeting is initially scheduled to start at time t.Each member has the starting time in his schedule and calendar remainders but in practice, the online meeting only begin when a certain number n of representative group leaders and group members will connect online and will be seated in these respective rooms.Thus, the effective starting time T of the online meeting is unknown and people organize their behavior as a function of ( t, n, T ).
Each group member can move from her office to the meeting room.The dynamics of agent i is simply given by ẋi = ui, where xi(0) ∈ D. Let n(t) be the number of people arrived (and seated) in the room before t.If the criterion is met (by all groups) before the initially scheduled time t of the meeting, this latter starts exactly at t.If on the other hand the criterion is met at a later time, T is determined by the self-consistency relation: + where ci are non-negative real numbers, and where h(Gn(xi)) ui 2 quantifies a congestion-dependent kinetic energy spent to reach the meeting room of her group.[T − t h ]+ quantifies the useless waiting time, [t h − T ]+ quantifies of the time for missing of beginning of the online meeting,[t h − t]+ quantifies the sensitivity to her reputation of being late at the meeting.Given the strategies (u1, . . ., ui−1, ui+1, . . ., un), of the other agents, the best response problem of i is: Even if h(.) is constant, the agents interact because of a common term: the starting time of the online meeting T, and n(T ) ≥ n.For this reason, the choice of the other agents matters.The best response of agent i solves the Pontryagin maximum principle will at arrive at position xroom, at time t h = 2 xroom−x i (0) Thus, the optimal payoff of agent i starting . The optimal payoff of agent i starting from x at time Knowing that the following two functions: ṽ1(x) = x, p * , with p * * = 1, and ṽ2(x) = c2 ± x − y , with x = y, solves the Eikonal equation, ṽx = 1, one deduces an explicit solution of the Bellman equation: vt and everyone arrives just before the meeting starts.However, the departing time is totally different depending the initially location for the agents.This proves the following result: Proposition 5: The tradeoff value to the meeting room starting from point x at time The next application uses MFTG theoretic modelling for smart cities.
Application 10 (Mobile CrowdSensing): The origins of crowdsourcing goes back at least to the nineteenth century and before [145], [146].Joseph Henry, the Smithsonian's first secretary, used the new networked technology of his day, the telegraph, to crowdsource weather reports from across the country, creating the first national weather map of the U.S. in 1856.Henry's successor, Spencer Baird, recruited citizen scientists to collect and ship natural history specimens to Washington, D.C. by the other revolutionary new technology of the day -the railroad -thus forming the bulk of the Institution's early scientific collections.
Today's mobile devices and vehicles not only serve as the key computing and communication device of choice, but it also comes with a rich set of embedded sensors, such as an accelerometer, digital compass, gyroscope, GPS, ambient light, dual microphone, proximity sensor, dual camera and many others (see all the available sensors on iPhone and Samsung Galaxy).Collectively, these sensors are enabling new applications across a wide variety of domains, creating huge data and give rise to a new area of research called mobile crowdsensing or mobile crowdsourcing [147].Crowd sensing pertains to the monitoring of large-scale phenomena that cannot be easily measured by a single individual.For example, intelligent transportation systems may require traffic congestion monitoring and air pollution level monitoring.These phenomena can be measured accurately only when many individuals provide speed and air quality information from their daily commutes, which are then aggregated spatio-temporally to determine congestion and pollution levels in smart cities.Such a collected data from the crowd can be seen (up to a certain level) as a knowledge, which in turn, can be seen as a public good [148].
A great opportunity exists to fuse information from populations of privately-held sensors to create useful sensing applications will be public good.On the other hand, it is important to model, design, analyze and understand the behavior of the users and their concerns such as privacy issues and resource considerations limit access to such data streams.We describe the main game principles of crowdsensing that offer mechanisms for sharing data from privately held sensors.We present below two MFTGs where each user decides its level of participation to the crowdsensing: (i) public good, (ii) information sharing.
The smartphones are battery-operated mobile devices and sensors suffer from a limited battery lifetime.Hence, there is a need for solutions that will limit the energy consumptions of such mobile Internet-connected objects.Such an involvement is translated into a energy consumption cost.All the data collected from these devices combine both voluntary participator sensing and opportunistic sensing from operators.The data is received by a network of cloud servers.For security and privacy concerns, several information are filtered, anonymized, aggregated and distributions (or mean-field) are computed.The model is a public good game with an extra reward for contributors.When decision-makers are optimizing their payoffs, a dilemma arises because individual and social benefits may not coincide.Since nobody can be excluded from the use of a public good, a user may not have an incentive to contribute to the public good.One way of solving the dilemma is to change the game by adding a second stage in which reward (fair) can be given to the contributors (non-free-riders).The strategic form game with incomplete information denoted by G0, is described as follows: A stochastic state of the environment is represented by x.There are n0 potential participant the mobile crowdsensing.The number n0 is arbitrary, and represent the number of users of the game G0.As we will see, the important number is not n0 but the number of active users (the ones with non-zero effort), who are contributing to the crowdsensing.Each mobile user i equipped with sensing capabilities, can decide to invest a certain level of involvement and effort ui ≥ 0. The action space of user i is Ui = R+.As we will see the degree of participation will be limited so that the action space can be included into a compact interval.The payoff of user i is additive and has three components: a public good component Ḡi(m − R(x)), a resource sharing component R(x) h i (u i ) n 0 j=1 h j (u j ) and a cost component p(x, ui).Putting together, the function payoff is where m = n 0 j=1 uj is the total contribution of all the users, where 1lB(x) is the indicator function which is equal to 1 if x belongs to the set B and 0 otherwise.This creates a discontinuous payoff function.The function Ḡi is a smooth and nondecreasing, R(x) is a random non-negative number driven by x.The discontinuity of the payoffs due the two branches {u : m ≥ R(x)} and {u : m < R(x)} can be handed easily by eliminated the fact that the actions in {u : m ≤ R(x)} cannot be an equilibrium candidates.
Using standard concavity assumption with the respect to owneffort, one can guarantee that the game has an equilibrium in pure strategies.We analyze the equilibrium for Ḡi(z) = aiz α , hi(z) = id(z) = z where ai ≥ 0, and α ∈ (0, 1].For any reward where m * ∈ arg max[ Ḡ(m) − m], there exists a design parameter (ai)i such that the "new" lottery based scheme provides the global optimum level of contribution in the public good.We collect mobile crowdsensing users to form a network in which secondary users who willing to share their throughput for the benefit of the society or their friends and friends' of friends.This can be seen as a virtual Multiple-Inputs-Multiple-Outputs (MIMO) system with several cells, multiple users per cell, multiple antennas at the transmitters, multiple antennas at the receivers.The virtual MIMO system is a sharing network represented by a graph (V, E), where V is the set of users representing the vertices of the social graph and E is the set of edges.To an active connection (i, j) ∈ E is associated a certain value ij ≥ 0. The term ij is strictly positive if j belongs to the altruistic outgoing network of i and i is concerned about the throughput of user j.The first-order outgoing neighborhood of i (excluding i) is Ni,−.Similarly, if i is receiving a certain portion from j then i ∈ Nj,− and ji > 0. In the virtual MIMO system, each user i gets a potential initial throughput T hpi,t during the slot/frame t and can decide to share/rent some portion of it to its altruism subnetwork members in Ni,−.User i makes a sharing decision vector ui,t = (uij,t)j∈N i , where uij,t ≥ 0. The ex-post throughput is therefore Since we are dealing with sharing decisions, the mathematical expressions are not necessarily needed if the output can be observed or measured.Given a measured throughput, A user can decide to share or not based its own needs/demands.The term j∈N i,+ uji,t represents the total extra throughput coming to user i from the other users in Ni,+ (excluding i).The term j∈N i,− uij,t represents the total outgoing throughput from user i to the other users in Ni,− (excluding i).In other word, user i has shared j∈N i,− uij,t to the others.If j / ∈ Ni,− then uij,t = 0 and for all i, uii,t = 0.The balance equation is i.e., the system total throughput ex-post sharing is equal to the system total throughput ex-ante sharing.This means that the virtual MIMO throughput is redistributed and sharing among the users through individual sharing decisions s.Some users may care about the others because he may be in their situation in other slot/day.For these (altruistic) users, the preferences are better captured by an altruism term in the payoff.We model it through a simple and parameterized altruism payoff.The payoff function of i at time t is represented by r1i(x, ui,t, u−i,t) = ri(T hpi,t+) Here, ij ≥ 0 and represents a certain weight on how much i is helping j.The matrix ( ij ) plays an important role in the sharing game under consideration since it determines the social network and the altruistic relationship between the users over the network.The throughput T hp depends implicitly the random variable x.
The static simultaneous act one-shot game problem over the network (V, E) is given by the collection G1, = (V, (R n 1 −1 + , r1i)i).The vector ui is in R n 1 + , but the i-th component is uii = 0. Therefore the choice vector reduces to be in R n 1 −1

+
. and is denoted by (ui,1, . . ., ui,i−1, 0, ui,i+1, . . ., ui,n 1 ) .An equilibrium of G1, in state w is a matrix s ∈ R We analyze the equilibria of G1, .Note that in practice the shared throughput cannot be arbitrary; it has to be feasible.Therefore, the set of actions can be restricted to where ui = (ui,1, . . ., ui,i−1, 0, ui,i+1, . . ., ui,n), and C > 0 is large enough.For example, C can be taken as the maximum system throughput j T hpj,0.This way, the set of sharing actions Ui of user i is non-empty, convex and compact.Assuming that the functions ri are strictly concave, non-decreasing and continuous, we get that the game has at least one equilibrium (in pure strategies).As highlighted above, the set of actions can be made convex and compact.Since ri are continuous and strictly convex, it turns out that, each payoff function ri is jointly continuous and is concave in the individual variable ui (which is a vector) when fixing the other variables.We can apply the well-known fixed-point results which give the existence of constrained Nash equilibria.As we know that G1, has at least one equilibrium, the next step is to characterize them.If the matrix u is an equilibrium of G1, then the following implications hold: uij > 0 =⇒ r i (T hpi,0+) = ij r j (T hpj,0+). ( The equilibria may not be unique depending on the network topology.This is easily proved and it is due to the fact that one may have multiple ways to redistribute depending on the network structure and several redistributions can lead to the same sum T hpi,0 + j uji − j uij.Even if we have a set of equilibria, the equilibrium throughput and the equilibrium payoff turn out to be uniquely determined.The set of equilibria has a special structure as it is non-empty, convex and compact.The ex-post equilibrium throughput increases with the ex-ante throughput and stochastically dominates the initial distribution of throughput of the entire network.For ri = − 1 θ e −θT hp i , θ > 0 let ij = where > 0.Then, the fairness is improved in the network as increases.The topology of the network matters.The difference between the highest throughput and the lowest throughput in the network is given by the geodesic distance (strength) of the multi-hop connection.

IV. TIME DELAYED STATES AND PAYOFFS
This section presents MFTGs with time-delayed state dynamics.Delayed dynamical systems and delayed payoffs appear in many applications.They are characteristic of past-dependence, i.e. their behavior at time t not only depends on the situation at t, but also on their past history and or time delayed state.Some of such situations can be described with controlled stochastic differential delay equations.Networked systems suffer from intermittent, delayed, and asynchronous communications and sensing.To accommodate such systems, time delays need to be introduced.Applications include • The Air Conditioning control towards a comfort temperature is influenced by integrated-state which represents the trend.• Transmission and propagation delay affect the performance of both wireline and wireless networks both delayed information processing and delayed information transmission occur.• In computer network security, the proportion of infected nodes at time t is a function of the delayed state, the topological delay, and the proportion of susceptible individuals and some time delay for the contamination period.• In energy markets, there is an observed phenomenon for the dynamics of the price, which comes with a delayed effect.

A. Time-delayed mean-field game
We consider a mean-field game where agents interact within the time frame T .The best-response of a generic player is where ) l≤I is the Integral state vector of the recent past state over [t − τ, t], This represents the trend of the state trajectory.The process φ l (t, s) is an Fs−adapted locally bounded process.λ is a positive and σ−finite measure.
The function γ : • The filtration Ft is the one generated by the union of events from W or N up time t.
The goal is to find or to characterize a best response strategy to mean-field (m1, m2) : u * ∈ arg maxu∈U G(u, m1, m2).
We will make the following assumption H1 in order to get a wellposed problem.
Hypothesis H1: The functions b, σ, g are continuously differentiable with the respect to (x, m).Moreover, b, σ, g and all their first derivatives with the respect to (x, y, z, m) are continuous in (x, m, u) and bounded.
We explain below why the existing solution approaches cannot be used to solve (26).First, the presence of y, z lead to a delayed integro-McKean-Vlasov and the stochastic maximum principle developed in [27]- [30] does not apply.The dynamic programming principle for Markovian mean-field control cannot be directly used here because the state dynamics is non-Markovian due to the past and time delayed states.Hence, a novel solution approach or an extension is needed in order to solve (26).A chaos expansion methodology can be developed as in [141] using generalized polynomial of Wick and Poisson jump process.The idea is to develop a finite-dimensional optimality equation for (26).In this respect, a stochastic maximum principle could be a good candidate solution approach.Under H1, for each control u ∈ U, m1 and m2 the state dynamics admits a unique solution, x(t) := x u (t).The non-optimized Hamiltonian is H(t, x, y, z, u, m1, m2, p, q, r, ω) : T × X 3 × U × X × U × R 2 × J × Ω → R where r(.) ∈ J and J is the set of functions on Θ such that Θ γ r(t, θ)µ(t, dθ) is finite.The Hamiltonian is H = g0 +bp+σq + Θ γ r(t, θ)µ(dθ).The first-order adjoint process (p, q, r) is time-advanced and determined by p(T ) = g1,x(x(T ), m1(T )).
We now discuss the existence and uniqueness of the first-order adjoint equation.Assuming the coefficients are L 2 , the first order adjoint (28) has a unique solution such that Moreover, the solution (p, q, r) can be found backwardly as follows: • Within the time frame • We fix p(T −τ ) from the previous step and solve (27) on interval and hence, (p(t+τ ), q(t+τ ), r(t+τ, θ)) is known from the previous step.However, p(t + τ ) may not be Ft−adapted.Therefore a conditional expectation with the respect to the filtration Ft is used.
If U is a convex domain, we know that the second-order adjoint processes of Peng's type are not required, and if (x * , u * ) is a best response to m1, m2 then there is a triplet of processes (p, q, r), that satisfy the first order adjoint equation such that H(t, x * , y * , z * , u * , m1, m2, p, q, r) −H(t, x * , y * , z * , u, m1, m2, p, q, r) ≥ 0, for all u ∈ U, almost every t and P−almost surely (a.s.).A necessary condition for (interior) best response strategy is therefore E[Hu | Ft] = 0 whenever Hu makes sense.A sufficient condition for optimality can be obtained, for example, in the concave case: g1, H are concave in (x, y, z, u) for each t almost surely.
Therefore p(t) = c4 on t ∈ [T − τ, T ].For t < T − τ, the processes q and r are zero and p is entirely deterministic and solves By assumption, su i (t, ui, ω) is decreasing in ui and from the above relationship it is clear that p is decreasing with τ.It follows that, if τ1 < τ2, p[τ1](t) > p[τ2](t).We would like to solve su i (t, ui, ω) = p[τ1](t) > p[τ2](t) By inverting the above equation one gets u * i [τ1] < u * i [τ2].Thus, the optimal strategy u * i increases if the time delay τ increases.
This proves the following result: Proposition 6: Time delay decreases the prosumer market price.The optimal strategy u * i increases if the time delay τ increases.

V. DECENTRALIZED INFORMATION AND PARTIAL OBSERVATION
Let F W t be the P-completed natural filtrations generated by W up to t. Set F W := {F W t , 0 ≤ t ≤ T } and F := {Ft, 0 ≤ t ≤ T }, where Ft = F W t ∨ σ(x0).An admissible control ui of agent i is an F W i -adapted process with values in a non-empty, closed and bounded subset (not necessarily convex) Ui of R d and satisfies E[ T 0 |ui(t)| 2 dt] < ∞.Those are nonanticipative measurable functionals of the Brownian motions.Since each agent has a different information structure (decentralized information), let Ui be the set of admissible strategies of i (with decentralized partial information) such that Gi,t ⊂ Fi,t, i.e., Ui := {ui ∈ L 2 G i,T ([0, T ], R d ), ui(t, .)∈ Ui P − a.s} Given a strategy ui ∈ Ui, and a (population) mean-field term m generated by other agents we consider the signal-observation x u i ,m i which satisfies the following stochastic differential equation of mean-field type to which we associate a best-response to mean-field [115], [139], [140]: sup u i ∈U i R(ui, m) subject to dxi(t) = b(t, xi(t), Exi(t), ui(t), m(t))dt +σ(t, xi(t), Exi(t), ui(t), m(t))dWi,t, xi(0) ∼ L(Xi,0), m(t) = population mean-field , g is the terminal cost and l is the running cost.Given m, any u * i ∈ Ui which satisfies R(u * i (•), m) = sup u i (•)∈U i R(ui, m) is called a pure best-response strategy to m, by agent i.In addition to the other coefficient we assume that γ satisfies H1.Under H1, the state dynamics admits a unique strong solution (see [142], Proposition 1.2.)Given m, we apply the SMP for risk-neutral mean-field type control from ( [143], Theorem 2.1) to the state dynamics x to derive the first order adjoint equation.Under the assumption H1, there exists a unique F-adapted pair of processes (p, q), which solves the Backward However, these processes (p, q) may not be adapted to decentralized information Gi,t.This is why their conditioning will appear in the maximum principle below.Again by ( [143], Theorem 2.1), there exists a unique F-adapted pair of processes (P, Q), which solves the second order adjoint equation {2bx(s)P (s) + σ 2 x P (s) + 2σx(s)Q(s) + Hxx(s)}ds such that E sup t∈[0,T ] |P (t)| 2 + T 0 |Q(t)| 2 dt < +∞.Note that in the multi-dimensional setting, the term 2bx(s)P (s) + σ 2 x P (s) + 2σx(s)Q(s) becomes b x P + P bx + σ x P σx + σ x Q + Qσx.

VI. LIMITATIONS AND CHALLENGES
The examples above show that the continuum of agents assumption is rarely observed in engineering practice.The agents are not necessarily symmetric and a single agent may have a non-negligible effect on the mean field terms as illustrated in the HVAC application.Without having a broad set of facts on which to theorize, there is a certain danger of mean-field game models that are mathematically elegant, yet have little connection to actual behavior observed in engineering practice.At present, our empirical knowledge is inadequate to the main assumptions of the classical mean-field game theory.This is why a relaxed version is needed in order to better capture wide ranges of behaviors and constraints observed in engineering systems.MFTG relaxations includes symmetry breaking, mixture between atomic and nonatomic agents, non-negligible effect on individual localized meanfield terms, and arbitrary number of decision-makers.In addition, behavioral and psychological factors should be incorporated for learning and information processes used by people-centric engineering systems.MFTG is still under development and is far from being a well-established tool for engineered systems.
Until now, MFTG was not focused on behavioral and cognitivelyplausible models of choices in humans, robots, machines, mobile devices and software-defined strategic interactions.Psychological and behavioral mean-field type game theories seem to explain behaviors that are better captured in experiments or in practice than classical game-theoretic equilibrium analysis.It allows to consider psychological aspects of the player in addition to the traditional "material" payoff modelling.The value depends upon choice consequences, mean-field states, mean-field actions and on beliefs about what will happen.The psychological MFTG framework can link cognition and emotion.It expresses emotions, guilt, empathy, altruism, spitefulness (maliciousness) of the players.It also include belief-dependent and other-regarding preferences in the motivations.It needs to be investigated how much the psychology of the people matters in their behaviors in engineering MFTGs.The answer to this question is particularly crucial when analyzing the quality-of-experience of the users in terms of MOS (opinion score) values.A preliminary result from a recent experiment conducted in [94], [149] with 47 people carrying mobile devices with WiFi direct and D2D technology shows that the participation in forwarding the data of the users is correlated with their level of empathy towards their neighbors.This suggests the use of not only material payoffs but also non-material payoffs in order to better capture users behaviors.Another aspect of the MFTGs is the complexity of the analysis (both equilibrium and non-equilibrium) when multiple players (and multiple mean-field terms) are involved in the interaction.

VII. CONCLUSION AND FUTURE WORK
The article presented basic applications of mean-field-type game theory in engineering, covering key aspects such as de-congestion in intelligent transportation networks, control of virus spread over network, multi-level building evacuation, next generation wireless networks, incentive-based demand satisfaction in smart energy systems, synchronization and coordination of nodes, mobile crowdsourcing and cloud resource management.It appears from the wide ranges of applications and coverage that mean-field-type game theory is a promising tool for engineering problems.However, the framework is still under development and needs to be improved to capture realistic behavior observed in practice.Possible extensions of the work described in this article include the study of mean-field-type games for risk engineering, and an integrated mean-field-type game framework for smarter cities ranging from transportation to water distribution with ICT (Information Communication Technology), big data and human-in-the-loop among several other interesting directions.

Fig. 1 :
Fig. 1: MFTG with engineering applications covered in this work.

Fig. 3 :
Fig. 3: Optimal cost trajectory as a function of time.

Fig. 4 :
Fig. 4: Optimal vs equilibrium state trajectory over time.The security level induced at equilibrium state is lower than the one induced at full cooperation.

Fig. 5 :
Fig. 5: Evolution of strategies of three agents over time.The imitative mean-field learning converges to a global optimum.

Fig. 6 :
Fig. 6: The imitative mean-field learning converges to a global optimum.

Fig. 10 :
Fig. 10: The two upper Figures plot the evolution of the number of remaining agents in the building.The number of agents in ground floor starts increasing because the flow is coming from first floor until certain time threshold and then decrease when agents start to exit.The lower Figure plots the evolution of the number of agents who have been evacuated safely.The plot has a typical shape of a cumulative distribution function.

Fig. 11 :
Fig.11: A typical large-scale network with regular nodes, relay nodes, primary users and jammers.The star sign represents a Jammer.The blue nodes are active secondary nodes, the nodes in circle are inactive secondary nodes, and the plus sign represents a primary user zone of transmission using MIMO millimeter wave wireless communication.

Fig. 13 :
Fig. 13: Optimal supply S * j of producer j obtained by means of inf-convolution of the Bellman operator

Fig. 14 :
Fig.14: Optimal Allocation k s jk (t) = S * j (t) between the two power stations of producer j at time period t

Fig. 15 :
Fig. 15: Markov chain representation: the parameters si are the complement of the other transitions.
h) , d = d θ +d θ and c = c θ +c θ .Then the limit of f n (m) is f (m).Notice that the sum of the all the components of f (m) is zero.Furthermore, if one of the components mj of m = (d θ , d θ , c θ , c θ , h) is zero then the corresponding drift function fj(m) ≥ 0. As a consequence, in the absence of birth and death process, the 4−dimensional simplex is forward invariant, meaning that if initially m(0) is in the simplex, then for any time greater than 0 the trajectory of m(t) stays in the simplex domain.Centralized control design : We minimize the proportion of node with states C or D by means of controlling u(.|), i.e., by adjusting (δm, δe) ∈ [0, 1] 2 .Since o(t) = c1 + c2 + d1 + d2 = 1 − h(t), minimizing o(t) is equivalent to maximize the proportion of susceptible node in the population.Therefore the optimization problem becomes    sup δe,δm h(T ) + T 0 h(t) dt ṁ = f (m), m(0) = m0 where, m = (c1, c2, d1, d2, h).

Fig. 17 :
Fig. 17: Evolution of Reward (Honest) for the control parameters δm = 0.9 and δm = 0.1.The smaller the meeting/opening rate is the larger the proportion of susceptible nodes.

Fig. 18 :
Fig. 18: Network-based virus propagation: each agent has a certain degree of connections without restriction on the location, capturing virus spread via internet or social media contacts.The average degree of the graph is 4.

1 which is positive for α ≤ 1 . 1 then
(iii) For α > 1, the activity (participation) of user j depends mainly of the aggregate of the others.u * j > 0 only if G ≤ G * and the number of active clients should be less than α α−1 .If n > α α−BRj = 0. (iv) With a participation constraint, the payoff at equilibrium (whenever it exists) is at least 0. (v) By choosing the price p * n = α (n−1)
and H. Tembine are with Learning and Game Theory Lab, New York University Abu Dhabi B. Djehiche is with the Department of Mathematics, KTH Royal Institute of Technology, Stockholm, Sweden

TABLE II :
Key limitations and differences between the game models What MFTGs can bring to the existing decision-making models The functions bi, σi, gi, ri are measurable functions.

TABLE III :
Probabilities , effects (D, C, H), actions and loss function.
− xi 2 )(v − vi)ρ(t − τi, dxdv) dt +c x∈B(x i , ) vρ(t − τi, X , dv) dt Delayed information transmission, where agent i compares its state to the information coming from its neighbor j after some time delay τi.Information transmission delays arise naturally in many dynamical processes on networks.dxi=wi + ρ(t − τi, dx) sin(x − xi(t)) + ui dt + σdWDelayed information transmission has direct applications in opinion dynamics and opinion formation on social graph: •