Large population games with interactions through controls and common noise: convergence results and equivalence between $open$--$loop$ and $closed$--$loop$ controls

In the presence of a common noise, we study the convergence problems in mean field game (MFG) and mean field control (MFC) problem where the cost function and the state dynamics depend upon the joint conditional distribution of the controlled state and the control process. In the first part, we consider the MFG setting. We start by recalling the notions of $measure$--$valued$ MFG equilibria and of approximate $closed$--$loop$ Nash equilibria associated to the corresponding $N$--player game. Then, we show that all convergent sequences of approximate $closed$--$loop$ Nash equilibria, when $N \to \infty,$ converge to $measure$--$valued$ MFG equilibria. And conversely, any $measure$--$valued$ MFG equilibrium is the limit of a sequence of approximate $closed$--$loop$ Nash equilibria. In other words, $measure$--$valued$ MFG equilibria are the accumulation points of the approximate $closed$--$loop$ Nash equilibria. Previous work has shown that $measure$--$valued$ MFG equilibria are the accumulation points of the approximate $open$--$loop$ Nash equilibria. Therefore, we obtain that the limits of approximate $closed$--$loop$ Nash equilibria and approximate $open$--$loop$ Nash equilibria are the same. In the second part, we deal with the MFC setting. After recalling the $closed$--$loop$ and $open$--$loop$ formulations of the MFC problem, we prove that they are equivalent. We also provide some convergence results related to approximate $closed$--$loop$ Pareto equilibria.


Introduction
Our primarily goal in this paper is to discuss the convergence problem of closed-loop Nash equilibria in the setting of mean field game of controls (MFGC) or extended mean field game. Let us briefly explained the mathematical framework that we consider. The full details explanation are given in Section 2.1. We consider that N players have private state processes X N := (X 1 , . . . , X N ) given by the stochastic differential equations (SDEs) system where T > 0 is a fixed time horizon, (B, W 1 , . . . , W N ) are independent Brownian motions where B is called the common noise, and α i is a Borel measurable function playing the role of the control of player i. An important feature here is the presence in the dynamics X i of player i of the empirical distribution ϕ N [α N ] of states and controls of all players. Given a strategy (α 1 , . . . , α N ), the reward to the player i is For ε N ≥ 0, the strategy (α 1 , . . . , α N ) will be called an ε N -closed-loop Nash equilibrium if for any admissible control β, and each i ∈ {1, · · · , N }, The presence of the term closed-loop indicates the fact that we consider controls which are Borel maps of [0, T ]×(R n ) N into U . The convergence problem here consists in characterizing the Nash equilibria when the number of players N goes to infinity. It is now well known that, when N tends to infinity, the Nash equilibria are related to the MFG here called mean field game of controls (MFGC) or Extended mean field game, which has the following structure (the precise definition is given in Section 2.2.1): for ε ≥ 0, a (σ{B s , s ≤ t}) t∈[0,T ] -predictable measure-valued process (µ t ) t∈[0,T ] is an ε-strong Markovian MFG equilibrium (or approximate strong Markovian MFG equilibrium) if for all t ∈ [0, T ], µ t = L(X t , α(t, X t , µ t )|B), where the state process X is governed by dX t = b t, X t , µ t , α(t, X t , µ t ) dt + σ t, X t dW t + σ 0 dB t , , t ∈ [0, T ] µ t := L(X t |B), where the optimization is over the solutions dX ′ t = b t, X ′ t , µ t , α ′ (t, X ′ t , µ t ) dt + σ t, X ′ t dW t + σ 0 dB t . (1.4) Discussion about the problematic and related studies The now well-known MFG was introduced in the seminal works of Lasry and Lions [45] and Huang, Caines, and Malhamé [35] as a way of studying the N -player game described here above, when the number of players N is very large. Since these pioneering works, this topic has been the subject of much research in the field of applied mathematics, in particular for its wide variety of application (see Carmona and Delarue [13] for examples of applications of MFG). The initial formulations of MFG did not consider the presence of the empirical distribution of controls like that formulated above in Equation (1.1).
Only the presence of the empirical distribution of states was envisaged. In order to deal with some modeling issues occurring in finance for instance, a natural "extension" of MFG known as extended MFG or MFG of controls has been formulated and studied by many studies these recent years, see Gomes and Voskanyan [31], Carmona and Lacker [15], Cardaliaguet and Lehalle [10], Gomes, Patrizi, and Voskanyan [32], Bonnans, Hadikhanloo, and Pfeiffer [8], Graber [33], Alasseur, Ben Taher, and Matoussi [1], Kobeissi [36]. Our study in this paper treats of extended MFG (or MFG of controls) by giving some convergence and equivalence results.
As mentioned in the beginning, our primarily goal is to make a rigorous connection between the N -player game and the extended mean field game. More precisely, ideally, we want to show two main results. First, the convergence result i.e. given an ε N -Nash equilibrium α N := (α 1,N , · · · , α N,N ), the sequence of empirical distribution of states and controls (ϕ N [α N ]) N ∈N * "converges" to a solution of the MFG of controls when N → ∞, with lim N →∞ ε N = 0. Second, the converse convergence result i.e. any solution of the MFG of controls is the "limit", when N → ∞, of a sequence of empirical distribution of states and controls (ϕ N [α N ]) N ∈N * associated to an ε N -Nash equilibrium α N := (α 1,N , · · · , α N,N ), for some ε N satisfying lim N →∞ ε N = 0.
Establishing this type of connection justifies the interpretation of MFG as the right limit formulation of the N -player game. Besides the presence of the empirical distribution of states and controls, another important feature of our setting is the consideration of closed-loop controls i.e. controls depending on the position of the players. Indeed, the other type of controls usually considered is the open-loop controls i.e. controls adapted to the filtration generated by the initial values and the Brownian motions. When N (≥ 2, the number of players) is fixed, the situation generated by these two concepts of equilibrium is very different, see the discussion in [13, Section 2.1.2].
In the setting of open-loop controls, the connection between N -player game and MFG (convergence result and converse convergence result) is now well-known and established. In the situation without the empirical distribution of controls, under relatively general assumptions, a complete picture has been proposed by Fisher [29] and Lacker [38] (for the case with common noise) using the notions of relaxed MFG equilibrium. For the extended MFG, while allowing the volatility to be controlled, a similar study to [38] is provided by Djete [23] thanks to the notion of measure-valued MFG equilibrium. With stronger assumptions but by providing convergence rates, Laurière and Tangpi [46] study these convergence problems in the situation without common noise using notably a notion of backward propagation of chaos.
The convergence problems in the case of closed-loop controls turn out to be much more problematic than in the case of open-loop controls. Indeed, as discussed in [13,Section 2.1.2], open-loop Nash equilibrium and closed-loop Nash equilibrium behave differently. Without taking into account the empirical distribution of states and controls (no extended in Theorem 2.10 proves that the notion of approximate strong Markovian MFG equilibrium is the correct infinite players formulation equivalent to the notion of measure-valued MFG equilibrium or more generally the correct infinite players formulation equivalent to any notion of weak MFG equilibrium. In other words, considering measure-valued MFG equilibrium (or weak MFG) is equivalent to considering approximate strong Markovian MFG equilibrium. The approximate strong Markovian MFG equilibrium has the advantage to be close to the usual notion considered in the literature. The only difference is for the approximate strong Markovian MFG equilibrium, the optimality is not an exact one but an ε-optimality (see above Equation (1.4)). In a second time, Theorem 2.12 shows that any measurevalued MFG equilibrium is the limit of approximate closed-loop Nash equilibria. Even in the framework of classical MFG, this result seems to be the first of this kind. Indeed, the converse convergence result of [42] needs to be done with a slight extension of the notion of closed-loop Nash equilibria which is called in [42] S-closed loop Nash equilibria. Our result shows that it is possible to avoid this extension. Mention that Theorem 2.10 and Theorem 2.12 are done without any use of convexity assumptions.
In the second part, we treat the case of extended mean field control problem and obtain Theorem 2.14 and Theorem 2.15. In Theorem 2.14, we show that the open-loop and the closed-loop formulation of the extended mean field control problem are exactly the same. A notable point in this result is the fact that there is no use of convexity assumptions. This appears to be the first such result, even for the classical mean field control problem. Next, in Theorem 2.15, we show some limit theory results, namely: any sequence of approximate Pareto equilibria (see Definition 2.3) is relatively compact and any limit point is the limit of sequence of approximate strong Markovian McKean-Vlasov (see Section 2.3.1).
In the assumptions we use in this paper, the condition: σ 0 is invertible needs to be underlined. Although we believe that there must be a way of proving our results without this condition, in this article, this condition is important. Indeed, under this assumption, the filtration generated by the conditional distribution of state (µ t ) t∈[0,T ] is very "close" to the filtration generated by the common noise (B t ) t∈[0,T ] . They are even equal when the control α has some regularities α (see for instance approximation in Proposition A.10 and Remark A.11). This fact is actually quite classic. Indeed, we know that when a volatility σ is non-degenerated, the unique strong solution X of dX t =b(t, X t )dt + σ(t, X t )dW t , X 0 = 0, has his natural filtration equal to the filtration generated by the Brownian motion W. This kind of phenomenon appears in our framework. Making the presence of a common noise obligatory allows us to prove our results especially the converse convergence result and the equivalence between closed-loop and open-loop formulations for the MFC problem. This effect of the common noise has been observed by many authors for various purposes (see for instance Delarue [19], Tchuendom [52], [3], Delarue and Vasileiadis [20]).
Outline of the paper The paper is structured as follows. After introducing some notations, Section 2 introduces both the MFG and MFC frameworks, defines all the notions and concepts of equilibrium, and states the main results of the article. Section 3 is devoted to the proofs. Namely, first, Section 3.1 provides, over a canonical space, an equivalence of our notions appropriate for the proofs. Then, Section 3.2 characterizes the limit of approximate closedloop Nash equilibria. Next, Section 3.3 characterizes the limit of approximate strong Markovian MFG equilibria and provides the converse convergence result. Finally, Section 3.4 gives the proofs relating to the McKean-Vlasov control problem.
Notations. (i) Given a Polish space (E, ∆) and p ≥ 1, we denote by P(E) the collection of all Borel probability measures on E, and by P p (E) the subset of Borel probability measures µ such that E ∆(e, e 0 ) p µ(de) < ∞ for some e 0 ∈ E. We equip P p (E) with the Wasserstein metric W p defined by [54,Theorem 6.18]). For any µ ∈ P(E) and µ-integrable function ϕ : E → R, we define and for another metric space (E ′ , ∆ ′ ), we denote by µ ⊗ µ ′ ∈ P(E × E ′ ) the product probability of any (µ, µ ′ ) ∈ P(E) × P(E ′ ).
Given a probability space (Ω, F , P) supporting a sub-σ-algebra G ⊂ F then for a Polish space E and any random variable ξ : Ω −→ E, both the notations L P (ξ|G)(ω) and P G ω • (ξ) −1 are used to denote the conditional distribution of ξ knowing G under P.
(ii) For any (E, ∆) and (E ′ , ∆ ′ ) two Polish spaces, we use C b (E, E ′ ) to denote the set of continuous functions f from E into E ′ such that sup e∈E ∆ ′ (f (e), e ′ 0 ) < ∞ for some e ′ 0 ∈ E ′ . Let N * denote the set of positive integers. Given non-negative integers m and n, we denote by S m×n the collection of all m × n-dimensional matrices with real entries, equipped with the standard Euclidean norm, which we denote by | · | regardless of the dimensions. We also denote S n := S n×n , and denote by 0 m×n the element in S m×n whose entries are all 0, and by I n the identity matrix in S n . Let k be a positive integer, we denote by C k b (R n ; R) the set of bounded maps f : R n −→ R, having bounded continuous derivatives of order up to and including k. Let f : R n −→ R be twice differentiable, we denote by ∇f and ∇ 2 f the gradient and Hessian of f . (1.5) We will say that a Borel measurable function h :

Setup and Main results
In this section, we first introduce a N -player game, and the definition of ε N -Nash and ε N -Pareto equilibria. Next, we formulate the notions of approximate strong Markovian and measure-valued MFG equilibria which will be essential to describe the limit of the Nash equilibria. Finally, we give the open-loop and closed-loop formulations of the McKean-Vlasov optimal control or MFC and use them to deal with the limit of Pareto equilibria.
The general assumptions used throughout this paper are now formulated. The dimension n ∈ N * , the nonempty Polish space (U, d), the horizon time T > 0 are fixed and P n U denotes the space of all Borel probability measures on R n ×U i.e. P n U := P(R n × U ). Also, we set p ≥ 2, ν ∈ P p ′ (R n ) with p ′ > p, and the probability space (Ω, H := (H t ) t∈[0,T ] , H, P) 1 . We give ourselves the following bounded Borel measurable functions Assumption 2.1. (i) σ 0 ∈ S n×n is an invertible constant matrix and U ⊂ R q , for q ∈ N * , is a compact convex nonempty set; (ii) The maps b and σ are Lipschitz in all their variables. Also, the maps L and g are s.t. for each t, (iii) Non-degeneracy condition: for some constant θ > 0, one has, for all (t, x) ∈ [0, T ] × R n , θI n ≤ σσ ⊤ (t, x); (iv) Separability condition: There exist Borel functions

Remark 2.2.
The previous assumptions are standard in the probabilistic approach of mean field game and control problems. The separability condition is more specific to the extended mean field game and control problems (see Carmona and Lacker [15], Cardaliaguet and Lehalle [10], Laurière and Tangpi [46], Possamaï and Tangpi [46], Djete [23]). It is mainly used for technical reasons. Notice that some conditions can be weakened. But, in order to avoid certain unnecessary technicalities in the proofs, we have chosen these.

The N-player games
Let N ∈ N * . We denote by A c N the collection of all progressively Borel measurable functions α : [0, T ]×(C n ) N → U. On the filtered probability space (Ω, H, H, P), let (W i ) i∈N * be a sequence of independent H-adapted R n -valued Brownian motions, B be an R n -valued H-adapted Brownian motion and (ξ i ) i∈N * a sequence of iid H 0 -random variables of law ν. Besides, (W i ) i∈N * , B and (ξ i ) i∈N * are independent. Then, given the control rule/strategy α := (α 1 , . . . , α N ) ∈ (A c N ) N , denote by X α := (X α,1 · , . . . , X α,N · ) the processes satisfying: The reward value of player i associated with control rule/strategy α := (α 1 , . . . , α N ) is then defined by Now, we give the precise definition of what we call approximate closed-loop Nash and Pareto equilibria.

Remark 2.4. (i)
Because of the fact that the controls (α 1 , · · · , α N ) are only Borel measurable, the strong existence and uniqueness of (2.1) are not standard. We refer to Veretennikov [53] for the well-posedness of this type of SDE when α i (t, . For the general situation, we consider existence and uniqueness in law of (2.1) by a classical application of the Girsanov's theorem. Let us mention that, as weak solution, X may not be defined only on (Ω, H, P). But for ease of reading and to avoid heavy notations, we assumed (Ω, H, P) in order to be able to define all our variables on this space.
(ii) While the existence of approximate Pareto equilibria is obvious, that of approximate Nash equilibria is unclear. As a consequence of our results, we will see that when ε > 0, the approximate Nash equilibria are well-defined under our assumptions.
(iii) Given our framework, the natural shape for our controls should be M arkovian that is to say α i (t, x 1 , · · · , x N ) = α i (t, x 1 (t), · · · , x N (t)) and not fully path dependent as we consider. In the MFG setting, although the establishment of the convergence of approximate Markovian Nash equilibria to MFG can be done with our techniques, we are unfortunately only able to establish the converse convergence result for fully path dependent controls (see Theorem 2.12). Therefore, we make the choice to present our article only with fully path dependent controls. This technical limitation seems to be a strange phenomenon highlighted in other articles (see [42; 41]), and only appears for the mean field game setting.

The mean field game of controls
We first formulate here the formulation of the extended mean field game (or mean field game of controls). We will call this formulation (approximate) strong Markovian MFG equilibrium. Second, inspired by [23], we give the notion of measure-valued MFG which generalizes the strong Markovian MFG equilibrium.

ε-strong Markovian MFG equilibrium
Let (B, W ) be an R n × R n -valued H-Brownian motion, and ξ be a F 0 -random variable ξ such that L(ξ) = ν. We denote by  [12; 14; 41]). Secondly, for the construction of approximate Nash equilibrium from MFG equilibrium, in MFG of control setting, this formulation gives a more natural way. Indeed, when α is regular enough (this will be our case), for each N ∈ N * , an approximate Nash equilibrium is constructed by defining: Due to the appearance of α i in both side of the equality, it is unclear that this construction is well defined.
(iii) The well-posedness of this notion of equilibrium is not clear at first sight. But, as we will see later (see Proposition 2.9 with Theorem 2.12), for any ε > 0, there exists an ε-strong Markovian MFG equilibrium.

Measure-valued MFG equilibrium
Definition 2.7 (measure-valued equilibrium). We say that a term one has

Limit theorem and converse limit theorem
We are now ready to formulate the main convergence results of this paper regarding the extended mean field game. We begin with the convergence of sequence of approximate Nash equilbria and approximate strong Markovian MFG equilibria towards the measure-valued MFG equilibria. The proof is given in Section 3.2.2 and Section 3.3.2. Theorem 2.10 (limit theorem). Let Assumption 2.1 hold true.
• For each N ∈ N * , let α N := (α 1,N , . . . , α N,N ) be an ε N -closed-loop Nash equilibrium, then the sequence ε N = 0, then for each limit point P ∞ there exists a measure-valued solution Ω, F , P, F, W, B, X, Λ, µ such that Remark 2.11. In the framework of extended mean field game, these convergence results seem to be the first of this type. Unlike the recent paper of [49] where the setting is without common noise, σ constant and strong assumptions, in our paper, we treat the case with common noise and the assumptions are less strong.   [41] and later in [42], in the presence of multiple MFG equilibria, this kind of converse result is delicate to prove without extension of the notion of closed-loop Nash equilibrium (see [42]). To bypass this difficulty, the condition σ 0 is invertible turns out to be quite useful.

The McKean-Vlasov control problem
Open-loop formulation Let A o be the collection of all U -valued processes α = (α s ) 0≤s≤T which are F-predictable. Then, given α ∈ A o , let X o,α be the unique strong solution of the SDE: Then, given α ∈ A c , let X c,α be the process satisfying: We will say that X c,α is an closed-loop McKean-Vlasov process associated to α. The closed-loop McKean-Vlasov control problem is given by (2.8)

Equivalence and limit theory results
We now provide here the equivalence result and the limit theory relating to the McKean-Vlasov control problem. The proofs are in Section 3.4.1 and Section 3.4.2.
McKean-Vlasov process associated to α ℓ , and one has Consequently, Recall that the definitions of ϕ N and ϕ N are given in Equation (2.1).

Measure-valued equilibrium: a canonical formulation
In this section, we present equivalent formulations of the measure-valued MFG equilibrium and McKean-Vlasov control problem over a canonical space. These formulations turn out to be those used in [23] and [25] to manage the open-loop framework. Therefore, we will see from our proofs that the set of limits of open-loop and closed-loop controls is the same. These formulations have the advantage to facilitate the presentation of the proofs.

Measure-valued control rules
Denote by M := M P n U the collection of all finite (Borel) measures q(dt, de) on [0, T ]×P n U , whose marginal distribution on [0, T ] is the Lebesgue measure ds i.e., q(ds, de) = q(s, de)ds for a measurable family (q(s, de)) s∈[0,T ] of Borel probability measures on P n U . Let Λ be the canonical element on M. We then introduce a canonical filtration . For each q ∈ M, one has the disintegration property: q(dt, de) = q(t, de)dt, and there is a version of the disintegration such that (t, q) → q(t, de) is F Λ -predictable.
The canonical element on Ω : with Λ ′ t∧· and Λ t∧· denote the restriction of Λ ′ and Λ on [0, t] × P n U (see definition 1.5). Notice that we can choose a version of the disintegration Λ(dν, dt) We consider L the following generator: for (t, x,ν, u) ∈ [0, T ] × R n × P n U × U , and ϕ ∈ C 2 (R n ) Also, for every f ∈ C 2 (R n ), let us define N t (f ) : and for each π ∈ P(R n ), the Borel set Z π by Z π := ν ∈ P n U :ν(dx, U ) = π(dx) .
Definition 3.1 (measure-valued control rule). We say that P ∈ P(Ω) is a measure-valued control rule if: (ii) (B t ) t∈[0,T ] is a (P, F) Wiener process starting at zero and for P-almost every ω ∈ Ω, N t (f ) = 0 for all f ∈ C 2 b (R n ) and every t ∈ [0, T ].
We shall denote P V the set of all measure-valued control rules.
Remark 3.3. The set of measure-valued control rules P V that we introduced is the same as the one used in [23]. However, as we will see later (see Proposition 3.6 and Remark 3.7), for the definition of measure-valued MFG equilibrium, only the case where B is (σ{µ t∧· , Λ t∧· }) t∈[0,T ] -adapted matters.
The following result is one of the key steps to understanding the measurability property satisfied by the Brownian motion B. Lemma 3.4. For any P ∈ P V , there exists a continuous function ϕ : Proof. Let ( Ω, F, P) be an extension of (Ω, F, P) supporting an R n -valued F-Brownian motion W, and a F 0 -random variable ξ s.t. L P (ξ) = ν. Besides, W, ξ and G T are independent. Given (µ, Λ, Λ ′ ), let X ′ be the solution of By the uniqueness of the previous equation, we can check that L P (X ′ t |G t ) = µ ′ t , P-a.e. for all t ∈ [0, T ] (see similar arguments in Proposition A.9 ). By taking the conditional expectation, we find that Therefore, the function ϕ is defined by This is enough to conclude.
The next Lemma is essentially an application of Proposition A.5. It basically indicates that we can replace Λ ′ , which plays the role of a control, by a sequence of more regular controls. This fact will be useful to show the canonical formulation of the measure-valued MFG equilibrium (see Proposition 3.6) and to deal with the convergence of Nash equilibria (see Proposition 3.10).

Lemma 3.5.
There exists a sequence of continuous functions ( β j ) j∈N * satisfying: for each j ∈ N * ,

s.t. if we define X ′j the solution of
Proof. It is enough to apply Proposition A.5 and using the information that P ′ B t = ϕ t (µ t∧· , µ t∧· , Λ t∧· , Λ t∧· ), t ∈ [0, T ] = 1 i.e. B is a continuous function of (µ, Λ). Now, using the measure-valued control rules, we give an equivalent definition of the measure-valued MFG equilibrium.

MFG equilibrium on the canonical space
Let us introduce the set of measure-valued equilibrium Proposition 3.6. The probability measure Q ⋆ belongs to S ⋆ if and only if Q ⋆ = P ⋆ • (µ, Λ) −1 where P ⋆ ∈ P V , and for every P ∈ P V such that L P ⋆ µ, Λ, B = L P µ, Λ, B , one has
Next, using the same measure-valued control rules, we define an equivalent formulation of the McKean-Vlasov optimal control.

Proposition 3.8. We have the following reformulation of
where J is defined in Equation

Limit of Nash equilibria
This section is devoted to the analysis of the behavior of sequence of closed-loop Nash equilibria when the number of players goes to infinity. At the end of this section, we will show that any limit point of sequence of Nash equilibria is a measure-valued MFG equilibrium.

Technical results
Recall that (Ω, H, P) is a filtered probability space supporting a H 0 -random variable ξ s.t. L P (ξ) = ν, and an R n × R n -valued H-Brownian motion (W, B). Besides, let µ be a H-adapted P(R n )-valued continuous process and Λ be a P(P n U )-valued H predictable process s.t. B is G := (G t ) t∈[0,T ] -adapted where G t := σ{µ t∧· , Λ t∧· }. Besides, (µ, Λ) is P-independent of (ξ, W ). For a continuous function let X be the unique strong solution of: a.e. and Λ ′ := δ µ ′ t (dν)dt. Notice that the process (3.8) is exactly the one used for the approximation of measure-valued control rule in Lemma 3.5. In what follows, we will show that it is possible to approximate this type of process by a sequence of interacting processes when a certain condition (see Equation (3.10)) is satisfied (see Lemma 3.9).
Proof. The proof is largely inspired by [41, Proof of Proposition 5.6]. We use some probability changes on (Ω, H, P). Notice that, Y i,i satisfies

Now, let us introduce
By uniqueness, we can check that Next, let us consider the sequence ( By using the fact that (b, σ) is bounded and the non-degeneracy of σ, with similar arguments to [16, Proposition A.1, Proposition A.2, Proposition B.1], it is straightforward to check that the sequence (Q N ) N ∈N * is relatively compact in W p (recall that L P (ξ) = ν ∈ P p ′ (R n ) with p ′ > p). Denote Q ∞ the limit of a sub-sequence, for sake of simplicity, we will use the same notation for the sequence and its sub-sequence. We now want to identify the limit. Mention that Af (s, x(s), w(s), z(s), b(s), π, q, m, u) q s (dm, du)ds.
We denote by Ω := C n × C n × C n × C 1 × C n W × M(P n U × U ), ( X, W , B, Z, µ, Φ, γ) the canonical processes, and F its canonical filtration. Also, we set (Π, µ, γ, B) the canonical element of Ω := P( Ω)×C n W ×M(P n U )×C n . Let 0 ≤ s ≤ t ≤ T and a continuous bounded function h : Ω → R, we define Let X α,i = X α,i − σ 0 B. By Itô formula, we know that df ( X α,i 1 2 Tr and Notice that for dt ⊗ dP-a.e. Then, in the sub-sequence (Q N ) N ∈N * , we can use ζ i, N,β t or ϕ N t [α] without affecting the limit Q ∞ . With all the previous observations, one can check that Notice that the bounded character of the coefficients (b, σ) and test functions (f, h) is important for passing to the limit. This is true for all (f, h). We can then use an appropriate countable set of maps of type f and g, and deduce that Q ∞ ω-a.e., for each ϕ bounded twice differentiable, Mϕ is a (Π(ω), and P ω • X, W , B, Z, µ(ω), Φ, γ(ω) −1 = Π(ω). Now, we define the following change of probability σ(r, X r ) −1 b r, X r ,ν, β(r, X r , µ(ω), γ(ω)) − b r, X r ,ν, u M (dν, du, dr).

Let us introduce
b r, X r ,ν, β(r, X r , µ(ω), γ(ω)) − b r, X r ,ν, u M (dν, du, dr), then, by Girsanov's theorem, V is a ( F, Q ω )-Brownian motion, and X satisfies (3.12) By Equation (3.11), one finds that By uniqueness of (3.12) (and Equation (3.8)), we deduce that This is true for any limit point Q ∞ , then we have the convergence of the entire sequence (Q N ) N ∈N * . Therefore Now, by combining the previous proposition and Lemma 3.5, we show that any measure-valued control rule satisfying a certain condition ( see Equation (3.13)) is the limit of a sequence of interacting processes.
Then by Lemma 3.9, if ( β) α,i is defined in (3.9), one has By combining our previous results, we get the proof of the proposition.

Limit set of approximate strong Markovian MFG equilibria and approximation of measure-valued MFG equilibrium
In this section, first, we will prove the second part of Theorem 2.10 namely showing that the limit set of approximate strong Markovian MFG solution is the measure-valued MFG equilibrium. Second, we will prove the results of Theorem 2.12.
To achieve these goals, we need to introduce the notion of approximate open-loop MFG equilibrium. Let F B := (F B t ) t∈[0,T ] be the natural filtration of B i.e. F B t := σ{B s : s ≤ t}. Let α be a (σ{X 0 , W t∧· , B t∧· }) t∈[0,T ] -predictable process. We denote by X the unique solution of dX t = b t, X t , µ t , α t dt + σ(t, X t )dW t + σ 0 dB t with X 0 = ξ, µ t := L(X t |F B t ) and µ t := L(X t , α t |F B t ). (3.14) Given µ, let α ′ be another (σ{X 0 , W t∧· , B t∧· }) t∈[0,T ] -predictable process and X ′ be the solution of a.e. Let ε ≥ 0, we say that the McKean-Vlasov process X associated to the control α is an ε-open-loop MFG equilibrium if : recall that J is defined in Equation (3.5),

Approximate strong Markovian MFG equilibrium as an approximate open-loop MFG equilibrium
The next Proposition show that any approximate strong Markovian MFG equilibrium can be seen as an approximate open-loop MFG equilibrium.
Proof. As X is a strong solution of Equation (2.2) and µ is F B -adapted, (α(t, X t , µ)) t∈[0,T ] is a (σ{X 0 , W t∧· , B t∧· }) t∈[0,T ]predictable process. To prove our proposition, it suffices to show the optimality condition. Let α ′ be a (σ{X 0 , W t∧· , B t∧· }) t∈[0,T ]predictable process and X ′ be the solution of Equation (3.15). Let us introduce a.e. Now, we can apply Proposition A.5. By Proposition A.5, there exists a sequence of functions (β k ) k∈N * satisfying: x, b, q) ∈ U is progressively measurable and Lipschitz in (x, b, q) uniformly in t such that if we let X ′k be the unique strong solution of: with µ ′k t := L(X ′k t |F B t ) and µ ′k t := L X ′k t , β k (t, X k t , B, Λ)|F B t then lim k→∞ µ ′k , δ µ ′k t (dν)dt = µ ′ , δ µ ′ t (dν)dt , P-a.e., for the Wasserstein metric W p .
With the same techniques used in Lemma 3.4, we can show that B is (σ{µ t∧··· , Λ t∧· }) t∈[0,T ] -adapted. As a result, we can find a progressively Borel measurable functionβ k : [0, T ] × R n × C n W × M → U such that β k (t, X ′k t , B, Λ) = β k (t, X ′k t , µ, Λ) a.e. Consequently, as µ is an ε-strong Markovian MFG equilibrium, one has This is enough to conclude.

Proof of Theorem 2.10 (limit set of approximate strong Markovian MFG equilibria)
Let (µ ℓ ) ℓ∈N * be a sequence s.t. for each ℓ ≥ 1, µ ℓ is an ε ℓ -strong Markovian MFG equilibrium for ε ℓ ≥ 0. By Proposition 3.11, µ ℓ can be seen as an ε ℓ open-loop MFG equilibrium. Let (P ℓ ) ℓ∈N * be the sequence defined by It is enough to apply [23,Theorem 2.12] to conclude that the sequence (P ℓ ) ℓ∈N * of approximate open-loop MFG equilibria is relatively compact in W p and that, when lim ℓ→∞ ε ℓ = 0, each limit point P is such that P • (µ, Λ) −1 ∈ S ⋆ . An idea of the proof of [23,Theorem 2.12] is provided in [23,Section 3]. First, there is an identification of any limit point as a measure-valued control rule satisfying µ = µ ′ and Λ = Λ a.e. under the corresponding measure. Second, any measure-valued control rule can be approximated by a sequence of controlled processes of type (3.15). The second point is used to show the optimality condition.

Approximation of approximate open-loop MFG equilibrium by approximate strong Markovian MFG equilibrium
In this part, we show that any approximate open-loop MFG equilibrium is the limit of a sequence of approximate strong Markovian MFG equilibria.
Let X be an ε-open-loop MFG equilibrium associated to the control α. Recall that µ and µ are defined in Equation (3.14).
Recall that (β 1,N , · · · , β N,N ) is defined in Equation (3.16). Proof. Let us define There exists a sequence of controls (κ By using the same technique as in the proof of Lemma 3.9, one has for each i ∈ {1, . . . , N }, N ,1 , . . . , X α i,N ,N ), and we introduce By uniqueness, we can check that where we used Proposition 3.14. A consequence of (3.17) is lim With the previous results, one has lim N →∞ Therefore, using all previous observations and as µ is an ε-strong markovian MFG equilibrium, so can be seen as an ε-open-loop MFG equilibrium, To obtain the result as formulated, observe that by symmetry ε N := c i,N = c 1,N . This is enough to conclude.
Proof of Theorem 2.12 (measure-valued MFG via approximate Nash equilibria) Let P ∈ P V such that P • (µ, Λ) −1 ∈ S ⋆ . By Theorem 2.12 (first part) (see also Proposition 3.13), there exist (ε k ) k∈N * ⊂ (0, ∞) with lim k→∞ ε k = 0 and a sequence (β k ) k∈N * s.t. µ k is an ε k -strong Markovian MFG equilibrium associated to the control β k with β k Lipschitz in (x, π) uniformly in t, and To conclude, it is enough to approximate approximate strong Markovian MFG equilibrium by approximate Nash equilibria. This is done by Proposition 3.15.

Proof of Theorem 2.15
Let ( P N ) N ∈N * ⊂ P(C n W × M × C n ) be the sequence defined by If we denote by ( µ, Λ, B) the canonical process of C n W × M × C n . Notice that P N = P N • ( µ, Λ) −1 , where P N is the sequence given in Theorem 2.15. By [25,Proposition 3.4], ( P N ) N ∈N * is relatively compact in W p , and for each limit point P ∞ , there exists a sequence of (σ{ξ, W t∧· , B t∧· }) t∈[0,T ] -predictable processes (α k ) k∈N * s.t. if X k is the solution of We see that, for each k ∈ N * , X k is an open-loop McKean-Vlasov process associated to α k . By Theorem 2.14 (see also Proposition 3.13), we can approximate this open-loop McKean-Vlasov process by a sequence of closed-loop McKean-Vlasov processes. To conclude the proof, it is enough to see that by Proposition 3.14, we can approach any closed-loop McKean-Vlasov process by a sequence of interacting processes. Proposition 3.14 we give exactly the sequence that we want for our Theorem 2.15.
, first of all, by the previous result, we observe that we have By Proposition 3.14, we can approximate any closed-loop McKean-Vlasov process by a sequence of interacting processes. Consequently, one gets As by Theorem 2.14, V o S = V c S , we can conclude.
As Φ 0 satisfies (A.1), this Fokker-Planck equation has a unique solution (see for instance [40,Theorem 2.3]). Therefore ν = µ 0 . This is true for any sub-sequence, then the all sequence converges. This is enough to conclude.
In the next part, in a more general framework than that considered in the article, we give an approximation of the controls. The result proved here is quite useful for the approximation through a sequence of closed-loop controls or Markovian controls.

We set
Remark A.8. Notice that, as we saw in the previous proof, we can take β Lipschitz in t as well. We chose the regularity mentioned in the proposition because it is largely enough to establish our results.

A.3 Uniqueness and equivalence of filtration
In this section, we first provide the uniqueness property of the stochastic Fokker-Planck equation that we use in this paper. In a second time, we show an approximation result of the Brownian motion B by a sequence of Fokker-Planck equations.
Let (X, µ) be a solution of dX t = Φ(t, X t , µ, B)dt + σ(t, X t )dW t + σ 0 dB t with µ t = L(X t | G t ) = L(X t | G T ). (A.14) Recall that the sub-filtration G is fixed. We are first interested in the uniqueness of the conditional law of X t given the σ-field G t i.e. L(X t | G t ). We say that there is uniqueness of the marginal law of (A.14) if for (X 1 , µ 1 ) and (X 2 , µ 2 ) s.t. for i = 1, 2, (X i , µ i ) is solution of (A.14) then µ 1 = µ 2 P-a.s. Proposition A.9. If Φ is uniformly Lipschitz in π, the uniqueness of the marginal law of (A.14) is true.