Correction to: Layered Networks, Equilibrium Dynamics, and Stable Coalitions

In this revised version, the authors have added a subsection describing the conditions for existence of equilibrium, and discussing the connection of the paper with the counterexamples on the existence of equilibrium in stochastic games in Levy and McLennan [24].

1 Introduction

A coalition is a group of players who, through their own actions, can realize some set of outcomes for its own members [38]. Here we will be interested in the equilibrium dynamics governing the formation and evolution of coalitions as well as the strategic forces which give rise to these dynamics. We will think of a coalition as a group of players belonging to the same club, and we will represent the prevailing club membership structure as a labeled, directed bipartite network. Because we will allow each player to be a member of multiple clubs, each player can be a member of multiple coalitions (see Page and Wooders [34]).¹ Each club network consists of three primitives: a set of players, a set of clubs, and a set of arc labels. In our network model, a player’s club membership is represented by a labeled directed arc from the node representing the player to the node representing the club. The arc label, which must be feasible for that player in that club, indicates the action chosen by the player to be taken in the chosen club. Thus, a player establishes a directed connection by choosing a club and a feasible club action. The set of all such player-specific directed club connections is the player’s club network and together the union of these player club networks constitute the club network. At each of infinitely many time points players, in light of the prevailing state and club network, are free to noncooperatively alter their club memberships as well as their corresponding club action profiles in accordance with the rules of network formation. We will assume that after players have altered their own club networks, each player receives a stage payoff, a function of the prevailing state-network pair; then, given the prevailing state and the new club network chosen by the players, a new state is generated in accordance with the law of motion. We will assume that players, in making their membership-action choices through discrete time, seek to maximize the discounted sum of their expected future payoffs. In particular, we will assume that players in forming their club networks play a discounted stochastic game of club network formation in which they seek to choose stationary Markov perfect membership-action strategies that maximize the discounted sum of their expected payoffs. Taken together, the players’ noncooperative network formation strategies (i.e., membership and club specific action strategies) determine a network formation process. We say that this process is an equilibrium network formation process if in the underlying discounted stochastic game (DSG) of network formation players noncooperatively choose stationary Markov perfect membership and club action strategies forming a Nash equilibrium.

Our main contributions, in addition to constructing a DSG model of layered club network formation, are (1) to show that our DSG model of network formation is approximable, and that as a consequence, it possesses a stationary Markov perfect equilibrium in players’ membership-action strategies; (2) to identify the assumptions on primitives which ensure that the induced equilibrium Markov state and club network formation process satisfies the Tweedie Conditions [37] and (3) to show that as a consequence of satisfying the Tweedie Conditions, the equilibrium Markov perfect network formation process generates a unique decomposition of the set of state-network pairs into a transient set together with finitely many basins of attraction. We then show that each basin, upon which resides a unique ergodic probability measure, has the property that if the Markov state-network process enters the basin, then the process will remain there for all future periods, visiting some unique, further subset of state-network pairs infinitely often. From a macroscopic perspective, it is these basin-specific club networks, visited infinitely often by the process (i.e., vio sets), that form the set of viable candidates for stable equilibrium club network. We show that if there is a basin having a vio set of states generating club networks each of whose induced coalition structure is invariant across the networks generated by the vio states, then this vio set of states generates club networks having a stable coalition structure.² This is what we mean by coalitional stability: persistent club network structures in which the underlying coalition structure is invariant across club networks generated by the set of vio states. We show that whether or not the vio states contained in a basin of attraction generate an invariant coalition structure depends on the distance between the club networks generated by the vio states (i.e., it depends on the distance between the vio networks). In particular, we show that closeness (across networks) leads to invariance in coalition structure across club networks. Thus, if each of finitely many basins has network vio sets containing club networks that are sufficiently close together (i.e., if networks are densely packed), then the equilibrium Markov process of club network formation will, in finite time with probability one, generate a stable coalition structure. While the coalition structures generated by the vio states in each basin are the same across the basin’s densely packed club networks, these signature coalition structures can differ across basins. In the long run, whether or not the equilibrium Markov process of network formation generates a stable coalition structure depends on whether or not each basin’s network vio set is densely packed. If all basin-specific network vio sets are sufficiently dense, then with probability 1 in finite time, the equilibrium network formation process will arrive at a basin-specific stable coalition structure. However, if some basins have network vio sets in which the club networks are not sufficiently close, while other basins possess densely packed network vio sets, then in equilibrium, there is a positive probability that a stable coalition structure will never be reached. What we conclude here is that if a basin’s vio set of networks is chainable, then a stable coalition structure will emerge.³ Thus, if club network vio sets are chainable, then in finite time with probability 1, there will emerge from the equilibrium process of club network formation, a basin-specific stable coalition structure.

As a running example we consider a DSG over time allocation networks (a class of club networks sufficiently simple to allow us to illustrate the ideas and results we obtain for our general DSG of network formation and equilibrium network dynamics). We show that because players’ payoff functions are naturally affine over the convex, compact feasible set of time allocation networks, players’ stationary Markov perfect equilibrium network formation strategies are bang-bang. Thus rather than diversifying their club time across several clubs, each player in each state spends all club time in one and only one club. Moreover, we show that all that is required for the equilibrium time allocation process induced by players’ bang-bang time allocation strategies to satisfy the Tweedie conditions is for the state space to be compact and the conditional densities over coming states to be continuous in the current state and time allocation network for almost all coming states (i.e., except potential coming states having probability zero of occurring).

2 Layered Club Networks

We begin with a formal definition of club networks and a discussion of their properties. The discussion here is based, in part, on prior work by the second author with M. Wooders (see Page and Wooders [32,33,34], Wooders and Page [38], and Page et al. [31]).⁴Multiple membership club networks, as defined in Page and Wooders [34], are examples of layered networks in which connections between layers are brought about by overlapping club memberships. In a club network where each player is the member of one and only one club, the induced club membership structure partitions the set of active players, making each club layer isolated—having no connections, via overlapping memberships, to other layers in the network.

In the club network model we construct here, the feasible action sets available to the players who are active in a particular club layer are subsets of a compact metric space of actions specific to that club layer—and these club specific action spaces can differ across layers. In Page and Wooders [34], the underlying metric space of player actions—whose subsets form the various player feasible sets—is the same across club layers. Here the heterogeneity of player action sets across club layers makes defining a metric to measure the distance between club networks a much more delicate task—but we do accomplish this, thereby providing us with a compact metric hyperspace of club networks in which to carry out our game theoretic analysis of the emergence of equilibrium-layered club network dynamics.

Finally, here unlike in Page and Wooders [32,33,34], and Page et al. [31], the game of network formation is dynamic with the equilibrium network dynamics being determined by the law of motion and the stationary Markov perfect equilibrium in behavioral network formation strategies which emerge from the game of network formation.

We begin by defining connections, layers, and networks.

2.1 Connections

[N-1] Assume the following:

(1) D is a finite set of n players equipped with the discrete metric $\eta _{D}$, having typical element d.⁵

(2) C is a finite set of m clubs equipped with discrete metric $\eta _{C}$ having typical element c .

(3) $A_{c}$ is a compact metric space of arc types, equipped with metric $\rho _{c}$ , having typical element a .

(4) $2^{A_{c}}$ is the collection of all closed subsets of $A_{c}$ (including the empty set).

(5) $(d,c)\longrightarrow A(d,c)\in 2^{A_{c}}$ is the feasible arc correspondence, with $A(d,c)\subset A_{c}$ for all c , with c-domain given by

$$\begin{aligned} \mathcal {D}(A(\cdot ,c)):=\{d\in D:A(d,c)\text { }\ne \varnothing \}. \end{aligned}$$

specifying for each club, the set of players who can join club c by indicating a nonempty feasible set of arcs club c will allow player d to use in connecting to club c.

A connection is an ordered pair, (a, (d, c)), consisting of an arc label, a, indicating player $d^{\prime }s$ action (arc) choice and a player-club pair, (d, c),  indicating player $d^{\prime }s$ club membership choice c. The action choice made by a particular player d must be feasible for the club player d has chosen, a requirement satisfied provided $a\in A(d,c)\subset A_{c}$.

We will refer to a player-club pair, (d, c), as a preconnection. A preconnection, (d, c), acquires the status of a connection only when player d has chosen a feasible arc and (a, (d, c)) becomes part of a network. For preconnection, (d, c), the set of all feasible connections is given by

$$\begin{aligned} \begin{array}{l} K_{dc}:=A(d,c)\times (\{d\}\times \{c\}). \end{array} \end{aligned}$$

The collection, $\{K_{dc}:(d,c)\in D\times C\}$, contains the basic building block of layered networks. Let $2^{K_{dc}}$ denote the collection of all closed subsets of $K_{dc}$ (including the empty set), and let $\mathbb {K} _{dc}\subset 2^{K_{dc}}$ be the collection of subsets of $K_{dc}$ of size at most 1.

[N-2] Assume the following:

(1) Each player, in considering whether or not to connect to a particular club either does not connect or connects in one and only one way.

(2) Each player or group of players can move freely and unilaterally from one club to another via feasible connections. Thus a player can drop his membership in any given club and, if allowed by the feasible arc correspondence, join any other club without bargaining and without seeking the permission of any player or group of players. ⁶

2.2 Layered Networks

If $G_{cd}\in$ $\mathbb {K}_{dc}$, then

$$\begin{aligned} \begin{array}{ll} G_{dc}= &{} \left\{ \begin{array}{lll} \left\{ (a_{dc},(d,c))\right\} \subset K_{dc} &{} \text {if }d\text { is a member of club }c &{} \\ &{} &{} \\ \varnothing &{} \text {if }d\text { is not a member of club }c\text {.} &{} \end{array} \right. \end{array} \end{aligned}$$

The feasible set of club networks, $\mathbb {K}:=(\mathbb {K}_{dc})_{dc}$, is given by an $n\times m$ array of feasible player-club connections (see, 1 below), where $\mathbb {K}_{c}:=(\mathbb {K}_{dc})_{d}$ is the feasible set of all possible c-layers (or club c layers), and where $\mathbb {K}^{d}:=(\mathbb {K}_{dc})_{c}$ is the feasible set of all possible d-layers (or player d layers). In array form we have

$$\begin{aligned} \left. \begin{array}{l} \mathbb {K}:=(\mathbb {K}_{dc})_{dc}\, \\ :=\left( \begin{array}{ccccc} \mathbb {K}_{11} &{} \cdots &{} \mathbb {K}_{1c} &{} \cdots &{} \mathbb {K}_{1m} \\ \vdots &{} &{} \vdots &{} &{} \vdots \\ \mathbb {K}_{d1} &{} \cdots &{} \mathbb {K}_{dc} &{} \cdots &{} \mathbb {K}_{dm} \\ \vdots &{} &{} \vdots &{} &{} \vdots \\ \mathbb {K}_{n1} &{} \cdots &{} \mathbb {K}_{nc} &{} \cdots &{} \mathbb {K}_{nm} \end{array} \right) _{n\times m}:=\underset{\text {{rows represent feasible d-layers} }}{\underbrace{\left( \begin{array}{lll} \cdots &{} \mathbb {K}^{1} &{} \cdots \\ &{} \vdots &{} \\ \cdots &{} \mathbb {K}^{d} &{} \cdots \\ &{} \vdots &{} \\ \cdots &{} \mathbb {K}^{n} &{} \cdots \end{array} \right) }} \\ \begin{array}{lll} &{} :=\underset{\text {{columns represent feasible c-layers}}}{ \underbrace{\left( \begin{array}{ccccc} \vdots &{} &{} \vdots &{} &{} \vdots \\ \mathbb {K}_{1} &{} &{} \mathbb {K}_{c} &{} &{} \mathbb {K}_{m} \\ \vdots &{} &{} \vdots &{} &{} \vdots \end{array} \right) }}.&\end{array} \end{array} \right\} \end{aligned}$$

(1)

Formally, we have the following definition of a feasible club network, $G\in \mathbb {K}:=(\mathbb {K}_{dc})_{dc}$, as an $n\times m$ array of feasible player-club connections.

Definition 1

(Feasible Club Networks, c-Layers, d-Layers, and Connection Arrays) A feasible club network $G\in \mathbb {K}$ is an $n\times m$ array of feasible player-club connections given by

$$\begin{aligned} \left. \begin{array}{l} G:=\left( \begin{array}{ccccc} G_{11} &{} \cdots &{} G_{1c} &{} \cdots &{} G_{1m} \\ \vdots &{} &{} \vdots &{} &{} \vdots \\ G_{d1} &{} \cdots &{} G_{dc} &{} \cdots &{} G_{dm} \\ \vdots &{} &{} \vdots &{} &{} \vdots \\ G_{n1} &{} \cdots &{} G_{nc} &{} \cdots &{} G_{nm} \end{array} \right) _{n\times m}:=\underset{\text {{rows represent d-layers} }}{ \underbrace{\left( \begin{array}{lll} \cdots &{} G^{1} &{} \cdots \\ &{} \vdots &{} \\ \cdots &{} G^{d} &{} \cdots \\ &{} \vdots &{} \\ \cdots &{} G^{n} &{} \cdots \end{array} \right) }} \\ \begin{array}{lll} &{} :=\underset{\text {{columns represent c-layers}}}{\underbrace{\left( \begin{array}{ccccc} \vdots &{} &{} \vdots &{} &{} \vdots \\ G_{1} &{} &{} G_{c} &{} &{} G_{m} \\ \vdots &{} &{} \vdots &{} &{} \vdots \end{array} \right) }}.&\end{array} \end{array} \right\} \end{aligned}$$

(2)

where for each player-club pair, (d, c),

$$\begin{aligned} G_{dc}\in \mathbb {K}_{dc}\subset 2^{K_{dc}}:=2^{A(d,c)\times (\{d\}\times \{c\})} \end{aligned}$$

is player $d^{\prime }s$ part of c-layer, $G_{c}\in \mathbb {K}_{c}$, given by the $c^{th}$ column in the club network, G, above, and where $G_{dc}\in \mathbb {K}_{dc}$ if and only if

$$\begin{aligned} \begin{array}{ll} G_{dc}= &{} \left\{ \begin{array}{lll} \left\{ (a_{dc},(d,c))\right\} \subset K_{dc} &{} \text {if }d\text { is a member of club }c &{} \\ &{} &{} \\ \varnothing &{} \text {if }d\text { is not a member of club }c\text {,} &{} \end{array} \right. \end{array} \end{aligned}$$

and where $G^{d}\in \mathbb {K}^{d}$ is player $d^{\prime }s$ connection choice profile, given by the $d^{th}$ row in the club network, G, above.

If we agree to the notational convention that $G_{dc}:=a_{dc}$ then club networks in $\mathbb {K}=(\mathbb {K}_{dc})_{dc}$ can be given a reduced form array representation—as an array of arc types (without loss of information)—as follows:

$$\begin{aligned} \left. \begin{array}{l} \,G:=\left( \begin{array}{ccccc} a_{11} &{} \cdots &{} a_{1c} &{} \cdots &{} a_{1m} \\ \vdots &{} &{} \vdots &{} &{} \vdots \\ a_{d1} &{} \cdots &{} a_{dc} &{} \cdots &{} a_{dm} \\ \vdots &{} &{} \vdots &{} &{} \vdots \\ a_{n1} &{} \cdots &{} a_{nc} &{} \cdots &{} a_{nm} \end{array} \right) _{n\times m}=\underset{\text {{reduced form rows represent d-layers} }}{\underbrace{\left( \begin{array}{lll} \cdots &{} a^{1} &{} \cdots \\ &{} \vdots &{} \\ \cdots &{} a^{d} &{} \cdots \\ &{} \vdots &{} \\ \cdots &{} a^{n} &{} \cdots \end{array} \right) }} \\ =\underset{\text {{reduced form columns represent c-layers}}}{ \underbrace{\left( \begin{array}{ccccc} \vdots &{} &{} \vdots &{} &{} \vdots \\ a_{1} &{} &{} a_{c} &{} &{} a_{m} \\ \vdots &{} &{} \vdots &{} &{} \vdots \end{array} \right) }}. \end{array} \right\} \end{aligned}$$

(3)

A club network, $G\in \mathbb {K}$ implicitly determines an arc selection, $(d,c)\longrightarrow G(d,c)$ given by

$$\begin{aligned} (d,c)\longrightarrow G(d,c)=\{a\in A(d,c):(a,(d,c))\in G\}, \end{aligned}$$

(4)

with domain given by

$$\begin{aligned} \mathcal {D}(G):=\{(d,c)\in D\times C:G(d,c)\text { }\ne \varnothing \}. \end{aligned}$$

If $G(d,c)=\varnothing$, then in network G, the preconnection (d, c) is not elevated to the status of a connection (i.e., player d is not a member of club c in network G either because d chose not to join club c or because player d was not allowed to join club c—i.e., $A(d,c)=\varnothing$). Alternatively, if $G(d,c)\ne \varnothing$, then preconnection (d, c) has been elevated to a connection. We note that $G(d,c)\subset A(d,c)$ is a singleton set for all preconnections $(d,c)\in D\times C$ and for all preconnections (d, c) in $\mathcal {D}(G)\subset \mathcal {D}(A(d,c))$, $G(d,c)\ne \varnothing$. Thus, a club network’s arc selection, $G(\cdot ,\cdot )$, is a selection from the feasible arc correspondence, $(d,c)\longrightarrow A(d,c)$ correspondence.

2.3 Coalition Structure

Each feasible club network, with its implied arc selection, determines a coalition structure. In particular, given arc selection $G(\cdot ,\cdot )$ determined by club network, $G\in \mathbb {K}$, each club c has a membership coalition given by the domain of the mapping $d\longrightarrow G(d,c)$ for fixed c. In particular, club c has membership coalition in network G, denoted by, $S_{cG}$, and given by

$$\begin{aligned} S_{cG}=\mathcal {D}(G(\cdot ,c)):=\{d\in D:G(d,c)\text { }\ne \varnothing \}\in 2^{D}\text {,} \end{aligned}$$

where $2^{D}$ is the collection of all subsets of D, including the empty set. Thus, the coalition structure determined by club network $G\in \mathbb {K }$ is given by

$$\begin{aligned} \begin{array}{l} \left. \begin{array}{l} (S_{1G},\ldots ,S_{cG},\ldots ,S_{mG}) \\ \text {where for }c=1,2,\ldots ,m \\ S_{cG}=\mathcal {D}(G(\cdot ,c)). \end{array} \right\} \end{array} \end{aligned}$$

(5)

In a club network, connections between layers are made through overlapping club memberships. Without this, each club layer is isolated. For example, in layered club network, $G\in \mathbb {K}$, if $S_{cG}\cap S_{c^{\prime }G}\ne \varnothing$, then each player $d\in$ $S_{cG}\cap S_{c^{\prime }G}$ is a member of club c as well as a member of club $c^{\prime }$. In this way, club layers $G_{c}$ and $G_{c^{\prime }}$ are connected in network G. Note that if in club network, G, players are members of one and only one club, then if the network has multiple nonempty layers, there are no connections between these layers—each layer is isolated precisely because there are no overlapping club memberships. In this case, the coalition structure induced by club network G given by

$$\begin{aligned} S_{G}:=(S_{1G},\ldots ,S_{mG})\in 2^{D}\times \cdots \times 2^{D}\text {,} \end{aligned}$$

(6)

is a partition of the active club members—and players are siloed by their club memberships.

3 Measuring the Distance Between Club Networks

In order to analyze the co-evolution of strategic behavior, club network structure and equilibrium dynamics, we require a topology for the space of club networks that is simultaneously coarse enough to guarantee compactness of the set of networks and fine enough to discriminate between differences across networks that are due to differences in the ways nodes are connected (via differing arc types) versus differences across networks that are due to the complete absence of connections. We resolve this topological dilemma by equipping the space club networks, $\mathbb {K},$ with the Hausdorff metric $h_{K}$—making the space of feasible club network connection arrays, a compact metric space (see Appendix 1 on the Hausdorff metric on the hyperspace of layered networks).

It is easy to show that if the Hausdorff distance between any pair club networks G and $G^{\prime }$ is less than $\varepsilon \in (0,1)$, then the networks can differ only in the ways a given set of player-club pairs are connected—and not in the set of player-club pairs that are connected. In particular, if for networks G and $G^{\prime }$, $h_{K}(G,G^{\prime })<\varepsilon <1$, then

$$\begin{aligned} (a,(d,c))\in G\text { if and only if }(a^{\prime },(d,c))\in G^{\prime }\text { } \end{aligned}$$

for arcs a and $a^{\prime }$ with $\rho _{A_{c}}(a,a^{\prime })<\varepsilon$. Thus, if two club networks are at $h_{K}$-distance $\varepsilon <1$, then both club networks G and $G^{\prime }$ have the same coalition structures, i.e.,

$$\begin{aligned} S_{G}:=(S_{1G},\ldots ,S_{mG})=(S_{1G^{\prime }},\ldots ,S_{mG^{\prime }}):=S_{G^{\prime }}\text {.} \end{aligned}$$

(7)

Such closeness will often occur and can only persist in network vio sets (sets of networks visited infinitely often by the equilibrium stochastic process of network formation) i.e., sets belonging to basins of attraction generated by the equilibrium dynamics governing the movements of club networks—all discussed in detail below.

We will equip the hyperspace of feasible club network connection arrays, $\mathbb {K}=(\mathbb {K}_{dc})_{dc}$, with the Hausdorff metric given by

$$\begin{aligned} h_{K}(G,G^{\prime }):=\sum _{d=1}^{n}\sum _{c=1}^{m}h_{K_{dc}}(G_{dc},G_{dc}^{\prime })\text {, for }(d,c)\text { connections }G_{dc}\text { and }G_{dc}^{\prime }\text { in } \mathbb {K}_{dc}\text {, } \end{aligned}$$

where

$$\begin{aligned} \begin{array}{ll} G_{dc}= &{} \left\{ \begin{array}{lll} \left\{ (a_{dc},(d,c))\right\} \subset K_{dc} &{} \text {if }d\text { is a member of club }c &{} \\ &{} &{} \\ \varnothing &{} \text {if }d\text { is not a member of club }c &{} \end{array} \right. \end{array} \end{aligned}$$

and where, due to the single-arc connection rule,

$$\begin{aligned} \begin{array}{ll} {\small h}_{K_{dc}}{\small (G}_{dc}{\small ,G}_{dc}^{\prime }{\small )=} &{} \left\{ \begin{array}{ll} {\small \rho }_{c}{\small (a}_{dc}{\small ,a}_{dc}^{\prime }{\small )} &{} \text {{if }}{\small G}_{dc}{\small \ne \varnothing }\text { { and} }{\small G}_{dc}^{\prime }{\small \ne \varnothing }\text {,} \\ &{} \\ {\small diam(A}_{dc}{\small )} &{} \text {{ if }}{\small G}_{dc}{\small \ne \varnothing ,G}_{dc}^{\prime }{\small =\varnothing }\text { { or }} {\small G}_{dc}{\small =\varnothing ,G}_{dc}^{\prime }{\small \ne \varnothing }\text {,} \\ &{} \\ {\small 0} &{} \text {{ if }}{\small G}_{c}{\small =\varnothing ,}\text { } {\small G}_{c}^{\prime }{\small =\varnothing }\text {.} \end{array} \right. \end{array} \end{aligned}$$

4 Example: Time Allocation Networks

Consider a space of club networks in which the set of arc types, $A_{c}$, is given by $A_{c}=[0,1]$ for all clubs, c, and the feasible arc correspondence, $(d,c)\longrightarrow A(d,c)$, is given by $A(d,c)=[0,1]$ for all player-clubs pairs. We have for each player-club pair, (d, c), that $K_{dc}:=[0,1]\times (\{d\}\times \{c\})$ and $2^{K_{dc}}$ is the hyperspace of all closed subsets of $K_{dc}$ (including the empty set) and $\mathbb {K} _{dc}$ is the collection of all subsets of $K_{dc}$ of size 1. Thus, $G_{dc}\in \mathbb {K}_{dc}$ if and only if

$$\begin{aligned} \begin{array}{ll} G_{dc}= &{} \left\{ \begin{array}{lll} \left\{ (a_{dc},(d,c))\right\} \subset K_{dc} &{} \text {if }d\text { spends fraction }a_{dc}\text { of club time in club }c &{} \\ &{} &{} \\ \varnothing &{} \text {if }d\text { is not a member of club }c &{} \end{array} \right. \end{array} \end{aligned}$$

But now we add a further condition given by the time allocation constraint. We will require that the d-layers, $\mathbb {K}^{d}$, in each club network $G\in$ $\mathbb {K}:=(\mathbb {K}_{dc})_{dc}$, having arc selection, $(d,c)\longrightarrow G(d,c)$, be such that

$$\begin{aligned} d\longrightarrow \mathop {\displaystyle \sum }\limits _{c\in \mathcal {D}(G(d,\cdot ))}G(d,c)=1\text {,} \end{aligned}$$

where $\mathcal {D}(G(d,\cdot ))$ is the club domain of player d—the clubs to which d belongs. The feasible set of club networks is,

$$\begin{aligned} \begin{array}{l} \begin{array}{l} \begin{array}{l} \mathbb {K}_{TA} \\ \\ :=\{G\in (\mathbb {K}_{dc})_{dc}:\forall (d,c)\text {, }G(d,c)=\{a_{dc}\}\text { , }a_{dc}\in [0,1]\text {, and }\forall \text { }d\text {,} \mathop {\displaystyle \sum }\limits _{c\in \mathcal {D}(G(d,\cdot ))}a_{dc}=1\}. \end{array} \end{array} \end{array} \end{aligned}$$

Thus, in a time allocation network, players freely and noncooperatively choose which clubs they want to join as well as their levels of participation in each club they join. Their levels of participation in each club are then expressed as fractions of their total club time—and these fractions must sum to one.

Figure 1 depicts a time allocation in which 5 players, $D=\{d_{1},d_{2},d_{3},d_{4},d_{5}\}$ allocate their club time across 3 clubs, $C=\{c_{1},c_{2},c_{3}\}$.

Fig. 1

Time allocation network G

Following the notational convention that

$$\begin{aligned} \left. \begin{array}{l} a_{dc_{k}}\in (0,1]\Longleftrightarrow G_{dc_{k}}=\{(a_{dc_{k}},(d,c_{k}))\} \\ \text {and} \\ a_{dc_{k}}=0\Longleftrightarrow G_{dc_{k}}=\varnothing \text {,} \end{array} \right\} \end{aligned}$$

(8)

the connections array in reduced form for network G in Fig. 1 is given by

$$\begin{aligned} \begin{array}{l} \left( \begin{array}{lll} 1 &{} 0 &{} 0 \\ 0 &{} a_{22} &{} a_{23} \\ 0 &{} 1 &{} 0 \\ 1 &{} 0 &{} 0 \\ 0 &{} 1 &{} 0 \end{array} \right) \end{array} \end{aligned}$$

Because the d-layers (the rows) must have arc labels, $a_{dc}\in [0,1]$, that sum to 1, and because players $d_{1},d_{3},d_{4},$ and $d_{5}$ spend all of their time in one and only one club, we know that in the case of player $d_{1}$, for example, $a_{11}=1$, while $a_{12}=a_{13}=0$. Player $d_{2}$ splits all of his club time between clubs $c_{2}$ and $c_{3}$. Thus, in the case of player $d_{2}$, we have that $a_{22}$ and $a_{23}$ are positive with $a_{22}+a_{23}=1$. Club $c_{2}$ has 3 members, $S_{c_{2}G}=\{d_{2},d_{3},d_{5}\}$, while club $c_{1}$ has 2 members, $S_{c_{1}G}=\{d_{1},d_{4}\}$, and club $c_{3}$ has 1 member, $S_{c_{3}G}=\{d_{2}\}$.

Again following notational convention (8) and writing out the connections long hand, the connections in the d-layers are given by

$$\begin{aligned} \begin{array}{l} G^{1}=(G_{11},G_{12},G_{13})=(\{(1,(d_{1},c_{1}))\},\varnothing ,\varnothing ), \\ G^{2}=(G_{21},G_{22},G_{23})=(\varnothing ,\{(a_{22},(d_{2},c_{2}))\},\{(a_{23},(d_{2},c_{3}))\}), \\ G^{3}=(G_{31},G_{32},G_{33})=(\varnothing ,\{(1,(d_{3},c_{2}))\},\varnothing ), \\ G^{4}=(G_{41},G_{42},G_{43})=(\{(1,(d_{4},c_{1}))\},\varnothing ,\varnothing ), \\ \text {and} \\ G^{5}=(G_{51},G_{52},G_{53})=(\varnothing ,\{(1,(d_{5},c_{2}))\},\varnothing ), \end{array} \end{aligned}$$

while the connections in the c-layers are given by

$$\begin{aligned} \begin{array}{l} G_{1}=(G_{11},G_{21},G_{31},G_{41},G_{51})=(\{(1,(d_{1},c_{1}))\}, \varnothing ,\varnothing ,\{(1,(d_{4},c_{1}))\},\varnothing ), \\ G_{2}=(G_{12},G_{22},G_{32},G_{42},G_{52})=(\varnothing ,\{(a_{22},(d_{2},c_{2}))\},\{(1,(d_{3},c_{2}))\},\varnothing ,\{(1,(d_{5},c_{2}))\}), \\ \text {and} \\ G_{3}=(G_{13},G_{23},G_{33},G_{43},G_{53})=(\varnothing ,\{(a_{23},(d_{2},c_{3}))\},\varnothing ,\varnothing ,\varnothing ). \end{array} \end{aligned}$$

We note that the time allocation club networks in $\mathbb {K}_{TA}$ and in general the club networks in $\mathbb {K}$ satisfy assumptions [N-1]. We will assume for the remainder of the paper that [N-2] holds. We will return to the example above later in the paper.

5 Discounted Stochastic Games of Club Network Formation

In order to address the questions of whether or not and under what conditions the strategic formation of club networks will lead to the emergence of dynamically stable coalition structures, we will show that our discounted stochastic game (DSG) of club network formation possesses Nash equilibria in stationary Markov perfect behavioral club network formation strategies. It is the players’ equilibrium behavioral network formation strategies which determine the equilibrium dynamics of club network formation. By identifying conditions under which stationary Markov perfect equilibria (SMPE) exist in such behavioral strategies and by showing that the resulting equilibrium state and club network dynamics are stable, we will be able to establish the conditions under which stable coalition structures will emerge and persist in the form of stable club networks. In this section we will construct a DSG model of club network formation and show that our model possesses SMPE in behavioral club network formation strategies. The SMPE existence problem in the setting considered here (with uncountable states and compact metric action spaces) is quite difficult and its solution—and counterexamples—are of independent interest (for details see Levy [23], Levy and McLennan [24], Page [29, 30], and Fu and Page [15]).

An m-player, nonzero sum, discounted stochastic game, DSG, over the product space of probability measures over player club networks (i.e., the convex, compact metric space of behavioral actions),

$$\begin{aligned} (\Delta (\mathbb {K}),\rho _{w_{ca}^{*}}):=(\mathop {\displaystyle \prod }\limits _{d}\Delta ( \mathbb {K}^{d}),\sum _{d}\rho _{w_{ca}^{*d}})\text {,} \end{aligned}$$

(9)

is given by the following primitives:

$$\begin{aligned} \begin{array}{l} DSG:=\left\{ \underset{\text {{ state space}}}{\underbrace{\left( \Omega ,B_{\Omega },\mu \right) }},\underset{\text {{ collection of one-shot games}}}{\underbrace{\left\{ \left( \Delta (\mathbb {K}^{d}),\Delta (\Phi _{d}(\omega )),\beta _{d},U_{d}(\omega ,v_{d},\cdot )\right) _{d\in D}\right\} _{{\small (\omega ,v)}}}},\underset{\text {{ law of motion}}}{\underbrace{q(\cdot |\omega ,\cdot )}}\right\} , \end{array} \end{aligned}$$

(10)

where $\Omega$ is the state space, $B_{\Omega }$ is the Borel $\sigma$-field of events, and $\mu$ is a probability measure. For each player d, $\mathbb {K}^{d}$ is the set of all possible player club networks available to player d, while $\Delta (\Phi _{d}(\omega ))$ is the convex, compact feasible set of behavioral action available to player d in state $\omega$. A feasible behavioral action available to player d in state $\omega$ is a probability measure $\sigma _{d}\in \Delta (\mathbb {K}^{d})$ with support contained in $\Phi _{d}(\omega )$ (i.e., $\sigma _{d}\in \Delta (\Phi _{d}(\omega ))$ if and only if $\sigma _{d}(\Phi _{d}(\omega ))=1$). Finally, $\beta _{d}\in (0,1)$ is player $d^{\prime }s$ discount rate and $U_{d}(\omega ,v_{d},\cdot )$ is player $d^{\prime }s$ payoff function in state $\omega$ given valuations (or prices) $v_{d}$, and $q(\cdot |\omega ,\cdot )$ is the law of motion in state $\omega$. If players holding value function profile $v=(v_{1}\ldots ,v_{n})$ choose feasible profile of behavioral actions,

$$\begin{aligned} \sigma =(\sigma _{1},\ldots ,\sigma _{n})\in \Delta (\Phi _{1}(\omega ))\times \cdots \times \Delta (\Phi _{n}(\omega ))=\Delta (\Phi (\omega )), \end{aligned}$$

(11)

in state $\omega$, then the next state $\omega ^{\prime }$ is chosen in accordance with probability measure $q(\cdot |\omega ,G)\in \Delta (\Omega )$ and player $d^{\prime }s$ stage payoff is given by

$$\begin{aligned} \begin{array}{l} \left. \begin{array}{l} U_{d}(\omega ,v_{d},\sigma ):=\int _{\mathbb {K}}\underset{u_{d}(\omega ,v_{d},G)}{\underbrace{\left[ (1-\beta _{d})r_{d}(\omega ,G)+\beta _{d}\int _{\Omega }v_{d}(\omega ^{\prime })q(\omega ^{\prime }|\omega ,G) \right] }}\pi (\sigma (dG)) \\ \\ :=\int _{\mathbb {K}}u_{d}(\omega ,v_{d},G)\pi \sigma (dG)\text {.} \end{array} \right\} \end{array} \end{aligned}$$

(12)

Here $\pi \sigma (dG):=\pi (\sigma _{1}(dG^{1}),\ldots ,\sigma _{m}(dG^{n})):=\otimes _{d=1}^{n}\sigma _{d}(dG^{d})$ is the product probability measure representing the random club network determined by the n-tuple of random player club networks, $(\sigma _{1}(dG^{1}),\ldots ,\sigma _{n}(dG^{n}))$, chosen by the players.

We will denote by, $\mathcal {G}_{{\small (\omega ,v)}}:=\left( \Delta (\Phi _{d}(\omega )),U_{d}(\omega ,v_{d},\cdot )\right) _{d\in D}$, the n-player ${\small (\omega ,v)}$-game in state $\omega$ underlying the DSG when players hold valuations (or state-contingent prices), $v:=(v_{1},\ldots ,v_{n})$.

5.1 Primitives and Assumptions

We will maintain the following assumptions throughout. Label these (i.e., assumptions (1)–(15) below) as [A-1]:

(1) $D=$ the set of players, consisting of n players indexed by $d=1,2,\ldots ,n$ and each having discount rate given by $\beta _{d}\in (0,1)$ .

(2) $(\Omega ,B_{\Omega },\mu )$, the state space where $\Omega$ is a complete separable metric spaces with metric $\rho _{\Omega }$, equipped with the Borel $\sigma$-field, $B_{\Omega }$, upon which is defined a probability measure, $\mu$ .

(3) $Y:=Y_{1}\times \cdots \times Y_{n}$ , the space of all possible players’ payoff profiles, $U:=(U_{1},\ldots ,U_{n})$ , such that for each player d , $Y_{d}:=[-M,M]$ and is equipped with the absolute value metric, $\rho _{Y_{d}}(U_{d},U_{d}^{\prime }):=\left| U_{d}-U_{d}^{\prime }\right|$ and Y is equipped with the sum metric, $\rho _{Y}:=\sum _{d}\rho _{Y_{d}}$ .

(4) $\mathbb {K}:=\mathbb {K}^{1}\times \cdots \times \mathbb {K}^{n},$ the set of player pure action profiles, $G:=(G^{1},\ldots ,G^{n})$ , where for each player d,

$$\begin{aligned} G^{d}:=(G_{d1},\ldots ,G_{dc},\ldots ,G_{dm}) \end{aligned}$$

is player $d^{\prime }s$ club network and where $\mathbb {K} ^{d}:=\mathbb {K}_{d1}\times \cdots \times \mathbb {K}_{dc}\times \cdots \times \mathbb {K}_{dm}$ is a compact metric space of feasible player club networks with typical element, $G^{d}$, equipped with metric, $h_{K^{d}}:=\sum _{c}h_{K_{dc}}$, and $\mathbb {K}$ is equipped with the sum metric, $h_{K}:=\sum _{d=1}^{n}\sum _{c=1}^{m}h_{K_{dc}}.$

(5) $\Delta (\mathbb {K}^{d})$ is the space of all probability measures, $\sigma _{d}$ , with supports contained in player $d^{\prime }s$ set of club networks, $\mathbb {K}^{d}$ , equipped with the compact metrizable weak star topology (a topology denoted by $w_{ca}^{*d}$ ) inherited from $ca(\mathbb {K}^{d})$ , the Banach space of finite signed Borel measures on $\mathbb {K}^{d}$ with the total variation norm. ⁷ We will equip $\Delta (\mathbb {K}^{d})$ with a metric, $\rho _{w_{ca}^{*d}}$ , compatible with the relative $w_{ca}^{*d}$ -topology on $\Delta (\mathbb {K}^{d})$ inherited from $ca(\mathbb {K}^{d})$ and we will refer to $\sigma _{d}$ as player $d^{\prime }s$ random player club network.

(6) $\Delta (\mathbb {K}):=\Delta (\mathbb {K}^{1})\times \cdots \times \Delta (\mathbb {K}^{n})$ , the space of player behavioral action profiles, $\sigma :=(\sigma _{1},\ldots ,\sigma _{n})$ , equipped with the sum metric, $\rho _{w_{ca}^{*}}:=\sum _{d}\rho _{w_{ca}^{*d}}$ , a metric compatible with the relative $w_{ca}^{*}$ -product topology on

$$\begin{aligned} \Delta (\mathbb {K}^{1})\times \cdots \times \Delta (\mathbb {K}^{n}) \end{aligned}$$

inherited from $ca(\mathbb {K}^{1})\times \cdots \times ca(\mathbb {K} ^{n})$ .

(7) $\omega \longrightarrow \Phi _{d}(\omega )$, is player $d^{\prime }s$ measurable action constraint correspondence, defined on $\Omega$ taking nonempty $h_{K^{d}}$-closed (and hence $h_{K^{d}}$-compact) network values in $\mathbb {K}^{d}$.

(8) $\omega \longrightarrow \Phi (\omega ):=\Phi _{1}(\omega )\times \cdots \times \Phi _{n}(\omega )$, players’ measurable action profile constraint correspondence, defined on $\Omega$ taking nonempty $h_{K}$-closed (and hence compact) network values in $\mathbb {K}.$

(9) $\omega \longrightarrow \Delta (\Phi _{d}(\omega ))$, is player $d^{\prime }s$ measurable behavioral action constraint correspondence, defined on $\Omega$ taking nonempty $w_{ca}^{*d}$-closed (and hence $w_{ca}^{*d}$-compact), convex random network values in $\Delta (\mathbb {K}^{d})$, containing all probability measures, $\sigma _{d}(\omega )$, with supports contained in player $d^{\prime }s$ feasible set of actions, $\Phi _{d}(\omega )\subset \mathbb {K}^{d}$, in state $\omega$.

(10) $\omega \longrightarrow \Delta (\Phi (\omega )):=\Delta (\Phi _{1}(\omega ))\times \cdots \times \Delta (\Phi _{n}(\omega ))$ , players’ measurable behavioral action profile constraint correspondence, defined on $\Omega$ taking nonempty $w_{ca}^{*}$ -closed (and hence $w_{ca}^{*}$ -compact), convex values in $\Delta ( \mathbb {K}^{1})\times \cdots \times \Delta (\mathbb {K}^{n}).$

(11) $\mathcal {L}_{Y_{d}}^{\infty }$ , the Banach space of all $\mu$ -equivalence classes of measurable (value) functions, $v_{d}(\cdot ),$ defined on $\Omega$ with values in $Y_{d}$ a.e. $[\mu ]$ , equipped with metric $\rho _{w_{d}^{*}}$ compatible with the weak star topology inherited from $\mathcal {L} _{R}^{\infty }$ . ⁸

(12) $\mathcal {L}_{Y}^{\infty }:=\mathcal {L}_{Y_{1}}^{\infty }\times \cdots \times \mathcal {L}_{Y_{m}}^{\infty }\subset \mathcal {L} _{R^{m}}^{\infty }$, the Banach space of all $\mu$-equivalence classes of measurable (value) function profiles, $v(\cdot ):=(v_{1}(\cdot ),\ldots ,v_{m}(\cdot ))$, defined on $\Omega$ with values in Y a.e. $[\mu ]$, equipped with the sum metric $\rho _{w^{*}}:=\sum _{d}\rho _{w_{d}^{*}}$ compatible with the weak star product topology inherited from $\mathcal {L}_{R^{m}}^{\infty }$.

(13) $\mathcal {S}^{\infty }(\Delta (\Phi _{d}(\cdot )))$, the set of all $\mu$-equivalence classes of measurable functions (selections), $\sigma _{d}(\cdot )\in \mathcal {L}_{\Delta (\mathbb { K}^{d})}^{\infty },$ defined on $\Omega$ such that in $\sigma _{d}(\omega )\in \Delta (\Phi _{d}(\omega ))$ a.e. $[\mu ]$ , and

$$\begin{aligned} \mathcal {S}^{\infty }(\Delta (\Phi (\cdot )))=\mathcal {S}^{\infty }(\Delta (\Phi _{1}(\cdot )))\times \cdots \times \mathcal {S}^{\infty }(\Delta (\Phi _{n}(\cdot ))) \end{aligned}$$

(13)

the set of all $\mu$-equivalence classes of measurable profiles (selection profiles), $\sigma (\cdot )=(\sigma _{1}(\cdot ),\ldots ,\sigma _{n}(\cdot ))$, defined on $\Omega$ such that in $\sigma (\omega )\in \Delta (\Phi (\omega )):=\Delta (\Phi _{1}(\omega ))\times \cdots \times \Delta (\Phi _{n}(\omega ))$ a.e. $[\mu ]$.

(14) $r_{d}(\cdot ,\cdot ):\Omega \times \mathbb {K}\longrightarrow Y_{d}$ is player $d^{\prime }s$ Caratheodory stage payoff function (i.e., for all $(\omega ,G)\in \Omega \times \mathbb {K}$ , $r_{d}(\omega ,\cdot )$ is $h_{K}$ -continuous on $\mathbb {K}$ and $r_{d}(\cdot ,G)$ is $(B_{\Omega },B_{Y_{d}})$ -measurable on  $\Omega$ ).

(15) $q(\cdot |\cdot ,\cdot ):\Omega \times \mathbb {K}\longrightarrow \Delta (\Omega )$ is the law of motion defined on $\Omega \times \mathbb {K}$ taking values in the space of probability measures on $\Omega$, having the following properties: (i) each probability measure, $q(\cdot |\omega ,G)$, in the collection

$$\begin{aligned} Q(\Omega \times \mathbb {K}):=\left\{ q(\cdot |\omega ,G):(\omega ,G)\in \Omega \times \mathbb {K}\right\} \end{aligned}$$

is absolutely continuous with respect to $\mu$ (denoted $Q(\Omega \times \mathbb {K})<<\mu$), (ii) for each $E\in B_{\Omega }$ , $q(E|\cdot ,\cdot )$ is measurable on $\Omega \times \mathbb {K}$, (iii) the collection of probability density functions,

$$\begin{aligned} H_{\mu }:=\left\{ h(\cdot |\omega ,G):(\omega ,G)\in \Omega \times \mathbb {K} \right\} \text {,} \end{aligned}$$

(14)

of $q(\cdot |\omega ,G)$ with respect to $\mu$ is such that for each state $\omega$, the function

$$\begin{aligned} G\longrightarrow h(\omega ^{\prime }|\omega ,G) \end{aligned}$$

(15)

is $h_{\mathbb {K}}$-continuous in G a.e. $[\mu ]$ in $\omega ^{\prime }.$

We note that the assumptions above are the usual assumptions underlying discounted stochastic game models (see Appendix 2 for technical notes on these assumptions).

6 Stationary Markov Perfect Equilibria in Club Network Formation Strategies

6.1 Existence

Let DSG be a discounted stochastic game of club network formation satisfying assumptions [A-1] above, with one-shot game,

$$\begin{aligned} \mathcal {G}_{\Omega \times \mathcal {L}_{Y}^{\infty }}:=\{\left( \Delta (\Phi _{d}(\omega )),U_{d}(\omega ,v_{d},\cdot )\right) _{d}\}_{(\omega ,v)\in \Omega \times \mathcal {L}_{Y}^{\infty }}. \end{aligned}$$

(16)

Definition 2

(Nash Equilibria in Behavioral Strategies) A feasible profile of probability measures over player club networks, $\sigma ^{*}:=(\sigma _{1}^{*},\ldots ,\sigma _{m}^{*})\in \Delta (\Phi (\omega ))$ is said to be a Nash equilibrium of the one-shot network formation game,$\left( \Delta (\Phi _{d}(\omega )),U_{d}(\omega ,v_{d},\cdot )\right) _{d}$, provided that for each player d,

$$\begin{aligned} U_{d}(\omega ,v_{d},\sigma _{d}^{*},\sigma _{-d}^{*})=\max _{\sigma _{d}\in \Delta (\Phi _{d}(\omega ))}U_{d}(\omega ,v_{d},\sigma _{d},\sigma _{-d}^{*}). \end{aligned}$$

(17)

Denote by $\mathcal {N}(\omega ,v)$ the set of all Nash equilibria belonging to $\left( \Delta (\Phi _{d}(\omega )),U_{d}(\omega ,v_{d},\cdot )\right) _{d}$, and by $\mathcal {P}(\omega ,v)$ the set of all Nash equilibria payoffs belonging to $\left( \Delta (\Phi _{d}(\omega )),U_{d}(\omega ,v_{d},\cdot )\right) _{d}$. Thus

$$\begin{aligned} \left. \begin{array}{l} \mathcal {P}(\omega ,v):=\{U\in Y:U=U(\omega ,v,\sigma )\text { for some } \sigma \in \mathcal {N}(\omega ,v)\} \\ \\ :=U(\omega ,v,\mathcal {N}(\omega ,v)).\text { } \end{array} \right\} \end{aligned}$$

(18)

Under assumptions, [A-1], we know that $\mathcal {N}(\omega ,v)$ is nonempty and $\rho _{w_{ca}^{*}}$-compact and therefore we know that $\mathcal {P} (\omega ,v)$ is nonempty and $\rho _{Y}$-compact. Moreover, applying optimal measurable selection results (e.g., Himmelberg et al. [19]) and Berge’s maximum theorem (e.g., see 17.31 in Aliprantis and Border [1]), we can show that the Nash correspondences, $\mathcal {N}(\cdot ,\cdot )$ and $\mathcal {P}(\cdot ,\cdot )$, are upper Caratheodory (also, see Proposition 4.2 in Page [28]). In particular, the Nash correspondence, $\mathcal {N}(\cdot ,\cdot )$, is jointly measurable in $(\omega ,v)$ and $\mathcal {N}(\omega ,\cdot )$ is upper semicontinuous in v for each $\omega$, and the Nash payoff correspondence, $\mathcal {P}(\cdot ,\cdot )$, is jointly measurable in $(\omega ,v)$ and $\mathcal {P}(\omega ,\cdot )$ is upper semicontinuous in v for each $\omega$.

Our main existence result is the following:

Theorem 1

(All DSGs satisfying [A-1] have SMPE in behavioral strategies) Let

$$\begin{aligned} DSG:=\left\{ \left( \Omega ,B_{\Omega },\mu \right) ,\left\{ \left( \Delta ( \mathbb {K}^{d}),\Delta (\Phi _{d}(\omega )),\beta _{d},U_{d}(\omega ,v_{d},\cdot )\right) _{d\in D}\right\} _{{\small (\omega ,v)\in }\Omega \times \mathcal {L}_{Y}^{\infty }},q(\cdot |\omega ,\cdot )\right\} \end{aligned}$$

be a discounted stochastic game of club network formation satisfying assumptions [A-1], with upper Caratheodory (uC) Nash correspondences, $\mathcal {N}(\cdot ,\cdot )$ and $\mathcal {P}(\cdot ,\cdot )$.

There exists a pair, $(v^{*},\sigma ^{*}(\cdot ))\in \mathcal { L}_{Y}^{\infty }\times \mathcal {S}^{\infty }(\Delta (\Phi (\cdot )))$ such that $\sigma ^{*}(\cdot )\in \mathcal {S}^{\infty }(\mathcal {N} _{v^{*}})$ is a Stationary Markov Perfect Equilibrium (SMPE) in behavioral club network formation strategies supported by Bellman prices $v^{*}\in \mathcal {S}^{\infty }(\mathcal {P}_{v^{*}})$- i.e., there exists a pair, $(v^{*},\sigma ^{*}(\cdot ))\in \mathcal {L}_{Y}^{\infty }\times \mathcal {S}^{\infty }(\Delta (\Phi (\cdot )))$ such that a.e. $[\mu ]$

$$\begin{aligned} \sigma ^{*}(\omega )\in \mathcal {N}(\omega ,v^{*})\text { and } v^{*}(\omega )=U(\omega ,v^{*},\sigma ^{*}(\omega ))\in \mathcal { P}(\omega ,v^{*})\text {.} \end{aligned}$$

For a formal proof we refer the reader to Fu and Page [15]. Informally, the proof proceeds along the following lines: Let $\mathcal {S}^{\infty }( \mathcal {P}(\cdot ,v))$ be the set of all $\mu$-equivalence classes of measurable functions, $u\in \mathcal {L}_{Y}^{\infty }$, such that $u(\omega )\in$ $\mathcal {P}(\omega ,v)$ a.e. $[\mu ]$. It follows from Blackwell’s Theorem [9], extended to DSGs, that our discounted stochastic game of club network formation (10) will have stationary Markov perfect equilibria in network formation strategies if and only if there exists a value function profile, $v^{*}\in \mathcal {L} _{Y}^{\infty }$, such that

$$\begin{aligned} v^{*}(\omega )\in \mathcal {P}(\omega ,v^{*})\text { a.e. }[\mu ]\text { ,} \end{aligned}$$

or equivalently if and only if the Nash payoff selection correspondence,

$$\begin{aligned} v\longrightarrow \mathcal {S}^{\infty }(\mathcal {P}(\cdot ,v)):=\mathcal {S} ^{\infty }(\mathcal {P}_{v}), \end{aligned}$$

has fixed points, i.e., has at least one value function profile, $v^{*}\in \mathcal {L}_{Y}^{\infty }$, such that $v^{*}\in \mathcal {S}^{\infty }(\mathcal {P}_{v^{*}})$. This is a very difficult fixed point problem because the measurable selection valued correspondence, $\mathcal {S}^{\infty }(\mathcal {P}_{(\cdot )})$, is neither convex valued nor closed value—nor of course is it upper semicontinuous. Until Fu and Page [15] no results were available to establish the existence of a fixed point for Nash payoff selection correspondences. Essentially what Fu and Page [15] show is that while the Nash payoff selection correspondence, $v\longrightarrow \mathcal {S} ^{\infty }(\mathcal {P}_{v})$, is badly behaved, under assumptions [A-1], its underlying upper Caratheodory Nash payoff correspondence, $\mathcal {P}(\cdot ,\cdot )$, always contains an upper Caratheodory sub-correspondence, $p(\cdot ,\cdot )$, that has $\varepsilon$-approximate Caratheodory selections for all $\varepsilon >0$, implying that the Nash payoff selection sub-correspondence, $v\longrightarrow \mathcal {S}^{\infty }(p_{v})$, has fixed points. Thus, Fu and Page [15] show that, in general, there exists a Nash payoff sub-correspondence, $\mathcal {S}^{\infty }(p_{(\cdot )})$, having fixed points, i.e., that there exists $v^{*}\in \mathcal {L} _{Y}^{\infty }$, such that $v^{*}\in \mathcal {S}^{\infty }(p_{v^{*}})\subset \mathcal {S}^{\infty }(\mathcal {P}_{v^{*}})$. More fundamentally, Fu and Page [15] are able to show that the upper Caratheodory Nash payoff sub-correspondence, $p(\cdot ,\cdot )$, has $\varepsilon$-approximate Caratheodory selections for all $\varepsilon >0$, because under assumption [A-1] the underlying upper Caratheodory Nash correspondence, $\mathcal {N}(\cdot ,\cdot )$, always contains an upper Caratheodory Nash sub-correspondence, $\eta (\cdot ,\cdot )$, that takes closed connected values. Thus, because

$$\begin{aligned} \begin{array}{ll} p(\omega ,v)&{} =(p_{1}(\omega ,v),\ldots ,p_{n}(\omega ,v)) \\ \\ &{} :=(U_{1}(\omega ,v_{1},\eta (\omega ,v)),\ldots ,U_{n}(\omega ,v_{n},\eta (\omega ,v))), \end{array} \end{aligned}$$

the continuity of $U_{d}(\omega ,v_{d},\cdot )$ in behavioral actions $\sigma$ for each player $d=1,2,\ldots ,n$ together with the closed connectedness of $\eta (\omega ,v)$ implies that

$$\begin{aligned} (\omega ,v)\longrightarrow p_{d}(\omega ,v)=U_{d}(\omega ,v_{d},\eta (\omega ,v)) \end{aligned}$$

is upper Caratheodory and interval valued for each player $d=1,2,\ldots ,n$. It then follows from Corollary 4.3 in Kucia and Nowak [22] that each player’s Nash payoff sub-correspondence, $p_{d}(\cdot ,\cdot )$, is Caratheodory approximable and this together with the unusual properties of Komlos convergence [21] of value functions allows us to show that there exists a value function profile,, $v^{*}\in \mathcal {L} _{Y}^{\infty }$, such that $v^{*}(\omega )\in p(\omega ,v^{*})$ a.e. $[\mu ]$ or equivalently such that

$$\begin{aligned} v^{*}\in \mathcal {S}^{\infty }(p_{v^{*}})\subset \mathcal {S}^{\infty }(\mathcal {P}_{v^{*}}). \end{aligned}$$

In order to complete our informal argument for existence we need only note that by implicit measurable selection (e.g., see Theorem 7.1 in Himmelberg [18]), there exists a profile, $\sigma ^{*}(\cdot )=(\sigma _{1}^{*}(\cdot ),\ldots ,\sigma _{m}^{*}(\cdot ))$, of a.e. measurable selections of $\omega \longrightarrow \eta (\omega ,v^{*})$, such that for each player $d=1,2,\ldots ,n$,

$$\begin{aligned} v_{d}^{*}(\omega )=U_{d}(\omega ,v_{d}^{*},\sigma ^{*}(\omega ))\in U_{d}(\omega ,v_{d}^{*},\eta (\omega ,v^{*})):=p_{d}(\omega ,v^{*})\text { a.e. }[\mu ], \end{aligned}$$

(19)

Thus, for each player d, the state-contingent prices given by value function, $v_{d}^{*}(\cdot )\in \mathcal {L}_{Y_{d}}^{\infty }$, incentivize the continued choice by each player d, of behavioral strategy, $\sigma _{d}^{*}(\cdot )$, and we have for the value function-strategy profile pair, $(v^{*},x^{*}(\cdot ))\in \mathcal {S} ^{\infty }(p_{v^{*}})\times \mathcal {S}^{\infty }(\eta _{v^{*}})$, that

$$\begin{aligned} v^{*}(\omega )=U(\omega ,v^{*},\sigma ^{*}(\omega ))\in p(\omega ,v^{*})\text { and }\sigma ^{*}(\omega )\in \eta (\omega ,v^{*}) \text { a.e. }[\mu ]\text {,} \end{aligned}$$

(20)

implying that

$$\begin{aligned} v^{*}(\omega )\in \mathcal {P}(\omega ,v^{*})\text { and }\sigma ^{*}(\omega )\in \mathcal {N}(\omega ,v^{*})\text { a.e. }[\mu ]\text {. } \end{aligned}$$

(21)

Thus, for value function-behavioral strategy profile pair, $(v^{*},\sigma ^{*}(\cdot ))$, we have for each player $d=1,2,\ldots ,n$ and for $\omega$ a.e. $[\mu ]$, that $(v^{*},\sigma ^{*}(\cdot ))$ satisfies the Bellman equation (1 below) and the Nash condition (2 below),

$$\begin{aligned} \left. \begin{array}{l} \text {(1) }v_{d}^{*}(\omega )=(1-\beta _{d})r_{d}(\omega ,\sigma _{d}^{*}(\omega ),\sigma _{-d}^{*}(\omega ))+\beta _{d}\int _{\Omega }v_{d}^{*}(\omega ^{\prime })q(\omega ^{\prime }|\omega ,\sigma _{d}^{*}(\omega ),\sigma _{-d}^{*}(\omega )), \\ \\ \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \text {and} \\ \\ \text {(2) }(1-\beta _{d})r_{d}(\omega ,\sigma _{d}^{*}(\omega ),\sigma _{-d}^{*}(\omega ))+\beta _{d}\int _{\Omega }v_{d}^{*}(\omega ^{\prime })q(\omega ^{\prime }|\omega ,\sigma _{d}^{*}(\omega ),\sigma _{-d}^{*}(\omega )) \\ \\ =\max _{\sigma _{d}\in \Delta (\Phi _{d}(\omega ))}\left[ (1-\beta _{d})r_{d}(\omega ,\sigma _{d},\sigma _{-d}^{*}(\omega ))+\beta _{d}\int _{\Omega }v_{d}^{*}(\omega ^{\prime })q(\omega ^{\prime }|\omega ,\sigma _{d},\sigma _{-d}^{*}(\omega ))\right] . \end{array} \right\} \end{aligned}$$

(22)

Thus, $\sigma ^{*}(\cdot )\in$ $\mathcal {S}^{\infty }(\mathcal {N} _{v^{*}})$ is a stationary Markov perfect equilibrium of a DSG satisfying assumptions [A-1], incentivized by state-contingent prices, $v^{*}\in$ $\mathcal {S}^{\infty }(\mathcal {P}_{v^{*}})$.

Thus here we have argued informally—and shown formally in Fu and Page [15]—that, under the usual assumptions specifying a discounted stochastic game (in this case a club network formation DSG), while the $DSG^{\prime }s$ Nash payoff selection is badly behaved, nonetheless, it naturally possesses (without additional assumptions) an underlying Nash payoff correspondence, $\mathcal {P}(\cdot ,\cdot )$, containing sub-correspondences, $p(\cdot ,\cdot )$, which are Caratheodory approximable implying that its induced selection correspondence, $v\longrightarrow \mathcal {S}^{\infty }(p_{v^{*}})$, has fixed points. He and Sun [16], by making an additional assumption (that the DSG is $\mathcal {G}$-nonatomic or has a coarser transition kernel) guarantee that $DSG^{\prime }s$ Nash payoff selection correspondence has a convex valued sub-correspondence—and therefore an approximable sub-correspondence.⁹ Moreover, He and Sun [16] show that Duggan [12] accomplishes the same thing by assuming that the DSG has a noisy state. In the negative direction, Levy [23] and Levy and McLennan [24] construct counterexamples showing that not all uncountable-finite DSGs have stationary Markov perfect equilibria. They accomplish this by constructing counterexamples in which the Nash correspondences are not approximable—which follows from the fact that in their counterexamples, there is an absence of fixed points. Because the club network formation DSG we analyze here is approximable, we avoid the Levy–McLennan counterexamples.

6.2 Example: DSGs over Time Allocation Networks

The one-shot game for a DSG satisfying assumptions [A-1] over time allocation networks, $\mathbb {K}_{TA}$, is given by

$$\begin{aligned} \mathcal {G}_{\Omega \times \mathcal {L}_{Y}^{\infty }}:=\{\left( \Delta (\Phi _{d}(\omega )),U_{d}(\omega ,v_{d},\cdot )\right) _{d}\}_{(\omega ,v)\in \Omega \times \mathcal {L}_{Y}^{\infty }}. \end{aligned}$$

(23)

In a $(\omega ,v)$-game each player d has preferences over the d-layer networks in $\mathbb {K}^{d}$ given by,

$$\begin{aligned} G^{d}\longrightarrow u_{d}(\omega ,v_{d},\underset{G}{\underbrace{ G^{d},G^{-d}}}), \end{aligned}$$

(24)

where $G^{d}=(G_{d1},\ldots ,G_{dm})$ and $G_{dc}=\{(a_{dc},(d,c))\}$ for $c\in \mathcal {D}(G(d,\cdot ))$, and $G_{dc}=\varnothing$ for $c\notin \mathcal {D}(G(d,\cdot ))$. Rewriting these preferences as preferences over d-layer networks in reduced from, we have

$$\begin{aligned} a^{d}\longrightarrow u_{d}(\omega ,v_{d},\underset{(G^{d},G^{-d})}{ \underbrace{a^{d},a^{-d}}}), \end{aligned}$$

(25)

where

$$\begin{aligned} G^{d}:=a^{d}=(a_{dc_{1}},\ldots ,a_{dc_{m}})\in \Delta ^{m}\subset [0,1]^{m}, \end{aligned}$$

(26)

and where

$$\begin{aligned} \Delta ^{m}:=\{(a_{dc_{1}},\ldots ,a_{dc_{m}})\in R_{+}^{m}:\sum _{k=1}^{m}a_{dc_{k}}=1\}, \end{aligned}$$

(27)

with notational convention, $a_{dc_{k}}\in (0,1]\Longleftrightarrow G_{dc_{k}}=\{(a_{dc_{k}},(d,c_{k}))\}$ and $a_{dc_{k}}=0\Longleftrightarrow G_{dc_{k}}=\varnothing$.

We will assume that for each player d and for each $(\omega ,v)\in \Omega \times \mathcal {L}_{Y}^{\infty }$, $u_{d}(\omega ,v_{d},\cdot )$ is affine on $\mathbb {K}_{TA}$. Thus, for time allocation networks, G and $G^{\prime }$ in $\mathbb {K}_{TA}$, we have for any $\lambda \in [0,1]$,

$$\begin{aligned} u_{d}(\omega ,v_{d},\lambda G+(1-\lambda )G^{\prime })=\lambda u_{d}(\omega ,v_{d},G)+(1-\lambda )u_{d}(\omega ,v_{d},G^{\prime }), \end{aligned}$$

(28)

where

$$\begin{aligned} \left. \begin{array}{l} \lambda G+(1-\lambda )G^{\prime } \\ =\lambda \left( \begin{array}{ccccc} a_{11} &{} \cdots &{} a_{1c} &{} \cdots &{} a_{1m} \\ \vdots &{} &{} \vdots &{} &{} \vdots \\ a_{d1} &{} \cdots &{} a_{dc} &{} \cdots &{} a_{dm} \\ \vdots &{} &{} \vdots &{} &{} \vdots \\ a_{n1} &{} \cdots &{} a_{nc} &{} \cdots &{} a_{nm} \end{array} \right) +(1-\lambda )\left( \begin{array}{ccccc} a_{11}^{\prime } &{} \cdots &{} a_{1c}^{\prime } &{} \cdots &{} a_{1m}^{\prime } \\ \vdots &{} &{} \vdots &{} &{} \vdots \\ a_{d1}^{\prime } &{} \cdots &{} a_{dc}^{\prime } &{} \cdots &{} a_{dm}^{\prime } \\ \vdots &{} &{} \vdots &{} &{} \vdots \\ a_{n1}^{\prime } &{} \cdots &{} a_{nc}^{\prime } &{} \cdots &{} a_{nm}^{\prime } \end{array} \right) \\ \\ =\left( \begin{array}{ccccc} {\small \lambda a}_{11}{\small +(1-\lambda )a}_{11}^{\prime } &{} \cdots &{} {\small \lambda a}_{1c}{\small +(1-\lambda )a}_{1c}^{\prime } &{} \cdots &{} {\small \lambda a}_{1m}{\small +(1-\lambda )a}_{1m}^{\prime } \\ \vdots &{} &{} \vdots &{} &{} \vdots \\ {\small \lambda a}_{d1}{\small +(1-\lambda )a}_{d1}^{\prime } &{} \cdots &{} {\small \lambda a}_{dc}{\small +(1-\lambda )a}_{dc}^{\prime } &{} \cdots &{} {\small \lambda a}_{dm}{\small +(1-\lambda )a}_{dm}^{\prime } \\ \vdots &{} &{} \vdots &{} &{} \vdots \\ {\small \lambda a}_{n1}{\small +(1-\lambda )a}_{n1}^{\prime } &{} \cdots &{} {\small \lambda a}_{nc}{\small +(1-\lambda )a}_{nc}^{\prime } &{} \cdots &{} {\small \lambda a}_{nm}{\small +(1-\lambda )a}_{nm}^{\prime } \end{array} \right) . \end{array} \right\} \end{aligned}$$

(29)

We will also assume that players’ one-shot payoff functions,

$$\begin{aligned} (\omega ,v_{d},a^{d},a^{-d})\longrightarrow u_{d}(\omega ,v_{d},a^{d},a^{-d}) \text {,} \end{aligned}$$

are measurable in $\omega$ for each $(v_{d},a^{d},a^{-d})$ and jointly continuous in $(v_{d},a^{d},a^{-d})$ for each $\omega$.

By Theorem 1 above (also see Fu and Page [15]), there exists a value function profile, $v^{*}\in \mathcal {L}_{Y}^{\infty }$ and an n-tuple of measurable function

$$\begin{aligned} (a^{*d_{1}}(\omega ),\ldots ,a^{*d_{n}}(\omega ))\in \underset{n\text {-times}}{\underbrace{\Delta ^{m}\times \Delta ^{m}\times \cdots \times \Delta ^{m}}}. \end{aligned}$$

where by notational convention,

$$\begin{aligned} \begin{array}{ll} a^{*d_{i}}(\omega ) &{}:=G^{*d_{i}}(\omega ) \\ \\ &{} =(G_{d_{i}c_{1}}^{*}(\omega ),\ldots ,G_{d_{i}c_{m}}^{*}(\omega )) \\ \\ &{} :=((a^*_{d_{i}c_{1}}(\omega ),(d_{i},c_{1})),\ldots ,(a^{*}_{d_{i}c_{m}}(\omega ),(d_{i},c_{m}))), \end{array} \end{aligned}$$

and where by convention,

$$\begin{aligned} \begin{array}{ll} (a_{d_{i}c_{k}}^*(\omega ),(d_{i},c_{k}))= &{} \left\{ \begin{array}{lll} \left\{ (1,(d_{i},c_{k}))\right\} &{} \text {if }d_{i}\text { spends }100\%\text { of club time in club }c_{k} &{} \\ &{} &{} \\ \left\{ (0,(d_{i},c_{k}))\right\} &{} \text {if }d_{i}\text { is not a member of club }c_{k}\text {.} &{} \end{array} \right. \end{array} \end{aligned}$$

In fact, given the assumptions that players’ one-shot payoff functions are affine over the convex set of time allocation networks (see 28 and 29 above), we know by Corollary 1.4 in Balder [4] that the m-tuple,

$$\begin{aligned} (a_{d_{i}c_{1}}^*(\omega ),\ldots ,a_{d_{i}c_{m}}^*(\omega )) \end{aligned}$$

consists of $1^{\prime }s$ and $0^{\prime }s$, because for each club $c_{k}$, $a_{d_{i}c_{k}}^*(\omega )$ is an extreme point of the interval [0, 1] a.e. $[\mu ]$. Thus, in each state a player spends $100\%$ of his time in one and only one club—i.e., each player’s stationary Markov perfect time allocation strategy is bang-bang. Thus we have by Theorem 1 above that

$$\begin{aligned} v^{*}(\omega )\in \mathcal {P}(\omega ,v^{*})\text { and }G^{*}(\omega )\in \mathcal {N}(\omega ,v^{*})\text { a.e. }[\mu ]\text {,} \end{aligned}$$

where by Corollary 1.3 in Balder [4], a.e. $[\mu ]$ in $\omega$, the stationary Markov perfect time allocation network, given by the arrays,

$$\begin{aligned} G^{*}(\omega ):=(G_{dc}^{*}(\omega ))_{dc}:=(a_{dc}^{*}(\omega ))_{dc}\text {,} \end{aligned}$$

has a reduced form $n\times m$ array, $(a_{dc}^{*}(\omega ))_{dc}$, consisting of $0^{\prime }s$ and $1^{\prime }s$. Moreover, in equilibrium, the state-contingent coalition of members of any club $c_{k}$ determined by the equilibrium club network $G^{*}(\cdot )$ is given by

$$\begin{aligned} S_{c_{k}G^{*}(\omega )}=\mathcal {D}(G^*(\omega )(\cdot ,c_{k}))=\{d:a_{dc_{k}}^{*}(\omega )=1\}. \end{aligned}$$

7 Equilibrium State-Network Dynamics and Stable Coalitions

Returning to our general model, under the profile of stationary Markov perfect equilibrium behavioral strategies

$$\begin{aligned} \sigma ^{*}(\cdot ):=(\sigma _{1}^{*}(\cdot ),\ldots ,\sigma _{n}^{*}(\cdot )), \end{aligned}$$

(30)

the equilibrium state and network formation process is given by,

$$\begin{aligned} \left\{ Z_{t}^{*}\right\} _{t=0}^{\infty }:=\left\{ (W_{t}^{*},G_{t}^{*})\right\} _{t=0}^{\infty }, \end{aligned}$$

(31)

with underlying probability space $(\Omega \times \mathbb {K},B_{\Omega }\times B_{\mathbb {K}},\mu \otimes \sigma ^{*}):=(\mathbb {Z},B_{\mathbb {Z }},P^{*})$ consists of random objects,

$$\begin{aligned} z\longrightarrow W_{t}^{*}(z):=W_{t}^{*}(\omega ,G)\text { and } z\longrightarrow G_{t}^{*}(z):=G_{t}^{*}(\omega ,G), \end{aligned}$$

such that in each state $z=(\omega ,G)\in \Omega \times \mathbb {K}$,

$$\begin{aligned} W_{t}^{*}(\omega ,G)=\omega \text { and }G_{t}^{*}(\omega ,G)=G, \end{aligned}$$

(32)

and where for any measurable set $E\in B_{\Omega }\times B_{\mathbb {K}}$,

$$\begin{aligned} P^{^{*}}(E)=\int _{\Omega }\sigma ^{*}(E_{\omega }|\omega )d\mu (\omega ), \end{aligned}$$

(33)

$E_{\omega }:=\{G\in$ $\mathbb {K}:(\omega ,G)\in E\}$ (see 2.6.2, the Product Measure Theorem, in Ash [3]).

The movements of the process, $\left\{ Z_{t}^{*}\right\} _{t=0}^{\infty }$, are governed by the equilibrium Markov transition kernel,

$$\begin{aligned} (\omega ,G)\longrightarrow \pi ^{*}(d(\omega ^{\prime },G^{\prime })|\omega ,G). \end{aligned}$$

For each $(\omega ,G)\in \Omega \times \mathbb {K}$, $\pi ^{*}(d(\omega ^{\prime },G^{\prime })|\omega ,G)$ is given by the product probability measure, $q(\cdot |\omega ,G)\otimes Q^{*}(\cdot |\omega ,G)$, where for each measurable rectangle, $R\times \mathbb {R}\in B_{\Omega }\times B_{ \mathbb {K}}$,

$$\begin{aligned} \left. \begin{array}{l} \pi ^{*}(R\times \mathbb {R}|\omega ,G):=q(R|\omega ,G)\otimes Q^{*}( \mathbb {R}|\omega ,G), \\ \\ :=q(R|\omega ,G)\otimes \int _{\Omega }\sigma ^{*}(\mathbb {R}|\omega ^{\prime })q(d\omega ^{\prime }|\omega ,G)\text {.} \end{array} \right\} \end{aligned}$$

(34)

Thus, if the current state-club network is $(\omega ,G)=(\omega ,G^{1},\ldots ,G^{n})$, with d-layers $(G^{d})_{d=1}^{n}$, then the probability that the coming state and coming club network lie in the set, $R\times \mathbb {R}\in B_{\Omega }\times B_{\mathbb {K}}$, is given by

$$\begin{aligned} \pi ^{*}(R\times \mathbb {R}|\omega ,G):=q(R|\omega ,G)\otimes Q^{*}( \mathbb {R}|\omega ,G) \end{aligned}$$

(35)

Whether or not a stable club network emerges depends on the stability properties of equilibrium state-network dynamics underlying club network formation. Our main objective is to say what we can regarding the emergence of stable club structures—and hence stable coalition structures—based on the stability properties of the equilibrium state-network formation dynamics. We would argue that this is one of the main aspects of network dynamics that has been missing from our understanding of network dynamics in various applied settings—the influence of strategic behavior on network dynamics. Here we present a first attempt. We will proceed as follows: First we state the Tweedie (Stability) Conditions [37] guaranteeing Markov stability. Second, we show that if we slightly strengthen assumptions [A-1] (2) and (15) in our discounted stochastic game model, then the stationary Markov perfect equilibrium (SMPE) of our discounted stochastic game of network formation gives rise to a Markov stable equilibrium state-network formation process satisfying the Tweedie Conditions. Finally, we summarize some of the main implications of the Tweedie Conditions and the resulting Markov stability properties of the equilibrium state-network formation process—such as the existence of finitely many basins of attraction and ergodic probabilities, and the implications of Markov stability for the emergence of stable coalition structures.

We begin by stating the Tweedie Conditions, and after strengthening assumptions [A-1] (2) and (15)(iii) in our discounted stochastic game model, showing that our strengthened discounted stochastic game model, in general, gives rise to an equilibrium state-network process satisfying the Tweedie conditions.

7.1 The Tweedie Conditions and Discounted Stochastic Games of Club Network Formation

We say that the Markov transition, $\pi ^{*}(\cdot |\cdot )$, satisfies the Tweedie conditions, [T], provided,

(1) (Drift Condition) There exists (i) a nonnegative-valued measurable function, $V(\cdot ):\Omega \times \mathbb {K}\longrightarrow [0,+\infty ]$, with $V(z^{0})<+\infty$ for some $z^{0}\in \Omega \times \mathbb {K}$; (ii) a subset $C\subset \Omega \times \mathbb {K}$, and (iii) a finite real number, $-\infty<b<+\infty$ , such that we have

$$\begin{aligned} \int _{\Omega \times \mathbb {K}}V(z^{\prime })\pi ^{*}(dz^{\prime }|z)\le V(z)-1+bI_{C}(z). \end{aligned}$$

(36)

(2) (Uniform Countable Additivity) For any sequence of measurable sets, $\{S_{n}\}_{n}\subset B_{\Omega }\times B_{\mathbb {K}}$ with $S_{n}\downarrow \varnothing$,

$$\begin{aligned} \lim _{n\longrightarrow +\infty }\sup _{z\in C}\pi ^{*}(S_{n}|z)=0. \end{aligned}$$

(37)

The intuition behind the Tweedie conditions can be described as follows: Rewriting first condition (36)—the drift condition—we have for each $z\in \Omega \times \mathbb {K}$

$$\begin{aligned} \left. \begin{array}{l} \int _{\Omega \times \mathbb {K}}V(z^{\prime })\pi ^{*}(dz^{\prime }|z)\le V(z)-1+bI_{C}(z) \\ \text {if and only if} \\ \underset{\text {{ drift}}}{\underbrace{\triangle V(z)}}\le \underset{ \text {{ state-contingent bound}}}{\underbrace{bI_{C}(z)-1},} \end{array} \right\} \end{aligned}$$

(38)

where

$$\begin{aligned} \triangle V(z):=\int _{\mathbb {Z}}[V(z^{\prime })-V(z)]\pi ^{*}(dz^{\prime }|z) \end{aligned}$$

is the drift operator evaluated at $z=(\omega ,G)\in \Omega \times \mathbb {K}$. The drift operator measures the expected drift (or movement) of the process away from states in C starting from state z. Drift is measured by the value taken by the drift operator, $\triangle V(\cdot )$, at z. We see from the state-contingent bound on the right-hand side of (38) that if we are measuring expected drift from a state z in C and if $b>1$, so that $b-1>0$, then condition (36) will tolerate some drift away from C. However, if we are measuring expected drift from a state z not in C, then condition (36) requires that the expected drift be back toward C. The first Tweedie condition requires that the equilibrium process be such that there exist such a function, $V(\cdot ):\Omega \times \mathbb {K}\longrightarrow [0,+\infty ]$, finite valued at least at one point, a set C, and a bounding constant, b, such (36) is satisfied. The second Tweedie condition requires that the collection of probability measures,

$$\begin{aligned} \pi ^{*}(\Omega \times \mathbb {K}):=\{\pi ^{*}(dz^{\prime }|z):z\in \Omega \times \mathbb {K}\}\text {,} \end{aligned}$$

determined by discounted stochastic game’s strengthened law of motion and the equilibrium behavioral strategies of the players be uniformly countably additive (37)—guaranteeing therefore that the expected drift of the process is a continuous function of state-network pairs (i.e., that there are no discontinuous jumps in the expected magnitude of the drift).

The required strengthening of our stochastic continuity assumptions [A-1](15) is the following:

[A-1](15)(iii)$^{*}$In addition to [A-1](15)(iii), we will assume that for any sequence of state-club network pairs, $\{(\omega ^{n},G^{n})\}_{n}$, converging to $(\omega ^{*},G^{*})$ under the metric $\rho _{\Omega \times \mathbb {K}}:=\rho _{\Omega }+h_{K}$ on the product of the state space $\Omega$ and the club network space $\mathbb {K}$,

$$\begin{aligned} q(R|\omega ^{n},G^{n})\longrightarrow q(R|\omega ^{*},G^{*})\text { } \end{aligned}$$

(39)

for any$R\in B_{\Omega }$, i.e., the sequence of probability measures, $\{q(\cdot |\omega ^{n},G^{n})\}_{n}$ converges setwise to $q(\cdot |\omega ^{*},G^{*})$.

Under the strengthened stochastic continuity assumptions, labeled [A-1](15)$^{*}$, because the sequence, $\{q(R|\omega ^{n},G^{n})\}_{n}$, converges to $q(R|\omega ^{*},G^{*})$ uniformly in $R\in B_{\Omega }$ and because $\sigma ^{*}(\mathbb {R}|\cdot )$ is a bounded measurable function for any $\mathbb {R\in }B_{\mathbb {K}}$, it follows from Delbaen’s lemma [11] that $\{Q^{*}(\cdot |\omega ^{n},G^{n})\}_{n}$ setwise converges to $Q^{*}(\cdot |\omega ^{*},G^{*})$, and in particular that,

$$\begin{aligned} \left. \begin{array}{l} Q^{*}(\mathbb {R}|\omega ^{n},G^{n}):=\int _{\Omega }\sigma ^{*}( \mathbb {R}|\omega ^{\prime })q(d\omega ^{\prime }|\omega ^{n},G^{n}) \\ \longrightarrow \\ \int _{\Omega }\sigma ^{*}(\mathbb {R}|\omega ^{\prime })q(d\omega ^{\prime }|\omega ^{*},G^{*})=Q^{*}(\mathbb {R}|\omega ^{*},G^{*}) \end{array} \right\} \end{aligned}$$

(40)

for any $\mathbb {R\in }B_{\mathbb {K}}$. Thus, given that $\{q(\cdot |\omega ^{n},G^{n})\}_{n}$ and $\{Q^{*}(\cdot |\omega ^{n},G^{n})\}_{n}$ setwise converge to $q(\cdot |\omega ^{*},G^{*})$ and $Q^{*}(\cdot |\omega ^{*},G^{*})$ for any sequence of state-club network pairs, $\{(\omega ^{n},G^{n})\}_{n}$, converging to $(\omega ^{*},G^{*})$, we have that

$$\begin{aligned} \pi ^{*}(R\times \mathbb {R}|\omega ^{n},G^{n})\longrightarrow \pi ^{*}(R\times \mathbb {R}|\omega ^{*},G^{*}) \end{aligned}$$

(41)

for any measurable rectangle, $R\times$ $\mathbb {R\in B}_{\Omega }\times B_{ \mathbb {K}}$ i.e., that $\{\pi ^{*}(\cdot |\omega ^{n},G^{n})\}_{n}$, setwise converges to $\pi ^{*}(\cdot |\omega ^{*},G^{*})$, for any sequence, $\{(\omega ^{n},G^{n})\}_{n}$, converging to $(\omega ^{*},G^{*})$. Moreover, we know that if $\{\pi ^{*}(\cdot |z^{n})\}_{n}$ setwise converges to $\pi ^{*}(\cdot |z^{*})$ for any sequence $\{z^{n}\}_{n}=\{(\omega ^{n},G^{n})\}_{n}$, converging to $z^{*}=(\omega ^{*},G^{*})$, then for any bounded and continuous function, $c(\cdot )\in C_{b}(\Omega \times \mathbb {K})$—or any bounded measurable function

$$\begin{aligned} \int _{\Omega \times \mathbb {K}}c(z^{\prime })\pi ^{*}(dz^{\prime }|z^{n})\longrightarrow \int _{\Omega \times \mathbb {K}}c(z^{\prime })\pi ^{*}(dz^{\prime }|z^{*}). \end{aligned}$$

(42)

i.e., $\{\pi ^{*}(\cdot |z^{n})\}_{n}$ converges weak star to $\pi ^{*}(\cdot |z^{*})$.

Up until now, we have assumed that the set of states, $\Omega$, is Polish (complete, separable, metric). Our strengthening of [A-1](2) is to assume that

[A-1](2)$^{*}$ In the probability space of states, $(\Omega ,B_{\Omega },\mu )$, $\Omega$, is a compact metric space.

Because the hyperspace of club networks, $\mathbb {K}$, is compact, with the strengthening of [A-1](2) making $\Omega$ compact, we have for any sequence, $\{(\omega ^{n},G^{n})\}_{n}$, in $\Omega \times \mathbb {K}$ that there is a convergent subsequence $\{(\omega ^{n_{k}},G^{n_{k}})\}_{k}$ with limit $(\omega ^{*},G^{*})\in \Omega \times \mathbb {K}$. Given the arguments and observations immediately above this means that the set of probability measures,

$$\begin{aligned} \Pi :=\{\pi ^{*}(\cdot |z):z\in \Omega \times \mathbb {K}\}, \end{aligned}$$

is weak star compact—because $Z:=\Omega \times \mathbb {K}$ is compact, any sequence $\{z^{n}\}_{n}$ has a subsequence converging to some $z^{*}\in Z$, implying that any sequence of probability measures, $\{\pi ^{*}(\cdot |z^{n})\}_{n}$, has a subsequence, $\{\pi ^{*}(\cdot |z^{n_{k}})\}_{k}$, setwise converging and therefore weak star converging to $\pi ^{*}(\cdot |z^{*})$. Moreover, by Proposition 1.4.2(a)(ii) in Hernandez-Lerma and Laserre [16], because $\Pi :=\{\pi ^{*}(\cdot |z):z\in \Omega \times \mathbb {K}\}$ is weak star compact, the equilibrium Markov transition kernel, $z\longrightarrow \pi ^{*}(dz^{\prime }|z)$, is uniformly countably additive on any closed subset C of $\Omega \times \mathbb {K}$. Thus, the equilibrium Markov kernel, $\pi ^{*}(\cdot |\cdot )$, governing the state-network process, $Z_{t},$ is such that for any closed subset C of $\Omega \times \mathbb {K}$,

$$\begin{aligned} \lim _{n\longrightarrow +\infty }\sup _{z\in C}\pi ^{*}(S_{n}|z)=0, \end{aligned}$$

(43)

for any sequence of states $\{S_{n}\}\subset B_{\Omega }\times B_{\mathbb {K} }$ with $S_{n}\downarrow \varnothing$.

We will denote by [A-1]$^{*}$ our list of assumptions [A-1] but with [A-1](2) and [A-1](15)(iii) strengthened as above. Our next result, which follows directly from the arguments above, states that any equilibrium state-network process determined by the stationary Markov perfect equilibrium of a DSG of club network formation satisfying assumptions [A-1] $^{*}$ satisfies the Tweedie conditions.

Theorem 2

(All DSGs satisfying [A-1]$^{*}$ induce equilibrium processes satisfying the Tweedie conditions) Any state-network Markov transition,

$$\begin{aligned} \begin{array}{l} \pi ^{*}(d(\omega ^{\prime },G^{\prime })|\omega ,G):=q(d\omega ^{\prime }|\omega ,G)\otimes Q^{*}(dG^{\prime }|\omega ,G), \\ \\ :=q(d\omega ^{\prime }|\omega ,G)\otimes \int _{\Omega }\sigma ^{*}(dG^{\prime }|\omega ^{\prime })q(d\omega ^{\prime }|\omega ,G)\text {,} \end{array} \end{aligned}$$

determined by a stationary Markov perfect behavioral club network formation strategy,

$$\begin{aligned} \sigma ^{*}(\cdot ):=(\sigma _{1}^{*}(\cdot ),\ldots ,\sigma _{n}^{*}(\cdot )), \end{aligned}$$

of a discounted stochastic game satisfying assumptions [A-1]$^{*}$ satisfies the Tweedie conditions. In particular, for $V(z)=1$ for all $z\in \Omega \times \mathbb {K}$, $b=2$, and $C=\Omega \times \mathbb {K}$ we have

$$\begin{aligned} \int _{\Omega \times \mathbb {K}}V(z^{\prime })\pi ^{*}(dz^{\prime }|z)\le V(z)-1+bI_{C}(z)\text { for all }z\in \Omega \times \mathbb {K}\text {, } \end{aligned}$$

and for any sequence of sets of states $\{S_{n}\}_{n}\subset B_{\Omega }\times B_{\mathbb {K}}$ with $S_{n}\downarrow \varnothing$

$$\begin{aligned} \lim _{n\longrightarrow +\infty }\sup _{z\in \Omega \times \mathbb {K}}\pi ^{*}(S_{n}|z)=0. \end{aligned}$$

In our DSG over time allocation networks discussed above, if we strengthen the stochastic continuity assumptions [A-1](15)(iii) to [A-1](15)(iii)$^{*}$ and if we assume that the state space, $\Omega$, is compact, then the equilibrium state-network process induced by players’ bang-bang strategies will satisfy the Tweedie conditions. Moreover, if $\Omega$ is compact and we assume that the set of conditional densities,

$$\begin{aligned} H_{\mu }:=\left\{ h(\cdot |\omega ,G):(\omega ,G)\in \Omega \times \mathbb {K} \right\} , \end{aligned}$$

of $q(\cdot |\omega ,G)$ with respect to $\mu$ is such that the function

$$\begin{aligned} (\omega ,G)\longrightarrow h(\omega ^{\prime }|\omega ,G) \end{aligned}$$

is $\rho _{\Omega \times \mathbb {K}}$-continuous ($\rho _{\Omega \times \mathbb {K}}:=\rho _{\Omega }+h_{K}$) in $\omega$ and G a.e. $[\mu ]$ in $\omega ^{\prime }$, then the Tweedie conditions will be satisfied in our DSG over time allocation networks.

8 Strategically Stable Club Networks and Coalition Structures

By Theorem 2 above, the equilibrium state-network transition, $\pi ^{*}(\cdot |\cdot )$, governing the movements of the state-network process, $\{Z_{t}^{*}\}_{t}$, through the space, $\Omega \times \mathbb {K}$, satisfies the Tweedie conditions. As a consequence, as shown by Tweedie [37], the space $\Omega \times \mathbb {K}$ can be decomposed in a unique way into a transient set of state-network pairs and a finite collection of largest absorbing sets (i.e., basins of attraction given by Harris sets) consisting of state-network pairs that might persist in the long run (and will persist in the long run if there is only one basin of attraction) and that exhaust all the possibilities for what will happen in the future with regard to the path of the network formation process.¹⁰

Restating Tweedie’s decomposition result for the strategically determined equilibrium state-network transition, $\pi ^{*}(\cdot |\cdot )$, we have the following:

Theorem 3

[37, Theorem 2] Let $\{Z_{t}^{*}\}_{t}$ be an equilibrium state-network Markov process governed by the transition kernel, $\pi ^{*}(dz^{\prime }|z)$, satisfying [T]. Then $\pi ^{*}(dz^{\prime }|z)$ uniquely partitions the space $\mathbb {Z}:=\Omega \times \mathbb {K}$ into a finite number of maximal Harris sets, $H_{i}\subset \mathbb {Z}$, and a transient set, $E\subset \mathbb {Z}$, and in particular,

$$\begin{aligned} \mathbb {Z}=\left[ \cup _{i=1}^{I}H_{i}\right] \cup E\text {.} \end{aligned}$$

(44)

The union of the Harris sets, $H_{I}:=\cup _{i=1}^{I}H_{i}$, has the property that starting from any transient $z\in E$, the state-network process $Z_{t}^{*}$ reaches $H_{I}$ (i.e., reaches some Harris set) in finite time with probability 1, and each Harris set, $H_{i}$ has an ergodic probability measure, $\lambda ^{i}$.

We denote the fact that starting from any transient state-network pair, $z=(\omega ,G)\in E$, the state-network process, $Z_{t}^{*}$, reaches $H_{I}$ in finite time with probability 1 by writing, $L(z,H_{I})=1$, where

$$\begin{aligned} L(z,H_{I}):=P_{z}^{*}(\tau _{H_{I}}<\infty )=P_{z}^{*}(Z_{t}^{*} \text { ever enters }H_{I}\text {),} \end{aligned}$$

(45)

denotes the first entry probability and where

$$\begin{aligned} \tau _{H_{I}}=\min \{t\ge 1:Z_{t}^{*}(z)\in H_{I}\} \end{aligned}$$

(46)

is the first enter time.

Because we are concerned with stability, one of the questions we want to answer is for any given subset S of state-network pairs, $z:=(\omega ,G)$, contained in a Harris set, $H_{i}$, how often does the state-network process, $Z_{t}^{*}$, visit state-network pairs in S? Let $\eta _{S}:=\sum _{t=1}^{\infty }I_{S}(Z_{t}^{*}(z))$ denote the number of visits of the process, $Z_{t}^{*}$, to the set of state-network pairs, S, where $I_{S}(\cdot )$ is the indicator function of the set S. We say that a state-network set, $S\subset \mathbb {Z}$, is a vio set (i.e., a set visited infinitely often) if the probability that $Z_{t}^{*}$, leaving from z, visits S infinitely often is 1—a fact denoted by writing

$$\begin{aligned} Q(z,S):=P_{z}^{*}(\eta _{S}=\infty )=1, \end{aligned}$$

where Q(z, S) denotes the vio set probability.

In order to answer the question of how many times the process visits various subsets of state-networks contained in a Harris set, $H_{i}$, the following further refinement of our partition is useful. To begin, because each Harris set $H_{i}$ is a largest absorbing set, if we restrict the process to the maximal Harris set, $H_{i}$, giving us a sub-process, $Z_{t}^{*H_{i}}:=Z_{t}^{*i}$, governed by the Markov kernel, $\pi _{H_{i}}^{*}(\cdot |\cdot ):=\pi _{i}^{*}(\cdot |\cdot )$, then the process, restricted to $H_{i}$, is $\lambda ^{i}(\cdot )$-irreducible (i.e., for all $S\subset H_{i}$ with $S\in B_{H_{i}}$ and $\lambda ^{i}(S)>0$, $L(z,S)>0$ for all $z\in H_{i}$). Because the process, $Z_{t}^{*}$, generates a finite set of largest absorbing sets with each having an ergodic measure, it follows from Theorem 2.18 and Corollary 2.19 in Costa and Dufour [10] that $Z_{t}^{*i}$, governed by the kernel, $\pi _{i}^{*}(\cdot |\cdot )$, is a $\lambda ^{i}(\cdot )$-irreducible, T-process (see p. 124 in Meyn and Tweedie [25], and Definitions 2.1–2.4 in Costa and Dufour [10]). By Theorem 9.3.6, in Meyn and Tweedie [25], each Harris set $H_{i}$ in expression (44) is positive Harris recurrent and by Theorem 9.3.5, in Meyn and Tweedie [25] can be further decomposed as,

$$\begin{aligned} H_{i}=R_{i}\cup E_{i} \end{aligned}$$

(47)

where $R_{i}$ is the set of topological Harris recurrent state-network pairs and $E_{i}$ is topologically transient (i.e., $z^{*i}\in E_{i}$ if and only if $E_{z^{*i}}(\eta _{O_{z^{*i}}})<\infty$).¹¹ Therefore, a state-network pair $z^{*i}=(\omega ^{*i},G^{*i})\in R_{i}$ if and only if

$$\begin{aligned} P_{z^{*i}}(\eta _{O_{z^{*i}}}=\infty )=1 \end{aligned}$$

(i.e., if and only if the process visits each state-network pair contained in any neighborhoods of $z^{*i}\in R_{i}\subset H_{i}$ infinitely often).¹² Essentially, when the process, $Z_{t}^{*}$, enters the maximal Harris set $H_{i}$ it stays there for all future periods and becomes a $\lambda ^{i}(\cdot )$-irreducible, T-process, visiting the topological Harris recurrent state-network pairs, $z^{*i}\in R_{i}$, infinitely often—passing through state-network pairs in $E_{i}$ on its way to state-network pairs in $R_{i}$. Thus, a refinement of the decomposition in Theorem 3 is given by

$$\begin{aligned} \mathbb {Z}=\left[ \cup _{i=1}^{I}(R_{i}\cup E_{i})\right] \cup E\text {.} \end{aligned}$$

(48)

If the process begins at some state-network pair in E, then in finite time with probability 1 the process will leave E and enter one of the basins, $H_{i}=R_{i}\cup E_{i}$, where it will remain, visiting each state-network pair in $R_{i}$ infinitely often with probability 1, and if the process visits a state-network pair in $E_{i}$ then with probability 1 it will leave that state-network pair in finite time never to return, perhaps visiting a different state-network pair in $E_{i}$ and with probability 1 leaving that state-network pair in finite time never to return.

Because the equilibrium Markov transition, $\pi ^{*}(\cdot |\cdot )$, governing the movements of the state-network process, $Z_{t}^{*}$, is strongly stochastically continuous (i.e., because $z^{n}$ converging to $z^{*}$ implies that $\pi ^{*}(\cdot |z^{n})$ converges setwise to $\pi ^{*}(\cdot |z^{*})$), $\pi ^{*}(\cdot |\cdot )$ has the strong Feller property (e.g., see Meyn and Tweedie [25], Section 6.1). Moreover, because $Z_{t}^{*i}$ restricted to $H_{i}$ is a $\lambda ^{i}(\cdot )$-irreducible, T-process, governed by the Markov kernel, $\pi _{i}^{*}(\cdot |\cdot )$, having the strong Feller property, it follows from Theorem 7.1(iii) in Tuominen and Tweedie [36] that because the state-network space is compact (see [A-2]$^{*}$), the set of topologically Harris recurrent (THR) state-network pairs, $R_{i}$ is closed (and hence compact) and $E_{i}$ is open.

Each set of topological Harris recurrent state-network pairs, $R_{i}\subset \mathbb {Z}:=\Omega \times \mathbb {K}$, determines a correspondence,

$$\begin{aligned} \omega \longrightarrow R_{i}(\omega ):=\{G\in \mathbb {K}:(\omega ,G)\in R_{i}\}\text {,} \end{aligned}$$

(49)

with range given by

$$\begin{aligned} \mathbb {R}_{i}:=\cup _{\omega \in \Omega }R_{i}(\omega )\text {.} \end{aligned}$$

(50)

For club network $G\in \mathbb {R}_{i}$, there is at least one state $\omega$ (and possibly many states) such that $(\omega ,G)\in R_{i}$. Because $R_{i}$ is compact, $\mathbb {R}_{i}$ is compact. Thus, for each $0<\varepsilon <1$, there is a finite set of club network contained in $\mathbb {R}_{i}$,

$$\begin{aligned} \{G_{\varepsilon h}^{*i}:h=1,2,\ldots ,N_{\varepsilon }^{i}\} \end{aligned}$$

and a corresponding finite set of open balls,

$$\begin{aligned} \{B_{h_{K}}(\varepsilon ,G_{\varepsilon h}^{*i}):h=1,2,\ldots ,N_{\varepsilon }^{i}\}\text {.} \end{aligned}$$

covering $\mathbb {R}_{i}$ such that any club network $G\in \mathbb {R}_{i}$ is contained in an open ball of networks, $B_{h_{K}}(\varepsilon ,G_{\varepsilon h}^{*i})$, with each network in $B_{h_{K}}(\varepsilon ,G_{\varepsilon h}^{*i})$ being at $h_{K}$-distance less than $\varepsilon \in (0,1)$ from network $G_{\varepsilon h}^{*i}$ for some $h=1,2,\ldots ,N_{\varepsilon }^{i}$. Given our prior observation concerning the properties of the Hausdorff metric on the hyperspace of club networks, we know that for $\varepsilon \in (0,1)$, all club networks contained in the open ball $B_{h_{K}}(\varepsilon ,G_{\varepsilon h}^{*i})$ have the same coalition structure as club network $G_{\varepsilon h}^{*i}$. Thus, for club networks G and $G^{\prime }$ contained in $B_{h_{K}}(\varepsilon ,G_{\varepsilon h}^{*i})$, for $\varepsilon \in (0,1)$,

$$\begin{aligned} \begin{array}{l} S_{G}:=(S_{1G},\ldots ,S_{mG})=(S_{1G^{\prime }},\ldots ,S_{mG^{\prime }}):=S_{G^{\prime }} \\ \\ =(S_{1G_{\varepsilon h}^{*i}},\ldots ,S_{mG_{\varepsilon h}^{*i}}):=S_{G_{\varepsilon h}^{*i}}, \end{array} \end{aligned}$$

where recall for each club-network pair, (c, G), 

$$\begin{aligned} S_{cG}=\mathcal {D}(G(\cdot ,c)):=\{d\in D:G(d,c)\ne \varnothing \}. \end{aligned}$$

What conditions are sufficient to guarantee that the finite set of networks,

$$\begin{aligned} \mathbb {G}_{\varepsilon }^{*i}:=\{G_{\varepsilon h}^{*i}:h=1,2,\ldots ,N_{\varepsilon }^{i}\}, \end{aligned}$$

underlying the finite $\varepsilon$-covering,

$$\begin{aligned} \{B_{h_{K}}(\varepsilon ,G_{\varepsilon h}^{*i}):h=1,2,\ldots ,N_{\varepsilon }^{i}\}\text {.} \end{aligned}$$

have the same coalition structure? Consider the following definition.

Definition 3

(Chainable THR Sets) Let $R_{i}$ be a compact topologically Harris recurrent (THR) set of state-network pairs with network part $\mathbb {R}_{i}$. We say that $\mathbb {R}_{i}$ is chainable if there is an $\varepsilon \in (0,1)$ and a finite $\varepsilon$-covering

$$\begin{aligned} \{B_{h_{K}}(\varepsilon ,G_{\varepsilon h}^{*i}):h=1,2,\ldots ,N_{\varepsilon }^{i}\}\text {,} \end{aligned}$$

with underlying set of networks, $\mathbb {G}_{\varepsilon }^{*i}:=\{G_{\varepsilon h}^{*i}:h=1,2,\ldots ,N_{\varepsilon }^{i}\}\subset \mathbb {R}_{i},$ such that for any two club network, G and $G^{\prime }$ in $\mathbb {R}_{i}$ there is a subset of the finite $\varepsilon$-covering, $\{B_{h_{K}}(\varepsilon ,G_{\varepsilon h}^{*i}):h=1,2,\ldots ,N_{\varepsilon }^{i}\}$ given by

$$\begin{aligned} \{B_{h_{K}}(\varepsilon ,G_{\varepsilon h_{k}}^{*i}):k=1,2,\ldots ,k^{\prime }\}\text {, }k^{\prime }<N_{\varepsilon ^{i}}\text { } \end{aligned}$$

with $G\in B_{h_{K}}(\varepsilon ,G_{\varepsilon h_{1}}^{*i})$ and $G^{\prime }\in B_{h_{K}}(\varepsilon ,G_{\varepsilon h_{k^{\prime }}}^{*i})$ such that

$$\begin{aligned} B_{h_{K}}(\varepsilon ,G_{\varepsilon h_{k}}^{*i})\cap B_{h_{K}}(\varepsilon ,G_{\varepsilon h_{k+1}}^{*i})\ne \varnothing \text {, }k=1,\ldots ,k^{\prime }-1. \end{aligned}$$

Thus, for any two club networks in $\mathbb {R}_{i}$, there is a $\varepsilon$-open ball path from G to $G^{\prime }$ consisting of pairwise intersecting open balls. Because consecutive open balls are intersecting, that is, because

$$\begin{aligned} B_{h_{K}}(\varepsilon ,G_{\varepsilon h_{k}}^{*i})\cap B_{h_{K}}(\varepsilon ,G_{\varepsilon h_{k+1}}^{*i})\ne \varnothing ,k=1,\ldots ,k^{\prime }-1\text {,} \end{aligned}$$

all the networks contained in $\cup _{k}B_{h_{K}}(\varepsilon ,G_{\varepsilon h_{k}}^{*i})$ have the same coalition structure. Because $\mathbb {R}_{i}$ is chainable such an $\varepsilon$-open ball path with intersecting consecutive open balls can be found between any two club networks in $\mathbb {R}_{i}$, implying that all the club networks in

$$\begin{aligned} \mathbb {R}_{i}=\cup _{h}B_{h_{K}}(\varepsilon ,G_{\varepsilon h}^{*i}) \end{aligned}$$

have the same coalition structure.

We can distill the arguments and conclusions above in the following Theorem:

Theorem 4

(Topological Harris Recurrence and Coalitional Homogeneity) Let $\{Z_{t}^{*}\}_{t}$ be an equilibrium state-network Markov process governed by the transition kernel, $\pi ^{*}(dz^{\prime }|z)$, satisfying [T] induced by the stationary Markov perfect equilibrium behavioral strategy profile, $\sigma ^{*}(\cdot )$, of a discounted stochastic game of club network formation satisfying assumptions, [A-1]$^{*}$. Let

$$\begin{aligned} \mathbb {Z}=\left[ \cup _{i=1}^{I}(R_{i}\cup E_{i})\right] \cup E\text {.} \end{aligned}$$

(51)

be the decomposition of the space $\mathbb {Z}:=\Omega \times \mathbb {K }$ generated by equilibrium state-network process, $\{Z_{t}^{*}\}_{t}$ into a finite number of maximal Harris sets, $H_{i}=R_{i}\cup E_{i}\subset \mathbb {Z}$, and a transient set, $E\subset \mathbb {Z}$ with $H_{i}=R_{i}\cup E_{i}$ being given by the union of an open topologically transient set $E_{i}$ and a compact topologically Harris recurrent (THR) set, $R_{i}$, with network part

$$\begin{aligned} \mathbb {R}_{i}:=\cup _{\omega \in \Omega }R_{i}(\omega )=\cup _{\omega \in \Omega }\{G\in \mathbb {K}:(\omega ,G)\in R_{i}\}. \end{aligned}$$

If $\mathbb {R}_{i}$ is chainable, then all club networks in $\mathbb {R}_{i}$ have the same underlying coalition structure. Moreover, all club networks in any chainable subset $\mathbb {C}_{i}$ of $\mathbb {R}_{i}$ contain networks with the same underlying coalition structure.

Proof

If $\mathbb {R}_{i}$ is chainable, then for any two club networks in $\mathbb {R}_{i}$, there is a $\varepsilon$-open ball path from G to $G^{\prime }$ consisting of pairwise intersecting open balls. Because consecutive open balls are intersecting, that is, because

$$\begin{aligned} B_{h_{K}}(\varepsilon ,G_{\varepsilon h_{k}}^{*i})\cap B_{h_{K}}(\varepsilon ,G_{\varepsilon h_{k+1}}^{*i})\ne \varnothing ,k=1,\ldots ,k^{\prime }-1\text {,} \end{aligned}$$

all the networks contained in $\cup _{k}B_{h_{K}}(\varepsilon ,G_{\varepsilon h_{k}}^{*i})$ have the same coalition structure. Because $\mathbb {R}_{i}$ is chainable such an $\varepsilon$-open ball path with intersecting consecutive open balls can be found between any two club networks in $\mathbb {R}_{i}$, implying that all the club networks in

$$\begin{aligned} \mathbb {R}_{i}=\cup _{h}B_{h_{K}}(\varepsilon ,G_{\varepsilon h}^{*i}) \end{aligned}$$

have the same coalition structure. $\square$

9 Summary and Conclusions

If we strengthen the assumptions underlying our discounted stochastic game of club network formation (from [A-1] to [A-1]$^{*}$), then because the equilibrium state-network transition kernel, $\pi ^{*}(dz^{\prime }|z)$, induced by the SMPE strategy profile, $\sigma ^{*}(\cdot )$, of the DSG satisfies the Tweedie conditions [T], we know by Theorem 2 in Tweedie [37] and further results by Meyn and Tweedie [25] which build on Tuominen and Tweedie [36], Tweedie [37], and Costa and Dufour [10], that the state-network space, $\mathbb {Z}:=\Omega \times \mathbb {K}$, can be uniquely partitioned into a finite number of maximal Harris sets, $H_{i}\subset \mathbb {Z}$, and a transient set, $E\subset \mathbb {Z}$, and in particular, we know that

$$\begin{aligned} \mathbb {Z}=\underset{H_{i}\text { a maximal Harris set}}{\underbrace{\left[ \cup _{i=1}^{I}(R_{i}\cup E_{i})\right] }}\cup E\text {.} \end{aligned}$$

where $H_{i}=R_{i}\cup E_{i}\subset \mathbb {Z}$, $E_{i}$ being an open topologically transient set and $R_{i}$ being a compact topologically Harris recurrent (THR) set with network part given by

$$\begin{aligned} \mathbb {R}_{i}:=\cup _{\omega \in \Omega }R_{i}(\omega )=\cup _{\omega \in \Omega }\{G\in \mathbb {K}:(\omega ,G)\in R_{i}\}. \end{aligned}$$

These conclusions all flow from the discounted stochastic game of club network formation under assumptions [A-1]$^{*}$. Our last result above tells us that if, in addition, the club network vio sets, $\mathbb {R} _{i}$, are chainable, then all the club networks in any given vio set $\mathbb {R}_{i}$ have the same coalition structure.

10 Addendum on Existence of SMPE

A condition sufficient to guarantee that a DSG satisfying the usual assumptions has stationary Markov perfect equilibria (SMPE) in behavioral strategies is that the upper Caratheodory (uC) Nash correspondence (belonging to the $DSG^{\prime }s$ underlying one-shot game) contains a continuum-valued uC Nash sub-correspondence. By the usual assumptions we mean the assumptions made by Nowak and Raghavan [27], made here and in Fu–Page [15], and the assumptions that Levy–McLennan [24] claim their DSG model satisfies. While the SMPE existence result stated here and taken from Fu–Page [15] would appear to be in conflict with the Levy–McLennan [24] counterexamples, we will show here that this is not the case because the Levy–McLennan [24] counterexamples do not apply to SMPE existence in behavioral strategies (considered here and called mixed strategies in Levy–McLennan [24]) in DSGs satisfying the usual assumptions (i.e., assumptions [A-1] above—the assumptions claimed by Levy–McLennan). To see that this is the case, we will construct a DSG model which includes the specific DSG model underlying the Levy–McLennan counterexamples and outline why all such DSGs have SMPE—a fact established formally in Fu–Page [15].

10.1 The Fu–Page SMPE Existence Result

Fu and Page [15] show that all convex DSGs (i.e., DSGs satisfying the usual assumptions where players have convex, compact metric action choice sets) have uC Nash correspondences containing continuum-valued uC Nash sub-correspondences. Thus, any DSG satisfying the usual assumptions with finite action sets (as in LM2015) played over the induced behavioral action sets (i.e., all probability measures over pure action sets)—making the induced DSG a convex DSG—has a Nash correspondence containing a continuum-valued uC Nash sub-correspondence. It then follows from the Fu–Page Fixed Point Theorem [15] that there exists an m-tuple of value functions, $v^{*}:=(v_{1}^{*},\ldots ,v_{m}^{*})\in \mathcal {L} _{Y}^{\infty }$ such that

$$\begin{aligned} v^{*}(\omega )\in \mathcal {P}(\omega ,v^{*})\text { a.e. }[\mu ] \end{aligned}$$

(52)

where $\mathcal {P}(\cdot ,\cdot )$ is the induced uC Nash payoff correspondence.

To see why a convex DSG always contains a continuum-valued uC Nash sub-correspondence, we begin by noting that the uC Nash correspondence is given by the composition of two mappings. In particular, we have for each $(\omega ,v)\in \Omega \times \mathcal {L}_{Y}^{\infty }$ that

$$\begin{aligned} (\omega ,v)\longrightarrow \mathcal {N}(\omega ,v)=N(K(\omega ,v)):=N\circ K(\omega ,v), \end{aligned}$$

(53)

where $N(\cdot )$ is the Ky Fan correspondence (i.e., the KFC) defined on the metric space of Ky Fan sets, $\mathbb {S}$, in $X\times X$ taking set values in the set of Nash equilibria, and $K(\cdot ,\cdot )$ is the collective security mapping (CSM) defined on the set of state-value function profile pairs, $(\omega ,v)\in \Omega \times \mathcal {L} _{Y}^{\infty }$, taking Ky Fan set values. Thus, for each $(\omega ,v)\in \Omega \times \mathcal {L}_{Y}^{\infty }$, $K(\omega ,v)\in$ $\mathbb {S}$ is a Ky Fan set, and for each Ky Fan set, $E\in$ $\mathbb {S}$, N(E) is the corresponding nonempty, compact set of Nash equilibria. Equipping the compact metric product space $X\times X$ with the sum metric,  $\rho _{X\times X}:=$ $\rho _{X}+\rho _{X}$, and then equipping the hyperspace, $P_{f}(X\times X)$, of nonempty closed subsets of $X\times X$ with the Hausdorff metric, $h_{\rho _{X\times X}}$, induced by the sum metric, $\rho _{X\times X}$, on $X\times X$, the KFC is an $h_{\rho _{X\times X}}$-$\rho _{X}$-upper semi-continuous correspondence defined on the compact metric hyperspace of Ky-Fan sets, $\mathbb {S}$, taking nonempty compact Nash equilibrium values (see Fu–Page [15]). In the language of Hola–Holy [20], the KFC is an USCO and as such contains minimal USCOs (i.e., minimal KFCs ),

$$\begin{aligned} n(\cdot ):\mathbb {S}\longrightarrow \text {Nash Equilibria.} \end{aligned}$$

(54)

More importantly, each such minimal KFC belonging to the KFC takes closed connected and minimally essential Nash equilibrium values (see Fu–Page [15]). Thus, any such continuum-valued minimal KFC, when composed with the $DSG^{\prime }s$ Ky Fan valued collective security mapping, $K(\cdot ,\cdot ):\Omega \times \mathcal {L}_{Y}^{\infty }\longrightarrow \mathbb {S}$, delivers a continuum-valued uC Nash sub-correspondence,

$$\begin{aligned} (\omega ,v)\longrightarrow n(K(\omega ,v)). \end{aligned}$$

(55)

It then follows from the Fu–Page Fixed Point Theorem [15], that there exists an m-tuple of value functions, $v^{*}:=(v_{1}^{*},\ldots ,v_{m}^{*})\in \mathcal {L}_{Y}^{\infty }$ such that for the induced uC Nash payoff sub-correspondence we have,

$$\begin{aligned} \left. \begin{array}{l} v^{*}(\omega )\in p(\omega ,v^{*}):=\{(u_{1}(\omega ,v_{1}^{*},x),\ldots ,u_{m}(\omega ,v_{m}^{*},x))\in Y:x\in n(K(\omega ,v^{*}))\} \\ \\ \subset \{(u_{1}(\omega ,v_{1}^{*},x),\ldots ,u_{m}(\omega ,v_{m}^{*},x))\in Y:x\in N(K(\omega ,v^{*}))\} \\ \\ =\{(u_{1}(\omega ,v_{1}^{*},x),\ldots ,u_{m}(\omega ,v_{m}^{*},x))\in Y:x\in \mathcal {N}(\omega ,v^{*})\}=\mathcal {P}(\omega ,v^{*})\text {, a.e. }[\mu ]\text {.} \end{array} \right\} \end{aligned}$$

(56)

The key fact making the proof of the Fu–Page [15] fixed point result possible and straightforward is that the composition of the continuum-valued uC Nash sub-correspondence, $n(K(\cdot ,\cdot ))$, with players’ m-tuple of Caratheodory payoff functions induces, for each player, an interval-valued uC Nash payoff sub-correspondence,

$$\begin{aligned} (\omega ,v)\longrightarrow p_{d}(\omega ,v):=u_{d}(\omega ,v_{d},n(K(\omega ,v))):=\{u_{d}(\omega ,v_{d},x)\in Y_{d}:x\in n(K(\omega ,v))\}. \end{aligned}$$

(57)

Thus, each player’s Nash payoff sub-correspondence $p_{d}(\cdot ,\cdot )$ is Caratheodory approximable (see Kucia and Nowak [22])—making possible a simple proof, using approximation methods, that there exists $v^{*}\in \mathcal {L}_{Y}^{\infty }$ such that $v^{*}(\omega )\in \mathcal {P} (\omega ,v^{*})$, a.e. $[\mu ]$.

In Fu–Page [15], the fixed point problem—i.e., showing that the Nash payoff selection correspondence has fixed points—takes center stage. This is not the case in Levy–McLennan [24]. In fact, in Levy–McLennan [24] the statement of the one-shot deviation principle is incomplete—more on this below.

10.2 The Levy–McLennan DSG Model

In LM2015, there are finitely many players, $d=A,B,C,C^{\prime },D,D^{\prime },E,F$, each with a finite action set, $A_{d}$. The state space is given by the unit interval, i.e., $\Omega =[0,1]$, equipped with the Borel $\sigma$-field, $B_{[0,1]}$. Each player’s stage payoff function is given by

$$\begin{aligned} r_{d}(\omega ,a)=(1-\omega )g_{d}(\varsigma _{d}(\omega ),a) \end{aligned}$$

where $a:=(a_{A},a_{B},a_{C},a_{C^{\prime }},a_{D},a_{D^{\prime }},a_{E},a_{F})\in A:=\mathop {\displaystyle \prod }\limits _{d=A}^{F}A_{d}$, $\varsigma _{d}(\cdot ):[0,1]\longrightarrow (-\frac{1}{2},\frac{1}{2})$ is a measurable function, and $g_{d}(\cdot ,a)$ is continuous on $(-\frac{1}{2},\frac{1}{2})$ for each pure action profile $a\in A$. The law of motion governing state transitions is given by

$$\begin{aligned} q(d\omega ^{\prime }|\omega ,a):=\widetilde{q}(\omega ,a)U_{\omega }+(1- \widetilde{q}(\omega ,a))\delta _{1}(\omega )\text {,} \end{aligned}$$

where $\widetilde{q}(\omega ,a)=(1-\omega )\frac{1}{64}h(a)$ and $h(a):=a_{C}+a_{C^{\prime }}+a_{D}+a_{D^{\prime }}$, and where $U_{\omega }$ distributes uniformly in $[\omega ,1]$ and $\delta _{1}(\cdot ):[0,1]\longrightarrow \{0,1\}$ is a Dirac measure concentrated at 1. The dominating measure $\mu$ is $\frac{1}{2}(U_{0}+\delta _{1})$.

To begin restating the LM2015 game over behavioral actions, let

$$\begin{aligned} \sigma \longrightarrow \pi \sigma :=\sigma _{A}(da_{A})\otimes \cdots \otimes \sigma _{F}(da_{F}):=\otimes _{d=A}^{F}\sigma _{d}(da_{d}) \end{aligned}$$

(58)

be the mapping from behavioral action profiles, $(\sigma _{d},\sigma _{-d})\in \Delta (A)$, into product probability measures $\pi \sigma :=\otimes _{d=A}^{F}\sigma _{d}(da_{d})$. Also, as in LM2015, let

$$\begin{aligned} \gamma _{\sigma d}(\omega _{0}):=E_{\omega }^{\sigma }\sum _{n=1}^{\infty }\beta ^{n-1}r_{d}(\omega ^{n},a^{n}) \end{aligned}$$

(59)

be the expected payoff vector under behavioral strategy profile

$$\begin{aligned} \left. \begin{array}{l} \omega \longrightarrow \sigma (\cdot |\omega ) \\ \\ :=(\sigma _{A}(\cdot |\omega ),\sigma _{B}(\cdot |\omega ),\sigma _{C}(\cdot |\omega ),\sigma _{C^{\prime }}(\cdot |\omega ),\sigma _{D}(\cdot |\omega ),\sigma _{D^{\prime }}(\cdot |\omega ),\sigma _{E}(\cdot |\omega ),\sigma _{F}(\cdot |\omega )) \end{array} \right\} \end{aligned}$$

(60)

in the game starting from state $\omega _{0}$. Restating stage payoffs over behavioral actions we have

$$\begin{aligned} r_{d}(\omega ,\sigma )=\int _{A}(1-\omega )g_{d}(\varsigma _{d}(\omega ),a)\pi \sigma (da), \end{aligned}$$

(61)

a Caratheodory function on $\Omega \times \Delta (A)$. Also, each player’s payoff function in the underlying one shot game (over behavioral actions) is given by

$$\begin{aligned} \left. \begin{array}{l} U_{d}(\omega ,v_{d},\sigma )=\int _{A}\underset{u_{d}(\omega ,v_{d},a)}{ \underbrace{\left[ (1-\beta )r_{d}(\omega ,a)+\beta \int _{\Omega }v_{d}(\omega ^{\prime })q(d\omega ^{\prime }|\omega ,a)\right] }}\pi \sigma (da) \\ \\ =\int _{A}u_{d}(\omega ,v_{d},a)\pi \sigma (da). \end{array} \right\} \end{aligned}$$

(62)

Thus, $U_{d}(\cdot ,\cdot ,\cdot )$ is Caratheodory (measurable in $\omega$ and jointly continuous in $v_{d}$ and $\sigma )$. Moreover, for each $(\omega ,v_{d},\sigma _{-d})$, $\sigma _{d}\longrightarrow U_{d}(\omega ,v_{d},\sigma _{d},\sigma _{-d})$ is affine.

It would seem that the LM2015 DSG model satisfies the usual assumptions, and therefore, because all such DSGs have Nash correspondences containing continuum-valued Nash sub-correspondences, the Levy–McLennan counterexamples do not apply to SMPE existence in behavioral strategies in these DSGs.

As noted above in the network formation DSG considered here, because each player’s uC Nash payoff sub-correspondence induced by a continuum-valued minimal uC Nash correspondence is Caratheodory approximable, the DSG considered here is approximable. The DSG underlying the Levy–McLennan SMPE existence counterexamples is not approximable. Why? Because the Nash payoff selection correspondence in the Levy–McLennan DSG is without fixed points, the DSG is not and cannot be approximable. Thus, if the DSG model underlying the Levy–McLennan counterexamples, in fact, satisfies the usual assumptions, then the Levy–McLennan counterexamples do not apply to SMPE existence in behavioral strategies.

10.3 Final Observations

We close by making two observations: First, concerning an observation made earlier about the one-shot deviation principle, for behavioral strategy, $\sigma (da|\cdot ),$ with $\sigma (da|\omega )$ a Nash equilibrium of the one-shot game $X_{\sigma }(\omega ,\cdot )$ in state $\omega$, where player $d^{\prime }s$ payoff function is

$$\begin{aligned} \begin{array}{l} \sigma (da|\omega )\longrightarrow X_{\sigma d}(\omega ,\sigma (da|\omega )) \\ \\ :=\int _{A}\left[ (1-\beta )r_{d}(\omega ,a)+\beta \int _{\Omega }\gamma _{\sigma d}(\omega ^{\prime })q(d\omega ^{\prime }|\omega ,a)\right] \pi \sigma (da|\omega )\text {,} \end{array} \end{aligned}$$

it is not automatic that $X_{\sigma d}(\omega ,\sigma (\omega ))=\gamma _{\sigma d}(\omega )$ for each player—as suggested in Levy–McLennan. The fixed point problem remains. In order for the strategy, $\sigma (da|\cdot ),$ with $\sigma (da|\omega )$ being Nash equilibrium of the one-shot game $X_{\sigma }(\omega ,\cdot )$, to be a stationary Markov perfect equilibrium, it must also be true that for each player d and for $\omega$ a.e. $[\mu ]$

$$\begin{aligned} \gamma _{\sigma d}(\omega )=\int _{A}\left[ (1-\beta )r_{d}(\omega ,a)+\beta \int _{\Omega }\gamma _{\sigma d}(\omega ^{\prime })q(d\omega ^{\prime }|\omega ,a)\right] \pi \sigma (da|\omega ), \end{aligned}$$

(63)

(i.e., in addition to the Nash condition, the Bellman condition above must hold).

Second, the steps in the construction of Levy–McLennan DSG model are highly abbreviated. Thus, it is difficult to determine whether or not in constructing their DSG model Levy and McLennan make new assumptions implicitly or modify existing assumptions. Here we show that all DSGs satisfying the usual assumptions are approximable. Levy–McLennan [24] start with a static strategic form base game with circular Nash equilibria (i.e., Nash equilibria homeomorphic to the unit circle) and via a sequence of modifications and additions to the base game (perhaps inadvertently making new assumptions or modifying the existing assumptions) construct a DSG that is not approximable and is without SMPE. One way to think about the Levy–McLennan results is that they show that in the class of nonapproximable DSGs, there exists DSGs having no stationary Markov perfect equilibria. Thus, while not all nonapproximable DSGs have SMPE, as shown by Levy and McLennan [24], all approximable DSGs do, as shown by Fu and Page [15]. Moreover, because all DSGs satisfying the usual assumptions have players with Caratheodory approximable Nash payoff sub-correspondences, all such DSGs escape the Levy–McLennan counterexamples, and as we state here (and show formally in Fu–Page, [15]), possess stationary Markov perfect equilibria.

Appendices

Appendix 1: The Hausdorff Metric on the Hyperspace of Layered Networks

To begin, equip the set of c-connections, $K_{c}:=\cup _{d}K_{dc}:=\cup _{d}A(d,c)\times (\{d\}\times \{c\})$, with the sum metric,

$$\begin{aligned} \begin{array}{l} \rho _{K_{c}}:=\rho _{c}+\eta _{D}+\eta _{C}\text {.} \end{array} \end{aligned}$$

(64)

Thus, the distance between c-connections, (a, (d, c)) and $(a^{\prime },(d^{\prime },c))$, is

$$\begin{aligned} \rho _{K_{c}}((a,(d,c)),(a^{\prime },(d^{\prime },c))):=\rho _{c}(a,a^{\prime })+\eta _{D}(d,d^{\prime })+\eta _{C}(c,c)=\rho _{c}^{*}(a,a^{\prime })+\eta _{D}(d,d^{\prime })\text {.} \end{aligned}$$

We will equip each hyperspace of c-layers, $2^{K_{c}}$, with the Hausdorff metric induced by the metric, $\rho _{K_{c}}$, on the set of c -connections. In defining the Hausdorff metric $h_{K_{c}}$ on $2^{K_{c}}$, we must allow for empty c-layers. For nonempty c-layer $G_{c}^{\prime }\in 2^{K_{c}}$ and connection $(a,(d,c))\in K_{c}$, we define the distance from (a, (d, c)) to the nonempty c-layer $G_{c}^{\prime }$ to be

$$\begin{aligned} dist((a,(d,c)),G_{c}^{\prime }):=\min _{(a^{\prime },(d^{\prime },c))\in G_{c}^{\prime }}\rho _{K_{c}}((a,(d,c)),(a^{\prime },(d^{\prime },c)))\text {; } \end{aligned}$$

(65)

and for c-layers $G_{c}$ $\ne \varnothing ,$ $G_{c}^{\prime }$ $\ne \varnothing$, we define the excess of $G_{c}$ over $G_{c}^{\prime }$ to be

$$\begin{aligned} e(G_{c},G_{c}^{\prime }):=\max _{(a,(b,c))\in G_{c}}dist((a,(d,c)),G_{c}^{\prime }). \end{aligned}$$

(66)

The Hausdorff distance between nonempty c-layers, $G_{c}$ and $G_{c}^{\prime }$ is given by

$$\begin{aligned} h_{K_{c}}(G_{c},G_{c}^{\prime })=\max \left\{ e(G_{c},G_{c}^{\prime }),e(G_{c}^{\prime },G_{c})\right\} \end{aligned}$$

(67)

while

$$\begin{aligned} \begin{array}{l} \left. \begin{array}{l} h_{K_{c}}(G_{c},\varnothing ):=h_{K_{c}}(\varnothing ,G_{c}^{\prime })=diam(K_{c}) \\ \text {and} \\ h_{K_{c}}(\varnothing ,\varnothing )=0. \end{array} \right\} \end{array} \end{aligned}$$

(68)

The diameter, $diam(K_{c})$, of the set of c-connection $K_{c},$is given by

$$\begin{aligned} \left. \begin{array}{l} diam(K_{c}) \\ \\ :=\max _{(a^{\prime },(d^{\prime },c))\text { and }(a^{\prime \prime },(d^{\prime \prime },c))\text { in }K_{c}\text { }}\rho _{K_{c}}((a^{\prime },(d^{\prime },c)),(a^{\prime \prime },(d^{\prime \prime },c))). \end{array} \right\} \end{aligned}$$

(69)

Thus, the Hausdorff metric on the hyperspace of c-layers, $2^{K_{c}}$, is given by

$$\begin{aligned} \begin{array}{ll} {\small h}_{K_{c}}{\small (G}_{c}{\small ,G}_{c}^{\prime }{\small )}= &{} \left\{ \begin{array}{ll} \max \left\{ {\small e(G}_{c}{\small ,G}_{c}^{\prime }{\small ),e(G} _{c}^{\prime }{\small ,G}_{c}{\small )}\right\} &{} \text {{ if }}{\small G}_{c}{\small \ne \varnothing ,}\text { }{\small G}_{c}^{\prime }{\small \ne \varnothing }\text {.} \\ &{} \\ {\small diam(K}_{c}{\small )} &{} \text {{ if }}{\small G}_{c}{\small \ne \varnothing ,G}_{c}^{\prime }{\small =\varnothing }\text { { or }} {\small G}_{c}{\small =\varnothing ,G}_{c}^{\prime }{\small \ne \varnothing }\text {.} \\ &{} \\ 0 &{} \text {{if }}{\small G}_{c}{\small =\varnothing ,}\text { }{\small G }_{c}^{\prime }{\small =\varnothing }\text {.} \end{array} \right. \end{array} \end{aligned}$$

(70)

Given that the basic building block of a club network array is the hyperspace $\mathbb {K}_{dc}$ of feasible c-connections belonging to player d, with an underlying set of connections given by $K_{dc}:=A(d,c)\times (\{d\}\times \{c\})$, and given that each player can take, at most one action in each club, we see that the Hausdorff metric $h_{K_{dc}}$ on $\mathbb {K}_{dc}$ reduces to

$$\begin{aligned} \begin{array}{ll} {\small h}_{K_{dc}}{\small (G}_{dc}{\small ,G}_{dc}^{\prime }{\small )=} &{} \left\{ \begin{array}{ll} {\small \rho }_{c}{\small (a}_{dc}{\small ,a}_{dc}^{\prime }{\small )} &{} \text {{ if }}{\small G}_{dc}{\small \ne \varnothing }\text { { and} }{\small G}_{dc}^{\prime }{\small \ne \varnothing }\text {.} \\ &{} \\ {\small diam(A}_{dc}{\small )} &{} \text {{ if }}{\small G}_{dc}{\small \ne \varnothing ,G}_{dc}^{\prime }{\small =\varnothing }\text { { or }} {\small G}_{dc}{\small =\varnothing ,G}_{dc}^{\prime }{\small \ne \varnothing }\text {.} \\ &{} \\ {\small 0} &{} \text {{ if }}{\small G}_{c}{\small =\varnothing ,}\text { } {\small G}_{c}^{\prime }{\small =\varnothing }\text {.} \end{array} \right. \end{array} \end{aligned}$$

(71)

Recall that $G_{dc}\in \mathbb {K}_{dc}$ if and only if

$$\begin{aligned} \begin{array}{ll} G_{dc}= &{} \left\{ \begin{array}{lll} \left\{ (a_{dc},(d,c))\right\} &{} \text {{ if} }{\small d}\text { { is a member of club} }{\small c} &{} \text {{ i.e., if }}(d,c)\in \mathcal {D}(G) \\ &{} &{} \\ \varnothing &{} \text {{ if }}{\small d}\text { { is not a member of club} }{\small c}\text {.} &{} \text {{ i.e., if} }(d,c)\notin \mathcal {D} (G){\small .} \end{array} \right. \end{array} \end{aligned}$$

(72)

The Hausdorff metric on the hyperspace of feasible club networks, $\mathbb {K} =(\mathbb {K}_{dc})_{dc}$, is given by

$$\begin{aligned} h_{K}(G,G^{\prime }):=\sum _{d=1}^{n}\sum _{c=1}^{m}h_{K_{dc}}(G_{dc},G_{dc}^{\prime })\text {.} \end{aligned}$$

(73)

for $G:=(G_{c})_{c\in C}$ and $G^{\prime }:=(G_{c}^{\prime })_{c\in C}$ in $\mathop {\displaystyle \prod }\limits _{c\in C}\mathbb {K}_{c}$. Because $(\mathbb {K}_{c},\rho _{K_{c}})$ is a compact metric space, we have by Proposition C.2 in Bertsekas and Shreve [6] that $(2^{K_{c}},h_{K_{c}})$ is a compact metric space of c-layers, and because $\mathbb {K}_{dc}$ is an $h_{K_{dc}}$-closed subset of the $h_{K_{dc}}$-compact subset of $2^{K_{dc}}$, $\mathbb {K}_{dc}$ is $h_{K_{dc}}$-compact—implying that $(\mathbb {K},h_{K})$, given by

$$\begin{aligned} (\mathbb {K},h_{K})=((\mathbb {K}_{dc})_{dc},\sum _{d=1}^{n} \sum _{c=1}^{m}h_{K_{dc}}), \end{aligned}$$

(74)

is a compact metric space.

Appendix 2: Technical Notes on Assumptions [A-1]

1.1 Weak Star Convergence of Random Player Club Networks

Under the metric, $\rho _{w_{ca}^{*d}}$, compatible with the relative $w_{ca}^{*d}$-topology on $\Delta (\mathbb {K}^{d})$, we have

$$\begin{aligned} \sigma _{d}^{n}\underset{\rho _{w_{ca}^{*d}}}{\longrightarrow }\sigma _{d}^{*}\text { if and only if }\int _{\mathbb {K}^{d}}c(G^{d})d\sigma _{d}^{n}(G^{d})\longrightarrow \int _{\mathbb {K}^{d}}c(G^{d})d\sigma _{d}^{*}(G^{d})\text { for all }c(\cdot )\in \mathcal {C}(\mathbb {K}^{d}) \text {, } \end{aligned}$$

(75)

where $\mathcal {C}(\mathbb {K}^{d})$ is the Banach space of continuous functions defined on the compact metric space, $\mathbb {K}^{d}$, with the sup norm. In fact, there exists a countable subcollection of continuous functions, $\mathcal {C}^{0}(\mathbb {K}^{d})$, such that

$$\begin{aligned} \sigma _{d}^{n}\underset{\rho _{w_{ca}^{*d}}}{\longrightarrow }\sigma _{d}^{*}\text { if and only if }\int _{\mathbb {K}^{d}}c_{i}(G^{d})d\sigma _{d}^{n}(G^{d})\longrightarrow \int _{\mathbb {K}^{d}}c_{i}(G^{d})d\sigma _{d}^{*}(G^{d})\text { } \end{aligned}$$

(76)

for all $c_{i}(\cdot )\in \mathcal {C}^{0}(\mathbb {K}^{d})$ (e.g., see Aliprantis and Border [1], Chapter 15).

By Theorem 3.2 in Billingsley [7], we know that

$$\begin{aligned} \text { }\pi \sigma ^{n}=\sigma _{1}^{n}\otimes \cdots \otimes \sigma _{m}^{n} \underset{w_{ca}^{*}}{\longrightarrow }\sigma _{1}^{*}\otimes \cdots \otimes \sigma _{m}^{*}=\pi \sigma ^{*}\text { if and only if for each player }\sigma _{d}^{n}\underset{w_{ca}^{*d}}{\longrightarrow } \sigma _{d}^{*}. \end{aligned}$$

(77)

Thus a sequence of behavioral action profiles $w_{ca}^{*}$-converges to a particular behavioral action profile if and only if each player’s sequence of behavioral actions $w_{ca}^{*d}$-converges to a particular behavioral action such that the $w_{ca}^{*}$-limit of behavioral action profiles is equal to the product of the $w_{ca}^{*d}$-limits of players’ behavioral action sequences. Unfortunately, the mapping $(\sigma _{d}(\cdot ),\sigma _{-d}(\cdot ))\longrightarrow \pi ((\sigma _{d}(\cdot ),\sigma _{-d}(\cdot )))$ is not jointly $\rho _{W_{ca}^{*d}}$-continuous. A good example of the failure of joint $\rho _{W_{ca}^{*d}}$-continuity can be found in Elliott et al. [14], Example 3.16.

1.2 Weak Star and K-Convergence of Value Functions

A sequence, $\{v^{n}\}_{n}\subset$ $\mathcal {L}_{Y}^{\infty }$, converges weak star to $v^{*}=(v_{1}^{*}(\cdot ),\ldots ,v_{m}^{*}(\cdot ))\in \mathcal {L}_{Y}^{\infty }$, denoted by $v^{n}\underset{\rho _{w^{*}}}{\longrightarrow }v^{*}$, if and only if

$$\begin{aligned} \int _{\Omega }\left\langle v^{n}(\omega ),l(\omega )\right\rangle _{R^{m}}d\mu (\omega )\longrightarrow \int _{\Omega }\left\langle v^{*}(\omega ),l(\omega )\right\rangle _{R^{m}}d\mu (\omega ) \end{aligned}$$

for all $l(\cdot )\in \mathcal {L}_{R^{m}}^{1}$.

A sequence, $\{v^{n}\}_{n}\subset$ $\mathcal {L}_{Y}^{\infty }$, K -converges (i.e., Komlos convergence—Komlos [21]) to $\widehat{v}\in \mathcal {L}_{Y}^{\infty }$, denoted by $v^{n}\underset{K}{\longrightarrow }\ \widehat{v}$, if and only if every subsequence, $\{v^{n_{k}}(\cdot )\}_{k}$, $of\ \{v^{n}(\cdot )\}_{n}\$has an arithmetic mean sequence, $\{\widehat{v }^{n_{k}}(\cdot )\}_{k}$, where

$$\begin{aligned} \widehat{v}^{n_{k}}(\cdot ):=\frac{1}{k}\sum _{q=1}^{k}v^{n_{q}}(\cdot ), \end{aligned}$$

such that

$$\begin{aligned} \widehat{v}^{n_{k}}(\omega )\underset{R^{m}}{\longrightarrow }\widehat{v} (\omega ){ {a.e. }}[\mu ]. \end{aligned}$$

The relationship between $w^{*}$-convergence and K-convergence is summarized via the following results from Balder [5]: For every sequence of value functions, $\{v^{n}\}_{n}\subset \mathcal {L}_{Y}^{\infty }$, and $\widehat{v}\in \mathcal {L}_{Y}^{\infty }$ the following statements are true:

$$\begin{aligned} \left. \begin{array}{l} \text {(i) If the sequence}\ \{v^{n}\}_{n}\ K\text {-converges to}\ \widehat{v}\text {, then }\{v^{n}\}_{n}\ w^{*}\text {-converges to } \widehat{v}.\text { } \\ \\ \text {(ii) The sequence }\{v^{n}\}_{n}\text { }w^{*} \text {-converges to } \widehat{v}\text { if and only if } \\ \text {every subsequence } \{v^{n_{k}}\}_{k} \text { of} \{v^{n}\}_{n} \text { has a further\ subsequence, } \{v^{n_{k_{r}}}\}_{r}\text {,} \\ K\text {-converging to } \widehat{v}\text {.} \end{array} \right\} \end{aligned}$$

(78)

For any sequence of value function profiles, $\{v^{n}\}_{n},$ in $\mathcal {L}_{Y}^{\infty }$ it is automatic that

$$\begin{aligned} sup_{n}\int _{\Omega }\left\| v^{n}(\omega )\right\| _{R^{m}}d\mu (\omega )<+\infty . \end{aligned}$$

Thus, by the classical Komlos Theorem [21], any such sequence, $\{v^{n}\}_{n}$, has a subsequence, $\{v^{n_{k}}\}_{k}$ that K-converges to some K-limit, $\widehat{v}\in \mathcal {L}_{Y}^{\infty }$.

1.3 Strong Stochastic Continuity of the Law of Motion

Under the stochastic continuity assumptions made above, [A-1](14), we have by Scheffee’s theorem (see Billingsley [8], Theorem 16.11) that for each $\omega \in \Omega$,

$$\begin{aligned} \left. \begin{array}{l} \sup _{E\in \mathcal {B}(\Omega )}\left| q(E|\omega ,G^{n})-q(E|\omega ,G^{*})\right| _{R} \\ \\ \le \int _{\Omega }\left| h(\omega ^{\prime }|\omega ,G^{n})-h(\omega ^{\prime }|\omega ,G^{*})\right| _{R}d\mu (\omega ^{\prime })\longrightarrow 0, \end{array} \right\} \end{aligned}$$

(79)

for any sequence of networks $\{G^{n}\}_{n}$ in $\Phi (\omega )$ converging to network $G^{*}\in \Phi (\omega )$ (i.e., for each $\omega \in \Omega$ the conditional density mapping, $G\longrightarrow h(\cdot |\omega ,G)$, is continuous in $L_{1}$ norm with respect to G). Thus, by Scheffee’s theorem the $L_{1}$ norm continuity of $G\longrightarrow h(\cdot |\omega ,G)$ with respect to network G in each state $\omega$ is equivalent to the continuity of $G\longrightarrow q(E|\omega ,G)$ in each state $\omega$ with respect to network G uniformly in $E\in B_{\Omega }$ (i.e., for each $\omega \in \Omega$, $q(E|\omega ,\cdot )$ is continuous in G,  uniformly with respect to $E\in B_{\Omega }$).

1.4 Convergence and Continuity

Under assumptions [A-1], for each $(\omega ,v)$ each player’s payoff function,

$$\begin{aligned} G\longrightarrow u_{d}(\omega ,v_{d},G):=(1-\beta _{d})r_{d}(\omega ,G)+\beta _{d}\int _{\Omega }v_{d}(\omega ^{\prime })q(\omega ^{\prime }|\omega ,G), \end{aligned}$$

(80)

is jointly continuous in $G=(G_{1},\ldots ,G_{m})$, and for any sequence of value function-network pairs, $\{(v^{n},G^{n})\}_{n}$, if $v^{n}\underset{ \rho _{w^{*}}}{\longrightarrow }v^{*}$ and $G^{n}\underset{h_{K}}{ \longrightarrow }G^{*}$ then for each $\omega$,

$$\begin{aligned} u(\omega ,v^{n},G^{n})\underset{\rho _{Y}}{\longrightarrow }u(\omega ,v^{*},G^{*})\text {,} \end{aligned}$$

(i.e., $u(\omega ,\cdot ,\cdot )$ is jointly continuous in (v, G)). Moreover, for each $\omega$, the v-parameterized collection of m-tuples of integrands given by

$$\begin{aligned} \left\{ u(\omega ,v,\cdot ):v\in \mathcal {L}_{Y}^{\infty }\right\} \text {,} \end{aligned}$$

(81)

where (see expression 80 above),

$$\begin{aligned} G\longrightarrow u(\omega ,v,G):=(u_{1}(\omega ,v_{1},G),\ldots ,u_{m}(\omega ,v_{m},G)) \end{aligned}$$

(82)

is uniformly equicontinuous (see Solan [35], Lemma 3.6).¹³ Thus, the Y-valued players’ payoff function, $u(\cdot ,\cdot ,\cdot )$, is a Caratheodory function: $\rho _{w^{*}\times h_{K}}$-continuous in (v, G) for each $\omega$, and $(B_{\Omega },B_{Y})$-measurable in $\omega$ on $\Omega$ for each (v, G).

Appendix 3: Hitting and Return Times

Let $\{W_{t}\}$ be the Markov process governed by the transition kernel, $p(d\omega ^{\prime }|\omega )$, where for each $t=0,1,2,3,\ldots$, we have for each $\omega \in \Omega$, $W_{t}(\omega )=\omega _{t}$. Recall that

$$\begin{aligned} p(B|\omega ):=prob\{W_{t+1}(\omega ^{\prime })\in B|W_{t}(\omega )=\omega \} \text {.} \end{aligned}$$

(83)

The number of visits of the process, $W_{t}(\cdot )$, to the set of states B is given by $\eta _{B}:=\sum _{t=1}^{\infty }I_{B}(W_{t}(\omega ))$. Let $\varphi$ be a nontrivial, $\sigma$-finite measure on $B_{\Omega }$.

We say that the process, $W_{t}(\cdot )$, is $\varphi$-irreducible if for all $B\in B_{\Omega }$ such that $\varphi (B)>0$,

$$\begin{aligned} U(\omega ,B)=E_{\omega }(\eta _{B})=\lim _{T\longrightarrow \infty }\sum _{t=1}^{T}p^{t}(B|\omega )=\sum _{t=1}^{\infty }p^{t}(B|\omega )>0\text { . } \end{aligned}$$

(84)

If the process, $W_{t}(\cdot )$, is $\varphi$-irreducible, then we know that there exists a (maximal) dominating measure, $\psi$, such that $W_{t}(\cdot )$ is also $\psi$-irreducible (see Proposition 4.2.2 in Meyn and Tweedie [25]).

Let

$$\begin{aligned} \left. \begin{array}{l} \tau _{B}=\min \{t\ge 1:W_{t}(\omega )\in B\}=\text {first return time,} \\ \qquad \qquad \qquad \qquad \qquad \quad \text {and} \\ \delta _{B}=\min \{t\ge 0:W_{t}(\omega )\in B\}=\text {first hitting time.} \end{array} \right\} \end{aligned}$$

(85)

Return time probabilities and recurrent time probabilities are given by

$$\begin{aligned} \left. \begin{array}{l} L(\omega ,B):=P_{\omega }(\tau _{B}<\infty )=P_{\omega }(W_{t}(\cdot )\text { ever enters }B\text {)} \\ \qquad \qquad \qquad \qquad \qquad \qquad \text {and} \\ Q(\omega ,B):=P_{\omega }(\eta _{B}=\infty )=P_{\omega }(W_{t}(\cdot )\text { enters } B\text { infinitely often)} \end{array} \right\} \end{aligned}$$

(86)

The set $B\in B_{\Omega }$ is Harris recurrent if $P_{\omega }(\eta _{B}=\infty )=1$ for all $\omega \in B$. The Markov process, $W_{t}(\cdot )$, is Harris recurrent if it is $\psi$-irreducible and every $B\in B_{\Omega }$ such that $\psi (B)>0$ is Harris recurrent. Thus a set B is Harris recurrent if when the Markov process $W_{t}(\cdot )$ starts at $\omega \in B$, it returns to B infinitely many times, except when the process starts at any state contained in a set of initial states having probability zero—thus, $P_{\omega }(\cdot )$-almost surely. In fact, for any Markov process $W_{t}(\cdot )$ that is Harris recurrent, $P_{\omega }(\eta _{B}=\infty )=1$ for all $\omega \in \Omega$ and for all $B\in B_{\Omega }$ such that $\psi (B)>0.$

Summarizing for all $B\in B_{\Omega }$ such that $\psi (B)>0$

$$\begin{aligned} \left. \begin{array}{l} U(\omega ,B):=E_{\omega }(\eta _{B})>0\Longleftrightarrow W_{t}(\cdot )\text { is }\psi \text {-irreducible},\text { } \\ U(\omega ,B):=E_{\omega }(\eta _{B})=\infty \Longleftrightarrow W_{t}(\cdot ) \text { is recurrent}, \\ Q(\omega ,B):=P_{\omega }(\eta _{B}=\infty )=1\Longleftrightarrow W_{t}(\cdot )\text { is Harris recurrent}.\text { } \end{array} \right\} \end{aligned}$$

(87)