Simple Uncoupled No-regret Learning Dynamics for Extensive-form Correlated Equilibrium

1.1 Related Work

The study of adaptive procedures leading to a CE dates back to at least the seminal works by Foster and Vohra [16], Fudenberg and Levine [17], ,19], and Hart and Mas-Colell [22], ,23]; see also the monograph by Fudenberg and Levine [18]. In particular, the work by Hart and Mas-Colell [22] proves that simple dynamics based on the notion of internal regret result in empirical frequencies of play that converge to the set of CEs in normal-form games. The strategy that the authors introduce—the so-called regret matching—is conceptually simple and guarantees that if all players follow this strategy, then the empirical frequency of play converges to the set of CEs (see also Cahn [5]). Other works describe extensions to the models studied in the aforementioned papers. For example, Stoltz and Lugosi [44] describe an adaptive procedure such that the resulting empirical frequency of play converges to the set of CEs in games with an infinite, but compact, set of actions, while Kakade et al. [29] consider efficient algorithms for computing correlated equilibria in graphical games.

In more recent years, a growing effort has been devoted to understanding the relationships between no-regret learning dynamics and equilibria in extensive-form games. These games pose additional challenges when compared to normal-form games, due to their sequential nature and the presence of imperfect information. While in two-player zero-sum extensive-form games it is widely known that no-regret learning dynamics converge to an NE—with the counterfactual regret minimization(CFR) algorithm and its variations being the state-of-the-art for equilibrium finding in such games [4, 33, 45, 46, 49]—the general case is less understood. Celli et al. [7] provide some variations of the classical CFR algorithm for n-player general-sum extensive-form games, showing that they provably converge to a normal-form coarse correlated equilibrium, which is based on a form of correlation that is less appealing than that of EFCE in sequential games. Indeed, normal-form coarse correlated equilibria require that the players commit to following all the recommendations issued by the mediator upfront before the beginning of the game, which is not realistic in practice.

Finally, we mention relevant literature subsequent to the conference version of this article. In a recent paper, Morrill et al. [37] conduct a study of different forms of correlation in extensive-form games, defining a taxonomy of solution concepts. Each of their solution concepts is attained by a particular set of no-regret learning dynamics, which is obtained by instantiating the phi-regret minimization framework [20, 21, 44] with a suitably defined deviation function. As part of their analysis, Morrill et al. [37] investigate some properties of the well-established CFR regret minimization algorithm [49] applied to n-player general-sum extensive-form games, establishing that it is hindsight-rational with respect to a specific set of deviation functions, which the authors coin blind counterfactual deviations. In subsequent recent work, Morrill et al. [36] extend their prior work [37] by identifying a general class of deviations—called behavioral deviations—that induce equilibria that can be found through uncoupled no-regret learning dynamics. Behavioral deviations are defined as those specifying an action transformation independently at each information set of the game. As the authors note, the deviation functions involved in the definition of EFCE do not fall under that category. A particular class of behavioral deviation functions—called causal partial sequence deviations—induces solution concepts that are (subsets of) EFCEs. Thus, their result begets an alternative set of no-regret learning dynamics that converge to EFCE, based on a different set of deviation functions than those we use in this article.

2.1 Mathematical Notation and Algorithmic Conventions

In this article, we adopt the following notational and algorithmic conventions:

• We denote the set of real numbers as \( \mathbb {R} \), the set of nonnegative real numbers as \( \mathbb {R}_{\ge 0} \), and the set \( \lbrace 1,2,\dots \rbrace \) of positive integers as \( \mathbb {N}_{\gt 0} \).

• The set \( \lbrace 1,\ldots ,n\rbrace \), where \( n\in \mathbb {N}_{\gt 0} \), is compactly denoted as \( [n] \); the empty set as \( \emptyset \).

• Given a set S, we denote its convex hull with the symbol \( \text {co}(S) \).

• Vectors and matrices are marked in bold.

• Given a discrete set \( S = \lbrace s_1,\dots ,s_n\rbrace \), we denote as \( \mathbb {R}^{S} \) (respectively, \( \mathbb {R}_{\ge 0}^{S} \)) the set of real (respectively, nonnegative real) \( |S| \)-dimensional vectors whose entries are denoted as \( \mathbf {x}[s_1], \dots ,\mathbf {x}[s_n] \). Given an element \( s \in S \), we denote by \( \mathbf {e}_s \in \mathbb {R}^{S} \) the canonical basis vector whose entries are all zeros except for \( \mathbf {e}_s[s] = 1 \).

• Similarly, given a discrete set S, we denote as \( \mathbb {R}^{|S|\times |S|} \) (respectively, \( \mathbb {R}_{\ge 0}^{|S|\times |S|} \)) the set of real (respectively, nonnegative real) \( |S|\times |S| \) square matrices \( \mathbf {M} \) whose entries are denoted as \( \mathbf {M}[s_r,s_c] \) (\( s_r,s_c\in S \)), where \( s_r \) corresponds to the row index and \( s_c \) to the column index.

• Given a discrete set S, we denote by \( \Delta ^{S} \) the simplex \( \Delta ^{S} := \lbrace \mathbf {x}\in \mathbb {R}_{\ge 0}^{S}: \sum _{s\in S} \mathbf {x}[s] = 1\rbrace \). The symbol \( \Delta ^n \), with \( n\in \mathbb {N}_{\gt 0} \), is used to mean \( \Delta ^{[n]} \).

• Given a discrete set S, we use the symbol \( \mathbb {S}^{S} \subseteq \mathbb {R}_{\ge 0}^{|S|\times |S|} \) to denote the set of stochastic matrices, that is, nonnegative square matrices whose columns all sum up to 1. The symbol \( \mathbb {S}^n \), where \( n\in \mathbb {N}_{\gt 0} \), is used to mean \( \mathbb {S}^{[n]} \).

• Given two functions \( f:X\rightarrow Y \) and \( g:Y\rightarrow Z \), we denote by \( g\circ f: X\rightarrow Z \) their composition \( \mathbf {x} \mapsto g(f(x)) \).

• Given a set S and a function f, the image of S via f is denoted as \( f(S) := \lbrace f(s): s\in S\rbrace \).

• Given a proposition \( \text {P} \), we denote with \( 1\!\!1 [\text {P}] \) the indicator function of that proposition: \( 1\!\!1 [\text {P}] = 1 \) if \( \text {P} \) is true, and \( 1\!\!1 [\text {P}] =0 \) if not.

• Given a partially ordered set \( (S, \prec) \) and two elements \( s,s^{\prime }\in S \), we use the standard derived symbols \( s \preceq s^{\prime } \) to mean that \( (s = s^{\prime }) \vee (s \prec s^{\prime }) \), \( s \succ s^{\prime } \) to mean that \( s^{\prime } \prec s \), and \( s \succeq s^{\prime } \) to mean that \( s^{\prime } \preceq s \). Furthermore, we use the crossed symbols \( \not\prec , \not\preceq , \not\succ \), and \( \not\succeq \) to mean that the relations \( \prec ,\preceq ,\succ \), and \( \succeq \) (respectively) do not hold.

• Several of the algorithms presented in this article take as input, give as output, or otherwise manipulate, linear functions. Therefore, to study the complexity of our routines, it is necessary to settle on a representation for such linear functions. Unless otherwise specified, we will always assume that a linear function f is stored in memory using coordinates relative to the canonical basis of their domain and codomain and call that representation the \( {canonical representation} \) of f, denoted \( \langle f\rangle \). Specifically:

  • If f is a linear function from \( \mathbb {R}^{S} \) (for some discrete set S) to \( \mathbb {R} \), then its canonical representation \( \langle f\rangle \) is the (unique) vector \( \mathbf {v} \in \mathbb {R}^{S} \) such that \( \begin{equation*} f(\mathbf {x}) = \mathbf {v}^\top \mathbf {x}\qquad \forall \ \mathbf {x}\in \mathbb {R}^{S}, \end{equation*} \) where \( \top \) denotes transposition.

  • If f is a linear function from \( \mathbb {R}^{S} \) to \( \mathbb {R}^{S} \) (for some discrete set S), then its canonical representation \( \langle f\rangle \) is the (unique) matrix \( \mathbf {M}\in \mathbb {R}^{|S|\times |S|} \) such that \( \begin{equation*} f(\mathbf {x}) = \mathbf {M} \mathbf {x} \qquad \forall \ \mathbf {x} \in \mathbb {R}^{S}. \end{equation*} \)

  • If F is a linear functional, mapping linear functions \( \phi :\mathbb {R}^{S}\rightarrow \mathbb {R}^{S} \) to reals, then its canonical representation \( \langle F\rangle \) is the (unique) matrix \( \mathbf {\Lambda } \in \mathbb {R}^{|S|\times |S|} \) such that \( \begin{equation} F(\phi) = \sum _{s_r, s_c\in S} \mathbf {\Lambda }[s_r, s_c]\cdot \langle \phi \rangle [s_r, s_c]\qquad \forall \ \phi :\mathbb {R}^{S}\rightarrow \mathbb {R}^{S}, \end{equation} \) where \( \langle \phi \rangle \) is the canonical representation of \( \phi \).

2.2 Extensive-form Games

In this subsection, we introduce some standard concepts, terminology, and notation that we will use to deal with extensive-form games. A summary of the notation we introduce can be found in Table 1. Examples 2.1 and 2.5 demonstrate some of the notation in a simple extensive-form game.

An extensive-form game is played on an oriented rooted game tree. We denote by \( \mathcal {H} \) the set of nodes of the game tree. Each node \( h\in \mathcal {H} \) that is not a leaf of the game tree is called a decision node and has an associated player that acts at that node. In an n-player extensive-form game, the set of valid players is the set \( [n]\cup \lbrace c\rbrace \), where c denotes the chance player—a fictitious player that selects actions according to a known fixed probability distribution and models exogenous stochasticity of the environment (for example, a roll of the dice or drawing a card from a deck). The player that acts at h is free to pick any one of the actions \( \mathcal {A}(h) \) that are available at h. For each possible action \( a\in \mathcal {A}(h) \), an edge connects h to the node to which the game transitions whenever action a is picked at h. Given a player \( i \in [n]\cup \lbrace c\rbrace \), we denote with \( \mathcal {H}^{(i)} \subseteq \mathcal {H} \) the set of all decision nodes that belong to Player i.

Leaves of the game tree are called terminal nodes and represent the outcomes of the game. As such, they are not associated with any acting player, and the set of actions is conventionally set to the empty set. The set of all terminal nodes in the game is denoted with the letter \( \mathcal {Z} \). So, the set of all nodes in the game tree is the disjoint union \( \mathcal {H}= \mathcal {H}^{(1)} \cup \dots \cup \mathcal {H}^{(n)}\cup \mathcal {Z} \). When the game transitions to a terminal node \( z\in \mathcal {Z} \), payoffs are assigned to each of the non-chance players by the set of functions \( \lbrace {u^{(i)}} :\mathcal {Z}\rightarrow \mathbb {R}\rbrace _{i\in [n]} \). Furthermore, we let \( p_c : \mathcal {Z}\rightarrow (0,1) \) denote the function assigning each terminal node \( z\in \mathcal {Z} \) to the product of probabilities of chance moves encountered on the path from the root of the game tree to z.

2.2.2 Sequences.

For any player \( i\in [n] \), and given an information set \( j\in {\mathcal{J}^{(i)}} \) and an action \( a\in \mathcal {A}(j) \), we denote as \( \sigma =(j,a) \) the sequence of Player i’s actions encountered on the path from the root of the game tree down to action a (included) at any node in information set j. In perfect-recall extensive-form games, such a sequence is guaranteed to be uniquely determined, because paths that reach decision nodes belonging to the same information set identify the same sequence of Player i’s actions. A special element \( \varnothing \) denotes the empty sequence of Player i. Then, the set of Player i’s sequences is defined as \( \begin{equation*} {\Sigma^(i)} := \lbrace (j,a): j\in {\mathcal{J}^{(i)}} , a\in \mathcal {A}(j)\rbrace \cup \lbrace \varnothing \rbrace . \end{equation*} \) Moreover, we let \( {\Sigma^(i)} _* := {\Sigma^(i)} \setminus \lbrace \varnothing \rbrace \) be the set of all sequences of Player i other than the empty one.

Given a node \( h\in \mathcal {H} \), we denote by \( \sigma ^{(i)}(j)\in {\Sigma^(i)} \) the last sequence (information set-action pair) of Player i encountered on the path from the root of the game tree to any node in j, also known as j’s parent sequence. If Player i does not act before h, then \( \sigma ^{(i)}(h) \) is set to the empty sequence \( \varnothing \). If \( \sigma ^{(i)}(j) = \varnothing \), then we say that information set j is a root information set of Player i, while, whenever \( \sigma ^{(i)}(j)=(j^{\prime },a) \), we say that \( j^{\prime } \) is the immediate predecessor of j, or equivalently that information set j is immediately reachable from sequence \( \sigma ^{(i)}(j) \). That nomenclature is supported by the observation that \( j^{\prime }\prec j \), and that Player i does not need to take other actions after choosing a at \( j^{\prime } \) to reach j. Analogously, for all \( z\in \mathcal {Z} \), we define \( \sigma ^{(i)}(z)\in {\Sigma^(i)} \) as the last sequence of Player i’s actions encountered the path from the root of the game tree to terminal node z (notice that \( \sigma ^{(i)}(z) = \varnothing \) whenever Player i never plays on the path from the root to z).

Just like information sets, there exists a natural partial ordering on sequences, which we also denote with the same symbol \( \prec \). For every \( i \in [n] \) and any pair of sequences \( \sigma ,\sigma ^{\prime } \in {\Sigma^(i)} \), the relation \( \sigma \prec \sigma ^{\prime } \) holds if \( \sigma =\varnothing \ne \sigma ^{\prime } \), or if the sequences are of the form \( \sigma =(j,a),\sigma ^{\prime }=(j^{\prime },a^{\prime }) \), and the set of Player i’s actions encountered on the path from the root of the tree to any node in \( j^{\prime } \) includes playing action a at one of information set j’s nodes. As for information sets, it is a direct consequence of the perfect recall assumption that the partially ordered set \( ({\Sigma^(i)} , \prec) \) is a forest. Finally, we introduce the overloaded notation \( \sigma \succeq j \) (or equivalently \( j \preceq \sigma \)), defined for any player \( i \in [n] \), information set \( j\in {\mathcal{J}^{(i)}} \), and sequence \( \sigma \in {\Sigma^(i)} \), to mean that the sequence of Player i ’s actions that is denoted by \( \sigma \) must lead the player to pass through (some node in) j; formally \( \sigma = (j^{\prime },a^{\prime }) \in {\Sigma^(i)} _* \wedge j^{\prime }\succeq j \). With that, we let \( {\Sigma^(i)} _j := \lbrace \sigma \in {\Sigma^(i)} : \sigma \succeq j\rbrace \subseteq {\Sigma^(i)} \) be the set of Player i’s sequences that terminate at j or any of its descendant information sets.

Example 2.1.

To illustrate some of the concepts and notation described so far, we consider the simple two-player extensive-form game in Figure 1, in which black round nodes belong to Player 1, and white round nodes belong to Player 2. The gray clusters of nodes identify the information sets. Since we chose different action numbers for different information sets, there exists a one-to-one correspondence between actions and sequences, and we will sometimes refer to sequences using the corresponding action number. For example, we will sometimes refer to sequence “3” to mean sequence \( (\text{B},{\mathsf{3}}) \), sequence “8” to mean sequence \( (\text {D},{\mathsf{8}}) \), and so on. Player 1 has four information sets—denoted a, b, c, and d—with two actions each. Player 2 only has two information sets, r and s, each with two actions. Information set d of Player 1 contains two nodes and models Player 1’s lack of knowledge of the action taken by Player 2 at information set s. The partial ordering between information sets for Player 1 is \( \text {A}\prec \text {B},\text {A}\prec \text {C},\text {A}\prec \text {D} \). Moreover, we have that \( \sigma ^{(1)}(\text {A})=\varnothing \), \( \sigma ^{(1)}(\text {B})=\sigma ^{(1)}(\text {C})={\mathsf{1}} \), and \( \sigma ^{(1)}(\text {D})={\mathsf{2}} \). For the terminal node z in the picture, \( \sigma ^{(1)}(z)={\mathsf{3}} \). Finally, we have that \( \Sigma ^{(1)}_{\text {A}}=\Sigma _*^{(1)}=\lbrace {\mathsf{1}},{\mathsf{2}},\dots ,{\mathsf{8}}\rbrace \), \( \Sigma ^{(1)}_{\text {B}}=\lbrace {\mathsf{3}},{\mathsf{4}}\rbrace \), \( \Sigma ^{(1)}_{\text {C}}=\lbrace {\mathsf{5}},{\mathsf{6}}\rbrace \), and \( \Sigma ^{(1)}_{\text {D}}=\lbrace {\mathsf{7}},{\mathsf{8}}\rbrace \).

Fig. 1.

View Figure

2.2.4 Deterministic Sequence-form Strategies.

Deterministic strategies are those that select, at each information set at which the player acts, exactly one action with probability one. Since the probability mass on each action is either 0 or 1, the set of deterministic sequence-form strategies for Player i—which we denote with the capital letter \( {\Pi^{(i)}} \)—corresponds exactly with the set of all sequence-form strategies whose components are all either 0 or 1.

Definition 2.4.

The set of deterministic sequence-form strategies for Player \( i \in [n] \) is the set \( \begin{equation*} {\Pi^{(i)}} := {\mathcal{Q}^{(i)}} \cap \lbrace 0,1\rbrace ^{{\Sigma^(i)} }. \end{equation*} \) Similarly, the set of deterministic sequence-form strategies for the subtree\( ^{{{2}}} \) rooted at j is \( \begin{equation*} {\Pi^{(i)}} _j := {{\mathcal{Q}^{(i)}} _j} \cap \lbrace 0,1\rbrace ^{{\Sigma^(i)} _j}. \end{equation*} \)

The set of deterministic sequence-form strategies corresponds one-to-one to the game-theoretic notion of reduced normal-form strategies (e.g., von Stengel [47, Section 4]). Furthermore, Kuhn’s Theorem [32] implies that \( \begin{equation*} {\mathcal{Q}^{(i)}} = \text {co}({\Pi^{(i)}}),\quad {\mathcal{Q}^{(i)}} _j = \text {co}({\Pi^{(i)}} _j) \qquad \quad \forall \,i\in [n], j\in {\mathcal{J}^{(i)}} . \end{equation*} \)

When it is important to emphasize that an arbitrary sequence-form strategy \( {\boldsymbol{q}} \in {\mathcal{Q}^{(i)}} \) (or \( {\boldsymbol{q}} \in {\mathcal{Q}^{(i)}} _j \) for some \( j\in {\mathcal{J}^{(i)}} \)) of Player \( i \in [n] \) need not be a deterministic sequence-form strategy, we will say that \( {\boldsymbol{q}} \) is a mixed sequence-form strategy.

Given a sequence-form strategy \( {\boldsymbol{q}} \in {\mathcal{Q}^{(i)}} \), it is possible to build an unbiased sampling scheme resulting in a (random) deterministic strategy \( \mathbf {\pi }\in {\Pi^{(i)}} \) such that \( \mathbb {E}[\mathbf {\pi }]={\boldsymbol{q}} \). A natural unbiased sampling procedure is the following: Start from any root information set of Player i, that is, an information set \( j \in {\mathcal{J}^{(i)}} \) such that \( \sigma ^{(i)}(j) = \varnothing \). Given any information set \( j \in {\mathcal{J}^{(i)}} \), an action \( a_j\in \mathcal {A}(j) \) is sampled with probability \( {\boldsymbol{q}} {}[(j,a_j)]/{\boldsymbol{q}} {}[\sigma ^{(i)}(j)] \); then, the same procedure is applied recursively to all information sets immediately reachable from sequence \( (j,a_j) \), that is, the information sets \( j^{\prime } \in {\mathcal{J}^{(i)}} \) such that \( \sigma ^{(i)}(j^{\prime }) = (j,a_j) \). The process is repeated for all the root information sets of Player i. The final deterministic sequence-form strategy \( \mathbf {\pi } \) is obtained by setting \( \mathbf {\pi }[(j,a_j)]=1 \) for each information set \( j \in {\mathcal{J}^{(i)}} \) visited during the procedure, and all other entries equal to 0.

Finally, we denote as \( \Pi := \times _{i\in [n]}{\Pi^{(i)}} \) the set of joint deterministic sequence-form strategies of all the players. Therefore, an element of \( \Pi \) is a tuple \( \mathbf {\pi } = (\mathbf {\pi }^{(1)},\dots ,\mathbf {\pi }^{(n)}) \) specifying a deterministic sequence-form strategy \( \pi^{(i)} \) for each player \( i \in [n] \).

Example 2.5.

Continuing Example 2.1, in Figure 2, we provide one (mixed) sequence-form strategy \( \mathbf {q}\in {\mathcal{Q}^{(i)}} [1] \) and five deterministic sequence-form strategies \( \lbrace \mathbf {q}_{{\mathsf{135}}},\mathbf {q}_{{\mathsf{136}}},\mathbf {q}_{{\mathsf{145}}},\mathbf {q}_{{\mathsf{27}}},\mathbf {q}_{{\mathsf{28}}}\rbrace \subseteq {\Pi^{(i)}} [1] \) for the small game in Figure 1 (left). One can check that these vectors are valid sequence-form strategies by verifying that the probability-mass-conservation constraints of Definition 2.2 hold. Let us consider the mixed sequence-form strategy \( {\boldsymbol{q}} \). There, \( {\boldsymbol{q}} [(\text {A},{\mathsf{1}})]={\boldsymbol{q}} [(\text {A},{\mathsf{2}})]=0.5 \), and therefore Player 1 will select between actions \( {\mathsf{1}} \) and \( {\mathsf{2}} \) at information set \( \text {A} \) uniformly at random. Suppose Player 1 selects action \( {\mathsf{1}} \). Then, if Player 1 reached information set b, then she would select actions \( {\mathsf{3}} \) and \( {\mathsf{4}} \) with probability \( 0.25 / 0.5=0.5 \) each. However, if Player 1 reached information set c, then she would choose action \( {\mathsf{5}} \) with probability \( 0.1/0.5=0.2 \), and action \( {\mathsf{6}} \) with probability \( 0.4/0.5=0.8 \). Analogously, if Player 1 played action \( {\mathsf{2}} \) at information set a, then upon reaching information set d she would play action \( {\mathsf{8}} \) with probability \( 0.5/0.5=1 \). In general, the probability of playing action a at a generic information set j can be obtained by dividing \( {\boldsymbol{q}} [(j,a)] \) by \( {\boldsymbol{q}} [\sigma ^{(i)}(j)] \). As a second example, consider the deterministic sequence-form strategy \( {\boldsymbol{q}} _{{\mathsf{136}}} \). When Player 1 plays according to that strategy, she will always choose action \( {\mathsf{1}} \) at information set a, action \( {\mathsf{3}} \) at information set b, and action \( {\mathsf{6}} \) at information set c. It is impossible for the player to reach information set d given her strategy at a and correspondingly \( {\boldsymbol{q}} _{{\mathsf{136}}}[{\mathsf{7}}] = {\boldsymbol{q}} _{{\mathsf{136}}}[{\mathsf{8}}] = 0 \).

Fig. 2.

View Figure

2.3 Extensive-form Correlated Equilibrium (EFCE)

Extensive-form correlated equilibrium has been proposed by von Stengel and Forges [48] as the natural counterpart to (normal-form) correlated equilibrium in extensive-form games. In an EFCE, before the beginning of the game the mediator draws a recommended action for each of the possible information sets that players may encounter in the game, according to some probability distribution defined over joint reduced normal-form strategies. These recommendations are not immediately revealed to each player. Instead, the mediator incrementally reveals relevant action recommendations as players reach new information sets. At any information set, the acting player is free to deviate from the recommended action, but doing so comes at the cost of future recommendations, which are no longer issued if the player deviates. In an EFCE, the recommended behavior is incentive-compatible for each player, that is, no player is strictly better off ever deviating from any of the mediator’s recommended actions.

Before introducing the formal definition of EFCE, let us mention that one could directly extend the definition of CE [1] to extensive-form games, thereby obtaining what is usually called normal-form correlated equilibrium(NFCE) of the extensive-form game. In this case, the mediator draws and recommends a complete reduced normal-form strategy to each player before the game starts. Then, before the beginning of the game, each player can decide whether to follow the recommended plan or deviate to an arbitrary strategy they desire. For arbitrary extensive-form games with perfect recall, the following inclusion of the set of equilibria holds: \( \text {NFCE}\subseteq \text {EFCE} \).

Multiple equivalent definitions of EFCE can be given. In this article, we follow the equivalent formulation given by Farina et al. [15] based on the concept of trigger agents introduced by Gordon et al. [20] and Dudík and Gordon [13]. In what follows, we will assume that an extensive-form game has been fixed:

An EFCE is a probability distribution \( \mathbf {\mu } \in \Delta ^{\Pi } \) over \( \Pi \) such that for any player \( i \in [n] \), trigger sequence \( \hat{\sigma } = (j,a) \in \Sigma _*^{(i)} \), and continuation strategy \( \hat{\mathbf {\pi }} \in {\Pi^{(i)}} _j \), the expected utility of the \( (\hat{\sigma },\hat{\mathbf {\pi }}) \)-trigger agent is not strictly greater than the expected utility that Player i would obtain by always following all of the mediator’s recommendations.

To turn the above condition into an analytic expression, it is useful to introduce the following additional quantities: Given a distribution \( \mathbf {\mu } \in \Delta ^{\Pi } \), we let \( r_{\mathbf {\mu }}(z) \) be the probability that the game ends in terminal node \( z \in \mathcal {Z} \) when all players follow recommendations issued by the mediator according to \( \mathbf {\mu } \); in particular, for every \( z \in \mathcal {Z} \), it holds: \( \begin{equation*} r_{\mathbf {\mu }}(z) := \sum _{\begin{matrix} ({\pi^{(1)}},\dots ,{\pi^{(n)}}) \in \Pi\\ {\pi^{(i)}}[\sigma ^{(i)}(z)] = 1 \,\,\, \forall i \in [n] \end{matrix}} \mathbf {\mu }[({\pi^{(1)}},\dots ,{\pi^{(n)}})], \end{equation*} \) where the summation is over all joint strategies \( (\pi^{(1)},\dots ,\pi^{(n)}) \in \Pi \) such that terminal node z is reachable when each player \( i \in [n] \) plays according to \( \pi^{(i)} \). Additionally, given a trigger sequence \( \hat{\sigma }= (j,a) \in {\Sigma^(i)} _* \) for a player \( i \in [n] \) and a continuation strategy \( \hat{\mathbf {\pi }} \in {\Pi^{(i)}} _j \), we let \( r^{(i)}_{\mathbf {\mu }, \hat{\sigma } \rightarrow \hat{\mathbf {\pi }}}(z) \) be the probability with which the \( (\hat{\sigma },\hat{\mathbf {\pi }}) \)-trigger agent reaches terminal node z. In particular, for every terminal node \( z \in \mathcal {Z} \) such that \( \sigma ^{(i)}(z) \succeq j \) it holds that: \( \begin{equation*} r^{(i)}_{\mathbf {\mu },\,\hat{\sigma } \rightarrow \hat{\mathbf {\pi }}}(z) := \left(\sum _{\begin{matrix} ({\pi^{(1)}},\dots ,{\pi^{(n)}}) \in \Pi\\ \pi [i^{\prime }][\sigma ^{(i^{\prime })}(z)] = 1 \,\,\, \forall i^{\prime }\ne i \\ {\pi^{(i)}}[\hat{\sigma }] = 1 \end{matrix}} \mathbf {\mu }[({\pi^{(1)}},\dots ,{\pi^{(n)}})] \right) \hat{\mathbf {\pi }} [ \sigma ^{(i)}(z) ] . \end{equation*} \)

We can now state the formal definition of EFCE and approximate EFCE.

2.4 Regret Minimization and Phi-regret Minimization

In this article, we will make heavy use of a mathematical object—one of the core abstractions in the field of online optimization—called a regret minimizer.

Calls to NextElement and ObserveUtility keep alternating to each other: First, the regret minimizer will output a point \( \mathbf {x}^1 \), then it will received feedback \( \ell ^1 \) from the environment, then it will output a new point \( \mathbf {x}^2 \), and so on. The decision making encoded by the regret minimizer is online, in the sense that at each time t, the output of the regret minimizer can depend on the prior outputs \( \mathbf {x}^1, \dots ,\mathbf {x}^{t-1} \) and corresponding observed utility functions \( \ell ^1,\dots ,\ell ^{t-1} \), but no information about future utilities is available. The objective for the regret minimizer is to output points so the cumulative regret (or simply regret) (3) \( \begin{equation} R^T := \max _{\mathbf {x}^*\in \mathcal {X}} \sum _{t=1}^T \Big (\ell ^t(\mathbf {x}^*) - \ell ^t(\mathbf {x}^t) \Big) \end{equation} \) grows asymptotically sublinearly in the time T. Many regret minimizers that guarantee a cumulative regret \( R^T = O(\sqrt {T}) \) at all times T for any convex and compact set \( \mathcal {X} \) are known in the literature (see, e.g., Cesa-Bianchi and Lugosi [9]).

A phi-regret minimizer is an extension of the concept of a regret minimizer introduced by Stoltz and Lugosi [44], building on previous work by Greenwald and Jafari [21].

In the special case of the set of constant transformations \( \Phi ^\text{const} := \lbrace \mathcal {X}\ni \mathbf {x} \mapsto \hat{\mathbf {x}}: \hat{\mathbf {x}}\in \mathcal {X}\rbrace \), the definition of cumulative phi-regret (4) reduces to that of cumulative regret given in (3). So, a regret minimizer is a special case of a phi-regret minimizer.

A general construction by Gordon et al. [20] gives a way to construct a \( \Phi \)-regret minimizer for \( \mathcal {X} \) starting from any regret minimizer (in the sense of Definition 2.8) for the convex hull \( \text {co}(\Phi) \) of the set of functions \( \Phi \). Specifically, let \( \mathcal {R}_\Phi \) be a deterministic ² regret minimizer for the set of transformations \( \text {co}(\Phi) \) whose cumulative regret grows sublinearly and assume that every \( \phi \in \text {co}(\Phi) \) admits a fixed point \( \phi (\mathbf {x}) = \mathbf {x}\in \mathcal {X} \). Then, a \( \Phi \)-regret minimizer \( \mathcal {R} \) can be constructed starting from \( \mathcal {R}_\Phi \) as follows:

• Each call to \( \mathcal {R}.\text {NEXTELEMENT} \) first calls NextElement on \( \mathcal {R}_\Phi \) to obtain the next transformation \( \phi ^t\in \text {co}(\Phi) \). Then, a fixed point \( \mathbf {x}^t = \phi ^t(\mathbf {x}^t) \) is computed and output.

• Each call to \( \mathcal {R}.\text {OBSERVEUTILITY}(\ell ^t) \) with linear utility function \( \ell ^t \) constructs the linear utility function \( L^t: \phi \mapsto \ell ^t(\phi (\mathbf {x}^t)) \), where \( \mathbf {x}^t \) is the last-output strategy, and passes it to \( \mathcal {R}_\Phi \) by calling \( \mathcal {R}_\Phi .\text {OBSERVEUTILITY}(L^t) \).²

The proof of correctness of the above construction is deceptively simple, and we recall it next. Since \( \mathcal {R}_\Phi \) outputs transformations \( \phi ^1,\phi ^2,\dots \) and receives utilities \( \phi \mapsto \ell ^1(\phi (\mathbf {x}^1)), \phi \mapsto \ell ^2(\phi (\mathbf {x}^2)), \dots \), its cumulative regret \( R_\Phi ^T \) is \( \begin{equation*} R_\Phi ^T = \max _{\phi ^*\in {\text {co}(\Phi)}} \sum _{t=1}^T \Big (\ell ^t(\phi ^*(\mathbf {x}^t)) - \ell ^t(\phi ^t(\mathbf {x}^t)) \Big). \end{equation*} \) Hence, since \( \mathbf {x}^t = \phi ^t(\mathbf {x}^t) \) is a fixed point of \( \phi ^t \), we can write (5) \( \begin{align} R_\Phi ^T = \max _{\phi ^*\in {\text {co}(\Phi)}} \sum _{t=1}^T \Big (\ell ^t(\phi ^*(\mathbf {x}^t)) - \ell ^t(\mathbf {x}^t) \Big) \ge {\max _{\phi ^*\in \Phi } \sum _{t=1}^T \Big (\ell ^t(\phi ^*(\mathbf {x}^t)) - \ell ^t(\mathbf {x}^t) \Big)}, \end{align} \) where the inequality follows from the observation that \( \text {co}(\Phi) \supseteq \Phi \). The right-hand side is exactly the cumulative \( \Phi \)-regret \( R^T \) incurred by \( \mathcal {R} \), as defined in (4). So, because the regret cumulated by \( \mathcal {R}_\Phi \) grows sublinearly by hypothesis, then so does the \( \Phi \)-regret cumulated by \( \mathcal {R} \).

4.1 Overview

Before delving into the details of the construction of our \( \psi^{(i)} \)-regret minimizer for the set of deterministic sequence-form strategies \( {\Pi^{(i)}} \) of a generic player \( i \in [n] \), we give an overview of the main logical steps that we use to solve Problem 1.

• In Section 4.2, we show that one can soundly move the attention from the set of deterministic sequence-form strategies \( {\Pi^{(i)}} \) to the set of mixed sequence-form strategies \( {\mathcal{Q}^{(i)}} = \text {co}({\Pi^{(i)}}) \). In particular, in the rest of the section, we will seek to construct a \( \psi^{(i)} \)-regret minimizer for the set \( {\mathcal{Q}^{(i)}} \) (as opposed to \( {\Pi^{(i)}} \)) that guarantees sublinear regret and polynomial-time implementation in the worst case.

• In Section 4.3, we show that the convex hull \( \text {co}(\psi^{(i)}) \) of the set of canonical trigger deviation functions possesses a combinatorial structure that can be leveraged to construct an efficient deterministic regret minimizer for it.

• Finally, in Section 4.4, we prove that given any \( \phi \in \text {co}(\psi^{(i)}) \), there exists a fixed-point sequence-form strategy \( \mathbf {q} \in {\mathcal{Q}^{(i)}} \) such that \( \phi (\mathbf {q}) = \mathbf {q} \), and that such a fixed-point strategy can be found in polynomial time in the number of sequences \( |{\Sigma^(i)} | \) of Player i.

Together, the last two steps enable us to apply the construction by Gordon et al. [20] described in Section 2.4 to obtain an efficient \( \text{co}(\Psi^{(i)}) \)-regret minimizer for the set of sequence-form strategies \( {\mathcal{Q}^{(i)}} \) with worst-case sublinear regret guarantees. We summarize that construction in Figure 4, which can serve as a reading aid for the section. Since \( \text {co}(\psi^{(i)}) \supseteq \psi^{(i)} \), such \( \text{co}(\Psi^{(i)}) \)-regret minimizer is also a \( \psi^{(i)} \)-regret minimizer, and the construction is complete.

Fig. 4.

View Figure

4.2 From Deterministic to Mixed Strategies

Suppose that a regret minimizer for a generic discrete set \( \mathcal {X} \) were sought, but only regret minimizers for the convex hull \( \text {co}(\mathcal {X}) \) were known. It seems natural to wonder whether one could take any regret minimizer for \( \text {co}(\mathcal {X}) \) and convert it into a regret minimizer for \( \mathcal {X} \) by sampling the outputs \( \bar{\mathbf {x}}^t \) of the former using an unbiased estimator \( \mathbf {x}^t \in \mathcal {X} \), with \( \mathbb {E}[\mathbf {x}^t] = \bar{\mathbf {x}}^t \). It is a folklore result, justified by a concentration argument, that this is indeed the case (see, for instance, Reference [9, page 192]). In particular, in the case of our interest where \( \mathcal {X}= {\Pi^{(i)}} \), the following can be shown:

Lemma 4.1 immediately implies that, to solve Problem 1, it is enough to solve the following problem:

The remainder of this section gives an algorithm that solves Problem 2 and, thus, indirectly, also Problem 1.

4.3 Regret Minimizer for the Convex Hull of the Set of Trigger Deviation Functions

In this subsection, we begin the construction of a phi-regret minimizer relative to the convex hull \( \text {co}(\psi^{(i)}) \) of the set of trigger deviation functions \( \psi^{(i)} \) for the set \( {\mathcal{Q}^{(i)}} \). Since \( \text {co}(\psi^{(i)}) \supseteq \psi^{(i)} \), any such \( (\text {co}(\psi^{(i)})) \)-regret minimizer is trivially also a \( \psi^{(i)} \)-regret minimizer for \( {\mathcal{Q}^{(i)}} \).

To obtain our \( (\text {co}(\psi^{(i)})) \)-regret minimizer, we will leverage the general framework due to Gordon et al. [20] that we recalled at the end of Section 2.4. In our particular case, that construction reduces to showing the following:

In this subsection, we will focus on point (1), while in the next subsection, we will focus on point (2). Specifically, the central result of this subsection, Theorem 4.6, will constructively establish the existence of an efficient deterministic regret minimizer \( \tilde{\mathcal {R}}^{(i)} \) for the set \( \text {co}(\psi^{(i)}) \).

The starting point of our approach is the observation that, because the convex hull operation is associative, \( \text {co}(\psi^{(i)}) = \text {co}(\lbrace {{\phi}^{(i)}_{\hat\sigma\rightarrow{\boldsymbol{y}}}} : \hat{\sigma }=(j,a)\in {\Sigma^(i)} _*, {\hat{\pi}} \in {\Pi^{(i)}} _j\rbrace) \) can be evaluated in two stages: First, for each sequence \( \hat{\sigma }= (j,a)\in {\Sigma^(i)} _* \) one can define the set \( \begin{equation*} {\bar\Psi^{(i)}_{\hat\sigma}} := \text {co}{\Big (}\big\lbrace {{\phi}^{(i)}_{\hat\sigma\rightarrow{\boldsymbol{y}}}} {}: \hat{\mathbf {\pi }} \in {\Pi^{(i)}} _{j} \big\rbrace {\Big)}; \end{equation*} \) and then, one can take the convex hull of all \( {\bar\Psi^{(i)}_{\hat\sigma}} \), that is, (24) \( \begin{equation} \text {co}(\psi^{(i)}) = \text {co}{\Big (}\big\lbrace {\bar\Psi^{(i)}_{\hat\sigma}} : \hat{\sigma }\in {\Sigma^(i)} _* \big\rbrace {\Big)}. \end{equation} \) Our construction of \( \tilde{\mathcal {R}}^{(i)} \) will follow a similar structure. First, for each \( \hat{\sigma }\in {\Sigma^(i)} _* \), we will construct a regret minimizer \( \tilde{\mathcal {R}}^{(i)}_{\hat{\sigma }} \) for the set of deviations \( {\bar\Psi^{(i)}_{\hat\sigma}} \) (Section 4.3.1). Then, we will combine all the regret minimizers \( \tilde{\mathcal {R}}^{(i)}_{\hat{\sigma }} \) into a composite regret minimizer \( \tilde{\mathcal {R}}^{(i)} \) for the set \( \text {co}(\psi^{(i)}) \) (Section 4.3.2).

4.3.1 Regret Minimizer for \( {\bar\Psi^{(i)}_{\hat\sigma}} \).

Fix any \( \hat{\sigma }=(j,a)\in {\Sigma^(i)} _* \). A deterministic regret minimizer for the set \( {\bar\Psi^{(i)}_{\hat\sigma}} \) can be constructed starting from any deterministic regret minimizer for the set \( {\mathcal{Q}^{(i)}} _j \). The crucial insight lies in the observation that the mapping \( \begin{equation*} h^{(i)}_{\hat{\sigma }}: \mathbb {R}^{{\Sigma^(i)} _j} \ni \mathbf {y} \mapsto \phi{^{(i)}_{\hat\sigma\rightarrow{\boldsymbol{y}}}} \end{equation*} \) is affine, since the entries in \( {\boldsymbol{M}^{(i)}_{\hat\sigma\rightarrow{\boldsymbol{y}}}} \) are defined using only constants and linear combinations of entries in \( \mathbf {y} \) (Definition 3.2). Hence, using the properties of affine functions, we can write \( \begin{equation*} \text {co}{\bigg (}\big\lbrace {{\phi}^{(i)}_{\hat\sigma\rightarrow{\boldsymbol{y}}}} : {\hat{\pi}} \in {\Pi^{(i)}} _j\big\rbrace {\bigg)} = \text {co}(h^{(i)}_{\hat{\sigma }}({\Pi^{(i)}} _j)) = h^{(i)}_{\hat{\sigma }}(\text {co}({\Pi^{(i)}} _j)) = h^{(i)}_{\hat{\sigma }}({\mathcal{Q}^{(i)}} _j). \end{equation*} \) So, we have just proved the following characterization of the set \( {\bar\Psi^{(i)}_{\hat\sigma}} \):

Lemma 4.3.

For all sequences \( \hat{\sigma }= (j,a)\in {\Sigma^(i)} _* \), \( {\bar\Psi^{(i)}_{\hat\sigma}} \) is the image of \( {\mathcal{Q}^{(i)}} _j \) under the affine mapping \( h^{(i)}_{\hat{\sigma }} \). In symbols, \( \begin{equation*} {\bar\Psi^{(i)}_{\hat\sigma}} = \big\lbrace \phi{^{(i)}_{\hat\sigma\rightarrow{\boldsymbol{q}\hat\sigma}}} : {\boldsymbol{q}} _{\hat{\sigma }}\in {\mathcal{Q}^{(i)}} _j\big\rbrace . \end{equation*} \)

As a consequence of Lemma 4.3, given any \( \hat{\sigma }=(j,a)\in {\Sigma^(i)} _* \), all transformations \( \phi \in {\bar\Psi^{(i)}_{\hat\sigma}} \) are of the form \( \phi =\phi{^{(i)}_{\hat\sigma\rightarrow{q_{\hat\sigma}}}} \) for some \( {\boldsymbol{q}} _{\hat{\sigma }}\in {\mathcal{Q}^{(i)}} _j \). Thus, the cumulative regret incurred by a generic sequence of transformations \( \phi ^1 = \phi{^{(i)}_{\hat\sigma\rightarrow{q^{(i)}_{\hat\sigma}}}}, \dots , \phi ^T = \phi{^{(i)}_{\hat\sigma\rightarrow{q^{(T)}_{\hat\sigma}}}} \) against generic linear utility functions \( L^1, \dots , L^T \) can be written as

(25)

Since \( L^t \) is linear and \( h^{(i)}_{\hat{\sigma }} \) is affine, their composition \( L^t\circ h^{(i)}_{\hat{\sigma }} \) is affine, and therefore the shifted function \( \begin{equation*} g^{(i),\,t}_{\hat{\sigma }} : \mathbb {R}^{{\Sigma^(i)} _j} \ni \mathbf {x} \mapsto L^t(h^{(i)}_{\hat{\sigma }}(\mathbf {x})) - L^t(h^{(i)}_{\hat{\sigma }}(\mathbf {0})) \end{equation*} \) is linear.² Furthermore, from (25) it follows that (26) \( \begin{equation} \max _{\phi ^* \in {\bar\Psi^{(i)}_{\hat\sigma}} } \sum _{t=1}^T L^t(\phi ^*) - L^t(\phi{^{(i)}_{\hat\sigma\rightarrow{q^{(t)}_{\hat\sigma}}}}) = \max _{\hat{\mathbf {q}}^* \in {\mathcal{Q}^{(i)}} _j}\sum _{t=1}^T g^{(i),\,t}_{\hat{\sigma }}(\hat{\mathbf {q}}^*) - g^{(i),\,t}_{\hat{\sigma }}({\boldsymbol{q}} _{\hat{\sigma }}^t). \end{equation} \)

Equation (26) suggests that if the continuation strategies \( {\boldsymbol{q}} _{\hat{\sigma }}^t \in {\mathcal{Q}^{(i)}} _j \) are picked by a deterministic regret minimizer \( \tilde{\mathcal {R}}^{(i)}_{\mathcal {Q}, \hat{\sigma }} \) that observes the linear utility functions \( g^{(i),\,t}_{\hat{\sigma }} \) at all times t, then the regret cumulated with respect to utility functions \( L^t \) by the corresponding elements \( \phi{^{(i)}_{\hat\sigma\rightarrow{q^{(t)}_{\hat\sigma}}}} \) grows sublinearly. We make that construction explicit in Algorithm 1.

Algorithm 1 can be instantiated with any deterministic regret minimizer \( \tilde{\mathcal {R}}^{(i)}_{\mathcal {Q},\,\hat{\sigma }} \) for the set of sequence-form strategies \( {\mathcal{Q}^{(i)}} _j \). The following proposition formalizes the cumulative regret guarantee when \( \tilde{\mathcal {R}}^{(i)}_{\mathcal {Q},\,\hat{\sigma }} \) is set to the CFR algorithm [49], which so far has arguably been the most widely used regret minimizer for sequence-form strategy spaces:

Proposition 4.4.

Let \( i\in [n] \) be any player and \( \hat{\sigma }=(j,a)\in {\Sigma^(i)} _* \) be any trigger sequence. Consider the deterministic regret minimizer \( \tilde{\mathcal {R}}^{(i)}_{\hat{\sigma }} \) (Algorithm 1), where \( \tilde{\mathcal {R}}_{\mathcal {Q},\,\hat{\sigma }} \) is set to be the CFR regret minimizer [49]. Upon observing a sequence of linear utility functions \( L^1,\dots ,L^T:\text {co}(\Psi^{(i)})\rightarrow \mathbb {R} \), the regret cumulated by the elements \( \phi ^1 = \phi{^{(i)}_{\hat\sigma\rightarrow{q^{(1)}_{\hat\sigma}}}},\dots ,\phi ^T = \phi{^{(i)}_{\hat\sigma\rightarrow{q^{(T)}_{\hat\sigma}}}} \) output by \( \tilde{\mathcal {R}}^{(i)}_{\hat{\sigma }} \) satisfies \( \begin{equation*} R^T = \max _{\phi ^*\in {\bar\Psi^{(i)}_{\hat\sigma}} } \sum _{t=1}^T L^t(\phi ^*) - L^t(\phi ^t) \le D\, |{\Sigma^(i)} _j|\, \sqrt {T}, \end{equation*} \) where D is the range of \( L^1, \dots , L^t \), that is, any constant such that \( \max _{\phi ,\phi ^{\prime }\in {\bar\Psi^{(i)}_{\hat\sigma}} }\lbrace L^t(\phi)-L^t(\phi ^{\prime })\rbrace \le D \) for all \( t=1,\dots , T \). Furthermore, the NextElement and the ObserveUtility operations run in \( O(|{\Sigma^(i)} |) \) time.

Proof.

As shown in (26), the regret cumulated by \( \tilde{\mathcal {R}}^{(i)}_{\hat{\sigma }} \) upon observing linear utility functions \( L^1,\dots ,L^t \) equals the regret cumulated by the CFR algorithm upon observing linear utility functions \( g^{(i),\,t}_{\hat{\sigma }} : \mathbb {R}^{{\Sigma^(i)} _j}\ni \mathbf {x} \mapsto L^t(h^{(i)}_{\hat{\sigma }}(\mathbf {x})) - L^t(h^{(i)}_{\hat{\sigma }}(\mathbf {0})) \). Furthermore, the range of \( g^{(i),\,t}_{\hat{\sigma }} \) satisfies the inequality \( \begin{align*} \max _{\mathbf {q},\,\mathbf {q}^{\prime }\in {\mathcal{Q}^{(i)}} _j} \big\lbrace g^{(i),\,t}_{\hat{\sigma }}(\mathbf {q}) - g^{(i),\,t}_{\hat{\sigma }}(\mathbf {q}^{\prime })\big\rbrace & = \max _{\mathbf {q},\,\mathbf {q}^{\prime }\in {\mathcal{Q}^{(i)}} _j} \big\lbrace L^t(h^{(i)}_{\hat{\sigma }}(\mathbf {q})) - L^t(h^{(i)}_{\hat{\sigma }}(\mathbf {q}^{\prime }))\big\rbrace \\ & = \max _{\phi ,\,\phi ^{\prime }\in {\bar\Psi^{(i)}_{\hat\sigma}} } \big\lbrace L^t(\phi)-L^t(\phi ^{\prime }) \big\rbrace \le D. \end{align*} \) So, applying the regret bound of the CFR algorithm (Theorems 3 and 4 of Zinkevich et al. [49]), \( \begin{equation*} R^T \le D {\begin{matrix} (\sum _{j^{\prime } \succeq j}\sqrt {|\mathcal {A}(j^{\prime })|}\end{matrix}})\sqrt {T} \le D{\begin{matrix} (\sum _{j^{\prime }\succeq j} |\mathcal {A}(j^{\prime })|\end{matrix}})\sqrt {T} = D|{\Sigma^(i)} _j|\sqrt {T}, \end{equation*} \) completing the proof of the regret bound.

The complexity analysis of the NextElement operation follows directly from the fact that CFR’s NextElement operation runs in linear time in \( |{\Sigma^(i)} _j| \). So, we focus on the complexity of the ObserveUtility operation. Fix any time t, and let \( \mathbf {\Lambda }^t := \langle L^t\rangle \) be the canonical representation (defined in Section 2.1) of the linear utility function \( L^t \). Since the canonical representation of \( h^{(i)}_{\hat{\sigma }}(\mathbf {x}) \) is the matrix \( \boldsymbol{M}{^{(i)}_{\hat\sigma\rightarrow{x}}} \) for all \( \mathbf {x}\in \mathbb {R}_{\ge 0}^{{\Sigma^(i)} _j} \), from (1), we obtain \( \begin{align*} L^t(h^{(i)}_{\hat{\sigma }}(\mathbf {x})) - L^t(h^{(i)}_{\hat{\sigma }}(\mathbf {0})) & = \sum _{\sigma _r,\sigma _c\in {\Sigma^(i)} _j} \mathbf {\Lambda }^t[\sigma _r,\sigma _c]\,{\begin{matrix} (\boldsymbol{M}{^{(i)}_{\hat\sigma\rightarrow{x}}}{}[\sigma _r,\sigma _c] - \boldsymbol{M}{^{(i)}_{\hat\sigma\rightarrow{0}}}\end{matrix}}) \\ & = \sum _{\sigma _r\succeq j} \mathbf {\Lambda }^t[\sigma _r,\hat{\sigma }]\,\mathbf {x}[\sigma _r], \end{align*} \) where the second equality follows from expanding the definitions of \( \boldsymbol{M}{^{(i)}_{\hat\sigma\rightarrow{x}}}{}[\sigma _r,\sigma _c] \) and \( \boldsymbol{M}{^{(i)}_{\hat\sigma\rightarrow{0}}}[\sigma _r,\sigma _c] \) given in (8). So, the canonical representation \( \langle g^{(i),\,t}_{\hat{\sigma }}\rangle \) of \( g^{(i),\,t}_{\hat{\sigma }} \) is the vector \( (\mathbf {\Lambda }^t[\sigma _r,\hat{\sigma }])_{\sigma _r\in {\Sigma^(i)} _j} \), which can be clearly computed and stored in memory in \( O(|{\Sigma^(i)} _j|) \) time. Using the fact that CFR’s ObserveUtility operation runs in linear time in \( |{\Sigma^(i)} _j| \), the complexity bound of the statement follows.□

4.3.2 Regret Minimizer for co(ψ⁽ⁱ⁾).

Recently, Farina et al. [14] showed that a regret minimizer for a composite set of the form \( \text {co}(\lbrace \mathcal {X}_1, \dots , \mathcal {X}_m\rbrace) \) can be constructed by combining any individual regret minimizers for \( \mathcal {X}_1,\dots ,\mathcal {X}_m \) through a construction—called a regret circuit—which we describe next.

Proposition 4.5

(Farina et al. [14], Section 4.3¹²)

Let \( \mathcal {X}_1, \dots , \mathcal {X}_m \) be a finite collection of sets, and let \( \mathcal {R}_1, \dots ,\mathcal {R}_m \) be any regret minimizers for them. Furthermore, let \( \mathcal {R}_\Delta \) be any regret minimizer for the m-simplex \( \Delta ^m := \lbrace (\lambda _1, \dots ,\lambda _m) \in \mathbb {R}_{\ge 0}^m, \sum _k\lambda _k = 1\rbrace \). A regret minimizer \( \mathcal {R}_\text{co} \) for the set \( \text {co}(\lbrace \mathcal {X}_1, \dots , \mathcal {X}_m\rbrace) \) can be constructed starting from \( \mathcal {R}_1,\dots ,\mathcal {R}_m \) and \( \mathcal {R}_\Delta \) as follows:

• \( \mathcal {R}_\text{co}.\text {NEXTELEMENT} \) calls NextElement on each of the regret minimizers \( \mathcal {R}_1,\dots ,\mathcal {R}_m \), obtaining elements \( \mathbf {x}_1^t, \dots , \mathbf {x}_m^t \). Then, it calls the NextElement operation on \( \mathcal {R}_\Delta \), obtaining an element of the simplex \( \mathbf {\lambda }^t = (\lambda _1^t, \dots ,\lambda _m^t) \). Finally, it returns the element \( \begin{equation*} \lambda _1^t \mathbf {x}_1^t + \dots + \lambda _m^t \mathbf {x}_m^t \in \text {co}(\lbrace \mathcal {X}_1, \dots , \mathcal {X}_m\rbrace). \end{equation*} \)

• \( \mathcal {R}_\text{co}.\text {OBSERVEUTILITY}(L^t) \) forwards the linear utility function \( L^t \) to each of the regret minimizers \( \mathcal {R}_1, \dots , \mathcal {R}_m \). Then, it calls the ObserveUtility operation on \( \mathcal {R}_\text{co} \) with the linear utility function \( (\lambda _1, \dots ,\lambda _m) \mapsto L^t(\mathbf {x}^t_1)\lambda _1 + \dots + L^t(\mathbf {x}_m^t)\lambda _m \).

In doing so, the regret \( R_\text{co}^T \) cumulated by \( \mathcal {R}_\text{co} \) upon observing any T linear utility functions relates to the regrets \( R_1^T, \dots , R_m^T, R_\Delta ^T \) cumulated by \( \mathcal {R}_1,\dots ,\mathcal {R}_m,\mathcal {R}_\Delta \), respectively, according to the inequality (27) \( \begin{equation} R_\text{co}^T \le R_\Delta ^T + \max \lbrace R_1^T, \dots , R_m^T\rbrace . \end{equation} \)

We apply the construction described in Proposition 4.5 to obtain our deterministic regret minimizer \( \mathcal {R}^{(i)} \) for the set \( \text {co}(\psi^{(i)}) = \text {co}(\lbrace {\bar\Psi^{(i)}_{\hat\sigma}} :\hat{\sigma }\in {\Sigma^(i)} \rbrace) \) starting from the deterministic regret minimizers \( \tilde{\mathcal {R}}^{(i)}_{\hat{\sigma }} \) (Algorithm 1), one for each sequence \( \hat{\sigma }\in {\Sigma^(i)} _* \), as well as any deterministic regret minimizer \( \mathcal {R}_\Delta ^{(i)} \) for the simplex \( \Delta ^{{\Sigma^(i)} _*} \). Pseudocode is given in Algorithm 2.

Combining Proposition 4.5 and Proposition 4.4, we obtain the following result:

Theorem 4.6.

Consider the regret minimizer \( \tilde{\mathcal {R}}^{(i)} \) (Algorithm 2), where \( \mathcal {R}_\Delta ^{(i)} \) is set to the regret matching algorithm, and \( \tilde{\mathcal {R}}^{(i)}_{\hat{\sigma }} \) is instantiated as described in Proposition 4.4. Upon observing a sequence of linear utility functions \( L^1,\dots ,L^T:\text {co}(\psi^{(i)})\rightarrow \mathbb {R} \), the regret cumulated by the transformations \( \phi ^1,\dots ,\phi ^T\in \text {co}(\psi^{(i)}) \) output by \( \tilde{\mathcal {R}}^{(i)} \) satisfies \( \begin{equation*} R^T = \max _{\phi ^* \in \text {co}(\psi^{(i)})}\sum _{t=1}^T L^t(\phi ^*) - L^t(\phi ^t) \le 2D\,|{\Sigma^(i)} |\,\sqrt {T}, \end{equation*} \) where D is the range of \( L^1, \dots , L^t \), that is, any constant such that \( \max _{\phi ,\phi ^{\prime }\in \text {co}(\psi^{(i)})}\lbrace L^t(\phi)-L^t(\phi ^{\prime })\rbrace \le D \) for all \( t=1,\dots , T \). Furthermore, the NextElement and the ObserveUtility operations run in \( O(|{\Sigma^(i)} |^2) \) time.

Proof.

At all t, the range of the linear utility function \( \mathbf {\lambda } \mapsto \sum _{\hat{\sigma }\in {\Sigma^(i)} _*} \mathbf {\lambda }[\hat{\sigma }]\,L^t(\phi{^{(i)}_{\hat\sigma\rightarrow{q^{(t)}_{\hat\sigma}}}}) \) is upper bounded by D. Hence, from the known regret bound of the regret matching algorithm [22, 49], the regret cumulated by \( \mathcal {R}_\Delta ^{(i)} \) after T iterations is upper bounded as \( \begin{equation*} R^T_\Delta \le D \sqrt {|{\Sigma^(i)} _*|}\sqrt {T}\le D |{\Sigma^(i)} |\sqrt {T}. \end{equation*} \) However, the regret bound in Proposition 4.4 shows that, for all \( \hat{\sigma }=(j,a)\in {\Sigma^(i)} _* \), the regret \( R^T_{\hat{\sigma }} \) cumulated by \( \tilde{\mathcal {R}}_{\hat{\sigma }}^{(i)} \) is upper bounded as \( R^T_{\hat{\sigma }} \le D |{\Sigma^(i)} _j| \sqrt {T} \). Applying (27) together with the fact that \( |{\Sigma^(i)} _j| \le |{\Sigma^(i)} | \) for all \( j\in {\mathcal{J}^{(i)}} \) yields the regret bound in the statement.

The complexity analysis of NextElement is straightforward: The regret matching algorithm produces elements in \( O(|{\Sigma^(i)} |) \) time, while each iteration of the loop over \( {\Sigma^(i)} _* \) requires \( O(|{\Sigma^(i)} |) \) time. Then, we focus on the complexity of ObserveUtility. There, the only operation whose analysis is not immediately obvious is the construction of the linear utility function \( \ell ^t_\lambda : \mathbf {\lambda } \mapsto \sum _{\hat{\sigma }\in {\Sigma^(i)} _*} \mathbf {\lambda }[\hat{\sigma }]\,L^t(\phi{^{(i)}_{\hat\sigma\rightarrow{q^{(t)}_{\hat\sigma}}}}) \), where it is necessary to check that its canonical representation (Section 2.1), given by the vector \( (L^t(\phi{^{(i)}_{\hat\sigma\rightarrow{q^{(t)}_{\hat\sigma}}}}))_{\hat{\sigma }\in {\Sigma^(i)} _*} \), can be computed and stored in memory in \( O(|{\Sigma^(i)} |^2) \) time. Fix any \( \hat{\sigma }\in {\Sigma^(i)} _* \). The canonical representation of \( \phi{^{(i)}_{\hat\sigma\rightarrow{q^{(t)}_{\hat\sigma}}}} \) is \( \boldsymbol{M}{^{(i)}_{\hat\sigma\rightarrow{q^{t}_{\hat\sigma}}}} \), which is a matrix with \( O(|{\Sigma^(i)} |) \) nonzero entries. Therefore, using (1), the evaluation of \( L^t(\phi{^{(i)}_{\hat\sigma\rightarrow{q^{(t)}_{\hat\sigma}}}}) \) via the canonical representations of \( L^t \) (given as input) and \( \phi{^{(i)}_{\hat\sigma\rightarrow{q^{(t)}_{\hat\sigma}}}} \) takes \( O(|{\Sigma^(i)} |) \) time. Thus, the representation of \( \ell _\lambda \) can be computed in \( O(|{\Sigma^(i)} |^2) \) time, confirming the analysis in the statement.□

4.4 Computation of the Next Strategy

In this subsection, we complete the construction of our \( \text{co}(\Psi^{(i)}) \)-regret minimizer for \( {\mathcal{Q}^{(i)}} \) started in Section 4.3 by showing that every transformation \( \phi \in \text {co}(\psi^{(i)}) \) admits a fixed point \( \phi (\mathbf {q}) = \mathbf {q} \in {\mathcal{Q}^{(i)}} \) and that such a fixed point can be computed in time quadratic in the number of sequences \( {\Sigma^(i)} \) of Player i.

As a key step in our algorithm, we will use the following well-known result about stationary distributions of stochastic matrices:

Several algorithms are known for computing fixed points of stochastic matrices (see, e.g., Reference [39] for a comparison of eight different methods). Since the particular choice of method is irrelevant, in this article, we will make the following assumption:

Our algorithm for computing a fixed point of \( \phi \in \text {co}(\psi^{(i)}) \) requires that the transformation \( \phi \) be expressed as a convex combination of elements from the sets \( \lbrace {\bar\Psi^{(i)}_{\hat\sigma}} \rbrace _{\hat{\sigma }\in {\Sigma^(i)} _*} \), that is, an expression of the form (28) \( \begin{equation} \phi = \sum _{\hat{\sigma }\in {\Sigma^(i)} _*}\lambda _{\hat{\sigma }}\, \phi{^{(i)}_{\hat\sigma\rightarrow{q_{\hat\sigma}}}},\quad \text{ where } \sum _{\hat{\sigma }\in {\Sigma^(i)} _*}\lambda _{\hat{\sigma }}=1, \text{ and }\lambda _{\hat{\sigma }} \ge 0,\ \ \mathbf {q}_{\hat{\sigma }}\in {\mathcal{Q}^{(i)}} _j \quad \! \forall \ \hat{\sigma }=(j,a)\in {\Sigma^(i)} _*, \end{equation} \) in accordance with the characterization of \( \text {co}(\psi^{(i)}) \) established by (24) and Lemma 4.3. Note that our regret minimizer \( \tilde{\mathcal {R}}^{(i)} \) for the set \( \text {co}(\psi^{(i)}) \) (Algorithm 2) already outputs transformations \( \phi \) expressed in the form above.

Our algorithm operates incrementally, constructing a fixed point sequence-form strategy \( \mathbf {q} \) for \( \phi \) information set by information set in a top-down fashion. To formalize this notion of top-down construction, we will make use of the two following definitions:

It follows from Definition 4.10 that a \( {\mathcal{J}^{(i)}} \)-partial fixed point of \( \phi \) is a vector \( \mathbf {q}\in {\mathcal{Q}^{(i)}} \) such that \( \mathbf {q} = \phi (\mathbf {q}) \). The following simple lemma establishes an \( \emptyset \)-partial fixed point for any transformation \( \phi \in \text {co}(\psi^{(i)}) \):

The key result that powers our algorithm to compute a fixed point of any \( \phi \in \text {co}{(}\psi^{(i)} {)} \) is that a J-partial fixed point can be cheaply promoted to be a \( (J\cup \lbrace j^*\rbrace) \)-partial fixed point, where \( j^* \in {\mathcal{J}^{(i)}} \setminus J \) is any information set whose predecessors are all in J. Algorithm 3 below gives an implementation of such a promotion: \( \text {EXTEND}(\phi , J, j^*, \mathbf {x}) \) starts with a J-partial fixed point \( \mathbf {x} \) of \( \phi \) and modifies all entries \( \mathbf {x}[(j^*,a)] \), \( a\in \mathcal {A}(j^*) \), so \( \mathbf {x} \) becomes a \( (J\cup \lbrace j^*\rbrace) \)-partial fixed point. Therefore, at a conceptual level, one can repeatedly invoke \( \text {EXTEND} \), growing the trunk J one information set at a time until \( J = {\mathcal{J}^{(i)}} \), starting from the \( \emptyset \)-partial fixed point \( \mathbf {x}_0 \) described in Lemma 4.11.

Before giving a proof of correctness and an analysis of the complexity of Extend, we illustrate an application of the algorithm in the simple extensive-form game of Figure 1.

To prove correctness of \( \text {EXTEND} \) in Proposition 4.14, we will find useful the following technical lemma:

Proof.

Fix any trigger sequence \( \hat{\sigma }= (j^{\prime },a^{\prime })\in {\Sigma^(i)} _* \). By expanding the matrix-vector multiplication between \( \boldsymbol{M}{^{(i)}_{\hat\sigma\rightarrow{\hat{q}_{\hat\sigma}}}} \) (Definition 3.2) and \( \mathbf {x} \), we have that for all \( \sigma \in {\Sigma^(i)} \), (31) \( \begin{align} \phi{^{(i)}_{\hat\sigma\rightarrow{q^{\hat\sigma}}}}(\mathbf {x})[\sigma ] & = \mathbf {x}[\sigma ]1\!\!1 [\sigma \not\succeq \hat{\sigma }] + \mathbf {q}_{\hat{\sigma }}[\sigma ] \mathbf {x}[\hat{\sigma }] 1\!\!1 [\sigma \succeq j^{\prime }]. \end{align} \) Therefore, for all \( \sigma \in {\Sigma^(i)} \),

as we wanted to show.□

A fixed point for \( \phi \in \text {co}(\psi^{(i)}) \) can be computed by repeatedly invoking \( \text {EXTEND} \) to grow the trunk J one information set at a time, until \( J = {\mathcal{J}^{(i)}} \), starting from the \( \emptyset \)-partial fixed point \( \mathbf {x}_0 \in \mathbb {R}_{\ge 0}^{{\Sigma^(i)} } \) introduced in Lemma 4.11. This leads to Algorithm 4, whose correctness and computational complexity is a straightforward corollary of Proposition 4.14.

4.5 The Complete Algorithm

In this subsection, we put together all the pieces we constructed in the previous subsections to build a \( \psi^{(i)} \)-regret minimizer that satisfies all requirements of Problem 1.

First, we provide in Algorithm 5 our \( \text{co}(\Psi^{(i)}) \)-regret minimizer. Its correctness follows from the correctness of the construction by Gordon et al. [20], described in Section 2.4, and by using Theorem 4.6 and Corollary 4.15.

Proof.

From the properties of Gordon et al.’s construction [20] (Section 2.4), the cumulative \( \psi^{(i)} \)-regret incurred by \( \bar{\mathcal {R}}^{(i)} \) is equal, at all times, to the cumulative regret incurred by the underlying regret minimizer \( \tilde{\mathcal {R}}^{(i)} \) for the set of deviations \( \psi^{(i)} \). So, the regret bound follows from the regret analysis of Theorem 4.6.

Similarly, the complexity analysis follows from combining the analysis of \( \tilde{\mathcal {R}}^{(i)} \) and of FixedPoint (Algorithm 4), together with the observation that the canonical representation \( \langle L^t \rangle \) of the linear utility function \( \text {co}(\psi^{(i)})\ni \phi \mapsto \ell ^t(\phi (\mathbf {q}^t)) \) is the matrix \( \langle \ell ^t\rangle (\mathbf {q}^t)^\top \), which can be trivially computed in \( O(|{\Sigma^(i)} |^2) \) time.

Since \( \text {co}(\psi^{(i)}) \supseteq \psi^{(i)} \), Algorithm 5 is in particular also a \( \psi^{(i)} \)-regret minimizer for the set \( {\mathcal{Q}^{(i)}} \), and thus Theorem 4.16 establishes that Algorithm 5 provides a solution to Problem 2.

To obtain our \( \psi^{(i)} \)-regret minimizer for the set of pure sequence-form strategies \( {\Pi^{(i)}} \) from Algorithm 5, we apply the construction described in Section 4.2. The resulting regret minimizer, \( \mathcal {R}^{(i)} \), is given in Algorithm 6. Applying Lemma 4.1 immediately yields the following corollary:

Therefore, Algorithm 6 provides a regret minimizer that satisfies all requirements of Problem 1.