Evolutionary and dynamical coherence resonances in the pair approximated prisoner's dilemma game

Stochasticity has recently emerged as being a potent promoter of cooperative behaviour in systems developed under the framework of evolutionary game theory. In the spatial prisoner's dilemma game, the fitness of players adopting the cooperative strategy was found to be resonantly dependent on the intensity of payoff fluctuations. Evidently, the phenomenon resembles classical coherence resonance, whereby the noise-induced order, or coherence, of the dynamics is substituted with the noise-induced prevalence of the ‘good’ strategy, thus marking a constructive effect of noise on the system. The connection between the former ‘dynamical’ coherence resonance and the latter so-called ‘evolutionary’ coherence resonance, however, has not yet been established. The two different definitions of coherence resonance appear to provoke some discomfort. The goal of the present paper is therefore, on one hand, to draw a clear line between the two different perceptions of coherence resonance, and on the other, to show that the two apparently disjoint phenomena, that are currently related only by name, can in fact be observed simultaneously, sharing an identical mechanism of emergence.


Introduction
Evolutionary game theory [1]- [3] is a vibrant subject across diverse disciplines of social and natural sciences. It provides a framework for studying the evolution of different strategies under various conditions that are modelled by the rules of games [4] to which players must conform. One of the most commonly applied games is the prisoner's dilemma game [5] that, in its original form, consists of cooperators and defectors, which compete for resources during the play. The dilemma results from the fact that although mutual cooperation yields the highest collective income, individuals do better if they decide to defect. Due to the fundamental principles of Darwinian selection, according to which only the most successful individuals prosper and reproduce, players possess an innate selfish drive that forces them to maximize their own income, thereby not paying any attention to the harm inflicted on the neighbour or the population in general. Hence, players decide to defect whereby none of them gets a profit. This unfavourable equilibrium is, however, often violated in real-life situations [6]- [9] and thus there emerges the need to explain the persistence of cooperative behaviour in the framework of the prisoner's dilemma game as well as evolutionary game theory in general.
One of the most famous mechanisms that is able to assure the survival of the cooperative strategy is the extension of the classical two-player prisoner's dilemma game to the multipleplayer game with agents located on vertices of a square grid [10]. The so-called spatial games or games on grids have become an inspiration for numerous studies trying to precise optimal conditions under which cooperators thrive best. A very promising approach has emerged to be the extension of the classical game not just from two-to a multi-player game but also from two-to the three-strategy game [11]. The so-called loners or volunteers, representing the third strategy besides cooperators and defectors, offer an escape hatch out of evolutionary stalemate by inducing a rock-paper-scissors cyclic dominance of all three strategies [12]. Thus, cooperators survive via oscillatory exchanges of dominance between the three possible strategies.
Recently, the addition of stochasticity has also emerged as being a potent promoter of cooperative behaviour in the spatial two-strategy prisoner's dilemma game if only the intensity of noise was fine-tuned to obtain the optimal result [13]. Since the prosperity of the cooperative strategy was found to have a resonant dependence on the intensity of introduced fluctuations, it was argued that the phenomenon is conceptually identical to coherence resonance observed previously in temporal or spatiotemporal dynamical systems, where noise alone can often induce or enhance order in the temporal [14,15], spatiotemporal [16] or spatial [17]- [19] dynamics of the system provided its intensity is carefully adjusted. Similar results were reported recently also for the spatial two-strategy prisoner's dilemma game on different types of random regular graphs [20] and for the noise-driven replicator dynamics with adaptive learning rates [21]. However, due to the fast progress and rapid accumulation of new results on this topic, a discomfort has emerged due to the seeming unrelatedness between the classical 'dynamical' coherence resonance phenomenon, describing noise-induced order or coherence of the temporal or spatial dynamics in a system [14]- [19], and the recently reported 'evolutionary' coherence resonances that so far have not yet been related to any particular enhancement of dynamical coherence, but reflect solely the constructive effect of noise in terms of the facilitation of the cooperative strategy or the cumulative payoff of the population.
The main objective of the present paper is to draw a clear line between the 'dynamical' and 'evolutionary' coherence resonances that can be observed in the framework of evolutionary game theory. For this purpose, we employ two dynamical systems derived by the pair approximation of the two-and three-strategy spatial prisoner's dilemma game [12], [22]- [24]. While the former comprises only steady state solutions of the frequencies of each strategy, the latter exhibits a more versatile dynamical behaviour, including a Hopf bifurcation to oscillatory changes of the three possible strategies. Importantly, while both models allow the observation of 'evolutionary' coherence resonances by varying the uncertainty level related to the strategy adoption process of participating players, the more complex three-strategy model enables also the observation of the classical 'dynamical' coherence resonance. In particular, by introducing Gaussian noise to the payoffs the observation of 'dynamical' coherence resonance in the system can be observed in the vicinity just before the Hopf bifurcation via the well-known mechanism of noise-induced anticipation of the oscillatory behaviour beyond the bifurcation point [25,26]. Fascinatingly, the resulting 'dynamical' coherence resonance is accompanied also by the 'evolutionary' coherence resonance, thus indicating the fact that under special conditions, such is the vicinity of a Hopf bifurcation point, the two seemingly unrelated phenomena are in fact intimately related and even share the same mechanism of emergence.

Pair approximated prisoner's dilemma game
We consider the two-and three-strategy prisoner's dilemma that is devised from the pair approximation [12,27] of the spatial version of the game [10]. The pair approximation tracks the frequencies of all possible strategy pairs in the game. The probability of finding an individual adopting strategy s accompanied by a neighbour adopting s is given by p s,s , where s, s ∈ {c, d} and s, s ∈ {c, d, l} in the two-and three-strategy prisoner's dilemma, respectively. Notations c, d and l indicate the strategies of cooperators, defectors and loners, respectively. To track the time development of the frequencies of all possible strategy pairs in the two-strategy prisoner's dilemma, we thus need to determine:ṗ c,c ,ṗ c,d ,ṗ d,c andṗ d,d . Because of the symmetry condition p c,d = p d,c and the constraint p c,c + p c,d + p d,c + p d,d = 1, we can describe the dynamics of the two-strategy game by only two differential equations: By taking into account the same symmetry conditions for mixed pairs as by the twostrategy prisoners dilemma, plus the constraint p c,c + p d,d + p l,l + 2p c,d + 2p c,l + 2p d,l = 1, the dynamics of the three-strategy game, on the other hand, can be described by five differential equations: In equations (1)- (7), the sums run over all possible strategies under consideration, n s (x, y, z) is the number of players adopting strategy s among the players x, y and z, while F s = s p s,s is the frequency or fitness of each particular strategy s, whereby s again runs over the set of all possible strategies. Importantly, note that F s establishes the formal connection between the mean-field theory and the pair approximation by converting pair configuration probabilities p s,s to the approximates of configuration probabilities of large clusters, thus warranting at least qualitatively identical results of both approaches. Moreover, W[P s (x, y, z) ← P s (u, v, w)] is the strategy adoption function to be specified below by equation (8), while P s (x, y, z) denotes the player adopting strategy s interacting with the neighbours adopting strategies x, y, z plus a player adopting strategy s . Analogously, P s (u, v, w) notates the player adopting strategy s interacting with the neighbours adopting strategies u, v, w plus a player adopting strategy s. For details regarding the derivation of individual differential equations, we refer the reader to [12] and the seminal work of Matsuda et al [27], where the pair approximation method is accurately described. Noteworthy, models devised in the framework of pair approximation possess a dynamical incompleteness, which is due to the fact that, in the simplest case, only a single pair of players is considered and corrections arising from the loops due to nearest-neighbour interactions are not taken into account. This results in the inability of the pair approximation to distinguish between different topological realizations of the spatial grid and less accurate predictions near extinction thresholds. Nevertheless, these difficulties can be surpassed by considering pair configuration probabilities of larger clusters, as exemplified in [20]. The dynamics of the resulting dynamical systems is governed by strategy changes of the players, and hence changes of the corresponding p s,s . Each player P s can change its strategy by comparing its payoff S s to the payoff S s of its neighbour P s in accordance with the strategy adoption function The payoffs of both players (S s , S s ), acquired during each integration step of the dynamical system, are calculated in accordance with the payoff matrix P s /P s c d l c 1 + ξ s /1 + ξ s 1 + r + ξ s /−r + ξ s δ + ξ s /δ + ξ s d −r + ξ s /1 + r + ξ s 0 + ξ s /0 + ξ s δ + ξ s /δ + ξ s l δ + ξ s /δ + ξ s δ + ξ s /δ + ξ s δ + ξ s /δ + ξ s 7 Institute of Physics ⌽ DEUTSCHE PHYSIKALISCHE GESELLSCHAFT Note that we use a rescaled version of the payoff matrix solely due to simplicity and clearness of the following calculations, but without any loss of generality. The payoff matrix is subjected to temporally and spatially white additive Gaussian noise [28], satisfying the correlation function ξ s (g)ξ s (h) = σ 2 δ ss δ gh , whereby indexes (s, s ) mark any of the two involved players adopting strategies s and s , respectively, while g and h index two consecutive integration steps. Moreover, r 0 determines the payoff of players, δ = 0.3 is the reward for voluntary participation, σ 2 is the variance of payoff variations, whereas K in equation (8) is the uncertainty related to the strategy adoption process [23]. 0 < K 1 implies that the better performing player is readily adopted. This condition corresponds to the so-called strong selection limit where the fitness F s of a particular strategy s is uniquely determined by the payoffs, while large values of K (e.g. K > 1) constitute the weak selection limit.
It is important to note that both parameter K and σ can be used to adjust the level of stochasticity in the game. In particular, while larger values of K smooth-out the strategy adoption function and thus render the adoption of the more successful strategy random, the parameter σ directly determines the intensity of payoff fluctuations, which again smears the deterministic nature of the game. However, while the introduction of stochasticity via different values of K is numerically more propitious, the introduced form of noise is 'static', meaning that the equilibrium states are obtained without the characteristic noise-induced fluctuations in the temporal traces. Therefore, different values of K qualify for the observation of 'evolutionary' coherence resonance, but not for the observation of its 'dynamical' counterpart. The latter, 'dynamical' coherence resonance can be observed only via adjustments of parameter σ that introduces dynamical noise to the system. In the next section, we will examine effects of different values of K and σ, separately from one another, for both studied dynamical systems as outlined in the Introduction.

Results
We start by studying effects of different values of K on the equilibrium frequency of cooperators for both studied dynamical systems and various values of r, while keeping the payoffs deterministic by setting σ = 0. Results presented in figure 1 clearly evidence that there exist an optimal amount of stochasticity, determined by the uncertainty K related to the strategy adoption process via equation (8), for which the frequency of the cooperative strategy F c is resonantly enhanced. The results are fairly robust against variations of r, although the magnitude of the phenomenon deteriorates substantially as r is increased. Indeed, Ohtsuki et al [29] have recently reported a very interesting and simple rule for the evolution of cooperation on graphs and social networks. For the weak selection limit, presently holding when K is large, the fulfilment of the condition [(1 + r)/r] > k, where k is the connectivity of each player, results in the persistence of the cooperative strategy even in the absence of reputation effects or strategic complexity. Our numerical calculations for the two-and three-strategy pair approximated prisoner's dilemma game fully comply with the simple rule [29], whereby we use k = 4 for the comparisons since each of the two central players in the pair approximation has four neighbours. In particular, note how the fraction of cooperators in figure 1 converges to zero convincingly as r and K become large. On the other hand, values of r smaller than 1/3, satisfying the condition, always keep cooperators alive. Remarkably, this result is valid for a broad range of topological realizations of the spatial grid and the pertaining pair approximations. Since the resonant dependence of F c on K marks a constructive effect of noise on the system (assuming here that the promotion of cooperation is indeed considered a constructive effect since it promotes the overall welfare of the whole population), it appears reasonable to term the phenomenon coherence resonance, as already argued in [13], where qualitatively identical results have been reported for the classical two-strategy prisoner's dilemma game. However, since different values of K just shift the steady state of the system there is, in fact, no noise-induced coherence of dynamical behaviour neither in the temporal nor spatial domain, as this is the case when studying dynamical systems [14]- [19]. Thus, the usage of the term coherence resonance to describe the phenomenon presented in figure 1 must be exercised with caution. Therefore, we propose the addition of the word 'evolutionary' prior to coherence resonance, solely to indicate the fact that it is only a particular evolutionary strategy that is resonantly promoted by noise, rather than the emergence of noise-induced order or coherence in the temporal dynamics of the studied system. However, since the pair approximation of the prisoner's dilemma game yields a nonlinear dynamical system, the observation of the classical 'dynamical' coherence resonance might still be feasible. Some well-known dynamical states that assure the observation of 'dynamical' coherence resonance are excitability [14,18,19,30] or the proximity of the system to special types of bifurcation points that mark the advent of dynamical instability, oscillatory regime or bistability [15,25,26]. In order to search for an appropriate dynamical state among the abovelisted, we perform a bifurcation analysis of both studied dynamical systems, whereby parameter r is considered as the bifurcation parameter. Moreover, we set σ = 0 and K = 0.001, which essentially corresponds to the deterministic setting of the game. Note that for K = 0.001, the strategy adoption function given by equation (8) is virtually step-like, thus effectively modelling the fully deterministic best-takes-over strategy adoption rule [31] corresponding to the strong selection limit. Results in figure 2 reveal that the two-strategy model comprises only steady states, and thus in the absence of excitability, does not qualify for the observation of 'dynamical' coherence resonance. On the other hand, the three-strategy model incorporates a Hopf bifurcation, occurring at r = 0.13, marking the transition from the steady state to the oscillatory dynamical regime. It is the proximity of this Hopf bifurcation we will try to exploit for the observation of the classical 'dynamical' coherence resonance in the studied three-strategy pair approximated prisoner's dilemma game.
Thus, we set r = 0.125 and introduce dynamical noise to the system via nonzero values of σ, while leaving the uncertainty related to the strategy adoption process constant at K = 0.001. In order to quantify the possible emergence of noise-induce coherence in the temporal domain, we calculate the characteristic correlation time, as proposed in [14], according to the formal equation where C(t) is the normalized autocorrelation function of the temporal trace of F c . Results in the left-hand panel of figure 3 clearly indicate that τ c exhibits a resonant dependence on σ, thus marking a classical 'dynamical' coherence resonance in the studied system. Remarkably, the phenomenon is also accompanied by the 'evolutionary' coherence resonance shown in the right-hand panel of figure 3. However, since nonzero values of σ induce fluctuations in the temporal traces of F c , we use the average of F c over a long time span (F c ) as the defining quantity for measuring the constructive effect of noise on the cooperative strategy. Indeed, by the same procedure as used to obtain results in the right-hand panel of figure 3, resonant curves in figure 1 can also be easily obtained by substituting K by nonzero values of σ (while setting K = 0.001, for example, to avoid a double source of stochasticity in the The strategy adoption uncertainty is fixed and equals K = 0.001, while r is set just prior to the Hopf bifurcation point at 0.125. Evidently, noisy disturbances are able to anticipate the dynamical behaviour of the system past the bifurcation point in a resonant manner depending on the intensity of noise. Strikingly, the average frequency of cooperatorsF c , calculated over a long time span after the transients have been discarded, also exhibits a resonant dependence on the intensity of noise, thus unifying the two conceptually different formulations of coherence resonance in the examined system. system, although higher values of K still yield a resonant dependence of F c on σ). Thus, we show that in the vicinity of special dynamical states, models of evolutionary game theory qualify also for the observation of 'dynamical' coherence resonance, and moreover, the latter can be accompanied also by the 'evolutionary' coherence resonance. Note though that in comparison to figure 1, the 'evolutionary' coherence resonance presented in the right-hand panel of figure 3 is expressed a little faintly (compare the values on the y-axis). This, however, is in accordance with the trend outlined already in figure 1, where it becomes instantly obvious that larger values of r decrease the potentially constructive effect of noise on the promotion of cooperation. Since the bifurcation point in the studied system occurs at a rather high value of r, the resulting magnitude of the 'evolutionary' coherence resonance at this point is thus small. Nonetheless, results in figure 3 succinctly show that 'dynamical' and 'evolutionary' coherence resonances, in models devised in the framework of evolutionary game theory, can be observed simultaneously, and that they also share the same mechanism of emergence; namely the noise-induced anticipation of the dynamical behaviour of the system beyond the Hopf bifurcation point.

Summary
In summary, we outline the difference between the resonantly dependent noise-induced facilitation of a particular game strategy, termed 'evolutionary' coherence resonance, and the resonantly dependent noise-induced order in the temporal dynamics of the studied system, Institute of Physics ⌽ DEUTSCHE PHYSIKALISCHE GESELLSCHAFT termed 'dynamical'coherence resonance [14]. We show that 'evolutionary'coherence resonances can be observed in simple two-strategy as well as in more complex three-strategy models, whereby the stochasticity introduced via the strategy adoption function yields the same results as dynamical noise introduced to the payoffs of players [28]. On the other hand, the classical 'dynamical' coherence resonance phenomenon can only be observed near special dynamical states, which can be found only in the three-strategy pair approximated prisoner's dilemma game. In the vicinity of the Hopf bifurcation incorporated in the model, dynamical noise is able to anticipate the oscillatory behaviour beyond the bifurcation point in a resonant manner, which results in a bell-shaped dependence of τ c on σ, thus marking 'dynamical' coherence resonance in the studied system. Importantly, the introduction of 'dynamical' noise is thereby essential, since noise introduced via the strategy adoption function yields smooth transitions to equilibrium states, thus essentially acting as 'static' or 'quenched' noise. Fascinatingly, the 'dynamical' coherence resonance in the three-strategy model is accompanied also by the 'evolutionary' coherence resonance. Thus, although the exact mechanism bringing about 'evolutionary' coherence resonances is not known in general, we show that at least near special dynamical states, the two types of noise-induced resonant behaviour can have the same mechanism of emergence and are hence intimately related.
Since the present study is to our knowledge the first to report 'dynamical' coherence resonance in a model developed within the framework of evolutionary game theory, we hope it will be an inspiration for future research in the apparently very fruitful and interesting combination of stochasticity and evolutionary game theory.