Evolution of Cooperation on Stochastic Dynamical Networks

Cooperative behavior that increases the fitness of others at a cost to oneself can be promoted by natural selection only in the presence of an additional mechanism. One such mechanism is based on population structure, which can lead to clustering of cooperating agents. Recently, the focus has turned to complex dynamical population structures such as social networks, where the nodes represent individuals and links represent social relationships. We investigate how the dynamics of a social network can change the level of cooperation in the network. Individuals either update their strategies by imitating their partners or adjust their social ties. For the dynamics of the network structure, a random link is selected and breaks with a probability determined by the adjacent individuals. Once it is broken, a new one is established. This linking dynamics can be conveniently characterized by a Markov chain in the configuration space of an ever-changing network of interacting agents. Our model can be analytically solved provided the dynamics of links proceeds much faster than the dynamics of strategies. This leads to a simple rule for the evolution of cooperation: The more fragile links between cooperating players and non-cooperating players are (or the more robust links between cooperators are), the more likely cooperation prevails. Our approach may pave the way for analytically investigating coevolution of strategy and structure.


Introduction
Cooperation is ubiquitous in the real world ranging from genes to multicellular organisms [1][2][3][4]. Most importantly, human society is based upon cooperation. However this cooperative behavior apparently contradicts natural selection [5]: Selfish behavior will be rewarded during competition between individuals, because selfish individuals enjoy the benefits from the cooperation of others, but avoid the associated costs. Therefore, the puzzle how natural selection can lead to cooperation has fascinated evolutionary biologists since Darwin.
Evolutionary game theory is an intuitive and convenient framework to study this puzzle. As a metaphor, the Prisoner's Dilemma (PD) has been widely used to investigate the origin of cooperation. In this game, two players simultaneously decide whether to cooperate (C) or to defect (D). They both receive R upon mutual cooperation and P upon mutual defection. A defector exploiting a cooperator receives T, and the exploited cooperator gets S. This can be formalized in the form of a payoff matrix, The PD is characterized by the payoff ranking TwRwPwS. For repeated games, the additional requirement 2RwTzS ensures that alternating between strategies is less lucrative than repeated mutual cooperation. In the one shot PD, it is best for a rational individual never to cooperate irrespective of the co-player's decision. Thus, defection is the Nash Equilibrium [6]. However, the two players would be better off if they both cooperated, hence the dilemma. In an evolutionary setting, where payoff determines reproductive fitness, defectors can reproduce faster based on their higher payoff and cooperation diminishes -defection is evolutionary stable [7,8]. Several mechanisms have been proposed to explain the persistence of cooperative behavior, including kin selection [9], direct [10,11] and indirect reciprocity [12,13], group selection [14,15] as well as the network reciprocity [16][17][18][19][20][21][22][23][24]. Furthermore, the relationship between these mechanisms receives an increasing attention [25][26][27][28]. Both in animal and human societies, individuals interact with a limited number of individuals. The interactions of individuals are often captured based on the network of contacts. Therefore, there has been an increasing interest in the influence of population structure on the evolution of cooperation.
Nowak and May first studied the PD game on regular lattices [16]. Subsequently, social dilemmas on regular graphs have been investigated [19][20][21]29]. Many authors have also considered more complex networks, such as scale-free and small-world afterwards [17,18,24]. It has been well recognized that network topologies can play a crucial role in the evolution of cooperation, in addition to the payoff matrix and the update mechanism.
Dynamical networks can significantly boost cooperation compared to static networks. On the one hand, cooperation thrives if individuals are able to promptly adjust their social ties, because this allows cooperators to escape from defectors [38]. Similarly, cooperation is more likely to occur if the favored relationships between cooperators (CC links) tend to be less fragile than adverse social ties (CD links) [37,43]. The latter result is consistent with our empirical intuitions and is widely observed in the real world. However, most of the works on this issue are investigated only by numerical methods and not by analytical approaches. This is mainly because it is difficult to describe the coevolution of strategy and structure of a network analytically.
Pacheco et al. approximate their linking dynamics by ordinary differential equations [39][40][41]43]. They found that fast linking dynamics leads to a transformation of the payoff matrix, such that e.g. cooperation in a Prisoner's Dilemma can be stabilized. This approach does not keep the total number of links constant. Moreover, the analytical approach does not take stochastic effects into account.
Here, we consider a linking dynamics described by a discrete stochastic model. The evolution of links can be described as a Markov chain, which is the starting point for our analytical considerations. We specify the conditions required for the payoff matrix to make cooperation stable. A simple rule is obtained when the linking dynamics proceeds sufficiently fast, which reveals quantitatively how the link breaking probabilities have to be chosen such that cooperation may gain a foothold. Furthermore, we show how our stochastic linking dynamics also results in a transformation of the payoff matrix as in [39].

Analysis
We consider the coevolution of strategy and structure in the PD game. Each player's strategy s can either be cooperation (C) or defection (D), denoted by (1,0) T and (0,1) T , respectively. Initially, the whole population of size N are situated on vertices of a regular graph with degree L, where nodes indicate individuals while edges denote the pairwise partnerships between individuals. We consider the case where the total number of agents N is much larger than the average degree L. The payoff of each individual is obtained by playing the PD game with all of its immediate neighbors: where N i represents the neighborhood set of player i and M is the payoff matrix. Instead of the general matrix of the Prisoner's Dilemma Eq. (1) with four parameters, we consider a simpler payoff matrix, where the parameter u, measuring how profitable unilateral defection is, ranges from zero to one. Note that this payoff matrix recovers the payoff ranking described above, TwRwPwS. We emphasize that Eq.(3) describes a special case of general PD games, but it is widely used in biology and sociology [1].
In each time step, an agent has the opportunity to change its strategy with probability w. With probability 1{w, a link in the network can be changed. For w~0, no strategy update takes place, hence the cooperation level stays unchanged and only the dynamical organization of cooperators and defectors can be observed [45]. For w~1, this model degenerates to a PD game on a static regular graph, which has been studied in great detail [16,19,22,29,48].
Let us first consider the dynamics of links (which occurs with probability 1{w). In each rewiring step, a link XY is selected from the network at random (XY~CC,CD or DD). The link remains intact with probability 1{k XY . With probability k XY , the link is broken. In this case, one of the two adjacent players is picked at random and switches to a random player who is not its immediate neighbor in the population (see Fig. (1)). In this way, link XY is broken and a new link XZ or YZ is introduced.
We consider the case of 0vk XY v1. In this case, linking dynamics does not assume rationality of agents: Adverse CD links may be kept and advantageous CC links may sometimes be broken.
In contrast with previous analytical work focusing on a dynamical number of links [39], here the total number of links LN=2 is constant in the evolution process as in [33,38]. This constraint can imply a limited resource and avoids that all individuals are linked to all others (for generic parameter choices).
In the beginning, each link is assigned a name j, where (j~1,2 Á Á Á ,LN=2). In each time step, we choose a link i t~i at random, where the superscript denotes the time. If the selected link i t does not break, we have i tz1~it . If the link breaks, a new social tie is introduced, denoted as i tz1 . We denote the type of link i t by T(i t ), where T(i t ) can be CC, CD or DD. Herein, we investigate how T(i t ) changes with time t.
The dynamics of T(i t ) can be captured by a Markov chain with transition matrix Q (XY )(ZW ) , which is the probability that an XY Figure 1. Linking dynamics. If the dashed link is selected in the topological evolution, it will be broken off with probability k CD . If the dashed link is broken, then either A or B is selected to establish a new link. If A is chosen, then he switches to a random individual of the population who is not his current neighbors (B, D, E, F, G or J). Otherwise, B is chosen, then he also switches to a random individual of the population who is not his current neighbors (A, F, G, H, I or J). doi:10.1371/journal.pone.0011187.g001 link transforms to a ZW link in one time step. According to the linking dynamics, the probability of moving between CC and DD is zero. So, we only have to calculate Q (XY )(XW ) or Q (XY )(ZY ) .
For instance, Q (CC)(CC) is the probability that i t of type CC transforms to i tz1 of type CC. This occurs in the following cases: (1) When i t is not selected in the linking dynamics (with probability (LN{2)=(LN)); (2) When i t is selected (with probability 2=(LN)), this happens either when the original CC link is not broken off (with probability 1{k CC ) or when the selected player of the original link switches to another cooperator provided the link is broken (with probability k CC x C where x C is the frequency of cooperators). Hence, Similar considerations for other links lead to the transition probability matrix where I 3 is the identity matrix and the matrix V is given by and x D~1 {x C is the frequency of defectors. We emphasize that the transition matrix is only an approximation, because it does not exclude the case that a player establishes a second link with one of its immediate neighbors. However, the approximation is very good when the degree of all links is much smaller than the population size.
Note that this Markov chain is irreducible and aperiodic when k CC k CD k DD x C =0, hence there exists a unique stationary distribution w~(w CC ,w CD ,w DD ) determined by equation wQ~w [49]. We find that where is a normalization factor. Here, w XY represents the probability that a link i t is of type XY in the stationary regime. Therefore, the average number of XY links N XY is given by: Thus, w XY also represents the average fraction of XY links in the whole population in the stationary regime of the linking dynamics.
Let us now consider the dynamics of strategies (which occurs with probability w). A player i with strategy s i is selected at random, subsequently player j with strategy s j is randomly selected among i's current neighbors. Player i compares the payoff with that of player j and takes strategy s j with probability [48,50] where P i and P j are the accumulated payoffs for i and j, respectively. The parameter b denotes the intensity of selection. For b%1, selection is weak and strategy changes are almost random. For b??, selection is strong and strategies of more successful agents are always adopted, whereas less successful agents are never imitated. In large, well mixed populations the dynamics can be approximated by [51] _ where j is the Gaussian white noise with variance 1, f C and f D denote the average fitness of a cooperator and a defector, respectively. For large population size N, the stochastic term vanishes [51] and we obtain Note that this equation has the same equilibrium properties as the usual replicator dynamics [8] _ If w is sufficiently small, the structure of the system is close to the stationary state when strategies change. In this case, the stationary distribution of linking dynamics determines the average fitness of individuals [39]. Then, we can employ the strategy dynamics from well mixed populations for our structured system. The average payoff of cooperators is given by The average payoff of defectors is Equating f C and f D or, equivalently, substituting them into Eq. (12), we find that an unstable equilibrium x Ã C [ (0,1) emerges when It is located at This critical value x Ã C determines the attraction basin of cooperation (x Ã C ,1: Cooperators take over when their initial frequency x C (0) is larger than this critical value, whereas defectors take over when x C (0) is less than this critical value. In other words, the evolutionary PD game with linking dynamics is similar to that of the coordination game in well mixed population where both cooperation and defection are best replies to themselves.
Let us show how the PD game transforms into a coordination game under linking dynamics. Substituting Eqs. (13) (14) into Eq. (12) yields where the first factors are always positive and f 0 C~k CD k DD x C and f 0 D~( 1zu)k CC k DD x C zuk CC k CD x D are the payoffs of cooperators and defectors in a modified game with payoff matrix In other words, the coevolution of strategy and structure transforms the original PD game into another one. In particular, M 0 turns to a coordination game when M 0 11 wM 0 21 , i.e., uv(k CD {k CC )=k CC . Thus, the PD game with linking dynamics corresponds to a coordination game in a well mixed population [39,40]. Cooperation is stable only when where r~(k CD {k CC )=k CC . The quantity r measures the propensity for cooperators to form clusters that supports cooperation [16]. Indeed, remembering that k XY indicates the probability with which an XY link breaks, (k CD =k CC )~rz1 characterizes the fragility ratio between CD and CC link. In particular, CD link are more fragile than CC links if r exceeds zero. In other words, a cooperator is more likely to play with cooperators than defectors and to sustain the social relationship when r is greater than zero. Therefore r also illustrates how likely a cooperator is to interact with a cooperator. The greater r, the more likely it is for cooperators to form clusters.
Increasing r allows cooperators to spread more effectively and can allow them to invade from initially small clusters [52]. The quantity r characterizes the propensity of cooperators to form clusters. Cooperation gains a foothold when r is sufficiently large. Precisely, r is sufficiently large when uvr by Eq. (19). In this case, cooperator clusters expand and take over the whole population.
We have explained intuitively how r enhances cooperation level. In addition, we can also show analytically that large r leads to cooperation by enlarging the cooperation attraction basin: Substituting k CD~( rz1)k CC to Eq. (16), we obtain: The quantity d dr x Ã C (r)~{k CC k DD u(1zu)=((k CC {k CD )uz (k CC uzk DD )r) 2 is always negative for all permitted parameters. Hence, x Ã C (r) is a decreasing function of r. Since (x Ã C ,1 is the attraction basin of cooperation. Accordingly, increasing r enlarges the attraction basin of cooperation. In other words, it requires fewer cooperators to take over the whole population with larger r. So far, it has been shown that the simple rule gives us an insight on how cooperation comes into being with linking dynamics. Furthermore, it can also be revealed that CC links should be less fragile [39] while CD ones should be easy to break in order to promote cooperation. Since r~k CD =k CC {1, the larger k CD or the smaller k CC , the greater r is. Thus cooperation is promoted when the probability to break CD links k CD is large or the probability to break CC links k CC is small. This is in line with previous numerical consideration [37,38]. However, r is independent of k DD . Does this mean that k DD has no impact on cooperation? In fact, it is not the case. On the contrary, k DD plays an important role in promoting cooperation when uvr holds. Actually, this simple rule only guarantees that the equilibrium x Ã C of Eq. (16) lies between zero and one, where it is defined. However it is not sufficient to make cooperation advantageous. Besides, the initial frequency of cooperators should lie in the attraction basin of cooperation (x Ã C ,1 to make cooperators gain a foothold in the population. Nevertheless, notice that Eq. (16) can be rewritten as: hence, x Ã C is a decreasing function of k DD provided k CD {k CC (1zu)w0, i.e. the simple rule holds. In this way, increasing k DD augments the attraction basin of cooperation (x Ã C ,1 (See Fig. (2)). Thus it is easier for cooperators to gain a foothold when k DD is larger.
In Fig. (3), we show that the simulation results are in agreement with our analytical predictions when the selection pressure is high, while the simulations deviate from the analytical results when the selection pressure is low. For strong selection, we find above the line k CC~( 1=(1zu))k CD , the cooperation level is low, which is consistent with our theoretical predictions. For weak selection, however, the cooperation level is almost 100% for the parameter region closely above the line for k CC between 0 and 0:6, where the cooperation level should be low based on our the simple rule.
These deviations are due to both the finite population effect and the approximation of linking dynamics by Eq. (5). On the one hand, as mentioned above, the transition matrix Eq. (6) is only an approximation based on the global frequency of cooperators, while they are also influenced by local frequencies in the simulations. On the other hand, we use the replicator equation to describe the strategy evolution. But the replicator equation is only an approximation of the strategy evolution when the population size is sufficiently large, which implies that small fitness differences can influence the dynamics. This explains why our theoretical predictions are less accurate for weak selection. Therefore, we focus on strong selection in the following.
We first investigate how u affects the evolution of cooperation. For each plot in Fig. (4), above the line k CC~( 1=(1zu))k CD , there is nearly no cooperation, while below the line, cooperation is possible. This is consistent with our simple rule. Furthermore, compared with the three plots in Fig. (4), we observe a decrease of the parameter region to sustain the cooperation when u increases. It indicates that only a small temptation to defect can sustain cooperation.
Let us further examine the role of k DD in the evolution of cooperation by simulation. It is observed clearly in Fig. (5) that the more fragile DD ties are, the easier it is for cooperators to wipe out defectors. Intuitively, for greater k DD , DD links are more likely to break and defectors are no longer trapped in their fruitless interactions and can instead seek new cooperators to exploit. Thus, it seems less likely to promote cooperation for large k DD . However, both analytical and simulation results show that high k DD promotes cooperation (See Eq. (21) and Fig. (2)). This is counter-intuitive. In fact, in this case, the quick partner-switching  between defectors induces the heterogeneity of the population, which results in cooperation. Similar results have been reported in [43].
Finally, we turn to investigate the role of w on the coevolution. Fig. (6) shows that for small w, the result is in good agreement with the theoretical prediction, while deviates from the simple rule for large w, as expected. Similar results have been reported in the analytical approach of Pacheco et al. [39]. Both analytical approaches are based on the time scale separation, i.e., all the links are almost in the stationary states when the strategy update occurs.

Results and Discussion
To sum up, we have established a discrete model to describe the stochastic linking dynamics analytically in terms of a Markov chain. Based on this linking dynamics, we have studied the coevolution of strategy and network structure. A simple condition for the evolution of cooperation is obtained analytically that becomes more accurate when selection is stronger. The rule shows that the less fragile CC links are, the easier cooperation emerges. The more fragile CD links are, the easier cooperation prevails.
Compared to Pacheco et al.'s work, time scales separation also plays an important part in our analysis. In Pacheco et al.'s work, time separation is used to ensure that the linking dynamics is in the stationary regime when the strategy evolution happens. But in contrast to Pacheco et al.'s work, our analytical approach explicitly considers stochastic effects in the linking dynamics. Further, when the population size is sufficiently large, this Markov chain describing the linking dynamics can be approximated by a different system of differential equations. Since the total number of links is constant in our approach, there are only two independent  variables describing the different kinds of links. In Pacheco et al.'s method, however, all the three variables are independent. In general, both methods lead to very similar qualitative results.
Regarding the coevolution of strategy and network, previous numerical work with constant number of links has assumed that dissatisfied ties are more likely to break off than satisfied ones. In this case, cooperation is more likely to be sustained. However, it has not been shown analytically that to what extent satisfied links are more stable than adverse ones to make cooperation gain a foothold. The simple rule uv(k CD {k CC )=k CC reveals such a relation between the payoff matrix and the parameters of the linking dynamics. It shows under which conditions cooperation may prevail, provided the linking dynamics is sufficiently fast. Furthermore, we have provided a series of numerical results to validate the analytical results. We find that numerical results are in agreement with the analytical results for strong selection, yet may deviate from the analytical results for weak selection.