Benefits of asynchronous exclusion for the evolution of cooperation in stochastic evolutionary optional public goods games

Mechanisms and conditions for the spontaneous emergence of cooperation in multi-player social dilemma games remain an open question. This paper focuses on stochastic evolutionary optional public goods games with different exclusion strategies. We introduce four strategy types in the population, namely, cooperation, defection, loner and exclusion. Synchronous and asynchronous exclusion forms have been compared in finite-sized, well-mixed and structured populations. In addition, we verify that the asynchronous exclusion mechanism is indeed better than the synchronous exclusion mechanism in all cases. The benefits of the asynchronous exclusion are measured by comparing the probability that the system chooses the cooperative states in the two situations. In the well-mixed population cases, only when the investment amplification factor is small and the probability of exclusion success is high will the asynchronous exclusion mechanism have a relatively large advantage in promoting cooperation. However, in the structured population cases, the range of the investment amplification factor, in which the asynchronous exclusion mechanism has relatively large advantages in promoting cooperation, is somewhat different and is mainly in the middle of the interval under our parameters. Our study further corroborated that when non-participation and exclusion strategies exist, a structured population does not necessarily promote cooperation compared with a well-mixed population for some parameter combinations. Thus, we acquire a good understanding of the emergence of cooperation under different exclusion mechanisms.

If punishment or reward is used as a type of strategy for the individuals, then, the conditions under which cooperative strategies can be emerging through systematic self-organization can be explored. Presently, various forms of punishments or rewards have been proposed under the direct reciprocity mechanism 36,37,43,[47][48][49][50][51][52][53][54][55][56][57][58][59][60][61] . An example is introducing punishing strategies to punish cooperators and defectors 52,59 ; introducing the moralists who cooperate whilst punishing non-cooperative behaviours and immoralists who defect whilst punishing other non-cooperative behaviours 60 ; self-organised punishment that allows players to adapt their punishment depending on the frequency of cooperation 49 ; tolerance-based punishments in which individuals punish their co-players based on social tolerance 61 ; conditional punishments with fine depending on the number of punishers 47 ; implicated punishments that punish all individuals in the group once a evildoer is caught 56 ; probabilistic sharing the cost of punishing defectors 48 ; and heterogeneous punishments that group punishers based on their willingness to bear the punishing 57 . Perc et al. 10 presented a thoroughly review on punishment mechanisms in evolutionary games. Notably, in the real world, a few shortcomings exist for punishment in promoting cooperation 62 . Apart from the efficiency loss that may be caused by implementing this behaviour, the promotion effect may also be affected by other factors owing to the increase in the process. For instance, Nikiforakis et al. 63 corroborated that the form of punishment and its feedback in the public goods game will have an impact on group cooperation behaviour through behavioural experiments, and only appropriate punishment feedback can promote cooperation in the population.
In the traditional form of punishment, the punishing strategy must generally pay a cost, and individuals who are punished must simultaneously pay a larger penalty cost. The exclusion strategy, which has been proposed recently, can be regarded as a new form of punishing strategy 64 . Unlike the traditional assumption that defectors who are punished must pay a fixed penalty cost, it assumes that defectors will be expelled from the group by excluders with a certain probability. Moreover, the deported individuals cannot share the cooperative benefits of the group. This mechanism has received widespread attention since its introduction [65][66][67] . For instance, Li et al. 66 extended the evolutionary public goods game model with exclusion strategies to a finite size population for the first time. Further, Liu et al. 67 simultaneously introduced prosocial punishment and exclusion type strategies and studied competitions between them, and they affirmed that exclusion can outperform punishment when they coexist. What's more, Li et al. 68 proposed the concept of sequential exclusion, also called asynchronous exclusion, and they contended that asynchronous exclusion is a more effective mechanism than synchronous exclusion when three strategy types exist, namely, cooperation, defection and exclusion. Whether this conclusion remains valid when other strategy types or structured populations are introduced requires investigation. In addition, how to measure the advantage of the asynchronous exclusion mechanism and how the advantage is affected by the system parameters entail further studies. Accordingly, this study aims to answer such arguments.
This study introduces four strategy types used in evolutionary public goods games in two population types, namely, finite-sized, well-mixed and structured populations. In addition, synchronous and asynchronous exclusion forms have been considered. In the well-mixed interactive population, similar to literature 37,69 , the population state is described by a Markov process. Different from them, the competitive evolution between different strategies is introduced. Based on the same evolutionary dynamic model, the stochastic stable equilibria of the system in the two exclusion cases and the effects of parameters on the probabilities of the system choosing different equilibria are analyzed. Thus, the benefits of the asynchronous exclusion mechanism can be measured by comparing the probability that the system chooses the cooperative states in the two situations. In the structured population, we also compare the evolutionary stable state of the system under the two exclusion mechanisms. Hence, the effects of parameters on the frequency of strategies in the stable state in the two exclusion situations can be obtained. We provide the exact range of parameters, which makes the asynchronous exclusion mechanism relatively efficient. Ultimately, these results elucidate the emergence of cooperation under different exclusion mechanisms.

Results
Synchronous and asynchronous exclusion in optional public goods games. We introduce four types of individuals in the public goods game (PGG), namely, cooperation (denote as C), defection (denote as D), non-participation (denote as L) and exclusion (denote as E). Cooperation type individuals invest in the public goods and share the benefits of their investment income, whereas defection type individuals do not invest in the public goods but can take a free ride of the cooperators' investment income. Without the loss of generality, we let the investment cost equals one. Non-participation type individuals, who are typically called loners, do not take part in the game but can obtain a fixed income σ, whereas exclusion type individuals not only participate in the investment in the public goods game, but also exclude defectors in the group. The exclusion behaviour is costly, which will bring additional cost to the excluders. However, it can prevent free-riders from sharing their investment income. The defectors who are excluded from the group cannot share the benefit of the public goods. In addition, we assume that the exclusion behaviour cannot successfully exclude defectors with certainty but only with probability β. Let c E denote the unit exclusion cost for an excluder. Evidently, the higher the probability β, the greater the cost c E ; thus, we assume that c E is a function of β with β > ′ c ( ) 0 E and β ″ > c ( ) 0 E . Furthermore, this study considers two types of exclusion mechanisms. The first is synchronous exclusion, under which each individual of the exclusion type independently and simultaneously expels all defectors. The other is asynchronous exclusion, and the exclusion process is sequential, which means that once a defector is expelled by an excluder, the latter excluders no longer have to spend the exclusion cost for this defector. Other parameters in our model are as follows. M is the population size (number of individuals). r is the amplification coefficient of the N-person public goods game (1 < r < N). κ is a parameter to describe individuals' reaction speed to the environment in their decision.
Numerical experiments in a finite size and well-mixed population. In this situation, each individual interacts with others with equal probability. Each time, N individuals are randomly sampled from the population to participate in the PGG. We focus on the stochastic stable states of the evolutionary system and the www.nature.com/scientificreports www.nature.com/scientificreports/ corresponding probability of the system to choose each stable state. We aim to compare the differences in the probability of the system selecting each evolutionary steady state under the two different exclusion mechanisms.  Limit probability of the system selecting each stochastic stable state under any parameter combinations of (r, σ) and fixed β = 0.1 in the asynchronous exclusion. In this situation, a large parameter region D 1 exists, leading the system to select the cooperative state with a low probability. Moreover, a corresponding region D 2 exists, leading the system to select the ' All L' state with a high probability. Notably, a small parameter area D 3 also exists corresponding to large values of r and small values of σ, which leads the system to select the ' All D' state with a high probability.
When β is small, the probability difference of the system selecting the cooperative states under the two exclusion mechanisms is small. Moreover, the points with relatively large differences in probability are located in a small region that changes with the increase in β. Given that the maximal probability difference does not exceed 0.1 when β ≤ 0.3, the advantage of the asynchronous exclusion mechanism is not evident. As β increases, the maximal probability difference also increases, and the benefits of the asynchronous exclusion mechanism slowly emerge.
its type, which is equivalent to all individuals being loners. Therefore, states (0, 0, M, 0),(0, 1, M − 1, 0) and (0, 0, M − 1, 1) can be collectively referred to as the ' All L' state. In the following, we fix parameters M = 20, N = 5, κ = 1 and let c E = 0.2 * 10 β to study the effects of parameters β, r, σ on the probability of the system to select each stable equilibrium. We provide results for larger size populations (M = 50 and 100) in Supplementary Information. The probability that the system selects some stable states will change significantly when M increases to 100, but for the main conclusions we present, there is no essential difference between M = 100 and M = 20. Figure 1 shows the limit probability of the system selecting each stochastic stable state under any parameter combinations of (r, σ) and fixed β = 0.1 in the asynchronous exclusion. It depicts that when β is small, a large range of parameters (r, σ) (denote this region as D 1 ) exists, in which the system selects the cooperative state with a low probability. Moreover, a corresponding region (denote as D 2 ) exists, leading the system to select the ' All L' state with a high probability. Notably, a small parameter area (denote as D 3 ) also exists corresponding to the large values of r and small values of σ, which leads the system to select the ' All D' state with a high probability. Notably, we notice that when β slowly increases, all three regions shrink rapidly. Moreover, the probability of the system selecting the ' All D' state also drop rapidly when parameters (r, σ) are in the D 3 region. More details can be found in Supplementary Information. The results are somewhat similar in the synchronous exclusion situation. However, the probability of the system reaching the cooperative state is lower than that of the asynchronous exclusion mechanism under the same combination of parameters. This finding further illustrates that the asynchronous exclusion mechanism works better for the evolution of cooperation. We use the probability difference to represent the benefit of asynchronous exclusion in promoting cooperation. Figure 2 shows the relationship between the probability difference and parameters (r, σ) when β is fixed at six different values, and it elucidates that when β is small, the probability difference of the system selecting the cooperative states under the two exclusion mechanisms is small. Moreover, the points with relatively large differences in probability are located in a small region D 4 (D 4 changes with the increase in β). Given that the maximal probability difference does not exceed 0.1 when β ≤ 0.3, the advantage of the asynchronous exclusion mechanism is not evident. As β increases, the maximal probability difference also increases, and the benefits of the asynchronous exclusion mechanism slowly emerge. Further, the points with relatively large differences in probability are located in an area with small values of r and small values of σ. To clearly compare the differences between the two exclusion situations, we fix the parameter σ = 0.1 to show the corresponding results. Figures 3 and 4 show the relationship between the limit probability of the system selecting each type of stable states and parameters r and β, respectively, for fixed σ = 0.1 under the two exclusion mechanisms. Only when r is small and β is large will the asynchronous exclusion mechanism have a relatively large advantage in promoting cooperation which is consistent with our theoretical The values of β in the first three subgraphs are relatively small, and the advantages of the asynchronous exclusion mechanism are not evident. The values of β in the latter three subgraphs are relatively large, and when r is small, the benefits of the asynchronous exclusion mechanism emerge. It verifies that only when r is small and β is large will the asynchronous exclusion mechanism have a relatively large advantage in promoting cooperation.
www.nature.com/scientificreports www.nature.com/scientificreports/ analysis. In fact, when the number of excluders in the population is l, the expected unit exclusion costs in the asynchronous situation c R and in the synchronous situation c E satisfy the following equation: . The difference between c R and c E increases as β increases. Moreover, when β → 0 + , no difference exists between the two costs.
Simulation results in a structured population. In this situation, each individual is located on a node of the square lattice, and plays the PGG with its four direct neighbors. Thus, each individual participates in five rounds of games to accumulate payoff. Exclusion individual E will exclude their adjacent defectors with certain probability β for each, whilst paying β-related unit cost c E . Defectors who are expelled from the group cannot obtain investment income in the corresponding single-round game. Moreover, we consider synchronous and asynchronous exclusion situations. Figure 5 shows the frequencies of all strategy types after the system reaches evolutionary stability in the two exclusion situations under two values of β = 0.1 and 0.8, respectively, when r changes from 1.5 to 4.5. For each simulation, the system iterates 10,000 to 50,000 rounds (10,000 times for a round), ensuring that the system has reached an evolutionary stable state. We take the average of the last 1000 rounds and obtain all the simulation results by the average results of 20 independent simulation experiments.
The simulation results also verify that the asynchronous exclusion works better than synchronous exclusion for promoting cooperation in the structured population. When β is small (β = 0.1), the advantage of the asynchronous exclusion is not evident. In this situation, a common r interval (3.1, 3.7) emerges in the two situations, so that the D strategy can emerge and the frequency of D peaks when r = 3.5 after the system becomes stable. Conversely, when β is large (β = 0.8), the asynchronous exclusion mechanism has a significant advantage mainly in the middle interval of r, the range of which is different from that in the well-mixed population. Two different r intervals (2.2, 2.9) and (2.5, 3.1) exist, which correspond to the asynchronous and synchronous exclusions, respectively, so that the D strategy can emerge. To observe the evolution of the spatial distribution of strategies under the two exclusion mechanisms, we choose four parameter combinations for comparison. Figures 6-9 show the spatiotemporal distribution of the four strategies in the PGG at different Monte Carlo steps (MCS) in the two exclusion situations.

Discussion
We verified that the asynchronous exclusion mechanism is indeed better than the synchronous exclusion mechanism for promoting cooperation in the well-mixed and structured populations. The benefits of the asynchronous exclusion are measured by comparing the probability that the system chooses the cooperative states in the two situations. In the well-mixed population cases, only when the investment amplification factor is small and the probability of exclusion success is high will the asynchronous exclusion mechanism have a relatively large advantage www.nature.com/scientificreports www.nature.com/scientificreports/ in promoting cooperation. However, in the structured population cases, the range of the investment amplification factor, in which the asynchronous exclusion mechanism has relatively large advantages in promoting cooperation, is somewhat different and is mainly in the middle of the interval under our parameters.
The three mechanisms of population structure, voluntary participation and exclusion can promote the evolution of cooperation. However, when these mechanisms exist simultaneously, we corroborate that within our parameters, an interval of r emerges, in which a large proportion of individuals choose defection after the system (b) β = 0.8. When β = 0.1, the advantage of the asynchronous exclusion is not evident, and a common r interval (3.1, 3.7) exists in the two exclusion situations, so that the D strategy can emerge and the frequency of D peaks when r = 3.5 after the system becomes stable. When β = 0.8, the asynchronous exclusion mechanism has a significant advantage mainly in the middle interval of r. Two different r intervals (2.2, 2.9) and (2.5, 3.1) exist, which correspond to the asynchronous and synchronous exclusions, respectively, so that the D strategy can emerge. www.nature.com/scientificreports www.nature.com/scientificreports/ becomes stable. Figure 5-(b) illustrates that when β = 0.8, the defection frequency reaches a peak value of approximately 0.4 at around r = 2.7 in the asynchronous exclusion situation; and reaches a peak value of roughly 0.8 at around r = 3.0 in the synchronous exclusion situation. The figure further depicts that in the well-mixed population under the same parameters, the frequency of defection is nearly zero. Thus, when non-participation and exclusion strategies exist, the population structure does not necessarily promote cooperation compared with the well-mixed population for some parameter combinations. These results can further enrich the existing conclusions on the voluntary and exclusion mechanisms for the evolution of cooperation. When the population has a heterogeneous network structure, the relevant conclusions need further verification, and we will explore this issue in the subsequent research.

Methods
Evolutionary dynamics in a finite size and well-mixed population. Suppose a finite size population consisting M individuals. Let variables X, Y, Z and W denote the numbers of cooperators, defectors, loners and excluders in the population, respectively. Each time step, N individuals are sampled randomly from the population to play the PGG. Let variables i, j, k and l denote the number of cooperators, defectors, loners and excluders, respectively, in a sampled group.
In the synchronous exclusion situation, the expected payoffs of the cooperation, defection, loner and exclusion type strategies are as follows. The details of analysis can refer to Supplementary Information.  In the synchronous exclusion situation, the system reaches the state of L, whereas in the asynchronous exclusion situation, the system reaches the stable state of C + D + L, that is, the three strategies coexist.
www.nature.com/scientificreports www.nature.com/scientificreports/  where In the asynchronous exclusion situation, the expected payoffs of cooperators, defectors and loners remain the same, but the expected payoff of exclusion-type strategy becomes When X, Y, Z and W take zero, respectively, the corresponding π C (2019) 9:8208 | https://doi.org/10.1038/s41598-019-44725-y www.nature.com/scientificreports www.nature.com/scientificreports/ on the network. Let π s i denote the cumulative payoff of individual i (with strategy s i ) participating in five rounds of the PGG. The system evolves according to the following rules. Each time, an individual i is randomly selected. Then, individual i chooses one of its neighbors at random. The chosen individual j imitates the strategy of individual i with probability → = π π τ + − p s s ( ) . When τ → + ∞, whether individual j imitates the strategy of individual i is completely random. In our simulation, we fix parameter τ to 0.1 and repeat the above process 10000 times as a round of iterations to ensure that each node has an opportunity to adjust its strategy on average in one iteration. The system can be stabilized after a sufficiently large number of iterations.