Evolution of cooperation in an n-player game with opting out

How cooperation could have evolved has been one of the central topics in evolutionary biology. When cooperators are likely to interact with other cooperators, and defectors are likely to interact with other defectors, positive assortment is created, facilitating the evolution of cooperation. Cooperation is observed not only in dyadic interactions but also, sometimes, in sizable groups. Previous studies have found that the opting out rule in which the group is disbanded if and only if the group is heterogeneous, especially facilitates the evolution of cooperation compared to the other opting out rules in multi-player games when the number of rounds is sufficiently large. However, the dynamics between the cooperators and defectors under such an opting out rule have been investigated only in the case where group size is relatively small (e.g., four). In addition, the effect of group size on the evolution of cooperation has not been explored, and considering that humans interact within larger groups, investigation in such large groups is essential. Here, through further algebraic analyses, it is newly revealed that there can be four internal equilibria when the group size is larger than four. In addition, while the effect of group size on cooperation is negative in the case of common goods, it is not straightforward (i.e., can be positive) in the case of public goods.

was predicted that the mean group size in humans would be of approximately 150 individuals (Dunbar, 1993). Křivan & Cressman (2020) have examined which opting out rule most facilitates the evolution of cooperation, not in a pair, but in a group. Let "the best opting out rule" denote an opting out rule which most facilitates the evolution of cooperation among a variety of opting out rules. The results revealed that the opting out rule in which the group is disbanded if and only if the group is heterogeneous, is the best opting out rule in multi-player games, when the number of rounds is sufficiently large.
However, at least two aspects were not examined in the above-mentioned study and they seem worthwhile investigating. Firstly, the analysis of the dynamics under the opting out rule was not conducted for a large group. Specifically, the dynamics were examined in the case where group size was four, revealing that there are at most two internal equilibria. Dunbar stated that the mean group size is approximately 60 individuals in apes. Further, in humans, a mean group size of approximately 150 individuals was estimated by Dunbar. It is unclear whether the dynamics when group size is four and those when it is larger than four (e.g., approximately 60 or 150) are the same. In the present study, analyses were conducted to reveal the dynamics in the case where the group size is >4. Secondly, Křivan & Cressman (2020) did not explore the effect of group size on the evolution of cooperation. Examining the effect of group size is significant in the field of evolutionary anthropology. The group size effect can depend on whether goods are rivalrous (i.e., common goods (e.g., fish stocks, open ocean fishing, and water)) or non-rivalrous (i.e., public goods (e.g., public information, national defense, and knowledge)); therefore, the group size effect in the case of common goods and that in the case of public goods should be examined, respectively.
The contents below are structured as follows: Section 2 describes a mathematical model of the evolution of cooperation in n-player games; Section 3 summarizes the results, illustrating the cooperation dynamics, exploring how the parameters affect them and the positions of internal equilibria; and Section 4 is devoted to the discussion.

Model assumption 2.1. Game
In the n-player Prisoner's Dilemma (PD) game, each of the n players cooperates or defects where ≥ 2.
(1) Table 1 summarizes what symbols mean. On one hand, when an individual cooperates, he/she pays cost c and contributes to a public good. In the present study, we dealt with non-excludable goods. Therefore, every n-player including him/her in the group receives an equal benefit . On the other hand, when an individual defects, he/she gives nothing. Let us consider the case where the number of cooperating individuals is (i.e., the number of defecting individuals is ). This assumption naturally leads to the fact that the payoffs to cooperating (C) and defecting (D) individuals are given by and , respectively (Boyd & Richerson, 1988). From the definition of the n-player PD game, the following inequality must be satisfied: J o u r n a l P r e -p r o o f .
(2) Rivalry plays a crucial role in the analysis of size effects (Taylor, 1987). On one hand, a benefit can be regarded as rivalrous if the usage of a benefit by an individual reduces its availability to others, and in the above model for a rivalrous good, does not change with . On the other hand, a benefit can be regarded as non-rivalrous if the usage of a benefit by an individual does not reduce its availability to others, and in the above model for a non-rivalrous good, does not change with ; thus linearly increases as group size ( ) increases. A common good is defined as a rivalrous non-excludable good, and a public good is defined as a non-rivalrous non-excludable good (Taylor, 1987).

Strategy
Two strategies are here introduced: the player adopting strategy C always cooperates, and the player adopting strategy D always defects. Correspondingly, the game is played by C and D players.
denotes the frequency of strategy in a population. Here, because an individual is either C or D, it follows the following equation:

Group formation
The population size is and is infinitely large ( ≫ ). In the first period, individuals are randomly selected and matched, and individuals in a group play the n-player PD game. Here, is a multiple of and not one group but groups are considered in this game. In later periods, the group composition depends on the outcomes of the previous period. It is assumed that a probability that a group breaks up in a round is determined by the number of cooperators in the group.
, where ≤ ≤ , denotes the probability that the interaction stops in the next round when of players are cooperators (i.e., of players are defectors). It is assumed that is either 1 or , where < < 1.
(4) The opting out rule can be expressed as { 0 , 1 , 2 , 3 , … , −1 , }. The number of opting out rules is 2 +1 . Křivan & Cressman (2020) found that when is sufficiently small, the best opting out rule, which most facilitates the evolution of cooperation among the 2 +1 opting out rules is: Under (5), when both cooperators and defectors are present in the group (i.e., the group is heterogeneous), the group is disbanded with probability 1. In contrast, when the group members are all cooperators or defectors (i.e., the group is homogeneous), the interaction stops with probability . The present study investigates the dynamics under this opting out rule (i.e., (5)). Individuals who did not terminate the interaction with partners in the previous period played the n-player PD game again with the same opponent players in the next period. In contrast, an individual who stopped the interaction in the previous period was paired with 1 individuals (who had been randomly selected from similar individuals who stopped interacting with opponents in a previous game). These players were thus paired for the n-player PD game. It was assumed that an individual who stopped interacting with partners could find the next opponents instantly without incurring a searching cost.
denotes the frequencies of a group in which cooperators and defectors are present at round . From the above assumption, the following equation can be derived The frequency of + 1 in a round for which a group was not dissolved in the previous round is given by (1 ). The ratio of individuals whose group was dissolved in the previous round to the whole population is ∑ =0 . The ratio of cooperators whose groups were dissolved to all the individuals whose groups were dissolved in the previous round is ( ). Because the matching process was randomly conducted, the group members' distribution is a binomial distribution. Using these equations and on the basis of the aforementioned discussion, it is revealed that the frequency of + 1 in a round for which a group was dissolved in the previous round is given by ∑ =0 Hence, + 1 is given by Here, Δ is introduced as can be interpreted as the changes of group frequencies between rounds. Based on (7) and (8), the following equation can be obtained: (9) It is assumed that such a group formation repeats infinite times. Under this condition, it is algebraically shown that there is just one equilibrium in the sense of group formation (see supplementary material A for proof). ̅̅̅̅ denotes the group frequencies at an equilibrium. It is numerically indicated that, after infinite repetitions, the group formation approaches the unique equilibrium (see supplementary material B). More specifically, we have lim →∞ = ̅̅̅̅ (10) From (8) and (10), the following equation can be obtained: lim →∞ Δ = . (11)

Dynamics
It is assumed that the changes in the strategies' frequencies are much slower than group formation. Therefore, each time there is a change in the strategies' frequencies, the group formation is at equilibrium. denotes the expected payoff of strategy from one round, while ̅ denotes the expected payoff of strategy from one round when the group formation reaches an equilibrium. Here, it is also assumed that the changes in the strategies' frequencies in the population can be described by the following replicator equations (Taylor & Jonker, 1978;Hofbauer & Sigmund, 1998): | denotes the probability that strategy belongs to a group in which there are cooperators and defectors. Here, using | , it is possible to describe and as: | ̅̅̅̅̅ denotes the probability that strategy belongs to a group in which there are cooperators and defectors when the group formation reaches an equilibrium. Here, using | ̅̅̅̅̅, it is possible to describe ̅̅̅ and ̅̅̅ as: The dynamics between strategies C and D are examined in the present study. By investigating the evolutionary dynamics, we will explore the evolution of cooperation in this paper.

Results
The subsections presented below examine the following aspects: 1) how influences the strategies' dynamics (e.g., when defectors dominate cooperators and when not) and the positions of internal equilibria (subsection 3.1); 2) how influences the strategies' dynamics (e.g., when defectors dominate cooperators and when not) and the positions of internal equilibria (subsection 3.2); 3) how group size influences the condition under which defectors dominate cooperators and the positions of internal equilibria when is constant (i.e., common good) (subsection 3.3); and 4) how group size influences the condition under which defectors dominate cooperators and the positions of internal equilibria when is constant (i.e., public good) (subsection 3.4).

Effect of
on the evolution of cooperation 3.1.1. The condition under which payoff to cooperators is larger than that to defectors. After algebraic calculations (see Supplementary material C for proof), the condition under which cooperators obtain a higher payoff than the defectors do is given by: J o u r n a l P r e -p r o o f Here, after algebra (see supplementary material C for proof), it turns out that there exists a unique so that: = denotes that satisfies (19). From (17), (18), and (19), it turns out that once , , , and are determined, it is determined which gets higher payoffs, cooperators or defectors. Figure 1 shows the relationship between and the critical value of . For a given , when and only when the actual value of is larger than the critical value of , the payoff to cooperators is larger than that to defectors. There are two kinds of shapes; one kind of shape is unimodal, and the other kind of shape is bimodal (see figure 1). In figure 1, it is observed that the figures in the case where is small and (or) is small are unimodal, while the figures in the case where is large and (or) is large are bimodal. Actually, algebraic calculation proves this observation. More specifically, it turns out that the figures with ( + 2 + 2 −1 4 ) ≤ + 2 are unimodal and the figures with ( + 2 + 2 −1 4 ) + 2 are bimodal (see supplementary material D for proof).

The effect of parameters on the number of internal equilibria
When ( + 2 + 2 −1 4 ) + 2 , there exists just one such that 1 = , which is presented in supplementary material E in the range of < < 1 2 (see supplementary material D for proof that such is unique). By 1 , we denote such . By the definition of 1 , we have < 1 < 1 2 .
(20) The condition under which there are no internal equilibria is given by The condition under which there are two internal equilibria is given by The condition under which there are four internal equilibria is given by and (24) (24) means that when ≤ 4, the number of internal equilibria cannot be 4.
From above, it turns out that the condition under which cooperators are not dominated by defectors (note that in such a case, there exist at least two internal equilibria) is given by , and affect the number of internal equilibria. We can confirm that when = 4, the number of internal equilibria cannot be 4.

The effect of parameters on the positions of internal equilibria
In the previous subsection (3.1.2.), we investigated the number of internal equilibria. In the current subsection (3.1.3.), we will investigate the positions of internal equilibria.
Additionally, we investigate what kind of bifurcation happens when the number of internal equilibria changes.
As shown in figure 2, when < 1 + , a blue-sky bifurcation occurs, and the stable and unstable equilibria emerge (see figure 3). The frequencies of cooperators at two internal equilibria are _ and _ℎ , respectively. Here, we have _ℎ = 1 _ .
(27) In figure 3, it is observed that as increases, _ (i.e., the frequency of cooperators at the unstable equilibrium) decreases, and _ℎ (i.e., that at the stable equilibrium) increases.
(28) As shown in figure 2, when < ℎ 1 , defectors dominate cooperators. When , there are just four internal equilibria, at which the frequencies of cooperators are _ , _ , _ℎ , and _ℎ , respectively. Here, we have . We pick up = .55 as a parameter satisfying this (figure 3). When = ℎ 1 , two stable and two unstable equilibria emerge (blue-sky bifurcation) (see figure 3). When = 1 + 2 −1 2 + 2 −2 , the lower stable equilibrium, at which the frequency of cooperators is _ , and the higher unstable equilibrium, at which the frequency of cooperators is _ℎ , coalesce (fold bifurcation) (see figure 3). In figure 3, it is observed that as increases, _ and _ℎ (i.e., the frequencies of cooperators at the two unstable equilibria) decrease, and _ and _ℎ (i.e., those at the two stable equilibria) increase.
No matter whether + 2 + 2 −1 4 ≤ + 2 or ( + 2 + J o u r n a l P r e -p r o o f Figure 3, it is observed that as increases, _ decreases and _ approaches 0 as approaches . Actually, it is shown (see supplementary material F for proof) that In addition, after algebraic calculation (see supplementary material F for proof), we can derive lim → _ = .
(33) Figure 4 shows the relationship between and the critical value of . For a given , when and only when the actual value of is smaller than the critical value of , the payoff to cooperators is larger than that to defectors. There are two kinds of shapes; one kind of shape is unimodal, and the other kind of shape is bimodal (see figure 4). In figure 4, it is observed that the figures in the case where is small and (or) is small are unimodal, while the figures in the case where is large and (or) is large are bimodal. Actually, algebraic calculation proves this observation. More specifically, it turns out that the figures with (1 2 ) ≤ 2 are unimodal and the figures with

Effect of on the evolution of cooperation 3.2.1. The condition under which payoff to cooperators is larger than that to defectors.
(1 2 ) 2 are bimodal.

The effect of parameters on the number of internal equilibria
By 2 , we denote such that = ℎ 1 . The condition under which there are no internal equilibria is given by The condition under which there are two internal equilibria is given by J o u r n a l P r e -p r o o f The condition under which there are four internal equilibria is given by and ( 1  2 ) 2.
(37) From above, it turns out that the condition under which cooperators are not dominated by defectors (note that in such a case, there exist at least two internal equilibria) is given by < ,

The effect of parameters on the positions of internal equilibria
In the previous subsection (3.2.2.), the number of internal equilibria was investigated. In the current subsection (3.2.3.), we will investigate the positions of internal equilibria. In addition, we investigate what kind of bifurcation alters the number of internal equilibria. Firstly, we investigate the case of (1 2 ) ≤ 2. . We pick up = 2.5 as a parameter satisfying this (figure 5). In figure 5, it is observed that when reaches the critical value (i.e., = , the stable and unstable internal equilibria coalesce (fold bifurcation). As increases, the frequency of cooperators at the unstable internal equilibrium ( _ ) increases, while that at the stable internal equilibrium ( _ℎ ) decreases. Thus, the impact of on the evolution of cooperation is negative. Secondly, we investigate the case of (1 2 ) 2. , two additional internal equilibria emerge (blue-sky bifurcation), one unstable and the other stable. When reaches 2 , the lower unstable and stable internal equilibria, and the higher unstable and stable internal equilibria coalesce (fold bifurcation), respectively. As increases, the frequency of cooperators at the lower unstable equilibrium ( _ ) and the frequency of cooperators at the higher unstable equilibrium ( _ℎ ) increase, while that at the lower stable equilibrium ( _ ) and that at the higher stable equilibrium ( _ℎ ) decrease. Thus, the impact of on the evolution of cooperation is negative. No matter whether (1 2 ) ≤ 2 or (1 2 ) 2, in Figure 5, it is observed that as approaches 0, _ approaches 0. Actually, after algebraic calculation (see supplementary material G), we have lim →0 _ = .
1 ( 2 ) is defined as a frequency of cooperators at which the lower (higher) unstable equilibrium and the lower (higher) stable equilibrium coalesce. Here, we have 2 = 1 1 .
(45) = 5 is chosen (figure 5). In Figure 5, it is observed that as approaches 1, 1 approaches a value, which is dependent on the group size ( ). It can be proved (see supplementary material H) that for satisfying ≥ 5, in the range of < < 1 2 , there is just one satisfying Here, the unique is denoted by . After algebraic calculation (see supplementary material I), we have lim →1 1 = , when ≥ 5.

J o u r n a l P r e -p r o o f
This subsection explores the effect of group size ( ) on the evolution of cooperation when is constant (i.e., common good).

From the perspective of the condition under which cooperators are not dominated by defectors
It is shown that the condition under which cooperators are not dominated by defectors is given by < , ,  Figure 6 (a), which is derived from (49) and (50), illustrates when defectors dominate cooperators and when they do not, showing that , decreases as increases. Figure 6 (a) also shows that the critical value of is smaller when is larger. Actually, it has been algebraically shown that lim →0 , = 1, lim →0 1− , = −1 2 −1 .
As increases, the right-hand side of (52), which is −1 2 −1 , increases; therefore, the left-hand side of (52), which is lim →0 1− , , increases. Hence, when is around 0, , decreases, and consequently, as increases, the condition under which cooperators are not dominated by defectors becomes more strict.
In addition, it has been algebraically (see supplementary material J for proof) shown that lim →1 , = 1 .
As increases, the right-hand side of (53), which is 1 decreases; therefore, the left-hand side of (53), which is lim →1 , , decreases. Hence, when is around 1, , decreases, and consequently, as increases, the condition under which cooperators are not dominated by defectors becomes more strict.
Thus, as the group size increases, the condition under which cooperators are not dominated by defectors becomes more strict. In other words, the evolution of cooperation becomes more difficult as group size increases.
In Figure 6 (a), it is expected that when the group size is very large, defectors dominate cooperators and there are no internal equilibria, irrespective of the values of and . Actually, it is possible to algebraically (see supplementary material K for proof) show that the following limiting values of , exist: lim →∞ , = .

From the perspective of the positions of internal equilibria
As observed in Figure 6 (a) and suggested by algebraic calculation, the impact of on whether cooperators are not dominated by defectors is monotonic. As increases, the condition under which cooperators are not dominated by defectors becomes more strict. When 1 1+ , defectors dominate cooperators, irrespective of the group size. When is large is above all the critical lines, depending on . In such cases, defectors dominate cooperators, irrespective of the group size. Figure 6 (a) indicates that the point corresponding to the case where is small and (or) is small is below some of the critical lines, depending on . In such cases, cooperators are not dominated by defectors when the group size is smaller than the critical value.
For some combinations of and , as the group size increases, the number of internal equilibria turns from 2 to 0. Figure 6(b) demonstrates that when is smaller than the critical value, there are two internal equilibria, one stable and the other unstable. As increases, the frequency of cooperators at the unstable equilibrium _ increases, while that at the stable equilibrium _ℎ decreases. Therefore, as the group size increases, the basin of attractions for the population consisting of only defectors (i.e., < < _ ) grows, and the frequency of cooperators in the case where cooperation is established _ℎ decreases. When the group size reaches a critical value, the unstable and stable equilibria coalesce (fold bifurcation). When is larger than the critical value, cooperators are dominated by defectors, and there are no internal equilibria. Thus, the effect of the group size on the evolution of cooperation is negative.
For other combinations of and , as the group size increases, the number of internal equilibria turns from 2 to 4, and then from 4 to 0. Figure 6(c) demonstrates that there are two critical values for , which are specifically denoted by 1 and 2 (where 1 < 2 ). When is smaller than 1 , there are two internal equilibria, one unstable and the other stable. It is also observed that the frequency of cooperators at the unstable internal equilibrium _ increases as increases, whereas the frequency of cooperators at the stable internal equilibrium _ℎ decreases as increases. When reaches 1 , two additional internal equilibria emerge (blue-sky bifurcation), one unstable and the other stable. For 1 < < 2 , there are four internal equilibria. As increases, the frequency of cooperators at the lower unstable equilibrium _ increases, and that at the lower stable equilibrium _ decreases. As increases, the frequency of cooperators at the higher unstable equilibrium _ℎ increases and that at the higher stable equilibrium _ℎ decreases. When reaches 2 , the two lower (unstable and stable) internal equilibria, and the two higher (unstable and stable) internal equilibria coalesce (fold bifurcation), respectively. When is larger than 2 , cooperators are dominated by defectors, and there are no internal equilibria. Thus, the impact of on the evolution of cooperation is negative. As observed in the above two cases, the frequency of cooperators at the unstable internal equilibrium _ increases as increases. Based on (43), it is J o u r n a l P r e -p r o o f revealed that when is around 0, the following relationship holds: which means that for larger , _ is larger, when is around 0.

Group size effects on cooperation when
is constant (i.e., public good) This subsection examines the effect of group size ( ) on the evolution of cooperation when is constant (i.e., public good).

From the perspective of the condition under which cooperators are not dominated by defectors
From (49), the condition under which cooperators are not dominated by defectors is given by , , Figure 7 (a), which is derived from (56) and (57) 5 .
(61) Hence, the evolution of cooperation is most likely when = 2 or = 3, while it is most unlikely when = 5. Thus, the impact of group size on the condition under which cooperators are not dominated by defectors is not monotonic.
In Figure 7 (  (63) In addition, after algebraic calculations (see supplementary material L for proof), when is constant, the following equation is obtained: Thus, when the group size is sufficiently large, defectors do not dominate cooperators in case of public goods (i.e., is constant.), if conditions are satisfied. This result means that the evolution of group-wise cooperation is possible if goods are public goods. This result is in stark contrast with the case of common goods.

From the perspective of the positions of internal equilibria
As observed in Figure 7 (a) and confirmed by algebraic calculation, the impact of on whether cooperators are not dominated by defectors is not monotonic. Besides, (56) means that when is smaller, cooperators are more likely dominated by defectors. In addition, (56) means that when is larger, cooperators are more likely dominated by defectors. Based on the range for group sizes for which cooperators are not dominated by defectors, the combinations for and can be classified into the three scenarios. Figure 7 (a) indicates that the point corresponding to the case where is small and (or) is large is below all the critical lines, depending on (i.e., the group size). This means that when is small and (or) is large, cooperators are dominated by defectors and there are no internal equilibria, irrespective of the group size. Figure 7 (a) indicates that a point, corresponding to a combination for and , is below a critical line corresponding to a group size and upper a critical line corresponding to another group size. For a combination for and , no internal equilibria exist and cooperators are dominated by defectors for a group size, whereas there exists internal equilibria and cooperators are not dominated by defectors for another group size. Figure 7(b)(c) illustrates how the positions of internal equilibria are influenced by . Actually, in Figure 7(b), this can be confirmed. In figure 7(b), it is observed that for some group sizes ( = 2 or 6 ≤ ≤ 12), there are no internal equilibria (and defectors dominate cooperators). For some group sizes ( = 3,4,5), there are two internal equilibria. For other group sizes (13 ≤ ≤ 2 ), there are four internal equilibria. Figure 7 (a) indicates that the point corresponding to the case where is large and (or) is small is upper all the critical lines, depending on (i.e., the group size). This means that when is large and (or) is small, cooperators are not dominated by defectors, and there are internal equilibria, irrespective of the group size. In Figure 7(c), it is observed that when group size is smaller than the critical value, there are two internal equilibria, while when it is at the critical value, a blue-sky bifurcation occurs and two additional internal equilibria emerge. Thus, when group size is larger than the critical value, there are four internal equilibria based on the frequency of cooperators achieved, as follows: unstable (lowest frequency), stable (second-lowest), unstable (third-lowest), and stable (highest).
In Figure 7(b)(c), it is observed that when group size increases, the frequencies of cooperators at the lower unstable and stable equilibria ( _ and _ , respectively) converge to zero, and the frequencies of cooperators at the higher unstable and stable equilibria ( _ℎ and _ℎ , respectively) converge to one. Figure 7(d) illustrates how the positions of internal equilibria and 1 are influenced by . In Figure 7(d), it is observed that as increases, 1 approaches 0. Actually, after algebraic calculation (see supplementary material M), we can derive lim →∞ _ = , (65) lim →∞ _ = , (66) lim →∞ 1 = .
(67) How do _ , _ , and 1 converge to zero? When in the range of , there exist exactly two so that denotes a smaller satisfying lim →∞ _ = + 1 1 .
(71) In Figure 7(b)(c), it is observed that the impact of group size on _ is not monotonic (see Appendix B for algebraic analysis supporting this non-monotonicity). Here, the basin of attraction for the population consisting of only defectors is < < _ . From the non-monotonicity, the group size impact on the basin of attraction for the population consisting of only defectors is not monotonic in the case of public goods. Křivan & Cressman (2020) have investigated the evolutionary dynamics of cooperation. Křivan & Cressman (2020) have shown that for sufficiently large number of rounds, the best opting out rule is the one in which the heterogeneous groups are disbanded and the homogeneous groups are not disbanded. The present study investigated the dynamics between the cooperators and defectors under the opting out rule, in more detail. While Křivan & Cressman (2020) studied the finite population, this paper studied the infinite population. In this model setting of infinite population, not only numerical computation but also algebraic analysis is possible. Therefore, we can discuss the case where the parameter values including the group size are general. Owing to this, we noticed that while there are at most two internal equilibria when the group size is equal to or smaller than four, there exists a combination for and that allows the emergence of four internal equilibria when the group size is larger than four. Thus, the cases in which group size is ≤ 4 or 4 are qualitatively different. Dunbar (1993) stated a mean group size in apes of approximately 60 individuals and predicted a mean group size in humans of approximately 150 individuals. As confirmed in figure 2 (d) (e) (see also (22)), although there exists a combination for and that allows the emergence of two internal equilibria, such regions are very tiny (almost invisible), when the group size is 60 or 150. Thus, the cases in which group size is ≤ 4 or = 6 15 are very different. In the present study, it was revealed that the evolution of cooperation is less likely as the group size increases (i.e., the group size effect is negative) when goods are common, while the impact of the group size on the evolution of cooperation is not straightforward (i.e., can be positive) in the case of public goods.

Discussion
This study examined the dynamics in an n-player game played by cooperators and defectors. The results showed that there are stable coexistences between cooperators and defectors when the cost-to-benefit ratio is sufficiently small and (or) the number of interactions is large enough; however, even in such cases, the population consisting of only defectors is stable against invasions by rare cooperators. Thus, opting out allows cooperation to be maintained once it is established; however, the rule does not facilitate the initial evolution of cooperation, which requires another mechanism. Křivan and Cressman (2020) revealed that the best opting out rule is the one in which the groups voluntarily stay together between rounds if and only if the groups are homogeneous (i.e., either all cooperators or all defectors) when the number of rounds is sufficient, while the best opting out rule is not so when the number of rounds is insufficient. The present study investigated the dynamics between cooperators and defectors in the above-mentioned opting out rule. However, this rule is the best only when the number of rounds is sufficient, and when this is not the case, another opting out rule, in which heterogeneous groups do not dissolve and the players belonging to such groups continue to interact, can be the best one (Křivan & Cressman, 2020). Moreover, the threshold of the number of rounds tends to increase as the group size increases, at least when the group size is small (two to six) (Křivan & Cressman, 2020). If this tendency is present with a larger group size, it may seem unlikely that this opting out rule would remain the best one when the size is approximately 60 or 150, which is the mean group size among apes or humans, respectively. Further research should be conducted to determine the best opting out rule when group size is large.

Appendices Appendix A Proof for (61)
Based on (59)  Based on (A.11), it is revealed that when is very large, the following relationship holds: ≈ −1 (1 1 2 ), (A.12) which indicates that increases as increases when is very large. Numerical computation (see supplementary material O) also suggests that for ≥ 7, the following relationship holds: + 1 .

Appendix B The impact of group size on
_ Actually, after algebraic calculation, we can find both positive impacts and negative impacts. Firstly, we introduce positive impacts. Based on (43), it is revealed that when is around 0, the following relationship holds:  The horizontal axis is , and the vertical axis is . The lines represent the thresholds of above which the payoff to cooperators is larger than that to defectors. The parameter values used are = .1 in the five left panels, = .5 in the five central panels, and = .9 in the five right panels; and = 4, = 5, = 7, = 5 , and = 15 in rows one, two, three, four, and five, respectively. Figure 1 is derived from (17), (18), and (19).    The horizontal axis is , and the vertical axis is . The lines represent the thresholds of below which the payoff to cooperators is larger than that to defectors. The parameter values used are = 2.9 in the three left panels, = 3.3 in the three central panels, and = 3.9 in the three right panels; and = 4, = 5, and = 6 in rows one, two, and three, respectively.   influences the internal stable and unstable equilibria when is constant (i.e., public goods) is shown. The relationship between and the internal equilibria-stable equilibrium (full) and unstable equilibrium (empty)-is shown with indicated on the horizontal axis and the frequency of cooperators on the vertical axis. The parameter values used are , = .2, .6 (circle) and , = .2, .73 (triangle), in (b) and (c), respectively. (d) How influences the internal stable and unstable equilibria when is constant (i.e., public goods) and 1 is shown. Circle represents internal equilibria in the case of , = .2, .6 . Triangle represents internal equilibria in the case of , = .2, .73 . + represents 1 and 2 in the case of = .2.
J o u r n a l P r e -p r o o f Frequency of cooperators at which higher unstable equilibrium and higher stable equilibrium coalesce

Highlights
•The evolution of cooperation was studied by examining an n-prisoner's dilemma game.
• The case where a group is disbanded, if and only if heterogeneous, was examined.
• When group size is larger than four, there can be four internal equilibria.
• For common goods, the effect of group size on cooperation is negative.
• For public goods, the effect of group size on cooperation is not simple.