Student-led replication studies in comparative politics: new findings by fresh eyes?

This research note presents two replication studies that were conducted during a research-oriented course at Heidelberg University in summer 2022. The first study, initially conducted by Korell, replicates the findings from Albert’s original contribution on rebel governance. Albert conducted an explorative study in analyzing the associations between rebel governance and rebel military capacity. Our replication study indicates support for Albert’s argument of the relevance of disaggregating rebel governance, while we also show that the original findings are mainly driven by Albert’s explorative model specifications. The second study, initially conducted by Reinecke, tests the robustness of Levin et al.’s study on the democratic peacekeeping hypothesis and expands upon the original study by directly testing the “selectorate-theoretic causal mechanism” for the democratic peacekeeping hypothesis. Overall, the replication studies presented in this research note provide interesting findings regarding empirical conflict studies and contribute to ongoing efforts to promote replication initiatives in the education of future political scientists.


Introduction
Replicating political science research has become more and more usual since King's instructive article on replication (1995).A series of articles have replicated empirical political science research, e.g. in the field of conflict research (e.g., Bell and Miller 2015;Dahl and Høyland 2012), comparative politics (e.g., Imai et al. 2021;Martel García 2014;Stout 2018;Goes 2023), and political behavior (e.g., Haglin 2017; Kobayashi et al. 2017;Busby and Druckman 2018), among others.While the replication of research has become more popular in recent decades, political science teaching often leaves replication as a tool of teaching aside.
In this research note, we present two replication studies in the field of conflict studies that have been conducted during a research-oriented course at Heidelberg University in summer 2022.In this course, presented in more detail by Pelke (2022), students in the bachelor's degree program in political science independently prepared replication studies in their field of interest under supervision of the seminar teacher.The research-oriented course took place as a block seminar with three attendance days of 7 h each with 16 students.In the time between the attendance dates, the students worked independently on their projects and presented their replication study on the last attendance date.In this first step of the course, the students discussed what constitutes good quantitative research and what is necessary for other researchers to be able to understand and replicate their own analyses.In particular, the students searched for suitable and published studies in their field of interest.In the second step of the course, the students first familiarized themselves with the data of the original study and decided which type of replication they would like to perform.The students carried out the necessary analysis steps independently or in a team and documented them in a comprehensible way.In the third step of the course, the students summarized their results in a research report and explained in particular what the contribution of their replication is for the concrete research field (value added).Furthermore, they documented all analysis steps transparently and made the corresponding software scripts and data available to their fellow students and the teacher.Overall, eight replication studies were submitted by the students as term papers, while all 16 students participated in the course conducted their own replication studies during the semester. 1Among the eight submitted term papers, two papers with the most value added and methodological rigor, which Reinecke and Korell wrote, were selected by the teacher for this research note.
By doing so, these students conducted empirical research on their own and followed King's advice to begin original research "by replicating some old work" (King 2006, p. 119).In sum, both replication studies presented here improve upon some aspects of the original research, which makes substantive difference in the empirical findings.The initial replication studies were independently prepared by students under the supervision of the seminar teacher.This research note was prepared jointly by the two student researchers and the course instructor afterwards.
In particular, the study conducted by Korell replicates the findings from Albert's original contribution titled "What is rebel governance?Introducing a new dataset on rebel institutions, 1945institutions, -2012institutions, ." Albert (2022) ) introduces the new Rebel Quasi-State Institutions dataset and argues in her explorative study that there is "value in disaggregating rebel governance and looking at institutions separately because not all governance institutions are correlated with strength in the same direction" (Albert 2022, p. 622).In his replication study, Korell addresses the problem of omitted variable bias that may occur when only correlations are analyzed as in the original study.However, the replication results support Albert's argument for the relevance of disaggregating rebel governance, since substantial differences in the effect sizes, statistical significance, and the direction of associations were found.
In a second replication study, Reinecke tests the robustness of Levin et al.'s study on the democratic peacekeeping hypothesis but also expands on the original study by directly testing the proposed theoretical mechanism.In particular, the original study finds that the effect of UN peacekeeping contributions of sending countries on coup risk varies by regime type.They argue that anocracies will likely experience reduced coup risk.Reinecke finds that these conclusions were driven by four important deficiencies in their original empirical analysis.In addition, the replication study provides-to our best knowledge-the first direct test of the selectorate-theoretic causal mechanism for the democratic peacekeeping hypothesis (Levin et al. 2021, p. 363) by calculating the average marginal effect of interaction between winning coalition size and amount of contributing peacekeepers on the predicted number of military coup attempts at home.In sum, we find empirical evidence against the selectorate theory argument proposed by Levin et al. (2021) that contributing troops to UN peacekeeping missions would impact the likelihood of a coup occurring at home.
In sum, both replication studies provide interesting findings in regard to empirical conflict studies but also how political science teaching can use replication studies to strengthen methodological rigor in student's methods training and to test the robustness of published research.A broader insight that emerges from this study is that replication studies (by students) substantially contribute to existing knowledge by communicating strengths and weak points of existing studies, as well as stimulate further research.Using replication studies in political science methods training not only benefits students, but may also lead to generating new knowledge and advancing the state of research, as we can show in two replication studies presented here.Lastly, both replication studies also contribute to the ongoing effort to promote replication initiatives in education of future political scientists (e.g.Political Science Replication Initiative 2023; Institute for Replication 2023).
2 Replication study: rebel governance and military capacity Albert's (2022) research explored the relationship between institutions that have been built by rebel governance and military capacity, building on Stewart's (2020) earlier work.Stewart found that rebel governance "appears to have either no relationship with rebel strength and sometimes even a negative and statistically significant relationship with rebel military capacity" (2020, p. 16), questioning the strategic motives of rebel governance.Albert (2022) created the new Rebel Quasi-State Institutions (RQSI) dataset and used it to investigate different rebel governance institutions in order to explain Stewart's findings and to disaggregate the association between individual rebel governance institutions and rebel strength.Albert's contribution utilizes two measures to operationalize rebel capacity and performs 34 linear regressions examining the relationship between 17 rebel institutions and two different measures of rebel capacity.She used a composite measure of rebel strength relative to government according to Cunningham et al. (2013).In addition, she operationalized the technology of rebellion variable by differentiating between the ability of rebel's capacity to fight conventionally, as it is important to rebel's ability to provide quasistate institutions.The technology of rebellion variable classifies a war "conventional when both the government and rebels fight conventionally, as irregular when rebels rely on guerrilla tactics but the government possesses conventional capability, and as symmetric nonconventional when neither side fights conventionally" (Albert 2022, p. 629; see also Kalyvas and Balcells 2010).The rebel governance institutions analyzed are diverse, ranging from the provision of health care and education to policing, the creation of a parallel national government, as well as the act of joining an international organization.The results found by Albert suggest that a disaggregated view of rebel institutions is necessary to understand their association with rebel military capacity, since institutions can have either a positive or a negative correlation with rebel capacity.However, her analyses were a first step in examining rebel's military strength, while her RQSI dataset give scholars the opportunity to investigate the effect of rebel governance on military capacity more granularly.We use the new RQSI dataset in this replication study to extend Albert's research by including additional control variables, country and year fixed effects, examining temporal and regional subsets, thereby testing the robustness of the association between rebel governance and rebel military strength.However, Albert only conducted an explorative study in analyzing the correlations between rebel governance and rebel military capacity and did not test any (causal) effects of rebel governance on rebel strength or the other way around.
The results presented in Fig. 1 differ from the original results (Fig. 1a,b) but further support Albert's argument of the relevance of disaggregating rebel governance, since substantial differences in effect size, statistical significance and the direction between the institutions and the relative rebel strength as well as irregular technology of rebellion were found.The use of country and year fixed effects as well as control variables is important to reduce omitted variable bias.In addition, by including control variables, we show that the relationship between some institutions and rebel strength is robust against omitted variables bias, while the majority is not, as can be seen by the comparison between the original estimates and the replication K estimates.While Albert conducted a first explorative analysis of the association of rebel governance and rebel strength without testing for causality, our analysis tries to test the (causal) connection between rebel governance and strength by using a twoway fixed effects designs and additional robustness tests.
Additionally, the results reveal important differences in the magnitude and direction of the effects between different regional and temporal subsets of rebel groups.Regional conflict characteristics suggest different governance styles for rebel institu-tions in Sub-Saharan Africa and Asia.Krause and Suzuki (2005), for example, find that rebel groups in Africa may have greater capacity due to their financial resources from oil exports, while rebel groups in Asia should focus on gaining autonomy and may be less resource-oriented due to ethnic fragmentation.Their findings imply that economic institutions, such as trade agreements and resource rights, would be more important in Africa, while political institutions, such as international representation or a constitution, would be more important in Asia.However, the results indicate differences across regions, but not in the pattern theoretically expected (See Supplementary Appendix A3).
In additional robustness tests, we find mainly the same patterns presented in Fig. 1 in analyses with region and year fixed effects (Figure A1 in the Supplementary Appendix).Neither the measure of rebel strength nor the technology of rebellion measure are statistically significant and substantially meaningful associated with one of the rebel governance institutions.In addition, Figure A2 and A3 test the relationship between rebel capacity and rebel institutions for the Cold War period and the period after the end of the Cold War.Both Figures show different patterns compared to Fig. 1, but overall there is no statistically significant association between rebel governance institutions and rebel strength.Figure A4 and A5 control for regional subsets.Neither for Asia nor for Sub-Saharan Africa a clear pattern emerged.For the Asian subsample, we find a statistically significant association of rebel capacity and elections, parallel government, policing, and justice, while these negative associations disappear when using the technology of rebellion measure.Lastly, we test for missing observations in the RQSI dataset by using multiple imputation techniques.The results are reported in Figure A6 and A7 in the Supplementary Appendix.The null findings presented in Fig. 1 remain robust, while we found more statistical significant associations between rebel military strength and rebel institutions.
Our replication also demonstrates the need for a reliable and comparable measure of rebel capability, as significant differences were found in the effect sizes and directions of the relationships between relative rebel strength and rebel technology across the same institutions.It also highlights the need to further develop the theory and adapt it to the availability of fine-grained quantitative data, since data on rebel governance has been scarce and is only now becoming more available to researchers.
In conclusion, this replication study further differentiates and increases the generalizability of Albert's findings, while also identifying and addressing limitations of the original study design.The Rebel Quasi-State Institutions dataset is shown to be a valuable contribution to research in this area, and our replication aims to increase the reliability of associations in this field of research.The results also emphasize the importance of using control variables and fixed effects to control for omitted variable bias and of analyzing temporal and regional subsets to understand the nuances of rebel governance institutions' associations with rebel's military capacity.
3 Replication study: winning coalition size and UN peacekeeping Levin et al. (2021) presented a systematic test of the coup risk as a function of countries' contributions to UN peacekeeping missions.They conducted a quantitative K analysis covering peacekeeping deployments, regime type, and coup risk since the end of the Cold War.In particular, they examined the effect of contributing military personnel to UN peacekeeping missions abroad on the risk of military intervention in politics at home based on the regime type.They built on findings from Lundgren (2018) and the influential selectorate theory model (Bueno de Mesquita et al. 1999Mesquita et al. , 2005) ) to explain how contributing to peacekeeping missions abroad might shape internal regime dynamics.Levin et al. (2021) argue that autocracies-often developing countries with low costs per soldier and a small winning coalition-would likely make a monetary profit and invest part of that profit back into their military by participating in UN peacekeeping missions.Overall, they argue that increasing military spending would lead to a higher risk of a military coup at home.Levin et al. (2021, p. 356) also argue that developed democracies have large winning coalitions and high per-soldier costs, in addition to militaries that are socialized against coups and would therefore see no increased coup risk.According to Levin et al. (2021) and a large literature in comparative politics, anocracies are intermediate or transitional regimes with large or growing selectorates but relatively small winning coalitions.The authors argue that anocracies try to present themselves as democracies in order to be able to participate in UN funding programs, and a coup would undermine access to said funds, which is why even the armed forces are incentivized to prevent coups.They systematically tested all three hypotheses and found empirical support for their claims.Anocracies have a reduced coup likelihood after peacekeeping deployments abroad, while autocracies have an increased coup-risk after contributing to UN peacekeeping operations.In democratic regimes, they find a null effect in terms of the interaction between peacekeepers and regime type.
However, we identified four important deficiencies in their original empirical analysis.First, there is a coding error in the democracy dummy variable named "Robust democracy" (see Supplementary Appendix B1).Second, the usage of rounded numbers by Levin et al. to calculate the average marginal effect (AME) of the interaction term between regime type and the number of peacekeepers abroad in the original Fig. 1 (Levin et al. 2021, p. 363) largely distorts the presented findings (see Supplementary Appendix B2).Moreover, the authors reported an incorrectly estimated quantity of interest by calculating the interaction term manually.By doing so, they interpret the marginal effects of the interaction term between regime types and number of peacekeepers without accounting for the effect of the regime type and the number of troops.By doing so, they interpret only one constitutive term (interaction term) as an unconditional marginal effect.We re-estimate all models by using the correct specified interaction term and their AME in Fig. 2. Modelling interaction terms in regression models are common in political science but researchers have often got in trouble with specification of these models and their interpretation (see Brambor et al. 2006).According to our analysis, Levin et al. made a frequent mistake: They specified and interpreted the constitutive terms "as if they are unconditional marginal effects" (Brambor et al. 2006, p. 64).They are not, as Fig. 2 shows empirically.To avoid such mistakes in future studies, researchers should always first answer what their main quantity of interest is (also called the "estimand", see Lundberg et al. 2021) and follow the checklist by Brambor et al. (2006).

Fig. 2 a
Incorrectly specified original estimates from Levin et al. (2021, p. 363).bAME of Regime Type, but incorrectly interpreted as the average marginal effects at the mean of the number of troops.c AME of regime type conditional on the number of peacekeepers abroad By correctly estimating the average marginal effect across,the AME of anocracy is reduced from around -0.5 to around 0.02, a positive effect, at the mean of peacekeepers abroad when correcting for the rounding errors.Third, their study relies on only one regime measure (Polity IV) without explaining the theoretical basis for said choice (see Supplementary Appendix B3 for additional analysis).Finally and most importantly, Levin et al. do not directly test their proposed causal mechanism, as they point out that they "do not specifically test the selectorate-theoretic causal mechanism" (Levin et al. 2021, p. 363).In this replication study, we present a direct test of the above-presented mechanism, in order to not only point to flaws of the original study.
The theoretical argument presented by Levin et al. (2021) relies strongly on the size of the selectorate and the size of the winning coalition in order to explain the effect of contributing troops to UN peacekeeping missions abroad have on the probability of experiencing a coup at home.Levin et al. assume that the varying winning coalition size is the main explanatory factor for the difference in the effect of interaction between regime type and peacekeepers.They test these mechanisms only indirectly via relatively uninformative regime type categories in terms of how these categories cover the concept of interest, the winning coalition size.The comparative politics literature argues that autocracies have a small, democracies a large, and anocracies an ambiguous-sized winning coalition.Since Levin et al. do not test this mechanism directly, the risk of accidentally identifying a spurious relationship as a causal one weakens their empirical findings.As we have shown in the Supplementary Appendix, there are serious concerns related to the robustness of their findings as well as the substantial interpretations of the authors.In the worst case, their conclusions are artefacts of the data being used and the questionable decision of rounding troop numbers in the erroneous marginal effects calculation.
However, we are providing-to the best of our knowledge-the first direct test of the "selectorate-theoretic causal mechanism" (Levin et al. 2021, p. 363) by calculating the average marginal effect of interaction between winning coalition size and contributing peacekeepers on the predicted number of military coup attempts at home.We do so by utilizing a new measure for selectorate theory's central concept, Fig. 3 Average marginal effect of peacekeepers by the size of the winning coalition.(We used the same data as Levin et al. (2021) for all variables, including the number of coup attempts and the number of troops contributed in a given year, with an exception for the winning coalition size.For this variable, we used the formula from Bueno de Mesquita and Smith (2022) to estimate a measure of winning coalition size with V-Dem data (Coppedge et al. 2023)) winning coalition size, presented in an article by Bueno de Mesquita and Smith (2022).
Figure 3 visualizes the average marginal effect of the interaction between the numbers of peacekeepers abroad and the size of the winning coalition.The figure clearly indicates that there is no statistically significant effect on the predicted number of coups.This finding questions the assumption that peacekeepers abroad and the size of the winning coalition are the mechanisms explaining the probability of coups.The evidence presented here is the first systematic test that directly uses data for the size of the winning coalition rather than relatively vague proxies as in earlier studies.Overall, we follow Levin et al. (2021) and test the selectoratetheoretic causal mechanisms utilizing the same data that Levin et al. (2021) used in their original study, with the edition of data for winning coalition size from Bueno de Mesquita and Smith (2022), who rely on fine-grained V-Dem Data.However, for future research it might be useful to examine the relationship with qualitative evidence to find the main causal mechanisms explaining the effects of peacekeepers abroad on coup probability.Our findings show, that the size of the winning coalition dependent on the number of peacekeepers abroad is clearly not an adequate theoretical explanation for the predicted number of coup attempts as presented in this study.In addition, as shown in Fig. 2, the original study incorrectly specified the interaction terms, which substantially affected the interpretation of the authors' findings.One interesting additional research lacuna would be to examine the association between budget profits from contribution to UN peacekeeping missions in lower-income countries and the likelihood of changes in the winning coalition size, composition, and the stability of the ratio between selectorate and winning coalition.In sum, the study by Levin et al. (2021) is only a first step in investigating the relationship between the contribution to UN peacekeeping missions and coup risk at home, since their study has far-reaching flaws.

Conclusion
This research note highlights the centrality of replication studies in teaching political science methods.By testing the empirical results of two studies related to the broad area of conflict research, this research note also indicates that student-led replication efforts are able to challenge and revise empirical findings from published research.In sum, student-led replications contribute to the state of research and are able to incentivize political scientists to provide better reproduction materials and to conduct robust research. 2n particular, our replication study extends Albert's explorative research design by providing additional evidence on the relationship between rebel governance institutions and rebel military strength.The results support Albert's argument that a disaggregated view of rebel institutions is necessary to understand their association with rebel military capacity.In addition, it provides a more nuanced understanding of the relationship between rebel governance institutions and rebel strength, while also revealing the need for a reliable and comparable measure of rebel capability and the importance of theory development.Overall, our replication study reinforces the importance of analyzing rebel governance institutions as individual factors rather than as a singular concept.The Rebel Quasi-State Institutions dataset created by Albert provides a valuable resource to further explore the relationship between rebel governance and rebel military capacity.
In a second replication study, we find that the evidence presented by Levin et al. (2021) for the (causal) connection between contributing troops to UN peacekeeping missions and the probability of experiencing a coup at home is lacking.Our direct test of the selectorate-theoretical mechanism, using a new measure for winning coalition size, found no statistically significant and substantial effect on the probability of coups.Thereby, our study shows the importance of rigorously testing theoretical mechanisms and using appropriate measures of the winning coalition and the selectorate.Utilizing a novel measurement to assess the size of the winning coalition (Bueno de Mesquita and Smith 2022), we challenged the theoretical assumptions of Levin et al. (2021).We presented robust empirical evidence against Levin et al.'s claim that contributing troops to UN peacekeeping missions, conditional on the size of the winning coalition, impacts the likelihood of a coup occurrence at home.Future research in this area should consider these issues and strive for more rigorous and transparent empirical analyses.Overall, our findings also underscore the importance of future replication studies and the centrality of providing reproducibility materials in political science.In sum, fresh students' eyes are able to contribute to the political science literature.

Fig. 1 a
Fig. 1 a, b Original estimates from Albert (2022).c, d Estimates from this replication study regressing rebel governance institutions on two measures of rebel capacity with country and year fixed effects and control variables