Do Public Works Programs Have Sustained Impacts? A Review of Experimental Studies from LMICs

The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.


Introduction
There is growing emphasis on expanding social protection around the world. The United Nations Sustainable Development Goals call upon countries to "implement nationally appropriate social protection systems for all" (United Nations, 2018). Cash transfers and public works are among the most widely used and debated social protection programs in low-and middle-income countries (LMICs) (e.g., World Bank, 2018;Banerjee, 2022). 1 Both types of programs provide cash for consumption support and poverty alleviation in the short run and offer promise for medium-run improvements in living standards facilitated by investments in human capital and productive assets.
The experimental literature on cash transfers goes back nearly 20 years, and their effectiveness is well-established (e.g., Fiszbein and Schady, 2009;Baird et al., 2014, Millań et al., 2019Kondylis and Loeser, 2021). While the experimental evidence on public works has been much thinner, several new studies have been completed in the past five years. This paper reviews experimental evaluations of public works programs from LMICs.
As such, we extend the scope of earlier reviews by Subbarao et al. (2013) and Gehrke and Hartwig (2018), which rely mostly on quasi-experimental evidence. We examine the impacts of public works on participants' short-term employment and earnings, and whether effects are sustained over the medium run. We also discuss effects on consumption, savings, assets, women's empowerment, and psychological well-being.
There are various types of public works programs, and their objectives link to both social protection and employment policies. For instance, several large-scale public works programs in low-and middle-income countries, such as those in Ethiopia or India, were established to address climatic shocks. 2 Humanitarian organizations like the World Food Programme also routinely deploy policy instruments with features of public works programs in response to shocks or seasonality (World Food Programme, 2016). Some public works programs in Latin America were implemented in response to severe macroeconomic instability and crisis, for instance, in Argentina. 3 Public works programs have also been adopted in fragile settings to respond to economic instability induced by conflict and spur post-conflict recovery, such as in the Central African Republic, Comoros, Côte d'Ivoire, Democratic Republic of Congo (DRC), and Sierra Leone. In these cases, they aim to provide public goods or rebuild community infrastructure destroyed during the 1 For instance, around 60 low-and middle-income countries are implementing workfare programs (World Bank, 2018). 2 In 2005, Ethiopia introduced the Productive Safety Net Program (PSNP) to address chronic food insecurity arising from drought shocks (Tesfay et al., 2017). Similarly, in the 1990s, one state in India, Maharashtra offered India's first public works program in response to a massive drought. Much later India introduced the world's largest public works program as a 'right to work', the Mahatma Gandhi National Rural Employment Guarantee Act (MGNREGA), which guarantees 100 days of temporary employment per year to every household residing in a rural district in India (Imbert and Papp, 2015;Mani et al., 2020). 3 The currency crisis in the 1990s led Argentina to introduce the Trabajar "To Work" program that provided short-term work to the poor (Ravallion and Jalan, 1999), but the deepening of the financial crisis in 2002 led to the expansion of the program to Jefes De Hogar (Galasso and Ravallion, 2004) wherein all unemployed heads of household became eligible to participate in the workfare program. In some cases, the workers were also required or encouraged to save some of their wage earnings. conflict and improve economic security and employment opportunities for individuals in conflict-affected regions. 4 Increasingly, the focus of public works programs is to improve medium-run employability, assets, and resilience to future shocks (Gehrke and Hartwig, 2018). Besides providing short-term employment or income support, these programs also aim to open pathways to productive employment after participants exit the program. This is often done by combining the public works experience with additional interventions, such as vocational training or micro-entrepreneurship support. 5 This paper reviews experimental evidence on public works programs, including a wave of recent studies, and provides directions for future research. We use a meta-analysis model to estimate the average impact of public works programs on individuals' earnings and employment. We use data from all countries for which a published paper, working paper, or project report was either publicly available or made available by the authors as of December 2021. As such, the review is based on 11 randomized control trials across 9 countries--Comoros (Gazeaud et al., 2019), Côte d'Ivoire (Bertrand et al., 2021), DRC (Brandily-Snyers et al., 2021;Mani and Mvukiyehe, 2021), Djibouti (Devoto et al., 2017), the Arab Republic of Egypt (World Bank, DECRG;Croke et al., 2023), Ethiopia (Abebe et al., 2021), the Lao People's Democratic Republic (World Bank, EAPGIL, 2020), Sierra Leone (Rosas and Sabharwal 2016), and Tunisia (Leight and Mvukiyehe, 2022) -see Appendix Table A1 for details. 6 We complement earlier work by Subbarao et al. (2013) and Gehrke and Hartwig (2018) in several important ways. First, we focus on public works experiments. Recent experimental studies provide causal evidence on the impacts of public works programs on individual participants and their households in diverse contexts, thus offering both greater internal and external validity. In contrast, existing reviews of public work programs are mostly based on quasi-experimental studies. This is a potential concern, as there are significant differences in findings generated by experimental and non-experimental evaluation methods (LaLonde, 1986;Smith and Todd, 2005;McKenzie, 2017;Card et al., 2018;Millán et al., 2019). The issue may be particularly salient for public works programs 4 Comoros rolled out a Social Safety Net Project (SSNP) that provided public works opportunities to poor households (Gazeaud et al., 2019). Public works programs in Sierra Leone, Côte d'Ivoire, DRC, and the Central African Republic were also introduced to facilitate post-conflict recovery (Rosas and Sabharwal, 2016;Bertrand et al., 2021;Brandily-Snyers et al., 2021;Mani and Mvukiyehe, 2021;Alik-Lagrange et al., 2023). 5 To achieve these broader objectives of transitioning workers from workfare to regular employment, during 1998-2000, Argentina's Proempleo experiment provided specialized training or vouchers that entitled employers of workfare participants to a sizable wage subsidy (Galasso et al., 2004). The public works in Côte d'Ivoire included complementary micro-entrepreneurship or job search skills training (Bertrand et al., 2021). In Urban DRC, the public works program was complemented with incentivized savings and skills training (Brandily-Snyers et al., 2021). Similarly, the World Food Programme increasingly implements food-forassets interventions that seek to put a greater emphasis on asset creation and environmental outcomes, together with providing complementary market access or livelihood support (World Food Programme, 2016). In recent innovations, digital public works have been piloted to also foster digital skill acquisitions. 6 A study from Malawi (Beegle et al., 2017) was not used, as it did not report results on employment and earnings in ways that are readily comparable to the other studies. A study from the Central African Republic did not have a report available at the time of the review, although it was recently published (Alik-Lagrange et al., 2023). that rely on participants' self-selection (Bertrand et al., 2021;Banerjee et al., 2022).
Second, we review evidence from studies that document impacts on outcomes at various points in time where impacts measured during program (4 studies) are classified as "short run," and post-program impacts (11 studies) are classified as "medium run." 4 of these 11 studies track respondents over two rounds. Overall, this allows us to assess not just the short-run effects of the public works programs, but also their medium-run effects, which remain underexplored in the literature until recently. Importantly, we pay close attention to some of the intermediary outcomes (e.g., asset accumulation, savings) that relate to mechanisms that may drive sustained program impacts over the medium run. Third, we draw lessons from programs targeting different types of public works participants (e.g., youth; urban and rural populations; women), as well as from diverse cultural and institutional contexts. Lastly, while we focus our discussion on economic outcomes, we also examine impacts on women's empowerment and psychological well-being, which remain understudied in the literature.
There are a few caveats to this review. First, we focus on documenting the impact on participants' outcomes. We are unable to comment on the general equilibrium effects or externalities of public works programs. This is not to suggest that these are not important, especially for programs that have a strong focus on the provision of public goods (e.g., roads, irrigation systems), the improvement of environmental outcomes (e.g., reforestation), or the generation of social externalities (e.g., social cohesion in post-conflict settings). However, only a few studies to date have been designed to identify general equilibrium effects or measure externalities (Abebe et al., 2021;Muralidharan et al., 2021;Leight and Mvukiyehe, 2022;Croke et al., 2023). Second, we are unable to directly compare the cost-effectiveness of the programs we study, as only two studies provide detailed cost data and cost-effectiveness analysis (Rosas et al., 2016;Bertrand et al., 2021).
Third, while comparing across countries is helpful for external validity, the body of experimental evidence is largely concentrated in Sub-Saharan and North Africa, highlighting the need for additional evidence from other regions. Lastly, it is beyond the scope of this review to comment on the data quality or limitations of specific studies, though we provide information on attrition or take-up that is relevant to interpret some of the country-specific findings.
The review documents substantial variation in the impacts of public works between studies and over time, but also highlights some consistent patterns and policy implications.
First, public works tend to significantly improve participants' employment and earnings during their participation in the program. Second, these short-term effects fade in the medium run, except in a few cases with particularly large short-term impacts on savings or investments in self-employment activities. Third, observed impacts on earnings are much lower than planned transfer amounts due to forgone earnings, which raises questions about cost-effectiveness. Fourth, there is little evidence of public works programs improving food consumption expenditure. Lastly, there is evidence of improvements in psychological well-being and women's empowerment in some cases, but it is not systematic, and measurement of these dimensions remains at times rather narrow.

Conceptual framework
This section outlines various channels through which public works may impact participants' outcomes in the short to medium runs. While it provides an overview of a broad range of mechanisms (including general equilibrium effects and externalities), the results section will later focus on a narrower set of outcomes and impacts for participants that are measured in the studies included in our review.
Public works programs can impact individual and household-level outcomes through a combination of short-run income and substitution effects. Access to a public works program in a community is likely to attract unemployed as well as underemployed individuals, giving them an opportunity to move away from low-paid farm and off-farm work (Subbarao et al., 2013;Filmer and Fox, 2014). This, in turn, may increase both temporary employment and income for participants. The income effect may also increase households' demand for food and non-food items (and hence consumption expenditures) in the short run (Ravi and Engler, 2015;Liu and Deininger, 2013;Azam, 2012). However, if program participants (or other members of their households) substitute other forms of employment (e.g., self-employment or low-paying wage jobs) to access public works jobs, then participation in public works could have little to no effect on overall household income and consumption due to forgone earnings from these other occupations (Bertrand et al., 2021;Brandily-Snyers et al., 2021). Depending on the type of labor substitution and the frequency of individuals' earnings from each occupation, it could even have a (temporary) negative effect on participants' household income and consumption, for instance, if public works payments are made less frequently or later than other income sources. The availability of short-term employment through public works may also allow households to smooth consumption expenditure and food security during the lean season when agricultural productivity is low (Zimmermann, 2014;Ravi and Engler, 2015;Beegle et al., 2017).
Since unemployment or underemployment is known to be positively correlated with poor mental health and unhappiness (Clark and Oswald, 1994;Farré et al., 2018), public works opportunities may improve an individual's psychological well-being (Bertrand et al., 2021, Hussam et al., 2021. At the same time, these opportunities might also create social stigma for participants (Ravallion, 2019). Several public works programs are specifically designed to encourage women's participation in the workforce (Afridi et al., 2016;Devoto et al., 2017;World Food Programme, 2021). The ability to earn income, which may otherwise be restricted or even unavailable to women in low-and middleincome countries, has the potential to improve economic empowerment and autonomy. This may, in turn, reduce domestic violence (Buller et al., 2018;, though, in some contexts, greater autonomy may increase gender-based violence (Hidrobo and Fernald, 2013).
Public works programs may also increase household income and consumption expenditure through savings, investment, and skill accumulation in the medium run, that is, after participation in the program ends. Income from public works used for savings and productive investments like the purchase of inputs (e.g., fertilizers) or assets (livestock, smartphones, etc.) may lead to subsequent improvements in household income and consumption expenditure (Bertrand et al., 2021;Mani and Mvukiyehe, 2021). On-the-job learning and soft skills accumulation facilitated by participation in public works programs could also increase employment and earnings for participants, although there is currently little empirical evidence to support this (Alik-Lagrange et al., 2017;Bertrand et al., 2021).
Public works programs may have broader externalities or general equilibrium effects that affect program participants or non-participants and can occur both in the short and medium runs. While the empirical analysis in this review does not consider these effects (as mentioned above), we briefly discuss them here. General equilibrium effects may operate via changes in private sector wages, public good creation in the community, or other social externalities. For example, public works programs may crowd out private sector employment and increase private sector wages, which in turn can increase household earnings and consumption expenditures among participants and non-participants alike (Azam, 2012;Imbert and Papp, 2015;Abebe et al., 2021;Muralidharan et al., 2021). The increase in local wage rates may also increase the overall price of goods, as such affecting the demand for food and non-food items. 7 Theoretically, the increase in private sector wages can also reduce the demand for labor among non-participants in the private sector.
Further, public works programs often involve the construction or maintenance of public infrastructure that can spur economic opportunities for both participant and nonparticipant households. For instance, public works programs may improve environmental outcomes and assets by planting trees or rehabilitating degraded lands (Adjognon et al., 2020). In drought-prone regions, the construction of canals and irrigation systems can increase returns on agricultural investments and activities (Subbarao et al., 2013).
Similarly, the construction or rehabilitation of roads can provide access to markets and better prices for agricultural commodities (Asher and Novasad, 2020).
Lastly, public works programs may also produce social externalities by facilitating positive interactions between participants and could reduce crime and other risky behaviors through shifting time use toward public works and increasing income (Fetzer, 2020;Bertrand et al., 2021). Increased intra-group contact in ethnically diverse communities could potentially contribute to social cohesion outcomes.

Program features
There is substantial variation in key design elements across the 11 public works programs included in our study. Below (and in Table 1), we summarize the following features: duration of employment, targeting, wages, maximum earnings, and type of work. Overall, the public works programs included in this review are qualitatively like those from other low-and middle-income countries described in Subbarao et al. (2013), except for those that allow for repeated participation every year, like India's Mahatma Gandhi National Rural Employment Guarantee Act (MGNREGA) or the Ethiopia rural Productive Safety Net Programme (PSNP), which are not represented here.

Duration of employment.
Public works programs are designed to offer employment for a limited duration: the median participant is offered the equivalent of approximately 4 months of work. The maximum duration is 2.5 months in Djibouti and up to 18 months in the Egypt Community program. The programs can provide consecutive months of employment, though some spread days worked within one year or over longer periods. For instance, the two programs in Comoros and Ethiopia offer a total of 9 months of work, but for 3 months per year over 3 years. The program in Lao PDR offers employment during specific times in the agricultural cycle, spreading 75 days (~3.75 months) of employment over two full rainy seasons (~1.5 years). 8 The maximum duration of employment for all programs included in our review is summarized in Table 1.
Targeting. All programs applied some form of geographic targeting. Four programs focused on urban areas, four on rural areas, and three covered both (Table 1). All programs used self-targeting, whereby the program is advertised, and individuals are invited to apply if they are interested. Many targeted specific groups such as youth (in Côte d'Ivoire and Sierra Leone), or women (in Djibouti, the Egypt Community program, and Lao PDR).
Wages. Daily wages differ widely, ranging from $2.5 in Sierra Leone to $14.7 in the Egypt Community program, with a median of $9.60 (see Table 1). 9 Setting a wage for a public works program involves trade-offs (Bertrand et al., 2021). On the one hand, setting a low wage can facilitate self-targeting by discouraging individuals who already work and have higher earnings from leaving their occupation to join the program. On the other hand, setting a higher wage can enhance take-up as well as lead to larger impacts on earnings among individuals most in need. Five of the 11 programs set wages within 20% of the minimum wage, with 2 programs further below and 2 programs further above the minimum wage (see Table 1 and additional details in Table A2). This again shows substantial variation between programs. In practice, however, it is not always clear that the minimum wage is an appropriate benchmark, because a large share of workers in low-income 8 There is some variation in the number of days offered within some programs. For instance, the Sierra Leone program offered 50-75 days of work and the Egypt Infrastructure program offered 1 week to 3 months of work. In these instances, we have used the maximum days allowed in the program in the calculations, since the average time worked is not always available. In the Egypt Community program, the duration of participation was set between 12-18 months; however, on average, individuals work up to 11.7 months. 9 All wages are converted to US dollars using the 2021 PPP conversion factor from the World Bank's International Comparison Program (https://data.worldbank.org/indicator/PA.NUS.PPP). economies are self-employed or engaged in informal wage employment for which minimum wages are not binding. 10 Maximum program earnings. Since both the duration of employment and the daily wages vary, there is also substantial variation in maximum earnings. Figure 1 shows the maximum monthly-equivalent value of earnings from the public works program by program duration in months. The total maximum monthly equivalent earnings are calculated assuming only 20 days of work is available in a month. For instance, the public works program in Comoros was offered for 60 days per year for three years, which means the maximum duration of the program was nine months (60/20 = 3 months x 3 years = 9 months). The potential value increases with the total number of workdays offered by the program. Median maximum earnings are US$96.5 per month of work, but maximum earnings range from as low as US$27 in Comoros to as high as US$294.70 in Egypt Community (an outlier both in total value and duration of participation).

Figure 1: Maximum earnings by maximum program duration
Notes: The x-axis shows the maximum duration of the public works program in months. The y-axis shows the maximum per month earnings (in USD PPP 2021) possible from participating in the public works program.
Type of work. Programs in urban areas include activities like street cleaning and garbage collection. In rural areas, work often contributes to larger infrastructure projects such as irrigation, afforestation, reforestation, or road construction and maintenance. Some 10 Several issues make the relevance of the minimum wage questionable as a benchmark in the study settings. First, the share of workers for which the minimum wage applies is very low, for instance in Sub-Saharan Africa or low-income economies where self-employment is prevalent but formal wage employment is rare (Bhorat et al., 2017;Filmer et al. 2014). Second, low-income countries (with the lower share of formal wage jobs) tend to set relatively higher minimum wages than middle-or upper-income countries (with a higher share of formal wage jobs) (Bhorat et al., 2017). Third, many countries have several minimum wages applying to different sectors, making it difficult to identify the right comparison (Bhorat et al., 2017). Lastly, minimum wage data are not documented consistently for all countries and years. programs explicitly tailor activities to their target population. For example, the Egypt Community program mainly targeted women and focused on activities that were not considered physically strenuous, such as cleanliness and environmental awareness campaigns, early childhood education, and literacy promotion. Similarly, in Djibouti, the program targeted only women and included light community works such as street rehabilitation or small artisanal projects.
Take-up: Program take-up is high in most studies, as shown in Table 1. Take-up rates were above 90 percent in Djibouti, Côte d'Ivoire, and Ethiopia. The lowest take-up rates occur in the Egypt infrastructure program (40 percent) and the Urban DRC program (56 percent).

Table 1: Program features
Notes: Duration of employment presents the maximum length of program available in a year except in Lao PDR where it's computed over 1.5 years. Further, in Comoros and Ethiopia, the program is offered for a few months every year for three consecutive years. In converting maximum days available on a public works project to months we assume that only 20 days of work is available in a month. See Table A2 for the calculation of the ratio of daily program wage to daily minimum wage.   (Bertrand et al., 2021). Since only these two studies contain multiple treatments, we do not attempt to draw more general conclusions on complementary program design features in this paper.

Evaluation sample
The median study sample size is 2,126. Individual-level randomized designs have a sample size ranging from 952 to 6,014. Cluster randomized designs have larger samples ranging from 1,017 to 19,442 with at least 30 clusters per treatment arm across studies. The median sample size per cluster is 31, comparable to the median sample size per cluster of 26 noted in Muralidharan and Niehaus (2017). 15 11 In Ethiopia, clusters were randomized into treatment and control, and, within treatment clusters, the participants were selected by local Ketena committees. 12 In Comoros the design varied the treatment intensity, 40 percent of eligible households received the intervention in treatment clusters and 20 percent of eligible households received the intervention in control clusters. 13 A few studies also offer public works opportunities to the control group in the post-evaluation period in a randomized phase-in design, namely, Egypt infrastructure, Sierra Leone, Ethiopia, and Djibouti. 14 In Comoros, a third level of randomization was carried out: in every treatment household, the gender of the participant was further randomly chosen within the household. 15 All individual level randomized experiments can detect at least a $1.50 improvement in consumption expenditure and earnings, assuming a standard deviation of $8 with 5 percent type I error rate and 80 percent power. Among the cluster RCTs, there is an improvement in consumption and earnings--the Egypt studies can detect at least a $1.40 improvement, Comoros $2.10, Ethiopia $1.70, Sierra Leone $1.00, and Tunisia $1.90. These are computed using the country specific cluster sizes noted in Table 2 and assuming a standard deviation of $8, intra-cluster correlation of 0.10, 5 percent type I error rate and 80 percent power. All these calculations also assume 100 percent take-up. In practice, however most programs did not have 100 percent take-up. Particularly low take-up rates in Urban DRC (56 percent) and Egypt infrastructure (40 percent) will reduce the power to detect significant effects substantially.
Follow-up survey attrition rates are generally small to modest in size. One exception is the study on the Egypt Community program, where attrition was 23 percent due to an administrative issue and not because of migration (Croke et al., 2023  varies between 3.9 and 6.9, and the median household size is 6.1. Except in Djibouti and Lao PDR, where the program targets only women, both men and women participate in most programs, but women account for less than half of participants in several countries, such as Comoros, Rural DRC, and Sierra Leone. In nine of the ten studies that report it, the median number of participants with no education is 33 percent.

Follow-up surveys and key outcomes
All studies administered at least one follow-up survey after the start of the intervention to collect detailed data across the treatment and control groups. Four studies

Figure 2: Timeline -Public works programs and surveys
We focus on five primary outcome variables that are measured consistently across studies.
These outcomes allow us to document the impact of public works programs on individual employment, earnings, household food consumption expenditure, savings, and assets. Table 4 provides the definitions. We also compare findings on two secondary outcomespsychological well-being and women's empowerment. These outcomes are measured less consistently across studies, see Appendix Table A3 for their definition in each case. Asset index Constructed using PCA where the list of physical assets (like lamps, air conditioners, TV, etc.), movable assets (like bikes, motorbikes, etc.), and durable assets varies country-by-country.

Results--Impact estimates from 11 RCTs of Public Works Programs
We now present the intent-to-treat (ITT) estimates of public works interventions on key outcomes. ITT estimates are based on the initial assignment to treatment status, regardless of whether individuals participated in the public works program. In most cases, governments or other parties that introduce programs can make them available to their target population, but they cannot mandate take-up, hence the ITT estimate is arguably the most policy-relevant parameter as it estimates the benefit of providing access to a program.
The ITT estimate is computed as the difference in average outcomes in the treatment group (randomly chosen public works participants, from within the population targeted by the program) and average outcomes in the control group (randomly chosen non-participants from within the same population). 16 We report results separately for each outcome, study, and, when applicable, followup survey. In addition, we aggregate results across studies and follow-up surveys using a random effects REML model (like the approach in McKenzie, 2021). The meta-analysis takes as inputs the point estimates and standard errors from each study. Higher weight is given to studies that have smaller standard errors.

Employment
Since public works programs offer cash in exchange of work, we first present their impacts on participants' employment. In Figure 3, we document the estimated impacts on individuals' employment from the different studies, along with the control mean for each case. The figure displays results by the timing of the survey with respect to program completion, with short-run impacts measured before individuals exit the program at the top (4 to 0 months before program completion), followed by impacts measured after individuals have exited the program in the medium run (from 1 to 66 months after program completion). Lastly, we also present estimates from a random-effects meta-analysis regression model that captures the short run, medium run, and overall impact of public works on employment.  16 All cluster randomized controlled trials included in this review, except the study on the Egypt Community program, compare randomly chosen participants in treatment clusters to non-participants in control clusters. In some cases, the sampling strategy used to sample non-participants in control clusters might differ from the sampling strategy used to sample participants in treatment clusters (e.g., Leight and Mvukiyehe, 2022). 85 percent of the control group has an occupation (in most cases working in informal jobs or self-employment). Even though the effect is not statistically significant in Lao PDR, the point estimate is non-negligible, suggesting a 15.5 percentage points increase, or 46 percent, relative to the control group. 17

Short
Next, we examine the effects measured after individuals exited the public works program. These post-program effects remain substantial in Sierra Leone at 1 month (11.2 percentage points, or 34 percent relative to the control mean), Egypt Community at 1 month group is relatively low (11.9 percent, 21.3 percent, and 9 percent, respectively). These large relative effects can be driven by the specific groups that are targeted in these cases, such as youth or women who are more likely to be unemployed or outside the labor force. They might also be explained by differences in the structure of the labor market across settings.
For instance, underemployment is more widespread in less formalized economies, where many individuals are engaged in low-paying occupations and few are formally unemployed.
The average effect on the likelihood of employment also hides substitution between occupations. In Côte d'Ivoire, a substantial share of treated youth substituted out of informal wage jobs and self-employment to take public works jobs, limiting the net effects on employment. In Ethiopia, the program increased employment in public works by 4.6 percentage points but also decreased employment in the private sector by 4.7 percentage points, resulting in an overall null result. 17 The authors also note that in an alternative specification there is a 16 percentage points increase in the likelihood of working in paid work. There is also an increase in being a 'regular earner'.
Both Urban and Rural DRC and Tunisia (at 12 months) are the only studies that find employment effects beyond 1-month post-intervention. In all three, there is an increase in the ownership of productive assets among participants (in the form of purchased land in Urban DRC, livestock in Rural DRC, and movable assets, livestock, and electronic equipment in Tunisia) that could explain the medium-run effects on employment (Brandily-Snyers et al., 2021;Mani and Mvukiyehe, 2021;Leight and Mvukiyehe, 2022).
There is only one study, Tunisia, that registers negative effects on employment measured 5.5 years after program completion. The authors speculate that the negative results on employment could be due to the program increasing wage expectations or increasing employment in informal occupations in the short-term, which may have limited improvements in employment outcomes over the longer-term.

Figure 3: Estimates of the impact of public works programs on employment
Notes: This figure presents the intent-to-treat effects on employment obtained from each study and followup survey. The effect size is the change in employment (in percentage points). Timing indicates the timing of the survey in months before/after program completion, where 0 indicates the time when the program was completed. The diamond shows the random effects meta-analysis estimate. The weight is from the random effects meta-analysis, with studies with smaller standard errors given larger weight.
Overall, the effects of public works on participants' employment are mixed. On the one hand, they seem promising as most of the studies (8 out of 11) find positive and significant effects on employment at some point in time. Consequently, the overall impact of public works programs on employment is 9.8 percentage points (95 percent CI: 1.9, 17.8). On the other hand, the average employment impacts during the program are 29.3 percentage points. This can be seen as quite low: it means that, for every 100 individuals taking on jobs, 71 would have also worked in the absence of the program. As such, while public works programs do increase the share of individuals employed in the short run, their net effect on the number of participants employed is smaller than the number of positions they offer. In addition, the evidence shows that public works programs do not produce much sustained impacts on employment: the overall impact after the programs end is small (3.4 percentage points) and not significant (p=0.14), with only three of the nine point estimates beyond one-month post-intervention revealing a positive impact. The overall medium-run (i.e., post-program) effect is an increase of 7.8 USD (95 percent CI: -1.3, 16.8), which is significant at the 10 percent level (p=0.09). Whether a study shows significant earnings gains after the program seems associated with participants' ability to acquire savings or productive assets for self-employment or business activities, which we will discuss further below.

Figure 4: Estimates of the impact of public works programs on monthly earnings
Notes: This figure presents the intent-to-treat effects on monthly earnings obtained from each study and follow-up survey. The effect size is the change in monthly earnings (in USD PPP 2021). Timing indicates the timing of the survey in months before/after program completion, where 0 indicates the time when the program was completed. The control mean is the mean of the study control group. The diamond shows the random effects meta-analysis estimate. The weight is from the random effects meta-analysis, with studies with smaller standard errors given larger weight.
From Figure 4, we can also examine the effect on monthly earnings relative to the monthly maximum possible value of earnings in the public works programs. The magnitude of forgone earnings is key for examining the cost-effectiveness of public works programs (e.g., Bertrand et al., 2021). At one extreme, if there are no forgone earnings, we would expect the program impacts on individuals' earnings during the program to be 100 percent of the value of public works wages. At the other extreme, if individuals fully substitute income from other sources for public works earnings, we expect impacts on individuals' earnings to be 0 during the program. As discussed above and documented in Figure 4, we see positive impacts on earnings in most cases during the program, ruling out a full substitution of income sources. However, Figure 5 below shows that the impacts on earnings are substantially below the maximum possible value of monthly earnings during the program, which points to large forgone earnings for participants. The ratio of impacts on earnings relative to the maximum potential monthly earnings during the program is 23 percent in Comoros, 54 percent in Côte d'Ivoire, 70 percent in Lao PDR, and 36 percent in Djibouti. On average, impacts on earnings are 46 percent of maximum potential program earnings, pointing to substantial forgone earnings. 19 The impacts on earnings relative to the maximum potential program earnings also decline over time and are low after the program, with an average of 7 percent (significant at the 10 percent level, p=0.09). While lower impacts on earnings relative to the value of program earnings are expected after the program, the rate at which impacts decay is key for calculating cost-effectiveness, because this affects the net present value of program impacts on earnings over time. Overall, the fact that effects on earnings are much smaller than the maximum possible earnings from public works during the program and quickly fade over time raises questions about the overall cost-effectiveness of public works programs. The results also suggest variations between programs, though few studies, aside from those in Côte d'Ivoire and Sierra Leone, provide the detailed information on program costs sufficient to conduct a full cost-effectiveness analysis.
Overall, the results show that the public works programs do lead to changes in earnings in the short run, but that these effects tend to fade in the medium run. In addition, the effects on earnings are small relative to the transfer amounts during the program, raising questions about cost-effectiveness.

Figure 5: ITT effects on monthly earnings relative to maximum monthly earnings
Notes: This figure presents the intent-to-treat effects on monthly earnings expressed relative to the monthly maximum possible earnings from the public works program. Timing indicates the timing of the survey in months before/after program completion, where 0 indicates the time when the program was completed. Control mean is the mean of the study control group. The diamond shows the random effects meta-analysis estimate. The weight is from the random effects meta-analysis, with studies with smaller standard errors given larger weight.

Food consumption expenditure
Public works programs are often conceived as a safety net intervention that can provide income support and facilitate consumption smoothing for poor households, particularly in the short run during program participation. Hence, we examine if the increases in earnings for participating individuals are associated with improvements in household food consumption expenditure. Figure 6 (Beegle et al., 2017). The null result on food consumption, despite the positive effects on employment and earnings, could mean that households are using the earnings to save or make productive investments, or that effects on individual participants' earnings are not sufficient to improve household-level welfare outcomes.

Figure 6: Estimates of the impact of public works programs on food consumption expenditure
Notes: This figure presents the intent-to-treat effects on monthly food expenditure obtained from the different papers. The effect size is the change in monthly food consumption expenditure (in USD PPP 2021). Timing indicates the timing of the survey in months before/after program completion, where 0 indicates the time when the program was completed. Control mean is the mean of the study control group. The diamond shows the random effects meta-analysis estimate. The weight is from the random effects meta-analysis, with studies with smaller standard errors given larger weight.

Savings and assets
In Figures 7 and 8, we present the estimated effects of public works programs on the probability of having any savings in the 12 months prior to the survey and a household asset index, respectively. 22 Savings and assets can be used to smooth consumption in case of a shock or to facilitate investments and increase income from agricultural and selfemployment activities. They are both important mechanisms through which the effects of the public works program could sustain in the medium run.
The observed increases in earnings 12 months after program completion (shown in Figure 4) are associated with a higher incidence of savings 23 (Rural DRC, Tunisia, and Côte d'Ivoire) and higher assets (Rural DRC and Tunisia). The short-run increase in earnings in Côte d'Ivoire stems from higher wages from the public works program.
However, the impacts 12 months post-program are due to higher profits from selfemployment activities. Public works wages were paid into a bank account, leading to a substantial increase in savings during the program that were partly used to invest in capital for these small business activities (Bertrand, et al., 2021). Similarly, in Rural DRC, the 12month impacts are associated with an increase in self-employment (by 8.5 percentage points), number of total hours worked (increase of almost 25 percent relative to the control mean), and increased ownership of productive assets such as livestock and physical assets such as refrigerators, televisions, air conditions, and motorbikes, that increased by 50 percentage points and 43.2 percentage points, respectively (Mani and Mvukiyehe, 2021).
Even though the Ethiopia study does not report any information on the likelihood of saving, households in Ethiopia also used the earnings from public works to save, with eligible households doubling their savings post-program.
Overall, the probability of having savings increases by 5.9 percentage points (95 percent CI: 0.60, 11.20), and the household asset index increases by 0.21 standard deviations (95 percent CI: 0.04, 0.37). 22 Note that, compared to employment and earnings, we slightly have less consistent measures on savings and assets across studies. 23 An increase in the amount of savings was also documented in Côte d'Ivoire and Ethiopia. Control mean is the mean of the study control group. The diamond shows the random effects meta-analysis estimate. The weight is from the random effects meta-analysis, with studies with smaller standard errors given larger weight.

Figure 8: Estimates of the impact of public works programs on a household asset index
Note: This figure presents the intent-to-treat effects on a household asset index obtained from the different papers. The effect size is the change in the asset index (in standard deviation units). Timing indicates the timing of the survey in months before/after program completion, where 0 indicates the time when the program was completed. Control mean is the mean of the study control group. The diamond shows the random effects meta-analysis estimate. The weight is from the random effects meta-analysis, with studies with smaller standard errors given larger weight.

Women's empowerment
Participation in public works can increase women's earned income, which is often limited in the study settings, and thus may improve women's autonomy and empowerment more broadly. The Egypt Community program and the programs in Lao PDR and Djibouti were specifically designed to encourage women's participation. They did so by either targeting only women (Djibouti and Lao PDR) or adjusting jobs to cater to women's comparative advantages or the needs of their families (Egypt Community). For instance, in Djibouti, the program targeted households with pregnant women and mothers with children less than two years old. The sub-projects under the Egypt Community program and Lao PDR entailed tasks that were not considered to be physically strenuous.
Seven of the 11 studies (or 9 of the 15 follow-up surveys) measure impacts on women's empowerment (see Table 5) in the form of increased economic engagement or decision-making in the household. 24 The measures vary across countries and tend to be relatively narrow, and only three of the 9 point estimates document a positive increase (Lao PDR during the program, Egypt Community at one-month post-program, and Tunisia at 12 months). In Lao PDR, impacts were found on measures such as intra-household decisionmaking and community participation. The increase in women's empowerment in Tunisia is driven by a large relative increase in the probability that women respondents have an income-generating activity. The Egypt Community study finds an increase in women's control over household resources 1-month post-program, but this effect does not persist and there is a reduction (which is not statistically significant) in women's control over household economic resources in the medium-run.  24 The Rural and Urban DRC studies also measure views on women's access to power and on gender-based violence and, in Comoros, the study measures women's experiences of physical, emotional, or economic violence in their household. None of these studies, however, finds significant impacts on women's empowerment. In a spin-off study in Lao PDR, no effect was found on gender-based violence (including no evidence of a backlash effect (Perova et al., 2021). 25 There is active research on the topic. For instance, Christian et al. (forthcoming) investigate whether public works can improve women's empowerment when designed specifically to increase women's access to earned income and use a much broader measure of women's empowerment than the studies included in this paper. Notes: These results are shown to reflect the sign of estimates instead of coefficient estimates or effect sizes because of differences in measurement. Point estimates that are not statistically significant are coded as 0. Countries are ordered by survey timing with respect to program completion, i.e., we first present countries that measure impacts during the program, followed by countries that measure 1 to 66 months post-program impacts. The Egypt Community study refers to psychological well-being as subjective well-being. See Appendix Table A3 for country specific variable definitions on women's empowerment and psychological wellbeing.

Conclusion: Lessons for policy and directions for future research
Public works programs in low-and middle-income countries are designed to offer shortrun employment and earnings opportunities to poor households and are often introduced to facilitate recovery from shocks (e.g., drought, conflict). These programs frequently also aim to sustainably raise living standards via, for example, improvements in assets, selfemployment, savings, and community infrastructure. This paper reviews experimental evidence from 11 public works programs that together offer both internal and external validity across diverse settings. Some important conclusions and policy lessons emerge from the review.
First, public works programs generate short-run improvements in employment and earnings. In some cases, the magnitude of impacts on participants' employment during the program can be large (such as in Comoros or Djibouti). Still, impacts on earnings tend to be small relative to the total program earnings, even during the program, raising questions about program cost-effectiveness. Second, increases in employment and earnings generally fade and do not persist much after individuals have exited the program. A few notable exceptions are found for programs that generate impacts on savings or productive investments linked to self-employment activities (such as in Rural DRC or Côte d'Ivoire).
Third, and perhaps surprisingly, there is little evidence of public works programs increasing food consumption expenditures across the studies we review. Finally, there is some evidence of improvements in women's empowerment and psychological well-being, though these are not found consistently, perhaps due to limitations in measurement.
Overall, the stylized message from the review is that public works programs have been relatively more successful in delivering on their short-run objectives to raise individuals' employment and earnings, rather than in achieving medium-run impacts on participants' economic outcomes or broader effects on household-level outcomes. These results also raise questions on how to design programs to achieve more robust impacts on outcomes related to broader policy objectives, for instance women's empowerment or psychosocial well-being.
Despite the rapid increase in public works experiments in low-and middle-income countries, much remains to be learned. First, we need better data to accurately measure program participation (ideally from administrative information systems) and outcomes at multiple points in time. None of the existing studies measures program take-up and participation and outcomes related to employment, consumption, assets, psychological well-being, and women's empowerment in the short-run (during the program) and after the program. Second, the duration of program participation and the level of wages vary widely across countries. It is unclear what the optimal program length or wage should be, but these parameters can affect impacts on short-run earnings, consumption, savings and investments, and those may in turn affect outcomes in the medium term. Third, it would be valuable to identify effective "public works plus" interventions that can facilitate transitions from temporary public works programs to more steady forms of employment, including in self-employment. Currently, there is very little evidence on the value-added of skills training or savings intervention in facilitating the transition from public works. In particular, given the recent success of multi-faceted economic inclusion programs (e.g., Banerjee et al., 2015, Bedoya et al., 2019Bossuroy et al., 2022) in poor settings, it would be useful for future studies to experiment with different modalities and bundles of savings facilitation, capital transfers, or skills interventions (e.g., vocational, socioemotional, or business training) as complements to the public works program. 26 Fourth, additional research using broader measures of women's empowerment is also critical to better understand intra-household dynamics, such as the conditions under which improvements in women participants' individual earnings improve household consumption and their decision-making power. Fifth, there is a need for more rigorous cost-effectiveness analysis.
Only two studies in the literature offer detailed cost-effectiveness analysis (Bertrand et al., 2021 andRosas et al., 2016). Future experiments in this area must pay close attention to collecting cost data to allow for a clear comparison between marginal benefits and marginal costs of the intervention. Lastly, while our review has not focused on general equilibrium effects, the value of public goods provided, or other social externalities from public works programs, are also first-order questions on which more evidence would be needed across settings. 26 In Côte d'Ivoire the basic public works program induced substantial savings, possibly facilitated by the payment of wages into bank accounts. Some participants were also offered a complementary basic entrepreneurship training to facilitate the transition to self-employment activities after their exit from the program, and other participants were offered a wage employment sensitization training to help them prepare and apply for wage jobs. Post-program impacts on being employed, total earnings, and the number of hours worked were not statistically different across the different treatment arms. Overall, the changes in skills and practices were small in magnitude and did not generate earnings beyond those induced by the basic public works program, suggesting limited value-added of the complementary interventions. In contrast, in Urban DRC, participants that received a combined package of public works programs, incentivized savings, and hard skills training experienced large impacts on savings and investment, indicative of medium-run improvements. Thus, the results suggest that a multifaceted approach may be most effective in supporting livelihoods in Urban DRC, where participation in the public works program alone did not significantly improve economic welfare outcomes in the medium run.  Index constructed using two measures of women's participation in household decisions: (1) whether women make decisions alone and (2) whether women make decisions jointly with other members.
In the first follow-up, women are asked: Did you alone decide how to use the money you earn? In the second follow-up, the question changes to: In your household, did you alone decide how to use the money you earn?
Index constructed based on: (1) whether women report any earned income over the past six months, (2) whether women alone decide how income will be used, (3) whether men decide alone how income will be used, and (4) whether women report any income-generating activity.
Measured as women's participation in household decision making and their voice in the communities Index constructed based on: (1) whether, in the household, the woman (also) makes decisions about how to use money; (2) views on women's access to power as measured by three statements (on the same rights and duties; the same chance to hold socio-administrative positions; and on their eligibility to sit as presidents of local management committees); (3) views on gender-based violence as measured by justification of wifebeating in eight specific situations.