What active labour market programmes work for immigrants in Europe? A meta-analysis of the evaluation literature

A growing body of programme evaluation literature recognises immigrants as a disadvantaged group in European labour markets and investigates the employment effects of Active Labour Market Programmes (ALMPs) on this subgroup. So far, however, there is no systematic review establishing which ALMPs are effective for immigrants. Using a meta-analysis, we condense 93 estimates from 33 empirical studies of the effectiveness of four types of ALMPs employed across Europe to combat immigrant unemployment: training, job search assistance, wage subsidies and subsidised public sector employment. We find that only wage subsidies can be confidently recommended to European policy-makers.


Introduction
Immigrants are under-represented in employment and over-represented in unemployment in most European countries. In 2009/2010 their employment rate was on average 2.9 percentage points lower than that of natives across all European OECD countries. At the same time their unemployment rate was 4.3 percentage points higher (see OECD, 2012).
To facilitate immigrants' labour market integration, European governments use a wide range of Active Labour Market Programmes (ALMPs). These include language and introduction courses, job search assistance, training programmes, and subsidised public and private sector employment and involve substantial government spending. While evidence on the effectiveness of single programmes exists, it has not been reviewed systematically to establish empirically which types of programmes actually facilitate immigrants' employment uptake. To shed light on this question we review the small but growing literature evaluating the employment effects of ALMPs on immigrants in Europe. By means of a meta-analysis, we try to identify which ALMPs work for immigrants and which ones do not. Our results should help policymakers employ activation measures more efficiently.
With respect to ALMPs' effects on all unemployed workers (natives and immigrants), recent analyses have strengthened a growing consensus: job search assistance and, to some extent, wage subsidies are effective in the short run while training works in the longer run; subsidised public sector employment (also known as public works), however, is generally ineffective (Heckman et al., 1999, Greenberg et al., 2003, Kluve, 2010, Card et al., 2010. Also, the findings of the ALMP evaluation literature on heterogeneous treatment effects on women or young workers have been reviewed (e.g., Bergemann andvan den Berg, 2008, Card et al., 2010).
For immigrants, only two surveys of the literature on the effect of ALMPs exist. Nekby (2008) provides a qualitative review of four studies evaluating labour market programmes for immigrants in the Nordic countries; she concludes that the same types of ALMPs work for immigrants as for the general population of unemployed workers. Rinne (2012) discusses the findings of three studies evaluating language/introduction courses designed for immigrants and eight recent evaluations of general labour market programmes' effects on immigrants. In a similar vein as Nekby, he suggests that "programs that are relatively closely linked to the labor market (e.g., work experience and wage subsidies) appear as the comparatively most effective programs" (Rinne, 2012, p 19). While both surveys present relevant evidence, neither of them formally aggregates the findings of the studies reviewed.
We provide an accessible quantitative summary of the existing empirical evidence. To this end, we collect the relevant studies following a search protocol and then condense the findings extracted from them in two steps: first, using descriptive analysis and second, performing a meta-analysis with sign and significance of the effect estimate as our outcome variable of interest.
As Stanley (2001) argues, "The most important strength of meta-analysis is that it moves literature reviews away from casual judgments about "good" studies that deserve attention and "poor" studies that should be set aside, and instead provides a replicable statistical framework for summarizing and interpreting the full range of evidence." The key ingredient of such an analysis is then an exhaustive data set of relevant studies. We find 33 micro-econometric papers that estimate 93 short-run treatment effects up to two years after programme start. The interventions evaluated were implemented in the Nordic countries (Norway, Sweden, Denmark, and Finland), Germany, the Netherlands and Switzerland between 1984 and 2007.
Our descriptive analysis looks at the distribution of impact estimates conditional on study characteristics in order to provide an absolute indication of the effectiveness of different types of ALMPs. Performing a meta-analysis of the same sample of effect estimates allows us to go beyond the descriptive analysis in two ways. First, we can control for different study characteristics when investigating which ALMP types are associated with significant or insignificant impact findings, simultaneously addressing such issues as methodological differences or changes in programme effectiveness over time. Second, we can provide a summary measure for whether the evidence suggests that one type of ALMP works better than some other.
We find that subsidised employment in the private sector is significantly more likely estimated to have a positive effect on immigrants' labour market outcomes than training. For the other ALMP types, our meta-analysis yields mostly insignificant results. The descriptive analysis shows that evaluations of training and job search assistance programmes produce predominantly insignificant effect estimates for immigrant participants. Public works seem to perform even worse, receiving many insignificant and negative evaluation results. Therefore, only subsidised employment in the private sector seems to be effective for immigrants.
The remainder of this paper consists of four parts: section 2 provides background information on immigrants on European labour markets; section 3 describes the data and presents some descriptive analysis; section 4 discusses the findings of our meta-analysis and performs a sensitivity check; section 5 concludes.

Immigrants in Europe and their labour market integration
On average, the share of immigrants (defined as foreign-born persons) among the total population amounted to 11.2% in European OECD countries in 2009/2010 (see Table A.1 in the appendix). 1 In almost every country, the share of immigrants in the working age population (age 15 to 64) is even larger than in the total population. It amounts to 13.3% on average across European OECD countries.
Despite substantial heterogeneity in immigrant origins, European countries share the problem of integrating immigrants into the labour market. Immigrants are usually under-represented in employment and over-represented in unemployment. Table 1 shows the employment and unemployment rates of immigrants in our sample of seven European OECD countries in 2009/2010 and how they compare to the respective rates of the native populations. On average, the employment rate for immigrants in these countries is 65.8%. It is 9.3 percentage points lower than the rate for natives. Differences in the employment rate are especially pronounced in Denmark, the Netherlands and Sweden, with a difference of more than 10 percentage points. Correspondingly, unemployment is more prevalent among immigrants than among natives. Immigrants' unemployment rate is 11.6% on average. It is more than twice as high as the natives' unemployment rate (5.4%). Across all European OECD countries, differences in the employment and unemployment rates between immigrants and natives are smaller than in the seven countries investigated here but still sizeable: 4.3 percentage points for unemployment and 2.9 percentage points for employment.
Include Table 1 about here To combat the high level of unemployment among immigrants and to foster their employment uptake governments use Active Labour Market Programmes (ALMPs). For immigrants, two different categories of ALMPs can be distinguished: first, programmes that are specifically 1 Note that this figure includes only the first generation of immigrants. Unfortunately, comparable data on immigrants across European countries including the second or third generation are not available. designed for and exclusively targeted at immigrants, and second, general programmes that are also used for the native population. In what follows we will refer to these categories as migrant-specific and general ALMPs, respectively.
General ALMPs comprise four types of interventions (see, e.g., Card et al., 2010): 1) Training: This includes all programmes that aim to enhance participants' skills needed for employment uptake (e.g. computer courses or courses providing specific occupational knowledge). Training programmes can be provided either on-the-job within a firm or off-the-job in a classroom.
2) Subsidised private sector employment: This category comprises programmes that generate incentives to increase job opportunities in the private sector. One example for such a programme is wage subsidies for employers who hire disadvantaged workers. Wage subsidies can also be paid to workers when they accept a job with a wage below their unemployment benefits or when they start their own business.
3) Subsidised public sector employment (public works): This type of intervention aims at offering temporary job opportunities outside the private sector, mainly for community services. Public-works programmes should be designed in such a way that they do not crowd out regular employment.

4) Job search assistance and sanctions:
This intervention type has the objective of making the job search process of participants more effective and efficient.
Job search assistance is predominantly provided by public employment services and includes counselling and monitoring of job search efforts. In case of a lack of job search effort, sanctions are intended to restore an appropriate level of compliance.
Migrant-specific programmes can be grouped into three categories: 1) Language training often not only improves participants' ability to communicate in the host country's main language but also provides information about history, culture and institutions of the host country. One example for such a course is the so-called orientation course in Germany (see, e.g., Liebig, 2007) 2) Introduction programmes provide a customised integration plan towards employment uptake. Targeted at newly arriving immigrants, they usually start with language training and continue with either training or subsidised em-ployment. Throughout the programme job search assistance is provided. See, e.g., Andersson Joona and Nekby (2012) and Sarvimäki and Hämäläinen (2012) for introduction programmes in Sweden and Finland, respectively.
3) General programmes exclusively for immigrants comprise general ALMPs other than language courses (training, subsidised private or public sector employment, job search assistance and sanctions) targeted at immigrants (and not at natives). One example for such an intervention is intensified job search assistance programmes, where immigrants are assigned to caseworkers whose caseload is reduced. That is, caseworkers have more time for the counselling and support of each individual. See, e.g., Aslund and Johansson (2011) for a programme of this kind in Sweden.
Whether general programmes or migrant-specific ones are more effective for the integration of immigrants in the labour market is a question of major policy interest. One might expect migrant-specific programmes to be more successful since they are designed for the needs of immigrants whereas general programmes address the needs of average participants, including mostly natives. However, the fact that in practice both programmes coexist in all European countries might be taken to suggest that neither of them is superior or that policymakers are not aware of which programmes work and which ones do not. 2 Empirical studies have not established an answer to this question either. We attempt to address it by means of our metaanalysis (see section 3).

Estimation Sample
To obtain an exhaustive sample of studies evaluating the effects of ALMPs on immigrants' labour market outcomes, we implemented the following search protocol: 1) Collect studies on ALMPs surveyed by Nekby (2008) We then identified those studies that met the following selection criteria: 1) Studies that estimate ALMP treatment effects for immigrants. 4 2) Studies that perform a micro-econometric evaluation of the intervention's effect on individual labour market outcomes, outlining the identification strategy.
3) Studies that evaluate an intervention that roughly fits into one of four ALMP categories (described more fully below): training, wage subsidy, public works, or services/sanctions. We also admitted studies that evaluate the aggregate effect of a country's ALMPs.
Applying these criteria yielded a sample of 34 studies estimating ALMP effects on immigrants' probability of or hazard to employment 5 in seven countries (Denmark, Finland, Germany, Netherlands, Norway, Sweden, Switzerland). Some studies evaluate several programmes or perform their analyses separately by gender or region as well as estimating effects for different points in time during follow-up. For comparability, we focus on short-run estimates, defined as effect estimates based on outcomes observed up to two years after programme start 6 . Where there is more than one such short-term estimate per gender-regiongroup combination, we choose the latest (most long-term) one. This gives 33 studies providing 93 short-run estimates. 7 The four ALMP categories we use are as follows: 3 Keywords, in different combinations: ALMP, labour market programmes, labor market programs, migrants, foreign, native, born, citizen, subgroup, sub-group, hetero; search engines used: Google Scholar, EconPapers and Econis. 4 The definition of immigrants varies across studies. It usually means those with foreign citizenship, the foreignborn or individuals whose parents or grandparents were foreign-born. Most studies estimate heterogeneous ALMP treatment effects for several subgroups, including immigrants. A few studies have a sample of only immigrants. 5 Instead of looking at employment, one study considers earnings. Another, evaluating the promotion of selfemployment, uses yet a different outcome variable: neither unemployed nor in receipt of unemployment. 6 While we do have information on longer-run outcomes (38 estimates), there is not enough variation in them to permit a separate econometric analysis. One study reports only long-term estimates for 36 and 50 months after the programme (Groß et al., 2006). This study is dropped from the analysis. 7 See the references for the list of the 33 studies analysed. 1) Classroom and on-the-job training (henceforth "training") 2) Subsidised private sector employment ("wage subsidy") 3) Subsidised public sector employment ("public works") 4) Job-search assistance and sanctions ("services/sanctions") These are taken from Card et al. (2010) but are fairly standard in the evaluation literature, as exemplified by analogous definitions in Calmfors (1994) and Kluve (2010). We allow for a fifth residual group of "other programmes" (including aggregate ALMP effects and programmes that combine several ALMP types in a single treatment). See also section 2 for a definition of the ALMP categories. 8 From our sample of 33 studies we extracted information about the programme evaluated and its geographic and chronological setting, the sample studied and the methods applied. We recorded programme type, duration and whether it was designed specifically for immigrants in order to characterise the nature of the treatment. To capture sample characteristics, we included information on whether an effect was estimated for males, females or a mixed group of participants as well as in what country and decade they received the treatment. As methodological proxies we documented the econometric technique used and whether the estimates came from a published paper or a working paper. 9

Summary Statistics
The first column of Table 2 summarises the distribution of the short-run estimates we focus on. First, consider the outcome variable: those evaluations finding no effect are most frequent (48 estimates), followed by ones finding significantly positive effects (32), with significantly negative effect estimates less frequent still (13 estimates).
Include Table 2 about here 8 Since only 6 out of the 33 studies analyse migrant-specific programmes, our econometric analysis cannot differentiate between these types of ALMP as outlined in section 2. Instead we define a dummy variable to indicate whether a programme is migrant-specific or general. We then classify migrant-specific programmes as training, wage subsidy, public works, services/sanctions or other programme, depending on their content. 9 We categorised PhD dissertations as published studies because of the similarities between PhD supervision and the referee process.
Next, lines 2a) to 2e) show that among ALMP types, training programmes dominate (30 estimates). Public works also feature prominently, contributing 23 data points; the third column reveals that this pattern is driven by evaluations for Germany. Both wage-subsidy (16) and services/sanctions (17) each provide about half as many observations as the largest category.
There are seven estimates in the residual category (other programmes). Only six of our 93 estimates are for migrant-specific programmes, whereas 87 are for general ones (see lines 2f) and 2g), respectively).
Lines 3a) through 3c) show that 25 estimates are for programmes with a duration of up to four months, while 20 effects are estimated for programmes of at least five months. However, most short-run estimates are for programmes of unknown or mixed duration (48), reflecting some heterogeneity in the level of detail on interventions given in the papers. Lines 4a) through 4c) reveal that about two thirds of estimates are from the 2000s. Next, lines 5a) through 5c) illustrate that matching approaches were the most popular method (55 estimates).
From column 3 it is clear that German estimates, based on matching procedures with only one exception, account for this distribution. Finally, lines 6a) and 6b) demonstrate that less than a third of the short-run estimates came from published papers (28), with Nordic evaluations accounting for disproportionately many publications (20) 10 .
Comparing estimates by origin reveals that the largest contributor, Germany, differs markedly from Denmark, Finland, Norway and Sweden (Nordic countries) and Switzerland and the Netherlands (other countries). Nordic estimates are relatively optimistic about programme effects while most German estimates are insignificant and effects tend to be more often negative in the other countries. There is more variety in the methods used to evaluate Nordic and Swiss/Dutch programmes than for German programmes. While training is the ALMP type evaluated most frequently in the Nordic countries and Switzerland/Netherlands, evaluations of public works dominate in Germany. 10 We do not intend to suggest that published papers meet different quality standards than working papers, given that we are agnostic about the relative quality of the various refereed journals and opt for estimates from working papers in some cases where the published version no longer presents all heterogeneous effect estimates, e.g., Gerfin and Lechner (2000).

Descriptive Analysis
In this subsection we present the distribution of the outcome variable (short-run effect: significantly negative, insignificant or significantly positive) conditional on the covariates we extracted from the studies (see Table 3). This serves a dual purpose: one is to provide a flavour of the potential results of the meta-analysis; another is to give some absolute indications of the effectiveness of the programme types evaluated. This is important because our metaanalysis, by virtue of its method, only allows conclusions about the relative effectiveness of different types of programme.
Include Table 3 about here Lines 1a) to 1e) of Table 3  These raw descriptive statistics indicate that wage subsidies seem to have positive employment effects; for the remaining ALMP categories, the evidence mostly points to an employment effect too close to zero to be significant. Because only six of the 93 estimates are for migrant-specific programmes (see lines 1f) and 1g), respectively), it is hard to draw reliable conclusions about their relative effectiveness. We effectively focus on the effect of general ALMPs on immigrants.
The next three lines, 2a) to 2c), seem to suggest that duration analysis is more optimistic about programme effectiveness than matching approaches are. Similarly, lines 3a) to 3c) may point to a deterioration of ALMP quality over time from the 1980s to the 2000s. While programme duration is unknown in most cases, short programmes may have been more effective than longer ones. Finally, published papers seem to find positive effects in a higher fraction of cases. Yet, regional and chronological differences may be confounding all of these potential relationships; multivariate analysis will help disentangle these effects.

Method
We perform an ordered probit analysis with sign and significance of the estimate as the outcome variable. This variable can take three values: significantly negative, insignificantly different from zero, and significantly positive. The explanatory variables of interest are dummies describing the type of ALMP. In addition, we include a number of variables to account for differences in evaluation technique and setting. We focus on the relationship between ALMP type and sign/significance of the short-run effect estimated for each study-gender-region cell.
The index model underlying our estimation is as follows: where , , and are dummy variables describing the programme type analysed in study (wage subsidy, public works, services/sanctions, or other programmes, with training being the omitted category), is a vector of control variables (study characteristics, programme characteristics, sample characteristics, contextual controls) and is an error term.
Taking into account the degrees of freedom underlying each effect estimate as well as a measure of effect size (such as a t-statistic) would be an attractive alternative to the sign/significance approach we take (see, e.g., Greenberg et al., 2003 andStanley, 2005). It would allow us to identify systematic differences in estimated effect size across types of programme, unconfounded by differences in precision. However, the plurality of the underlying econometric estimation techniques makes this unfeasible. 11 Card et al. (2010) show that the approach we follow would be invalid if the pattern of estimate sign and significance were generated by differences in precision rather than differences in effect size. They also demonstrate that the sign/significance approach is approximately valid when the effective sample size is constant, i.e., when larger samples are offset by more demanding designs. They present evidence that this is the case in their sample of studies and indeed find that the sign/significance approach and an effect size-based analysis on a subsample of studies that use the probability of employment as outcome variable yield similar results.
11 It is not straightforward how test statistics from different types of models can be transformed into a common distribution so that the test statistics can be compared directly. For instance, test statistics from a duration model and a matching model will have different distributions (and degrees of freedom).
While we cannot perform such a check for our smaller set of studies, we can partly rely on their finding in that there is some overlap between our samples of evaluation studies.
Recent theoretical work on meta-analyses has stressed the importance of checking for publication bias (see, e.g., Stanley, 2005). We cannot rule out publication bias as the test statistics of the studies we analyse are not directly comparable. In this paper, however, our focus is the relative effectiveness of different types of ALMPs. As long as the presence of publication bias does not interact with the type of programme evaluated, it will not distort our findings on the relative effectiveness of different types of ALMPs.

Estimation results
We estimate six specifications of the ordered probit model outlined above, gradually introducing groups of control variables. Specification 1 includes only the type of programme, omitting training. Specification 2 adds study characteristics: whether the study employed duration analysis or some other econometric technique (omitted: matching), and whether the paper is published (baseline: working paper). Specification 3 introduces programme characteristics, namely whether the intervention was designed for immigrants and whether the treatment was short, that is, no longer than four months. In specification 4, sample characteristics enter the equation: participant gender (baseline: pooled estimation for men and women) and treatment in the 2000s (omitted: 1980s or 1990s). Specification 5 adds the unemployment rate in the year that the evaluated programme started as a proxy for the macroeconomic context. Alternatively, specification 6 uses GDP growth as a contextual control. 12 Table 4 presents the results. In specification 1, which includes only programme type, we obtain positive coefficients for wage subsidy and services/sanctions. The interpretation for these positive coefficients is that studies evaluating wage subsidies and services/sanctions are more likely to find positive employment effects than studies evaluating training. However, the estimated coefficients are not statistically significant. The coefficients on public works and other programmes are negative, though only the coefficient for other programmes is marginally significant. There is no meaningful interpretation for the coefficient on "other programmes" as this is a residual category.
When including study characteristics (method and publication status) in specification 2, the results remain very similar. None of the ALMP types are significant except for the residual category. Specification 3's programme features cause the wage subsidy coefficient to grow and become significant; all other ALMP type coefficients remain insignificant. Including information about the sample on the right-hand side in specification 4 does little to wage subsidy (still significant) and public works or services/sanctions (still insignificant). When we add contextual control variables in specification 5 (unemployment rate) and 6 (GDP growth rate), the coefficient on wage subsidy remains positive and significant, though only marginally in specification 6. The parameter estimates on public works and services/sanctions are still insignificant.
Based on the Akaike information criterion we choose specification 5 as our preferred one. Include Table 4 about here It is worth re-iterating that our ordered probit analysis only permits relative, not absolute, conclusions on the effectiveness of ALMP types. Thus, our meta-analysis suggests that wage subsidies work better than training. Because the corresponding coefficients are insignificant, we cannot claim with confidence that public works are less effective than training or that ser-  (2008) and Rinne (2012) arrived at in their qualitative reviews.

Robustness analysis
To address the potential criticism that our selection criteria were to some extent arbitrary we vary these criteria and re-estimate our preferred specification on the sample of estimates this gives. This provides a simple check on the robustness of our results. Our variation is to tighten the definitions of ALMPs and of the short run.
First, we drop estimates for programmes that, strictly speaking, are not ALMPs. That is, we exclude evaluation studies of temporary agency work, which like wage subsidy programmes make hiring cheaper and may facilitate employer-worker matching, and exclude transfer reduction programmes, which work much like sanctions. Moreover, we exclude evaluations of aggregate ALMP effects. Second, we define the short run as up to twelve months after programme start rather than 24 months.
This gives an alternative sample of 86 estimates from 27 studies. As Table 5 illustrates, our variation results in a very similar pattern of coefficient estimates, providing evidence for the robustness of our findings (see Table A.2 in the appendix for the full results). In the appendix we also include the results of both main and robustness analysis when standard errors are clustered at the study level 14 . As Tables A.3 and A.4 illustrate, these are practically the same as those presented here.
Include Table 5 about here

Conclusion
Immigrants constitute an important group on European labour markets in terms of both the risks they face and the potential they harbour: they are numerous and over-represented in unemployment on the one hand but younger than the native population on the other. This highlights the importance of immigrants' labour market integration. While the full range of ALMPs is used in practice, there is little empirical guidance for policy-makers seeking to facilitate immigrants' employment take-up. In other words, there is not yet a clear indication of what programmes work for immigrants.
To answer this question, we provide a quantitative synthesis of the evidence on ALMPs' effect on immigrants. Using 93 effect estimates extracted from 33 relevant evaluation studies, we perform a meta-analysis of the evaluation results. An ordered probit analysis based on sign and significance of short-run effect estimates suggests that wage subsidies work better for immigrants than training programmes. Public works may be less effective while job search assistance programmes (services/sanctions) may be more effective but estimated coefficients are insignificant. To help interpret these relative statements, we present a detailed descriptive analysis: there, effect estimates for wage subsidy programmes are mostly positive, suggesting that wage subsidies are indeed a promising measure to increase employment rates of immigrants. They should be used more often than public works which reduce employment chances or result in insignificant employment effects at best. The short-run effects of services/sanctions on employment prospects are mostly insignificant. The same is true for training.
At this point, only wage subsidies can be confidently recommended to policymakers trying using general ALMPs to improve immigrants' labour market integration. The reasons for the 14 Several studies present estimates for the effects of multiple programmes. These estimates are based on different samples of people though. This is why we regard the estimates as independent. It may be argued that author's individual research strategies introduce correlation between multiple estimates of one study -in which case clustering at the study level would be appropriate.
disappointing effects of the other programmes need to be clarified in further research. Furthermore, migrant-specific interventions such as language courses and introduction programmes, on which the evidence is still scarce, may be promising; further research in this area is warranted, too.   The table displays absolute numbers. Short-run estimates are defined as effect estimates based on outcomes observed up to two years after programme start. Where there are more than one such short-term estimates, the latest (most long-term) one is sampled. Nordic countries include Denmark, Finland, Norway and Sweden. Other countries include the Netherlands and Switzerland.  The numbers relate to short-run estimates which are defined as effect estimates based on outcomes observed up to two years after programme start. Where there are more than one such short-term estimates, the latest (most long-term) one is sampled. coefficients of ordered probit models. Heteroskedasticity-robust standard errors in parentheses. The dependent variable takes value 1 for significantly positive estimates, 0 for insignificant and -1 for significantly negative estimates. Unemployment and GDP growth rates are annual rates in % for the year the evaluated programme started. "Other countries" includes Netherlands and Switzerland. *** denotes p < 0.01, ** denotes p < 0.05 and * denotes p < 0.1.  (5) in Table 4. "Variation" tightens the definition of ALMP and the short run, eliminating 6 studies. "Other programmes" includes aggregate ALMP effects and combined programmes. UE is for unemployment rate. *** denotes p < 0.01, ** denotes p < 0.05 and * denotes p < 0.1.   (5) in Table 4. "Variation" tightens the definition of ALMP and the short run, eliminating 6 studies. "Other programmes" includes aggregate ALMP effects and combined programmes. *** denotes p < 0.01, ** denotes p < 0.05 and * denotes p < 0.1. The table displays estimated coefficients of ordered probit models. Standard errors (clustered by study) in parentheses. The dependent variable is an indicator taking value 1 for significantly positive estimates, 0 for insignificant and -1 for significantly negative estimates. Unemployment and GDP growth rates are annual rates in % for the year the evaluated programme started. "Other countries" includes Netherlands and Switzerland. *** denotes p < 0.01, ** denotes p < 0.05 and * denotes p < 0.1.  Table A.3. Variation tightens the definition of ALMP and the short run, eliminating 6 studies. "Other programmes" includes aggregate ALMP effects and combined programmes. *** denotes p < 0.01, ** denotes p < 0.05 and * denotes p < 0.1.