– Ceasefire : The Effects of “ Pulling Levers ” Focused Deterrence Strategies on Crime

Law enforcement agencies use offender-based policing strategies to address crime by focusing resources and efforts on the small percentage of persons committing the greatest percentage of crimes. These “focused deterrence” or "pulling levers" strategies rely on research that has shown that a relatively small number of offenders are responsible for a large number of the crimes that are committed. Focused deterrence/pulling levers strategies are policing strategies that follow the core principles of deterrence theory. Deterrence theory posits that crime can be prevented if potential offenders believe the costs of committing a crime outweigh the benefits, especially if consequences for committing crime are swift and certain. The logic behind these strategies is that violent, gun, and drug crime can be reduced by identifying and targeting these offenders; and once identified, the offenders receive a direct and explicit message intended to deter them from future criminal behavior. This message informs the offender that police, criminal justice stakeholders, and community partners are aware of their illegal activities and if they continue to engage in this behavior, there will be swift and certain legal sanctions. If the targeted offenders respond positively, they are provided with a variety of social service opportunities. If they fail to heed the deterrence message, they are subject to an array of enforcement actions. These enforcement actions are the "levers" that can be "pulled" in response to the offenders' continued criminal behavior.


SEARCH STRATEGY
Several strategies were used to perform an exhaustive search for literature fitting the eligibility criteria. First, a keyword search was performed on an array of online abstract databases. Second, we reviewed the bibliographies of past narrative and empirical reviews of literature that examined the effectiveness of pulling levers focused deterrence programs. Third, we performed forward searches for works that have cited seminal focused deterrence studies. Fourth, we searched bibliographies of narrative reviews of police crime prevention efforts and past completed Campbell systematic reviews of police crime prevention efforts. Fifth, we performed hand searches of leading journals in the field.

DATA COLLECTION AND ANALYSIS
For our ten eligible studies, we complete a narrative review of effectiveness and a formal meta-analysis of the main effects of these programs on reported crime outcomes.

MAIN RESULTS
Based on our narrative review, we find that nine of the ten eligible evaluations reported statistically significant reductions in crime. It is important to note here that all ten evaluations used nonrandomized quasi-experimental designs. No randomized controlled trials were identified by our search strategies. Our metaanalysis suggests that pulling levers focused deterrence strategies are associated with an overall statistically-significant, medium-sized crime reduction effect.

CONCLUSIONS
We conclude that pulling levers focused deterrence strategies seem to be effective in reducing crime. However, we urge caution in interpreting these results because of the lack of more rigorous randomized controlled trials in the existing body of scientific evidence on this approach.

Background
Deterrence theory posits that crimes can be prevented when the costs of committing the crime are perceived by the offender to outweigh the benefits (Gibbs, 1975;Zimring and Hawkins, 1973). Most discussions of the deterrence mechanism distinguish between "general" and "special" deterrence (Cook, 1980). General deterrence is the idea that the general population is dissuaded from committing crime when it sees that punishment necessarily follows the commission of a crime. Special deterrence involves punishment administered to criminals with the intent to discourage them from committing crimes in the future. Much of the literature evaluating deterrence focuses on the effect of changing certainty, swiftness, and severity of punishment associated with certain acts on the prevalence of those crimes (see, e.g. Apel and Nagin, 2011;Blumstein, Cohen, and Nagin, 1978;Cook, 1980;Nagin, 1998;Paternoster, 1987).
In recent years scholars have begun to argue that police interventions provide an effective approach for gaining both special and general deterrence against crime. A series of experimental and quasi-experimental studies have shown that the police can be effective in preventing crime (Braga, 2001(Braga, , 2005Skogan and Frydl, 2004;Weisburd and Eck, 2004), and that such crime prevention benefits are not offset by displacement of crime to areas near to police interventions (Braga, 2001;Weisburd et al., 2006). Durlauf and Nagin have drawn from this literature to argue that "increasing the visibility of the police by hiring more officers and by allocating existing officers in ways that heighten the perceived risk of apprehension consistently seem to have substantial marginal deterrent effects" (2011: 14). Indeed, they conclude that crime prevention in the U.S. would be improved by "shifting resources from imprisonment to policing" (2011: 9-10).
A recent innovation in policing that capitalizes on the growing evidence of the effectiveness of police deterrence strategies is the "focused deterrence" framework, often referred to as "pulling-levers policing" (Kennedy, 1997(Kennedy, , 2008. Pioneered in Boston as a problem-oriented policing project to halt serious gang violence during the 1990s (Kennedy, Peel, and Braga, 1996), the focused deterrence framework has been applied in many U.S. cities through federally sponsored violence prevention programs such as the Strategic Alternatives to Community Safety Initiative and Project Safe Neighborhoods (Dalton, 2002). Focused deterrence strategies honor core deterrence ideas, such as increasing risks faced by offenders, while finding new and creative ways of deploying traditional and non-traditional law enforcement tools to do so, such as directly communicating incentives and disincentives to targeted offenders (Kennedy, 1997(Kennedy, , 2008. In its simplest form, the approach consists of selecting a particular crime problem, such as youth homicide; convening an interagency working group of law enforcement, social-service, and community-based practitioners; conducting research to identify key offenders, groups, and behavior patterns; framing a response to offenders and groups of offenders that uses a varied menu of sanctions ("pulling levers") to stop them from continuing their violent behavior; focusing social services and community resources on targeted offenders and groups to match law enforcement prevention efforts; and directly and repeatedly communicating with offenders to make them understand why they are receiving this special attention (Kennedy, 1997(Kennedy, , 2006. The focused deterrence approach is also consistent with recent theorizing about police innovation, which suggests that approaches that seek to both create more focus in application of crime prevention programs and that expand the tools of policing are likely to be most successful (Weisburd and Eck, 2004).
In the United States, without the support of a formal evaluation, Boston's Operation Ceasefire pulling levers strategy was hailed as an unprecedented success (see, e.g. Butterfield, 1996;Witkin, 1997). These claims followed a surprising large decrease in youth homicide after the strategy was fully implemented in mid May 1996. However, more rigorous examinations of youth homicide in Boston soon followed. A U.S. Department of Justice-sponsored quasi-experimental evaluation of Operation Ceasefire that the intervention was associated with a 63% decrease in monthly number of Boston youth homicides (Braga et al., 2001). Other examinations of violence trends in Boston, however, were more skeptical of the violence prevention value of Boston's approach (see, e.g., Fagan, 2002;Rosenfeld et al., 2005).
Since the publication of the Boston Ceasefire evaluation, several evaluations of similar focused deterrence strategies implemented in other U.S. cities, such as Chicago (Papachristos et al., 2007), Los Angeles , and Indianapolis (McGarrell et al., 2006), have been published. These evaluations reported similar noteworthy crime reduction effects associated with the focused deterrence approach. The basic principles of the focused deterrence approach have also been applied to overt drug market problems (Kennedy, 2009) and repeat offending by substanceabusing probationers (Hawken and Kleiman (2009) with positive crime prevention gains reported.
Given the growing popularity of pulling levers focused deterrence strategies in the United States, as well as the conflicting views on its crime prevention value, a systematic review of the empirical evidence on the effects of focused deterrence interventions on crime is necessary to assess the value of this approach to crime prevention.

Objectives
The objective of this review is to synthesize the existing published and nonpublished empirical evidence on the effects of pulling levers focused deterrence strategies on crime and to provide a systematic assessment of the preventive value of this approach. Often, these new approaches represent a specific application of deterrence strategies within a problem-oriented policing framework (Kennedy, 2006;Braga, 2008a; see also Goldstein, 1990;Eck and Spelman, 1987). In short, many focused deterrence strategies use the iterative problem-oriented policing process (scanning, analysis, response, and assessment) to frame an interagency response to deter groups of chronic offenders from continuing their ongoing violent conflicts. More recently, however, these strategic applications of focused deterrence principles have been applied to a wider range of problems such as overt drug markets (Kennedy, 2009). This review will include this wider range of focused deterrence strategies.

Types of Studies
Only studies that used comparison group designs involving before and after measures were eligible for the main analyses of this review. In several well-known pulling levers focused deterrence evaluations (e.g. Braga et al., 2001;McGarrell et al., 2006), the comparison group experienced routine modern police responses to crime that includes a blend of traditional police responses (e.g., random patrol, rapid response, and ad-hoc investigations) and opportunistic community problemoriented responses. While strategic interventions developed from community and problem-oriented policing initiatives may be present in certain control areas, none of the comparisons implemented focused deterrence strategies to address crime problems. The comparison group study had to be either experimental or quasiexperimental (nonrandomized) (Campbell and Stanley, 1966;Cook and Campbell, 1979).

Types of Units of Analysis
Pulling levers strategies attempt to influence the criminal behavior of individuals through the strategic application of enforcement and social service resources to facilitate desirable behaviors. However, existing reviews of the crime prevention value of pulling levers strategies have noted that published evaluations only report aggregated measures of underlying levels of criminal behaviors in targeted areas (Wellford et al., 2005). For example, in Boston, the pulling levers intervention targeted violent behavior among gang-involved offenders. The evaluators measured behavioral change among gang-involved offenders by examining city-wide trends in aggregated measures of serious violence (Braga et al., 2001).

Types of Interventions
To be eligible for this review, interventions had to be identified as a focused deterrence strategy. As described by Kennedy (2006: 156-157), pulling levers operations have tended to follow this basic framework: • Selection of a particular crime problem, such as youth homicide or street drug dealing.
• Pulling together an interagency enforcement group, typically including police, probation, parole, state and federal prosecutors, and sometimes federal enforcement agencies. • Conducting research, usually relying heavily on the field experience of frontline police officers, to identify key offenders -and frequently groups of offenders, such as street gangs, drug crews, and the like -and the context of their behavior. • Framing a special enforcement operation directed at those offenders and groups of offenders, and designed to substantially influence that context, for example by using any and all legal tools (or levers) to sanction groups such as crack crews whose members commit serious violence. • Matching those enforcement operations with parallel efforts to direct services and the moral voices of affected communities to those same offenders and groups. • Communicating directly and repeatedly with offenders and groups to let them know that they are under particular scrutiny, what acts (such as shootings) will get special attention, when that has in fact happened to particular offenders and groups, and what they can do to avoid enforcement action. One form of this communication is the "forum," "notification," or "call-in," in which offenders are invited or directed (usually because they are on probation or parole) to attend face-to-face meetings with law enforcement officials, service providers, and community figures.
We used this basic framework to assist in our determination whether particular programs engaged the focused deterrence approach. It is important to note here, however, that certain programs that were determined to be eligible for this review did not necessarily follow the very specific pulling levers steps identified by Kennedy (2006). Pulling levers focused deterrence strategies are often framed as problemoriented exercises where specific recurring crime problems are analyzed and responses are highly customized to local conditions and operational capacities. As such, we fully anticipated a variety of pulling levers focused deterrence strategies to be identified by our systematic review.

Types of Outcome Measures
Eligible studies must measure the effects of the focused deterrence intervention on officially recorded levels of crime at places or crime by individuals. Appropriate crime measures included crime incident reports, citizen emergency calls for service, and arrest data. Particular attention was paid to studies that measured crime displacement effects and diffusion of crime control benefit effects. For instance, Kennedy (2006) described a place-based application of pulling levers focused on a disorderly drug market operating in High Point, North Carolina. Crime prevention strategies focused on specific locations have been criticized as resulting in displacement (see Repetto, 1976). More recently, academics have observed that crime prevention programs may result in the complete opposite of displacementthat crime control benefits were greater than expected and "spill over" into places beyond the target areas (Clarke and Weisburd, 1994). The quality of the methodologies used to measure displacement and diffusion effects, as well as the types of displacement (spatial, temporal, target, modus operandi) examined, was also assessed.

SEARCH STRATEGIES FOR IDENTIFICATION OF STUDIES
Several strategies were used to perform an exhaustive search for literature fitting the eligibility criteria. First, a keyword search was performed on an array of online abstract databases (see lists of keywords and databases below). Second, we reviewed the bibliographies of past narrative and empirical reviews of literature that examined the effectiveness of pulling levers focused deterrence programs (Kennedy, 2006(Kennedy, , 2008Wellford et al., 2005;Skogan and Frydl, 2004). Third, we performed forward searches for works that have cited seminal pulling levers focused deterrence studies (Kennedy et al., 1996;Kennedy, 1997;Braga et al., 2001;McGarrell et al., 2006;Papachristos et al., 2007). Fourth, we searched bibliographies of narrative reviews of police crime prevention efforts (Braga, 2008a;Braga, Kennedy, and Tita, 2002;Sherman, 2002;Weisburd and Eck, 2004) and past completed Campbell systematic reviews of police crime prevention efforts (Braga, 2007;Mazerolle et al., 2007;Weisburd et al., 2008). Fifth, we performed hand searches of leading journals in the field. 1 Our searches were all completed between May and September 2010. Thus, our review only covers studies published in 2010 and earlier. Sixth, after finishing the above searches and reviewing the studies as described later, we e-mailed the list of studies meeting our eligibility criteria in September 2010 to leading criminology and criminal justice scholars knowledgeable in the area of pulling levers focused deterrence strategies. These 90 scholars were defined as those who authored at least one study which appeared on our inclusion list, anyone involved with the National Academy of Sciences reviews of police research (Skogan and Frydl, 2004) and firearms research (Wellford et al., 2005), and other leading scholars identified by the authors (see Appendix A). This helped us identify studies the above searches left out as these experts were able to refer us to studies we missed, particularly unpublished studies. Finally, we consulted with an information specialist at the outset of our review and at points along the way in order to ensure that we used The following terms were used to search the fifteen databases listed above: 1) Pulling levers AND police 2) Problem-oriented policing 3) Police AND repeat offenders 4) Police AND gangs 5) Police AND guns 6) Gang violence prevention 7) Focused deterrence 8) Deterring violent offenders 9) Strategic gang enforcement 10) Crackdowns AND gangs 11) Enforcement swamping 12) Drug market intervention

DETAILS OF STUDY CODING CATEGORIES
All eligible studies were coded (see coding protocol attached in Appendix B) on a variety of criteria including:

STATISTICAL PROCEDURES AND CONVENTIONS
Analysis of outcome measures across studies were carried out in a uniform manner and, when appropriate and possible, involved quantitative analytical methods. We used meta-analyses of program effects to determine the size and direction of the effects and weighting effect sizes based on the variance of the effect size and the study sample size (Lipsey and Wilson, 2001). In this systematic review, we used the standardized mean difference effect size (also known as Cohen's d; see Rosenthal, 1994). We employed the Effect Size Calculator, developed by David B. Wilson and available on the Campbell Collaboration's web site, to calculate standardized mean difference effect sizes for reported outcomes in each study. 3

Determination of Independent Findings
We then used Biostat's Comprehensive Meta Analysis Version 2.2 to conduct the meta-analysis of effect sizes. The specific approaches used to calculate effect sizes for each outcome in the eligible studies are described in the meta-analysis section.
One problem in conducting meta-analyses in crime and justice is that investigators often did not prioritize outcomes examined. This is common in studies in the social sciences in which authors view good practice as demanding that all relevant outcomes be reported. For example, the Boston Operation Ceasefire evaluation presents an array of outcome measures including youth homicides, shots fired calls for service, gun assault incidents, and youth gun assault incidents in one high-risk district (Braga et al., 2001). However, the lack of prioritization of outcomes in a study raises the question of how to derive an overall effect of treatment. For example, the reporting of one significant result may reflect a type of "creaming" in which the authors focus on one significant finding and ignore the less positive results of other outcomes. But authors commonly view the presentation of multiple findings as a method for identifying the specific contexts in which the treatment is effective. When the number of such comparisons is small and therefore unlikely to affect the error rates for specific comparisons such an approach is often valid.
We analyze the studies using three approaches. The first is conservative in the sense that it combines all reported outcomes reported into an overall average effect size statistic. The second represents the largest effect reported in the studies and gives an upper bound to our findings. It is important to note that in some of the studies with more than one outcome reported, the largest outcome reflected what authors thought would be the most direct program effect. This was true for the Boston Operation Ceasefire evaluation, which examined a wider range of serious gun violence outcome measures, but assumed that the largest program effects given the intervention would be found in the case of youth homicide incidents (Braga et al., 2001). Finally, we present the smallest effect size for each study. This approach is the most conservative and likely underestimates the effect of pulling levers focused deterrence on crime. We use it here primarily to provide a lower bound to our findings.

SELECTION OF STUDIES
Search strategies in the systematic review process generate a large number of citations and abstracts for potentially relevant studies that must be closely screened to determine whether the studies meet the eligibility criteria (Farrington and Petrosino, 2001). The screening process yields a much smaller pool of eligible studies for inclusion in the review. The four search strategies produced 2,473 distinct abstracts. The contents of these abstracts were reviewed for any suggestion of an evaluation of pulling levers interventions. 93 distinct abstracts were selected for closer review and the full-text reports, journal articles, and books for these abstracts were acquired and carefully assessed to determine whether the interventions involved pulling levers focused deterrence strategies and whether the studies used randomized controlled trial designs or nonrandomized quasiexperimental designs. Ten eligible studies were identified and included in this review:

CHARACTERISTICS OF ELIGIBLE STUDIES
The ten selected studies examined pulling levers focused deterrence interventions that were implemented in small, medium, and large U.S. cities (Table 1). 4 Three of the eligible evaluations (Cincinnati, Nashville, and Newark) were not published at the time the review of abstracts was completed. 5 The two crime reduction strategies focused on individuals represented deviations from the classic pulling levers focused deterrence approach developed in Boston and defined by Kennedy (2006). However, after a careful review of program elements, we determined that the necessary components of an eligible study were present. As an additional check on our selections, we contacted academics associated with each of the two programs and reviewed our decisions with them.
All ten included evaluations that were released after 2000 and seven were completed after 2007. Six studies evaluated the crime reduction effects of pulling levers strategies on serious violence generated by street gangs or criminally-active street groups (Boston, Cincinnati, Indianapolis, Los Angeles, Lowell, and Stockton). Two studies evaluated strategies focused on reducing crime driven by street-level drug markets (Nashville and Rockford) and are generally called "Drug Market Intervention" (DMI) pulling levers focused deterrence strategies. Two studies evaluated crime reduction strategies that were focused on individuals (Chicago and Newark).
activities, and (4) social marketing of deterrence and social norms messages through offender notification meetings (Papachristos et al., 2007).
All ten eligible studies used quasi-experimental designs to analyze the impact of pulling levers focused deterrence strategies on crime. Seven evaluations used quasiexperimental designs with non-equivalent comparison groups (Boston, Cincinnati, Indianapolis, Lowell, Nashville, Rockford, and Stockton). Two evaluations used quasi-experimental designs with comparison groups created through matching techniques (Chicago and Newark). One evaluation used a quasi-experimental design that included both non-equivalent comparison groups and comparison groups created through matching techniques (Los Angeles). Table 2 provides a brief summary of the treatment, units of analysis, and research design used by the ten eligible studies.
Five studies evaluated the crime reduction effects of focused deterrence strategies by comparing trends in key outcome variables in a targeted geographic area (identified as a neighborhood, policing district, or well-defined zone) to trends in key outcome variables in comparison areas. The Chicago study used propensity score matching techniques to identify very similar comparison policing districts to compare against the targeted policing districts. The Los Angeles study used two non-equivalent comparisons (the target area relative to the remainder of the larger neighborhood; the targeted neighborhood relative to the surrounding larger geographic community area). The Los Angeles study also used propensity score matching techniques to identify similar Census block groups to compare against the Census block groups that comprised the targeted area. The Newark evaluation used crime mapping technology and simple matching techniques to identify a comparison gun hot spot area that was similar to the targeted Ceasefire zone in terms of gunshot wounding incidents, geographic size, and socio-demographic characteristics. The Nashville and Rockford studies compared crime trends in targeted neighborhoods relative to crime trends in the surrounding County and city areas, respectively.
Five studies evaluated the crime reduction effects of citywide pulling levers interventions. The Boston, Indianapolis, Lowell, and Stockton quasi-experimental designs compared citywide trends in key outcomes to citywide trends in key outcomes in sets of non-equivalent cities that did not experience a pulling levers intervention during the study time period. The Cincinnati evaluation compared citywide trends in homicides involving members of criminally-active groups targeted by the pulling levers intervention relative to trends in homicides that did not involve members of criminally-active groups.
Three studies examined possible immediate spatial crime displacement and diffusion of crime control benefits that may have been generated by the pulling levers interventions (Los Angeles, Nashville, and Newark). Only one study noted potential threats to the integrity of the treatment. Tita et al. (2003) reported that the Los Angeles pulling levers intervention was not fully implemented as planned.
The implementation of the Ceasefire program in the Boyle Heights neighborhood of Los Angeles was negatively affected by the well-known Ramparts LAPD police corruption scandal and a lack of ownership of the intervention by the participating agencies.

NARRATIVE REVIEW OF THE EFFECTS OF PULLING LEVERS FOCUSED DETERRENCE STRATEGIES ON CRIME
This section provides a brief narrative review of the effects of the eligible pulling levers focused deterrence interventions on crime. Table 3 summarizes the main effects of the intervention on crime outcomes and, when measured, crime displacement and diffusion of crime control benefits effects. A more detailed narrative review of the focused deterrence strategies contained in the eligible studies is provided in Appendix C. Nine of the ten evaluations of pulling levers focused deterrence strategies reported statistically-significant crime reduction effects associated with the approach. While the authors did report a noteworthy nonstatistically significant reduction in gunshot wound incidents, only the evaluation of Newark's Operation Ceasefire did not report any statistically significant crime prevention benefits generated by the violence reduction strategy.
Evaluations of focused deterrence strategies targeting gangs and criminally-active groups reported large statistically-significant reductions in violent crime; these results included: a 63% reduction in youth homicides in Boston, 44% reduction in gun assault incidents in Lowell, 42% reduction in gun homicides in Stockton, 35% reduction in homicides of criminally-active group members in Cincinnati, 34% reduction in total homicides in Indianapolis, and significant short-term reductions in violent crime in Los Angeles. The two drug market intervention evaluations also reported statistically significant crime reductions. In Nashville, the drug market intervention generated a 55% reduction in illegal drug possession incidents. In Rockford, the drug market intervention generated a 22% reduction in non-violent offenses. While Newark's strategy did not generate statistically significant crime control gains when high-rate offenders were targeted, the other two individualfocused programs reported statistically significant crime reductions. In Chicago, the PSN intervention was associated with a 37% reduction in homicide.
Two of the three studies that measured possible crime displacement and diffusion effects reported noteworthy diffusion of crime control benefits associated with the focused deterrence intervention. Consistent with the absence of a treatment effect, the Newark evaluation did not report any statistically significant crime displacement or diffusion effects. The Nashville evaluation reported statistically significant reductions in drug offenses and total calls for service in the non-treated area immediately adjoining the targeted drug market area. The Los Angeles evaluation found statistically-significant reductions in violent crime in treatment census block groups surrounding the treatment census block groups as well as noteworthy reductions in violent offending by non-treated gangs that were "socially-tied" to treatment gangs.
To test the statistical significance of the observed distribution of crime reduction effects reported by the ten eligible studies, we used an application of the binomial distribution known as the sign test (Blalock, 1979). This simple test examines the probabilities of getting an observed proportion of successes from a population of equal proportions of successes and failures. Nine of the ten studies (90.0%) reported noteworthy crime reductions associated with the pulling levers approach.
According to the sign test, this result was statistically significant (exact binomial two tailed probability = .0215).

META-ANALYSIS OF THE EFFECTS OF PULLING LEVERS STRATEGIES ON CRIME
It is important to note here that, while the impacts of the pulling levers focused deterrence programs were measured at larger areal units (city, policing area, targeted zone, census unit), these interventions were highly focused on a small number of risky individuals and risky groups who tend to commit their crimes at a relatively small number of high crime places within those larger areas. Pulling levers focused deterrence strategies are highly targeted interventions that are not broadly diffused across large populations or large areas. Computation of effect sizes in the studies was not always direct. The goal was to convert all observed effects into a standardized mean difference effect size metric. None of the studies we examined calculated standardized effect sizes, and indeed, it was sometimes difficult to develop precise effect size metrics from published materials. This reflects a more general problem in crime and justice with "reporting validity" (Farrington, 2006;Lösel and Köferl, 1989), and has been documented in recent reviews of reporting validity in crime and justice studies (see Perry and Johnson, 2008;Perry et al., 2010).
As described earlier, David B. Wilson's Effect Size Calculator was used to calculate the standardized mean difference effect sizes for all outcomes in the eligible studies. For the Boston, Lowell, and Stockton studies, we calculated standardized mean effect sizes based on newly-estimated Ordinary Least Squares regression models that used the same covariates and modeling choices as the count regression models in the original evaluations. 7 7 We could not find a satisfactory method of calculating effect sizes based on the reported treatment coefficients estimated by the Poisson and negative binomial regression models used in these evaluations. Braga was involved as the lead author of the Boston, Lowell, and Stockton studies and ran the new OLS regressions from the original study data. The treatment dummy variables in the new models produced an "adjusted" mean difference for the intervention effect for each of the outcome variables. The unstandardized regression coefficients, standard deviation of the dependent variables, and treatment and control Ns were entered into the Effect Size Calculator to calculate the standardized mean difference effect size. It is important to note that, despite the change from count regression For the Chicago, Cincinnati, Indianapolis, Nashville, Newark, and Rockford, we calculated standardized mean effect sizes based on the ttest results reported for the intervention variables' effects on the outcome variables. 8 Using the mean effect criterion for all eligible studies, the forest plots in Figure 1 show the standardized difference in means between the treatment and control or comparison conditions (effect size) with a 95 percent confidence interval plotted around them for all eligible studies. Points plotted to the right of 0 indicate a treatment effect; in this case, the study showed a reduction in crime or disorder.
Points to the left of 0 indicate a backfire effect where control conditions improved relative to treatment conditions. Since the Q statistic which was significant at the p < .05 level (Q = 41.752, df = 9), we used a random effects model to estimate the overall mean effect size based on a heterogeneous distribution of effect sizes. The metaanalysis of effect sizes suggests a strongly significant effect in favor of pulling levers focused deterrence strategies. The overall effect size for these studies is .604 (see Cohen, 1988). This is above Cohen's standard for a medium effect of .50 and below that of a large effect at .80 (Cohen, 1988). Nonetheless, the overall effect size is relatively large compared to assessments of interventions in crime and justice work more generally.
For the Los Angeles study, we acquired the original evaluation data, calculated the mean gain scores, pre-test and post-test standard deviations, and paired t-tests, and entered these data into the Effect Size Calculator to estimate the standardized mean difference effect sizes. In Appendix C, we provide effect sizes for each outcome for the ten eligible studies.
All studies reported effect sizes that favor treatment conditions over control conditions, with only the Cincinnati, Los Angeles, and Newark studies not reporting statistically significant effect sizes. The Lowell (1.186) and Indianapolis (1.039) studies reported the largest statistically significant effect sizes while the Chicago study (.181) reported the smallest statistically significant effect size. The forest plots in Figures 2 and 3 present the meta-analyses of the largest and smallest effect sizes for each study, respectively. 9 models to OLS models, there were no substantive changes in the direction, size, or statistically significance of the treatment estimates for these studies.
For the largest effect size meta-analysis, the overall standardize mean difference effect size was large (.806) and statistically significant at the p < .05 level. For the smallest effect size meta-analysis, the overall standardize mean difference effect size was medium (.474) and statistically significant at the p < .05 level.

PROGRAM TYPE AND RESEARCH DESIGN AS EFFECT SIZE MODERATORS
Our narrative review documented that pulling levers focused deterrence strategies have been directed at reducing crime by street gangs and criminally-active groups, drug markets, and high-risk individuals in high-crime areas. These programs represent differing applications of pulling levers strategies to control distinct types of problems. Moderator variables help to explain and understand differences across studies in the outcomes observed. Program type could be an influential moderator of the observed effect sizes in our overall meta-analysis. Figure 4 presents a random effects model examining the three different program types identified in the narrative review. It is important to note that the Q-statistic associated with the between group variation was large and statistically significant (Q = 20.430, df = 2, p <.05), suggesting that program type is influential in determining effect sizes. The gang / group intervention programs were associated with the largest within-group effect size (.770, p <.05), followed by the drug market intervention (DMI) programs (.661, p <.05) and the high-risk individuals programs (.186, p <.05). When program type was included as a moderator, the meta-analysis estimated a more modest overall effect size (.306, p<.05).
Pulling levers focused deterrence strategies directed at high-risk individuals in highcrime areas generated a smaller within-group effect size when compared to the DMI and gang / group intervention strategies. These smaller effect sizes may, in part, stem from methodological decisions to analyze outcomes at the area level rather than the individual level. As described in our narrative review, the Chicago PSN quasi-experimental evaluation conducted additional analyses of the effects of the intervention on individual recidivism patterns. As an exploratory exercise, we calculated an effect size for the Chicago (.434, SE = .050, p < .05). This medium and statistically significant effect size suggests a program impact similar to area-level impacts generated by the DMI and gang / group interventions.
Given the important distinction in methodological quality between the nonequivalent quasi-experiments and the quasi-experiments that used matching techniques to identify comparison groups, we also examined research design as a moderator variable. Figure 5 presents a random effects model examining the two different classes of quasi-experimental designs included in this review. 10 10 For the overall model, Q = 41.725, df = 9, p < 0.0001.
It is important to note that the Q-statistic associated with the between group variation was large and statistically significant (Q = 31.039, df = 1, p <.05), suggesting that research design is influential in determining effect sizes. Consistent with prior research suggesting that weaker designs are more likely to report stronger effects in crime and justice studies (Weisburd et al., 2001;Welsh et al., 2011), the nonequivalent quasi-experimental designs were associated with a much larger withingroup effect size (.766, p <.05) relative to the matched quasi-experimental designs (.196, p <.05). When research design type was included as a moderator, the metaanalysis estimated a more modest overall effect size (.312, p<.05). Lipsey (2003) cautions that analyses that include single moderator variables can be misleading as these types of analyses can be confounded by the absence of other relevant variables. In this analysis, "research design type" is highly correlated with "program type." Five of the six gang / group intervention studies used nonequivalent quasi-experimental designs to analyze program effects. With the noteworthy exception of the Los Angeles evaluation, the five gang / group intervention programs attempted to influence ongoing feuds among gangs and groups in conflict networks that spanned the urban landscape. For instance, the authors of the Boston Ceasefire evaluation reported that, given the nature of their intervention and the dynamics of the problem, it was not possible to set aside within-city control gangs or comparison areas (Braga et al., 2001).

PUBLICATION BIAS
Publication bias presents a strong challenge to any review of evaluation studies (Rothstein, 2008). Campbell reviews, such as ours, take a number of steps to reduce publication bias, as represented by the fact that four of the 10 eligible studies in our review came from unpublished sources (one government report and three unpublished reports). Wilson (2009) has argued moreover that there is often little difference in methodological quality between published and unpublished studies suggesting the importance of searching the "grey literature." As a first step in investigating potential publication bias, we compared mean effect sizes for unpublished vs. published studies. Using random effects models, the mean effect size for published studies is 0.713 (p < .05) and for unpublished studies, the average effect is 0.384 (p < .05). While the published studies report larger effect sizes, the similarity in the direction and statistical significance in the mean effect sizes between the published and unpublished literature suggests that publication bias may not have major impact on the outcomes of this review.
We then used the trim-and-fill procedure (Duval and Tweedie 2000) to estimate the effect of potential data censoring, such as publication bias, on the outcome of the meta-analyses. The diagnostic funnel plot is based on the idea that, in the absence of bias, the plot of study effect sizes should be symmetric about the mean effect size. If there is asymmetry, the trim-and-fill procedure imputes the missing studies, adds them to the analysis, and then re-computes the mean effect size. A visual inspection of the resulting funnel plot indicated some asymmetry with more studies with a large effect and a large standard error to the right of the mean than the left of the mean. The trim-and-fill procedure determined that three studies should be added to create symmetry. The funnel plot with imputed studies is presented in Figure 4.
These additional studies modestly altered the mean effect size estimate. The mean random effect decreased from 0.604 (95% CI = 0.349, 0.859) to 0.437 (95% CI = 0.200, 0.637). Indeed, the 95 percent confidence intervals overlap, suggesting that the mean effect sizes may actually be the same.

Conclusion
The available scientific evidence on the crime reduction value of focused deterrence strategies has been previously characterized as "promising" but "descriptive rather than evaluative" (Skogan and Frydl, 2004: 241) and as "limited" but "still evolving" (Wellford et al., 2005: 10) by the U.S. National Research Council's Committee to Review Research on Police Policy and Practices and Committee to Improve Research Information and Data on Firearms, respectively. Our systematic review identified ten evaluations of focused deterrence strategies; nine of these evaluations were completed after the National Research Council reports were published. A betterdeveloped base of scientific evidence now exists to assess whether crime prevention impacts are associated with this approach.
The basic findings of our review are very positive. Nine out of ten eligible studies reported strong and statistically significant crime reductions associated with the approach. Nonetheless, we are concerned with the lack of rigorous randomized experimental evaluations of this promising approach. While the biases in quasiexperimental research are not clear (e.g. Campbell and Boruch, 1975; Wilkinson and Task Force on Statistical Inference, 1999), recent reviews in crime and justice suggest that weaker research designs often lead to more positive outcomes (e.g. see Weisburd, Lum, and Petrosino, 2001;Welsh et al., 2011). This does not mean that non-experimental studies cannot be of high quality, but only that there is evidence that non-experimental designs in crime and justice are likely to overstate outcomes as contrasted with randomized experiments. In his review of situational crime prevention evaluations, Guerette (2009) finds that the conclusions of randomized evaluations were generally consistent with the majority conclusion of the nonrandomized evaluations. While our narrative review is consistent with Guerette's (2009) conclusion, our calculated effect sizes reveal that less rigorous focused deterrence evaluation designs were associated with stronger reported effects. As such, we think that caution should be used in drawing conclusions regarding population effect sizes for the pulling levers intervention.
At the same time, the effects observed in the studies reviewed were often very large, and such effect sizes are evidenced as well in those studies using strong comparison groups (e.g. Papachristos, Meares, and Fagan, 2007). Our review provides strong empirical evidence for the crime prevention effectiveness of focused deterrence strategies. Even if we assume that the effects observed contain some degree of upward bias, it appears that the overall impact of such programs is noteworthy.
These findings are certainly encouraging, and point to the promises of this approach.
We certainly believe that the positive outcomes of the present studies indicate that additional experimental evaluations, however difficult and costly, are warranted. The potential barriers are real, especially in regards to identifying valid treatment and comparison areas. But existing evidence is strong enough to warrant a large investment in multi-site experiments (Weisburd and Taxman, 2000). Such experiments could solve the problem of small numbers of places in single jurisdictions, and would also allow for examination of variation in effectiveness across contexts.
Despite our concerns over the lack of randomized experiments, we believe that the findings of eligible focused deterrence evaluations fit well within existing research suggesting that deterrence-based strategies, if applied correctly, can reduce crime (Apel and Nagin, 2011). The focused deterrence approach seems to have the desirable characteristic of altering offenders' perceptions of sanction risk. Our findings are also supported by the growing body of scientific evidence that suggests police departments, and their partners, can be effective in controlling specific crime problems when they engage a variety of partners, and tailor an array of tactics to address underlying criminogenic conditions and dynamics (Braga, 2008a;Weisburd and Eck, 2004). Indeed, our study suggests that Durlauf and Nagin (2011) are correct in their conclusion that imprisonment and crime can both be reduced through the noteworthy marginal deterrent effects generated by allocating police officers, and their criminal justice partners, in ways that heighten the perceived risk of apprehension.
While the results of this review are very supportive of deterrence principles, we believe that other complementary crime control mechanisms are at work in the focused deterrence strategies described here that need to be highlighted and better understood (see Weisburd, 2011). In Durlauf and Nagin's (2011) article, the focus is on the possibilities for increasing perceived risk and deterrence by increasing police presence. Although this conclusion is warranted by the data and represents an important component of the causal mechanisms that have increased the effectiveness of focused deterrence strategies, we believe it misses an important part of the story. In the focused deterrence approach, the emphasis is not only on increasing the risk of offending, it is also on decreasing opportunity structures for violence, deflecting offenders away from crime, increasing the collective efficacy of communities, and increasing the legitimacy of police actions. Indeed, we suspect that the large effects we observe come precisely from the multi-faceted ways in which this program influences criminals.
A number of scholars have focused on the mechanism of "discouragement" when discussing crime prevention benefits of interventions (see, e.g. Clarke and Weisburd, 1994). Discouragement emphasizes reducing the opportunities for crime and increasing alternative opportunity structures for offenders. In this context, situational crime prevention techniques are often implemented as part of the core pulling levers work in focused deterrence strategies (Braga and Kennedy, 2012;Skubak Tillyer and Kennedy, 2008). For instance, the Cincinnati Initiative to Reduce Violence used civil forfeiture techniques to close down a highly problematic bar that generated recurring serious violence (Engel, Corsaro, and Skubak Tillyer, 2010). Extending guardianship, assisting natural surveillance, strengthening formal surveillance, reducing the anonymity of offenders, and utilizing place managers can greatly enhance the range and the quality of the varying enforcement and regulatory levers that can be pulled on offending groups and key actors in criminal networks (see, e.g. Welsh and Farrington, 2009). The focused deterrence approach also seeks to redirect offenders away from violent crime through the provision of social services and opportunities. In all the gang / group interventions reviewed here, gang members were offered job training, employment, substance abuse treatment, housing assistance, and a variety of other services and opportunities.
Aspects of "broken windows" theory may also be relevant for our understanding of how and why focused deterrence programs reduce crime (Wilson and Kelling, 1982). Broken windows theory argues that intensive efforts by police to reduce social and physical disorder can reverse the breakdown of community social controls that accompanies untended and unrestrained violations of social order. Thus, crime is reduced in part because of efforts by the police and in part because of increased vigilance by community members. Kleiman and Smith (1990: 88) describe the potential benefits of an intensive police effort to reduce drug crime and disorder by noting "a dramatic police effort may call forth increased neighborhood efforts at selfprotection against drug dealing activity; given police resources such self-defense may be essential to long-run control of drug dealing." Sampson, Raudenbush, and Earls (1997) emphasize the capacity of a community to realize common values and regulate behavior within it through cohesive relationships and mutual trust among residents. They argue that the key factor determining whether crime will flourish is a sense of the "collective efficacy" of a community. A community with strong collective efficacy is characterized by "high capacities for collective action for the public good" (St. Jean, 2007: 3). Focused deterrence enhances collective efficacy in communities by emphasizing the importance of engaging and enlisting community members in the strategies developed. The High Point DMI strategy, for example, drew upon collective efficacy principles by engaging family, friends, and other "influential" community members in addressing the criminal behaviors of local drug dealers (Kennedy, 2009).
Finally, the focused deterrence approach takes advantage of recent theorizing regarding procedural justice and legitimacy. The effectiveness of policing is dependent on public perceptions of the legitimacy of police actions (Skogan and Frydl, 2004;Tyler, 1990Tyler, , 2004. Legitimacy is the public belief that there is a responsibility and obligation to voluntarily accept and defer to the decisions made by authorities (Tyler, 1990(Tyler, , 2004. Recent studies suggest that when procedural justice approaches are used by the police citizens will not only evaluate the legitimacy of the police more highly, they will also be more likely to obey the law in the future (see, e.g. Paternoster et al., 1997). Advocates of focused deterrence strategies argue that targeted offenders should be treated with respect and dignity (Kennedy, 2008(Kennedy, , 2009, reflecting procedural justice principles. The Chicago PSN strategy, for instance, sought to increase the likelihood that the offenders would "buy in" and voluntarily comply with the pro-social, anti-violence norms being advocated by interacting with offenders in ways that enhance procedural justice in their communication sessions (Papachristos, Meares, and Fagan, 2007).
In closing, we think it is important to recognize that focused deterrence strategies are a very recent addition to the existing scholarly literature on crime control and prevention strategies. While the evaluation evidence needs to be strengthened and the theoretical underpinnings of the approach needs further refinement, we believe that jurisdictions suffering from gang violence, overt drug markets, and repeat offender problems should add focused deterrence strategies to their existing portfolio of prevention and control interventions. The existing evidence suggests these new approaches to crime prevention and control generate noteworthy crime reductions.

Plans for Updating the Review
The authors expect to update the review every five years.

Acknowledgements
This research was supported through funds provided by the National Policing Improvement Agency, United Kingdom. We would like to thank David B. Wilson, Terri Piggott, Charlotte Gill, Michael Maxfield, and several anonymous reviewers for their advice and helpful comments on earlier versions of this review. We would also like to thank Phyllis Schultze of the Gottfredson Library at Rutgers University for her help in the identification of potentially eligible studies for this review. Nicholas Corsaro, Danielle Wallace, Andrew Papachristos, George Tita, and Greg Ridgeway deserve special thanks for sharing data and providing requested additional analyses. Points of view or opinions expressed in this research are those of the authors and do not necessarily represent the official view of the National Policing Improvement Agency.

OPERATION CEASEFIRE IN BOSTON, MASSACHUSETTS
The Boston Gun Project was a problem-oriented policing enterprise expressly aimed at taking on a serious, large-scale crime problem -homicide victimization among young people in Boston in the 1990s. The trajectory of the Boston Gun Project, and the resulting Operation Ceasefire intervention, is by now well known and extensively documented (Braga et al., 2001;Kennedy, 1997Kennedy, , 2006Kennedy et al., 1996). Briefly, a working group of law enforcement personnel, youth workers, and Harvard researchers diagnosed the youth violence problem in Boston as one of patterned, largely vendetta-like hostility amongst a small population of chronic offenders, and particularly among those involved in 61 loose, informal, mostly neighborhood-based "gangs." These 61 gangs consisted of some 1,300 members, representing less than 1 percent of the city's youth between the ages of 14 and 24. Although small in number, these gangs were responsible for more than 60 percent of youth homicide in Boston.
The Operation Ceasefire focused deterrence strategy was designed to prevent violence by reaching out directly to gangs, saying explicitly that violence would no longer be tolerated, and backing up that message by "pulling every lever" legally available when violence occurred (Kennedy, 1997). The chronic involvement of gang members in a wide variety of offenses made them, and the gangs they formed, vulnerable to a coordinated criminal justice response. The authorities could disrupt street drug activity, focus police attention on low-level street crimes such as trespassing and public drinking, serve outstanding warrants, cultivate confidential informants for medium-and long-term investigations of gang activities, deliver strict probation and parole enforcement, seize drug proceeds and other assets, ensure stiffer plea bargains and sterner prosecutorial attention, request stronger bail terms (and enforce them), and bring potentially severe federal investigative and prosecutorial attention to gang-related drug and gun activity. Simultaneously, youth workers, probation and parole officers, and later churches and other community groups offered gang members services and other kinds of help.
These partners also delivered an explicit message that violence was unacceptable to the community and that "street" justifications for violence were mistaken. The Ceasefire Working Group delivered this message in formal meetings with gang members (known as "forums" or "call-ins"), through individual police and probation contacts with gang members, through meetings with inmates at secure juvenile facilities in the city, and through gang outreach workers. The deterrence message was not a deal with gang members to stop violence. Rather, it was a promise to gang members that violent behavior would evoke an immediate and intense response. If gangs committed other crimes but refrained from violence, the normal workings of police, prosecutors, and the rest of the criminal justice system dealt with these matters. But if gang members hurt people, the Working Group concentrated its enforcement actions on their gangs.
The Ceasefire "crackdowns" were not designed to eliminate gangs or stop every aspect of gang activity, but to control and deter serious violence. To do this, the Working Group explained its actions against targeted gangs to other gangs, as in "this gang did violence, we responded with the following actions, and here is how to prevent anything similar from happening to you." The ongoing Working Group process regularly watched the city for outbreaks of gang violence and framed any necessary responses in accord with the Ceasefire strategy. As the strategy unfolded, the Working Group continued communication with gangs and gang members to convey its determination to stop violence, to explain its actions to the target population, and to maximize both voluntary compliance and the strategy's deterrent power.
The DOJ-sponsored evaluation of the impact of Operation Ceasefire used a nonrandomized quasi-experimental design to compare youth homicide trends in Boston to youth homicide trends in other major cities in the United States and large New England cities (Braga et al., 2001). 11 The Ceasefire evaluation concluded that the program was associated with statistically significant reductions in youth homicide and the other indicators of The key outcome variable was the monthly number of homicide victims ages 24 and under between January 1, 1991 andMay 31, 1998. The within-Boston program impact assessment was supplemented by analyses of Ceasefire's effect on the monthly number of citywide gun assault incidents, citywide shots-fired calls for service, and youth gun assault incidents in one high-risk policing district. Poisson and negative binomial regression models, controlling for secular trends, seasonal variations, Boston youth population trends, Boston employment rate trends, robbery trends, adult homicide trends, and youth drug arrest trends, were used to estimate the effect of Ceasefire on the outcome variables. The impact of Ceasefire was estimated using a dummy variable with June 1996 selected as the commencement of the post-implementation period. serious gun violence in Boston. Controlling for the other covariates, the evaluation reported that Ceasefire was associated with a 63 percent reduction in the monthly count of youth homicides, a 25 percent reduction in the monthly count of citywide gun assault incidents, a 32 percent reduction in the monthly count of citywide shotsfired calls for service, and a 44 percent reduction in the monthly count of youth gun assaults in selected high-risk district (Braga et al., 2001; see also Piehl et al., 2000). In a companion paper, Piehl et al. (2003) closely analyzed the monthly counts in the youth homicide time series to determine whether the timing of the implementation of Ceasefire coincided with the start of the significant decrease in Boston youth homicides. The authors developed an econometric model that evaluated all possible monthly break points in the time series to identify the maximal monthly break point associated with a significant structural change in the trajectory of the time series. Controlling for trends and seasonal variations, the timing of the "optimal break" in the time series was in the summer months after Ceasefire was implemented.
The DOJ-sponsored evaluation then conducted a comparative analysis of youth homicide trends in 39 of the most populous cities in the United States and 29 New England cities with populations of more than 60,000 residents (Braga et al., 2001). Using count regression models that controlled for trends, seasonal variations, and serial autocorrelation, the research found that only three cities (Dallas, TX; Jacksonville, FL; and Virginia Beach, VA) had significant reductions in the monthly count of youth homicides that coincided with the implementation of Ceasefire in Boston and an additional four cities (Los Angeles, CA; New York City, NY; Philadelphia, PA; and Tucson, AZ) had significant reductions in the monthly count of youth homicides at some point within the time series. Further examination of the youth homicide trends in these cities, however, revealed trajectories that looked distinct from the trajectory of Boston youth homicide over the same time period. As such, the researchers concluded that Boston's youth homicide reduction associated with Operation Ceasefire was distinct when compared to trends in most major U.S. cities.
The DOJ evaluation has been reviewed by a number of researchers and the relationship between the implementation of Ceasefire and the trajectory youth homicide in Boston during the 1990s has been closely scrutinized. Fagan (2002) suggested that some of the decrease in homicide may have occurred without the Ceasefire intervention in place as violence was decreasing in most major U.S. cities. In support of this perspective, Fagan's (2002) presented a simple time-series graph on youth gun homicide in Boston and in other Massachusetts cities that suggested a general downward trend in gun violence may have existed before Operation Ceasefire was implemented. Using growth-curve analysis to examine predicted homicide trend data for the 95 largest U.S. cities during the 1990s, Rosenfeld and his colleagues (2005) found some evidence of a sharper youth homicide drop in Boston than elsewhere but suggest that the small number of youth homicide incidents precludes strong conclusions about program effectiveness based on their statistical models. However, in his review of their analysis, Richard Berk (2005) raised a number of statistical and methodological concerns with the analysis developed by Rosenfeld and his colleagues. Ludwig (2005) suggested that Ceasefire was associated with a large drop in youth homicide but, given the complexities of analyzing city-level homicide trend data, there remained some uncertainty about the extent of Ceasefire's effect on youth violence in Boston. Morgan and Winship's (2007) review of the DOJ evaluation concluded that the analysis was a "very highquality example" of how to conduct an interrupted time series analysis of program impact and further noted "they offer four types of supplemental analysis… which can be used to strengthen the warrant for causal assertion" (252).
The National Academies' Panel on Improving Information and Data on Firearms (Wellford et al., 2005) concluded that the Ceasefire evaluation was compelling in associating the intervention with the subsequent decline in youth homicide. However, the Panel also suggested that many complex factors affect youth homicide trends and it was difficult to specify the exact relationship between the Ceasefire intervention and subsequent changes in youth offending behaviors. While the DOJsponsored evaluation controlled for existing violence trends and certain rival causal factors such as changes in the youth population, drug markets, and employment in Boston, there could be complex interaction effects among these factors not measured by the evaluation that could account for some meaningful portion of the decrease. The evaluation was not a randomized, controlled experiment. Therefore, the non-randomized control group research design cannot rule out these internal threats to the conclusion that Ceasefire was the key factor in the youth homicide decline.

INDIANAPOLIS VIOLENCE REDUCTION PARTNERSHIP IN INDIANAPOLIS, INDIANA
The Indianapolis Violence Reduction Partnership (IVRP) working group was comprised of Indiana University researchers and federal, state, and local law enforcement agencies (McGarrell and Chermak, 2003). During the problem analysis phase, the researchers examined 258 homicides from 1997 and the first 8 months of 1998 and found that a majority of homicide victims (63%) and offenders (75%) had criminal and/or juvenile records. Those with a prior record often had a substantial number of arrests. The working group members followed the structured qualitative data gathering exercises used in Boston to gain insight on the nature of homicide incidents. The qualitative exercise revealed that 59% of the incidents involved "groups of known chronic offenders" and 53% involved drug-related motives such as settling business and turf disputes (McGarrell and Chermak, 2003). It is worth noting that the terminology "groups of known chronic offenders" was initially used because, at that point in time, there was not a consensual definition of "gang" and the reality of much gang activity in Indianapolis was of a relatively loose structure (McGarrell and Chermak, 2003).
The working group developed two sets of overlapping strategies. First, the most violent chronic offenders in Indianapolis were identified and targeted for heightened arrest, prosecution, and incarceration (McGarrell and Chermak, 2003). Second, the working group engaged the pulling levers approach to reduce violent behavior by gangs and groups of known chronic offenders (McGarrell and Chermak, 2003). The IVRP strategy implemented by the Indianapolis working group closely resembled the Boston version of pulling levers. The communications strategy, however, differed in an important way. The deterrence and social services message was delivered in meetings with high-risk probationers and parolees organized by neighborhoods. Similarly, home visits by probation and parolees were generally organized by neighborhood. As the project progressed, when a homicide or series of homicides involved certain groups or gangs, the working group attempted to target meetings, enforcement activities, and home visits on the involved groups or gangs (McGarrell and Chermak, 2003). The impact of the IVRP strategy was estimated using a dummy variable with April 1999 selected as the commencement of the post-intervention period (McGarrell et al., 2006). The ARIMA analyses of the Indianapolis homicide time series estimated that the IVRP intervention was associated with a statistically significant 34 percent reduction in monthly numbers of homicides. The ARIMA models analyzing the other cities' homicide time series did not report any statistically significant associations between the timing of IVRP and subsequent decreases in monthly homicide numbers. In a subsequent analysis of Indianapolis homicide time series data, Corsaro and McGarrell (2009) used ARIMA models to analyze the impact of IVRP on gang and non-gang homicides. The analyses found a statistically significant 38 percent reduction in gang homicides following the implementation of IVRP and did not find a statistically significant reduction in the non-gang homicides during the post-intervention time period. Since IVRP was explicitly designed to reduce gang violence, the authors concluded that these results support the position that the intervention was indeed having the desired effects on violent gang offending.

OPERATION PEACEKEEPER IN STOCKTON, CALIFORNIA
Beginning in mid-1997, criminal justice agencies in Stockton began experimenting with the pulling levers approach to address a sudden increase in youth homicide. The Stockton Police Department and other local, state, and federal law enforcement agencies believed that most of the youth violence problem was driven by gang conflicts and that the pulling levers approached used in Boston might be effective in reducing Stockton's gang violence problem. The strategy was implemented by the Stockton Police Department's Gang Street Enforcement Team and grew into what is now known as "Operation Peacekeeper" as more agencies joined the partnership (Wakeling, 2003).
The Peacekeeper intervention was managed by a working group of line-level criminal justice practitioners; social service providers also participated in the working group process as appropriate. When street gang violence erupted or when it came to the attention of a working group member that gang violence was imminent, the working group followed the Boston model by sending a direct message that gang violence would not be tolerated, pulling all available enforcement levers to prevent violence, continuing communications, and providing social services and opportunities to gang members who want them.
The Operation Peacekeeper evaluation used a nonrandomized quasi-experimental design to compare gun homicide trends in Stockton to gun homicide trends in eight other midsized California cities (Braga, 2008b). The eight comparison California cities included Anaheim, Bakersfield, Fresno, Long Beach, Oakland, Riverside, Sacramento, and Santa Ana. For each of the nine cities included in the evaluation, the key outcome variable was the monthly number of gun homicide victims between January 1, 1990 andDecember 31, 2005. The evaluation carefully analyzed the distributions of the dependent variables for each city's time series to determine the appropriate regression models for the impact assessment. Ordinary Least Squares (Santa Ana), maximum likelihood with an AR(1) autoregressive component (Long Beach, Oakland), negative binomial (Anaheim, Bakersfield, Fresno, Riverside, Stockton), and Poisson (Sacramento) regression models were used to analyze the city time series data.
Stockton's Operation Peacekeeper intervention was implemented in September 1997 and was operational until it was discontinued in December 2002 (Braga, 2008b). Multiple category dummy variables indicating the time periods when the Stockton Peacekeeper intervention was present or not were included in the regression models to estimate the trajectory of the monthly counts of gun homicide in each of the time series after Stockton implemented its gun violence reduction initiative. Controlling for existing linear and non-linear trends, seasonal variations, and violent crime trends, the negative binomial regression analyzing the Stockton gun homicide time series estimated that the intervention was associated with a statistically significant 42 percent reduction in the monthly count of gun homicides. None of the comparison cities experienced a statistically significant reduction in the monthly count of gun homicides that coincided with the implementation of the Peacekeeper intervention in Stockton.

PROJECT SAFE NEIGHBORHOODS IN LOWELL, MASSACHUSETTS
Supported by funds from the U.S. Department of Justice-sponsored Project Safe Neighborhoods initiative, an interagency task force implemented a pulling levers focused deterrence strategy to prevent gun violence among Hispanic and Asian gangs in Lowell, Massachusetts in 2002 (Braga, McDevitt, and Pierce, 2006). The Lowell authorities used a pulling levers focused deterrence strategy that replicated Boston's Operation Ceasefire to prevent violence among Hispanic gangs. However, from the outset, they felt much less confident about their ability to prevent Asian gang violence by applying the same set of criminal justice levers to Asian gang members. During the intervention time period, the Lowell Police Department (LPD) had little reliable intelligence about Asian gangs in the city (Braga et al., 2006). The LPD had attempted to develop informants in the past but most these efforts had been unsuccessful.
Through PSN, the LPD increased its efforts to develop intelligence about the structure of the city's Asian gangs and particularly the relationship between Asian gang violence and ongoing illegal gambling that was being run by local Asian businesses. In Lowell, Cambodian and Laotian gangs were comprised of youth whose street activities were influenced by "elders" of the gang (Braga et al., 2006). Elders were generally long-time gang members in their 30s and 40s that no longer engaged in illegal activities on the street or participated in street-level violence with rival youth. Rather, these older gang members were heavily involved in running illegal gambling dens and informal casinos that were operated out of cafes, video stores, and warehouses located in the poor Asian neighborhoods of Lowell. The elders used young street gang members to protect their business interests and to collect any unpaid gambling debts. Illegal gaming was a very lucrative business that was much more important to the elders than any ongoing beefs the youth in their gang had with other youth (Braga et al., 2006). In contrast to acquiring information on individuals responsible for gun crimes in Asian communities, it was much easier to detect the presence of gambling operations through surveillance or a simple visit to the suspected business establishment.
The importance of illegal gaming to influential members of Asian street gangs provided a potentially potent lever to law enforcement in preventing violence. The authorities in Lowell believed that they could systematically prevent street violence among gangs by targeting the gambling interests of older members. When a street gang was violent, the LPD targeted the gambling businesses run by the older members of the gang. The enforcement activities ranged from serving a search warrant on the business that houses the illegal enterprise and making arrests to simply placing a patrol car in front of the suspected gambling location to deter gamblers from entering. The LPD coupled these tactics with the delivery of a clear message, "when the gang kids associated with you act violently, we will shut down your gambling business. When violence erupts, no one makes money" (Braga et al., 2006: 40). Between October 2002 and June 2003, the height of the focused attention on Asian gangs, the LPD conducted some 30 search warrants on illegal gambling dens that resulted in more than 100 gambling-related arrests (Braga et al., 2006).
The evaluation of the PSN gang violence reduction strategy used a nonrandomized quasi-experimental design to compare fatal and non-fatal gun assault incident trends in Lowell to fatal and non-fatal gun assault incident trends in seven other Massachusetts cities and the entire State of Massachusetts (Braga et al., 2008). The seven comparison Massachusetts cities included Boston, Brockton, Fall River, Lynn, New Bedford, Springfield, and Worcester. For the State of Massachusetts and the eight cities included in the evaluation, the key outcome variable was the monthly number of gun assault incidents between January 1, 1996 andDecember 31, 2005. The evaluation carefully analyzed the distributions of the dependent variables for each time series to determine the appropriate regression models for the impact assessment. Maximum likelihood with an AR(1) autoregressive component (Boston, Springfield, and State of Massachusetts), negative binomial (Brockton, Lynn, New Bedford, Worcester), and Poisson (Lowell, Fall River) regression models were used to analyze the city time series data.
The impact of Lowell's PSN strategy was estimated using a dummy variable with October 2002 selected as the commencement of the post-intervention period (Braga et al., 2008). Controlling for existing linear and non-linear trends, seasonal variations, population changes, and violent crime trends, the Poisson regression model reported that the Lowell PSN intervention was associated with a statistically significant 44 percent reduction in the monthly count of gun assault incidents. Neither the comparison cities nor the State of Massachusetts experienced a statistically significant reduction in the monthly count of gun homicides that coincided with the implementation of the PSN intervention in Lowell.

CINCINNATI INITIATIVE TO REDUCE VIOLENCE IN CINCINNATI, OHIO
In response to a disturbing increase in homicides between 2001 and 2006, Cincinnati's political leadership partnered with law enforcement officials, academics, medical professionals, street advocates, and community and business leaders, to form the Cincinnati Initiative to Reduce Violence (CIRV; Engel et al., 2010). Problem analyses suggested that violent street groups of active criminal offenders generated the bulk of homicides and shootings in Cincinnati. As described by Engel et al. (2010), Cincinnati implemented a group violence reduction strategy that was modeled after the pulling levers focused deterrence strategy implemented in Boston and included law enforcement consequences for violence, along with social service opportunities and community engagement. In face-to-face offender notification meetings, police, community activists, political figures, civil rights activists, ex-offenders, parents of murdered children, social service providers, medical personnel, and business, civic and religious leaders told members of violent groups that the violence must stop, that there would be law enforcement consequences for the entire group if it did not, and that the community would support these consequences (Engel et al., 2010). The working group partners also told violent group members that there was social service help for all who wanted it.
In series of unpublished reports, researchers from the Policing Institute at the University of Cincinnati completed ongoing preliminary evaluations of the impact of the CIRV interventions (Engel et al. 2008;Engel et al., 2009;Engel et al., 2010). The CIRV intervention was implemented in July 2007; members of criminally-active street groups in Cincinnati were the target population for the pulling levers focused deterrence strategy. The research team used Poisson regression models controlling for trends and month seasonal variations to estimate the effects of the CIRV treatment on monthly counts of group-member-involved (GMI) homicides in Cincinnati. As a comparison, they also ran the same Poisson model on non-GMI homicides in Cincinnati. The authors also used a similar negative binomial regression model to examine the CIRV on total shootings in Cincinnati. The citywide shooting data were not further parsed to conduct a comparative analysis of GMI and non-GMI shootings over time. Using a dummy variable to indicate the presence or absence of the CIRV treatment, the pre-intervention period comprised January 2004 through June 2007 and the post-intervention period comprised July 2007 through July 2010.
These analyses revealed that the CIRV strategy was associated with a statistically significant 35 percent reduction in the monthly count of GMI homicides between the pre-intervention and post-intervention time periods (p<.10). Non-GMI homicides experienced a non-statistically significant 35 percent increase between the preintervention and post-intervention time periods. The analyses also revealed a statistically significant 21 percent reduction (p<.05) in the monthly count of citywide total shootings associated with the implementation of the CIRV strategy.

OPERATION CEASEFIRE IN NEWARK, NEW JERSEY
In an unpublished report, researchers from the Violence Institute of New Jersey at the University of Medicine and Dentistry of New Jersey (UMDNJ) evaluated the Operation Ceasefire gang violence strategy in Newark, New Jersey (Boyle et al., 2010). The Newark Ceasefire strategy focused on preventing gun violence by individual gang members in a targeted "Ceasefire Zone." According to Boyle et al. (2010), the Newark strategy blended the law enforcement actions developed by the Boston Ceasefire pulling levers strategy (Kennedy et al., 1996) with the public health violence prevention activities developed by CeaseFire Chicago (Skogan et al., 2008). Shooting teams of detectives from the Newark, Irvington, and New Jersey State police departments aggressively investigated fatal and non-fatal shootings in the Ceasefire Zone. Parole officers also closely monitored high-risk individuals in the targeted area.
Drawing on the Chicago approach, Newark Ceasefire addressed risk and protective factors for individual gang members through five program components: public education, community mobilization, faith-based leader involvement, youth outreach, and criminal justice system engagement. Ceasefire youth outreach workers attempted to change the way gang members thought about and reacted to violence and to connect them with available services and opportunities. While there were not any formal offender-notification strategies in place, the participating law enforcement agencies, community groups, and outreach workers actively communicated with individual gang members to prevent retaliatory shootings and disrupt ongoing conflicts.
The Ceasefire Zone was a roughly two-square mile section of Newark that experienced elevated levels of gun homicides and shooting incidents. The Newark Ceasefire intervention was implemented on May 11, 2005. The evaluation team used ARIMA models to examine non-fatal gunshot wound trends in the Ceasefire Zone, a comparison zone, and the remainder of the City of Newark minus the Ceasefire Zone (Boyle et al., 2010). The comparison zone was identified through spatial analyses of non-fatal gunshot wounds to identify an area of similar size with similar levels of gun violence in Newark and also matched to the Ceasefire Zone based on 2000 Census data on the number of block groups in each area, population, resident race and ethnicity, median resident age and household income, concentrated poverty, and vacant housing units.
In the Newark Ceasefire evaluation, the key outcome variable was the weekly number of non-fatal gunshot wound victims treated at the Trauma Center at University Hospital in Newark in the Ceasefire Zone, comparison zone, and remainder of Newark between January 1, 2004and December 31, 2006(Boyle et al., 2010. These victimizations were geocoded by the location of gunshot wounding and then aggregated into weekly counts in the larger areas in which the events were contained. The ARIMA model estimated that the Newark Ceasefire intervention was associated with a non-statistically significant decrease in the weekly number of nonfatal gunshot wound victims in the Ceasefire Zone. The comparison zone also experienced a smaller, non-significant decrease in the weekly number of non-fatal gunshot victims and the remainder of Newark experienced a non-significant increase in the weekly number of non-fatal gunshot victims. As such, the evaluators concluded that Newark Ceasefire was not associated with any significant reductions in non-fatal gunshot wounds.
The researchers also used crime mapping software to examine potential crime displacement and diffusion effects in the areas immediately surrounding the Ceasefire Zone. While the researchers noted changes in the spatial distributions of non-fatal gunshot woundings in the areas surrounding the Ceasefire Zone, they concluded that their analyses could not link the development of new gun violence hot spots and "cold spots" to displacement and/or diffusion processes associated with the Ceasefire intervention.

OPERATION CEASEFIRE IN LOS ANGELES, CALIFORNIA
In March 1998, NIJ funded the RAND Corporation to develop and test strategies for reducing gun violence among youth in Los Angeles. In part, the goal was to determine which parts of the Boston Gun Project might be replicable in Los Angeles.
In designing the replication, RAND drew a clear distinction between the process governing the design and implementation of the strategy (data-driven policy development; problem solving, working groups) and the elements and design (pulling levers, collective accountability, retailing the message) of the Boston model. Processes, in theory, can be sustained and adaptive, and as such can be utilized to address dynamic problems. By singling out process as an important component, the RAND team hoped to make clear that process can affect program effectiveness independently of the program elements or the merits of the actual design (see Tita, Riley, and Greenwood, 2003).
The Los Angeles replication was unique in several important ways. First, the implementation was not citywide, but only within a single neighborhood (Boyle Heights) within a single Los Angeles Police Department Division (Hollenbeck). The project site, Boyle Heights, had a population that was relatively homogenous. Well over 80% of the residents were Latinos of Mexican origin. The same was true for the gangs, many of which were formed prior to the Second World War. These gangs were clearly "traditional" gangs, with memberships exceeding a hundred members or more. The gangs were strongly territorial, contain age-graded sub-structures, and are inter-generational in nature .
Unlike other cities where gang and group-involved violence was a rather recent phenomenon, Los Angeles represented an attempt to reduce gun violence in a "chronic gang city" with a long history of gang violence, and equally long history of gang reduction strategies. The research team had to first convince members of the local criminal justice and at large community that the approach we were espousing differed in important ways from these previous efforts to combat gangs. And in fact it does -the RAND project was not about "doing something about gangs," but rather "doing something about gun violence" in a community where gang members committed an overwhelming proportion of gun violence. The independent analysis of homicide files confirmed the perception held by police and community alike that gangs were highly over-represented in homicidal acts. From 1995-1998, 50% of all homicides had a clear gang motivation. Another 25% of the homicides could be coded as "gang related" because they involved a gang member as a victim or offender, but were motivated for reasons other than gang rivalries.
Given the social organization of violence in Boyle Heights, the multi-disciplinary working group fully embraced the pulling levers focused deterrence strategy developed in Boston. A high-profile gang shooting that resulted in a double homicide in Boyle Heights triggered the implementation of the Operation Ceasefire intervention in October 2000. The processes of retailing the message were formally adopted, though it was mostly accomplished through personal contact rather than in a group setting. Police, probation, community advocates, street gang workers, a local hospital and local clergy were all passing along the message of collective accountability for gangs continuing to commit gang violence. Unfortunately, Tita, Riley, Ridgeway et al. (2003) reported that the Los Angeles pulling levers intervention was not fully implemented as planned. The implementation of the Ceasefire program in the Boyle Heights was negatively affected by the well-known Ramparts LAPD police corruption scandal and a lack of ownership of the intervention by the participating agencies.
Despite the implementation difficulties, the RAND Corporation evaluated the Operation Ceasefire pulling levers strategy to reduce gun violence among gangs in the Boyle Heights area of Los Angeles . In their evaluation, RAND researchers examined the effects of the pulling levers gang violence reduction strategy on violent crime (homicides, attempted homicides, robberies, assaults, and kidnappings), "gang crime" (violent crime and terrorist threats, firearm discharge, vandalism, and graffiti committed by gang members), and gun crime (any of the above crimes that involved use of a firearm).
The RAND evaluation analyzed changes in their key outcome variables for three time periods across three comparison areas . The three time periods were the six months prior to the triggering event-the pre-intervention period; the four months in which all parts of the intervention were applied-the suppression period; and the two months in which only selected parts of the intervention were applied, such as heightened patrol of public housing units in the area and greater enforcement of probation and parole regulations-the deterrence period. The three comparison areas were (1) Boyle Heights compared with the remainder of the Hollenbeck area, (2) the five police reporting districts where the intervention was targeted compared with the remainder of Boyle Heights, and (3) the Census block groups comprising the turf of the targeted gangs compared with a group of Census block groups scattered throughout Hollenbeck that most closely matched the characteristics of the targeted area based on a propensity score analysis. In explaining the rationale for their research design, the RAND researchers reported: A reduction in crime in the treatment areas greater than those in the comparison areas during the suppression period would help show the effects of all measures combined, whereas continuing reductions in the deterrence period would suggest that the intervention may have had some long-term effects in changing behavior, or that short-term application of some resources can produce a long-term deterrence effect (although we recognize our measure of deterrence is confounded by the continuation of some suppression activities). (Tita et al., 2003: 24 -25).
The evaluation used a variety of approaches to detect the effects of the Ceasefire intervention across the study time period . The RAND researchers used Bayesian analyses of the count-based distributions of the outcome variables across these time periods in Boyle Heights relative to the remainder of Hollenbeck comparison and in the five targeted reporting districts relative to the remainder of Boyle Heights comparison. In their analyses of targeted Census block groups relative to matched comparison Census block groups, the RAND researchers used Probit and step-wise linear regression models to define the matched comparison Census block groups. They then used a slightly more rigorous version of the "difference in differences" approach that assumed the level of crime followed a Poisson distribution and considered serial time trends to evaluate the effects of the Ceasefire intervention across these time periods .
Their statistical analyses revealed that gang crime in Boyle Heights decreased significantly compared with other regions of Hollenbeck during the suppression period of the intervention, and violent, gang, and gun crime all decreased significantly in the deterrence period. The analyses suggested that the significant reduction in gang crime may have begun in the suppression period. Violent crime, however, did not decrease significantly in the suppression period. In the five targeted police reporting districts, violent crime decreased significantly in comparison with the rest of Boyle Heights in the suppression and the deterrence periods, and gang crime decreased significantly in the suppression period. Neither gang crime in the deterrence period nor gun crime in the deterrence or suppression periods decreased significantly in comparison with the remainder of Boyle Heights. The RAND evaluation also reported that, in the Census block groups overlapping the targeted reporting districts, violent crime decreased significantly compared with the matched blocks . Their analyses also suggested that some of this significant reduction may have persisted into the deterrence period.
In addition to their analyses of the main effects of the intervention, RAND researchers examined the effects of the intervention on neighboring areas and gangs. Their analyses suggested a strong diffusion of violence prevention benefits emanating from the targeted areas and targeted gangs . In the six months after the intervention, the researchers reported in the 6 targeted Census block groups that violent crime had decreased by 34 percent, gang crime decreased by 28 percent, and gun crime decreased by 26 percent. In the 11 Census block groups immediately surrounding the targeted block groups, violent crime had decreased by 33 percent, gang crime decreased by 44 percent, and gun crime decreased by 28 percent. The RAND research team also examined gang crime by gangs not targeted by the Ceasefire intervention that were "socially tied" through conflicts and alliances to the target gangs. After the Ceasefire intervention was implemented, gang crimes committed by the targeted gangs and the non-targeted, socially-tied gangs decreased by a matching 26 percent.

PROJECT SAFE NEIGHBORHOODS IN CHICAGO, ILLINOIS
The PSN was implemented in two adjacent police districts in Chicago's West Side where rates of murder and gun violence were more than four times higher than the city average in 2002. As described by Papachristos, Meares, and Fagan (2007), the PSN team reasoned that the best way to address Chicago's homicide and gun violence problem was to craft intervention strategies focused on the population with a very high risk of being a victim or offender of gun violence in the targeted neighborhoods. Two principles guided the design and development of PSN interventions: (1) that enforcement efforts be highly specified and targeted to those most at risk of being a victim and offender of gun violence, and (2) that serious efforts should be directed towards changing the normative side of gun violence, i.e., the reasons young men use guns and their attitudes towards the law and law enforcers. Following these principles the PSN team devised several law enforcement, community outreach, and offender notification forums and follow-up re-entry programs. The PSN interventions were implemented in May 2002.
A quasi-experimental design was used to evaluate the impact of the various PSN programs on neighborhood-level homicide rates in Chicago (Papachristos et al., 2007). As described, two adjacent police districts were nonrandomly selected from the city's 25 police districts as PSN treatment districts and, via propensity score matching procedures, two other police districts selected as near-equivalent controls.
Monthly and quarterly counts of homicide incidents between January 1999 and December 2004 were identified as the key outcome variables (Papachristos et al., 2007;Meares et al., 2009); however, the evaluation also analyzed monthly and quarterly counts of gun homicide incidents, gang homicide incidents, and aggravated assault incidents in the treatment districts relative to the control districts.
The research team analyzed the overall effects of the PSN treatment as well as the four interventions that comprised the PSN treatment: (1) increased federal prosecutions for convicted felons carrying or using guns, (2) the length of sentences associated with federal prosecutions, (3) supply-side firearm policing activities (gun recoveries by ATF-CPD gun teams), and (4) social marketing of deterrence and social norms messages through justice-style offender notification meetings. In these offender notification meetings, randomly selected gun-and gang-involved recently-released former prison inmates returning to the treatment districts were informed of their vulnerability as felons to federal firearms laws, with stiff mandatory minimum sentences; offered social services; and addressed by community members and exoffenders. Using individual growth curve regression models, the research team found that the PSN treatment was associated with a statistically significant 37 percent reduction in the number of homicides in the treatment district relative to the control districts. The overall PSN treatment was also associated with statistically significant decreases in gun homicide incidents and aggravated assault incidents, and a non-statistically significant decrease in gang homicide incidents.
The PSN intervention that generated the largest, statistically significant effect on decreased homicide in the treatment districts relative to control districts was the offender notification forums. In short, the greater the proportion of offenders who attended the forums, the greater the decline in treatment district levels of homicide. Increased federal prosecutions and the number of guns recovered by the gun teams were associated with modest but statistically significant declines in homicides in the treatment districts relative to the control districts. Getting more guns off the street and prosecuting more offenders federally for gun crimes were associated with small but meaningful homicide decreases. The length of sentences associated with federal prosecutions was not associated with the observed homicide decreases.
In a supplemental unpublished analysis, Fagan et al. (2008) analyzed recidivism rates of individuals who participated in the PSN notification forums. Using survival analyses, the authors found that those who attended a PSN forum were 30 percent less likely to be rearrested relative to a comparison group of similar recentlyreleased individuals from the same neighborhood. The program diminished recidivism levels for both gang and non-gang members and seemed to be particularly effective for individuals who had only one prior felony conviction.

DRUG MARKET INTERVENTION IN NASHVILLE, TENNESSEE
In an unpublished report, Corsaro and McGarrell (2009) evaluated the impact of a pulling levers focused deterrence strategy to reduce crime and disorder problems associated with an illegal drug market operating in the McFerrin Park neighborhood of Nashville, Tennessee. Drawing on similar intervention conducted in High Point, North Carolina (Kennedy, 2009), the project employed a joint police-community partnership to identify individual offenders, notify them of the consequences of continued dealing, provide supportive services through a community-based resource coordinator, and convey an uncompromising community norm against drug dealing. This application of focused deterrence is generally referred to as the "Drug Market Intervention" (DMI) strategy.
The DMI seeks to shut down overt drug markets entirely (Kennedy, 2009). Enforcement powers are used strategically and sparingly, employing arrest and prosecution only against violent offenders and when nonviolent offenders have resisted all efforts to get them to desist and to provide them with help. Through the use of "banked" cases, 12 The strategy was implemented in March 2008 and the evaluation examined outcome data for the March 2005 through April 2009 time period. The evaluation measured the effects of the DMI intervention on five outcome variables: violent crime (the aggregated number of homicide, forcible rape, robbery, and aggravated assault incidents), property crime (the sum of burglaries, larcenies, and motor vehicle thefts), illegal drug possession offenses, drug equipment offenses (the monthly number of charges for drug-paraphernalia and related crimes), and total calls for service. The researchers analyzed the aggregated monthly number of these outcome variables for the following Nashville areas: 1) the McFerrin Park target neighborhood to assess the local effect; 2) adjoining, contagious areas to the McFerrin Park neighborhood to assess whether a local displacement or a diffusion of benefits occurred; and 3) the remainder of Davidson County, once the target and adjoining areas were subtracted from the county totals for general trend comparison purposes.
the strategy makes the promise of law enforcement sanctions against dealers extremely direct and credible, so that dealers are in no doubt concerning the consequences of offending and have good reason to change their behavior. The strategy also brings powerful informal social control to bear on dealers from immediate family and community figures. The strategy organizes and focuses services, help, and support on dealers so that those who are willing have what they need to change their lives. Each operation also includes a maintenance strategy.
ARIMA models with a dummy variable to represent the implementation of the DMI strategy were used to analyze trends in the treatment, adjoining, and comparison areas. The analyses reported that the DMI intervention was associated with a statistically significant 28.4 percent reduction in property offenses, a statistically significant 55.5 percent reduction in illegal drug possession offenses, and a statistically significant 36.8 percent reduction in drug equipment offenses. The analyses also found that the DMI intervention was associated with non-statistically significant reductions in violent crime incidents and total calls for service. The evaluation did not find any significant reductions in any of the key outcome variables in the remainder of Davidson County. The researchers did find, however, statistically significant reductions in illegal drug possession offenses, drug equipment offenses, and total calls for service in the adjoining area. This suggested that the DMI intervention was associated with a noteworthy diffusion of crime control benefits beyond the McFerrin Park target neighborhood.

DRUG MARKET INTERVENTION IN ROCKFORD, ILLINOIS
Corsaro et al. (2010) evaluated the impact of a pulling levers focused deterrence strategy to reduce crime and disorder problems associated with an illegal drug market operating in the Delancey Heights neighborhood of Rockford, Illinois. Like the Nashville strategy described above, this research and development study was a replication of the High Point, North Carolina DMI (Kennedy, 2009). The strategy was implemented in May 2007 and the evaluation examined outcome data for the June 2006 through June 2008 time period. The evaluation measured the effects of the DMI intervention on two outcome variables: violent crime (the aggregated number of homicide, rape, kidnapping, robbery, and aggravated assault incidents) and nonviolent crime (the aggregate number of property, drug, and nuisance crime incidents). The researchers analyzed the aggregated monthly number of these outcome variables for the Delancey Heights neighborhood and for the remainder of Rockford without Delancey Heights.
Hierarchical generalized linear growth curve regression models with a dummy variable to represent the implementation of the DMI strategy were used to analyze trends in the treatment and comparison areas. The analyses reported that the DMI intervention was associated with a statistically significant 22 percent reduction in nonviolent offenses and a non-statistically significant reduction in violent offenses in the Delancey Heights target neighborhood. The evaluation did not find any significant reductions in either violent offenses or nonviolent offenses in the remainder of Rockford. Corsaro et al. (2010) also presented qualitative data from interviews with 34 adult residents from the Delancey Heights neighborhood. The authors reported that the majority of the residents interviewed noted considerable crime and disorder improvements in their neighborhood after the DMI was implemented. whether they formally entered the program or not. Of the eligible probationers, two thirds were assigned to the HOPE treatment (n = 330) and one third were assigned to the control group (n = 163). Ninety-three percent of the probationers assigned for treatment appeared for their initial HOPE warning hearing and participated in the intervention. The experiment commenced in October 2007 and the intervention period lasted for one year.
Based on their analyses of the experimental data, Hawken and Kleiman (2009) concluded that HOPE was very effective in changing the behaviors of substanceabusing probationers. Only 21 percent of HOPE probationers experienced new arrests as compared to 47 percent of control probationers (p<01). HOPE probationers outperformed control probationers on a number of other performance measures such as missed probation appointments (treatment = 9%, control = 23%), positive urine drug test results (treatment = 13%, control = 46%), revocation rates (treatment = 7%, control = 15%), and the number of days sentenced to incarceration (treatment = 138 days, control = 267 days).
Based on our selection criteria, HOPE was not included in our final review. However, as stated earlier, several scholars believed that HOPE does fit within the general framework of pulling levers focused deterrence strategies. We agree that it is broadly similar to another evaluation included in our systematic review that is focused on a corrections population -Chicago's Project Safe Neighborhoods intervention (Papachristos et al., 2007). The key elements of Chicago PSN strategy are administered by the Illinois Department of Correction and the U.S. Attorney's Office (the call-in session is given to returning parolees to selected neighborhoods). The contribution of the Chicago Police Department is limited to increasing their gun policing efforts in the selected neighborhoods. The CPD does not select the returning parolees for the intervention nor do they run the communications strategy. Their only role is to increase gun recoveries and arrest those who commit violent crimes in these neighborhoods.
Moreover, probation has a central role in all of the gang / group-based focused deterrence interventions included in our review. Monitoring offenders in the community to ensure they are abiding by probation conditions, changing conditions, and revoking probation are key levers that are pulled in the application of focused deterrence strategies to gangs and criminally-active groups. In interagency working group settings, all involved agencies govern the shape and content of the pulling levers interventions. While the police convene the working group meeting, they share governance with the other criminal justice agencies, social service providers, and community members in the group. Probation is involved as a key decision maker in the process.
Most applications of pulling levers focused deterrence strategies have therapeutic elements (e.g. Braga et al., 2001;Papachristos et al., 2007). Indeed, the working group has social service providers, street outreach workers, and community members as core members. A vital part of the communications strategy to pair threats of sanctions with offers of help (job training / placement, education, substance abuse counseling, etc.). All targeted gangs and groups are offered services throughout the entire process. Under HOPE, probationers are sentenced to drug treatment only if they continue to test positive for drug use, or if they request a treatment referral. HOPE is distinct from drug courts in economizing on treatment and court resources (probationers appear before a judge only when a violation is detected).
If HOPE is included in our review, the substantive findings do not change (see Braga and Weisburd, 2012). Our narrative review of program effects would change to report that there were 10 out of 11 eligible evaluations with positive findings (twotailed p = 0.0117). Our main meta-analysis of the effects of pulling levers focused deterrence strategies would still be limited to the ten studies that evaluated the impact of these interventions on crime outcomes at the area-level. As described by Lipsey and Wilson (2001), it is problematic to combine effect sizes from studies with very different units of analysis (such as combining studies focused on people with studies focused on places). We can, however, separately calculate and examine effect sizes for the HOPE intervention's impact on recidivism by individual probationers and the Chicago PSN intervention's impact on recidivism by individual recentlyreleased prison inmates.
As already described in our systematic review, the Chicago PSN quasi-experimental evaluation conducted additional analyses of the effects of the intervention on individual recidivism patterns. The HOPE randomized controlled experiment also examined the effects of the treatment on individual recidivism patterns in Honolulu. Both studies measured recidivism as a new arrest by treatment and control individuals. The HOPE effect size was calculated by using the raw data from Hawken and Kleiman (2009: 63-64) to construct a two by two table and entering the cell counts in the Effect Size Calculator. The more conservative Logit method was used to calculate the effect size. We calculated effect sizes for the Chicago (.434, SE = .050, p < .05) and Honolulu (.666, SE = .115, p < .05) studies. When the effect sizes for these studies were combined via a fixed effects meta-analysis model (Q = 3.391, df = 1, p = 0.066), the overall effect size was medium and statistically significant (.471, SE = .046, p < .05), suggesting a program impact similar to area-level impacts generated by the DMI and gang / group interventions.